Skip to content

Contributing

Prerequisites

Install all the required tools:

Install the pre-commit hooks.

In the project root run make precommit. This will install all the relevant pre-commit hooks across the entire codebase.

Running the API

To run the API locally, you will need to set the following environment variables. They will correspond to rAPId resources already created in AWS and can be taken from the environment of your rAPId ECS task:

AWS_ACCOUNT=
AWS_REGION=
COGNITO_USER_POOL_ID=
DATA_BUCKET=
DOMAIN_NAME=
RESOURCE_PREFIX=
ALLOWED_EMAIL_DOMAINS=

You can then run the following commands from within the root project directory:

make api-run runs Batect to bring up a locally running version of the API within a Docker container.

make api-run-dev runs Batect to bring up a locally running version of the API within a Docker container using a "hot-reload" mode so you can keep this running during development.

Passing an AWS Role

As the application connects to AWS services you will need to provide the application with AWS credentials.

To do this, you will need to add the following environment variables to the .env file of rAPId.

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_SESSION_TOKEN=

For security, we recommend that these are only the temporary credentials of an assumed role.

Running the UI

You can run the UI locally for development either against an already running rAPId instance that has been deployed or a locally running instance of rAPId.

  1. Install npm, we recommend using nvm
  2. Install the all the required packages make ui-setup

To connect the UI with your rAPId instance you will need to set the following environment variables:

NEXT_PUBLIC_API_URL=
NEXT_PUBLIC_API_URL_PROXY=

The UI uses a proxy system that allows API requests to be made to the rAPId server without them getting blocked CORS issues. You can set the NEXT_PUBLIC_API_URL to the api suffix of the rAPId instance /api and set NEXT_PUBLIC_API_URL_PROXY to be the full domain name, such as https://myrapid.co.uk.

Running make ui-run-dev will then launch the UI in development mode with hot reloading.

RAT Token

The UI looks for a cookie with the name rat (rAPId Access Token), to authenticate with the API. This is a Cognito token that is generated by the UI and passed to the API in the Authorization header. The API then validates the token and extracts the user information from it.

For the authentication to work when running the local UI, you need to login to your deployed version of rAPId, copy the value of the rat cookie. Navigate back to your locally running version of the UI and create a new rat cookie with the value you copied.

Testing

Every rAPId module other than the infrastructure has complementary tests.

API

To run the API tests locally, you will need to set the following environment variables:

DATA_BUCKET=the-bucket
AWS_ACCOUNT=123456
AWS_REGION=eu-west-2
RESOURCE_PREFIX=rapid
DOMAIN_NAME=example.com
COGNITO_USER_POOL_ID=11111111
ALLOWED_EMAIL_DOMAINS=example1.com,example2.com
LAYERS=raw,layer

Then run the following command from within the root project directory:

make api-test

UI

To test the UI run the following command from within the root project directory:

make ui-test

SDK

To test the SDK run the following command from within the root project directory:

make sdk-test

Security

Code vulnerabilities

To scan for security vulnerabilities run make security-check

bandit is in use to check for common python vulnerabilities.

Infrastructure flaws

To launch a static scan on the infrastructure, you can run make infra-scan.

checkov is used to check for flaws in the AWS architecture.

Scanned image vulnerabilities

The API image is scanned on a daily basis for vulnerabilities. New releases cannot then be made until image the vulnerabilities are addressed.

To address the vulnerability, reviewed the content of it and assess if the application is likely to be affected by it.

If it is deemed to be acceptable, you can ignore it by adding the CVE to a new line in the vulnerability-ignore-list.txt file. Ensure you commit with a meaningful error message, the URL and your reasoning for ignoring it. By doing this the pipeline will accept that vulnerability and not fail the build.

If the vulnerability is not acceptable, then another base image will need to be used.

Secrets

We are using the python package detect-secrets to analyse all files tracked by git for anything that looks like a secret that might be at risk of being committed.

There is a .secrets.baseline file that defines any secrets (and their location) that were found in the repo, which are not deemed a risk and ignored (e.g.: false positives)

To check the repo for secrets during development run make detect-secrets. This compares any detected secrets with the baseline files and throws an error if a new one is found.

Firstly, REMOVE ANY SECRETS!

However, if the scan incorrectly detects a secret, run: make ignore-secrets to update the baseline file. This file should be added and committed.

The secret detection step is run as part of the make precommit target.

Releasing

rAPId has several core components that are all versioned when a release is created:

  1. A API image that can be pulled to run the python application.
  2. A zip package containing the static UI.
  3. A public pypi package of the built rAPId-sdk that provides an easy and pythonic way to interact with the API.
  4. A terraform module that can be used to create the AWS services necessary for running the various components of rAPId.

Performing a release involves tagging the repository with a new version number so that the API image, UI, SDK and Terraform all get versioned. Note that the API and SDK can be versioned and released independently, so you can conserve the API version number if just releasing an update to the SDK.

Prerequisites

  • Download the GitHub CLI with brew install gh
  • Then run gh auth login and follow the steps

Steps

  1. Decide on the new version number for the API and the UI and/or the SDK following the semantic versioning approach.
  2. Update and commit the Changelog (you can follow the template). You'll need to separate SDK and API changes into their respective changelogs, under docs/changelog.
  3. Bundle API, UI and terraform changes as part of the API changelog.
  4. Insert SDK changes into the SDK changelog.
  5. Run make release commit=<commit_hash> type=<sdk|api> version=vX.X.X

Ensure the version number follows the format vX.X.X with full-stops in the same places for both API and SDK changes.

Now the release pipeline will run automatically, build all images and packages off that version of the code and tag it within GitHub.