Contributing
Prerequisites
Install all the required tools:
- jq (use Homebrew)
- Git
- pre-commit
- Make
- AWS CLI
- Homebrew
- Docker
- Java version 8+ (for Batect, future releases will remove this dependency)
- Python (v3.10) - Only for local development without Docker (ideally manage via pyenv)
Install the pre-commit hooks.
In the project root run make precommit
. This will install all the relevant pre-commit hooks across the entire codebase.
Running the API
To run the API locally, you will need to set the following environment variables. They will correspond to rAPId resources already created in AWS and can be taken from the environment of your rAPId ECS task:
AWS_ACCOUNT=
AWS_REGION=
COGNITO_USER_POOL_ID=
DATA_BUCKET=
DOMAIN_NAME=
RESOURCE_PREFIX=
ALLOWED_EMAIL_DOMAINS=
You can then run the following commands from within the root project directory:
make api-run
runs Batect to bring up a locally running version of the API within a Docker container.
make api-run-dev
runs Batect to bring up a locally running version of the API within a Docker container using a "hot-reload" mode so you can keep this running during development.
Passing an AWS Role
As the application connects to AWS services you will need to provide the application with AWS credentials.
To do this, you will need to add the following environment variables to the .env
file of rAPId.
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_SESSION_TOKEN=
For security, we recommend that these are only the temporary credentials of an assumed role.
Running the UI
You can run the UI locally for development either against an already running rAPId instance that has been deployed or a locally running instance of rAPId.
- Install
npm
, we recommend using nvm - Install the all the required packages
make ui-setup
To connect the UI with your rAPId instance you will need to set the following environment variables:
NEXT_PUBLIC_API_URL=
NEXT_PUBLIC_API_URL_PROXY=
The UI uses a proxy system that allows API requests to be made to the rAPId server without them getting blocked CORS issues. You can set the NEXT_PUBLIC_API_URL
to the api suffix of the rAPId instance /api
and set NEXT_PUBLIC_API_URL_PROXY
to be the full domain name, such as https://myrapid.co.uk
.
Running make ui-run-dev
will then launch the UI in development mode with hot reloading.
RAT Token
The UI looks for a cookie with the name rat
(rAPId Access Token), to authenticate with the API. This is a Cognito token that is generated by the UI and passed to the API in the Authorization
header. The API then validates the token and extracts the user information from it.
For the authentication to work when running the local UI, you need to login to your deployed version of rAPId, copy the value of the rat
cookie. Navigate back to your locally running version of the UI and create a new rat
cookie with the value you copied.
Testing
Every rAPId module other than the infrastructure has complementary tests.
API
To run the API tests locally, you will need to set the following environment variables:
DATA_BUCKET=the-bucket
AWS_ACCOUNT=123456
AWS_REGION=eu-west-2
RESOURCE_PREFIX=rapid
DOMAIN_NAME=example.com
COGNITO_USER_POOL_ID=11111111
ALLOWED_EMAIL_DOMAINS=example1.com,example2.com
LAYERS=raw,layer
Then run the following command from within the root project directory:
make api-test
UI
To test the UI run the following command from within the root project directory:
make ui-test
SDK
To test the SDK run the following command from within the root project directory:
make sdk-test
Security
Code vulnerabilities
To scan for security vulnerabilities run make security-check
bandit is in use to check for common python vulnerabilities.
Infrastructure flaws
To launch a static scan on the infrastructure, you can run make infra-scan
.
checkov is used to check for flaws in the AWS architecture.
Scanned image vulnerabilities
The API image is scanned on a daily basis for vulnerabilities. New releases cannot then be made until image the vulnerabilities are addressed.
To address the vulnerability, reviewed the content of it and assess if the application is likely to be affected by it.
If it is deemed to be acceptable, you can ignore it by adding the CVE
to a new
line in the vulnerability-ignore-list.txt
file. Ensure you commit with a meaningful error message, the URL and your
reasoning for ignoring it. By doing this the pipeline will accept that vulnerability and not fail the build.
If the vulnerability is not acceptable, then another base image will need to be used.
Secrets
We are using the python package detect-secrets to analyse all files tracked by git for anything that looks like a secret that might be at risk of being committed.
There is a .secrets.baseline
file that defines any secrets (and their location) that were found in the repo, which are
not deemed a risk and ignored (e.g.: false positives)
To check the repo for secrets during development run make detect-secrets
. This compares any detected secrets with the
baseline files and throws an error if a new one is found.
Firstly, REMOVE ANY SECRETS!
However, if the scan incorrectly detects a secret, run: make ignore-secrets
to update the baseline file. This file
should be added and committed.
The secret detection step is run as part of the make precommit
target.
Releasing
rAPId has several core components that are all versioned when a release is created:
- A API image that can be pulled to run the python application.
- A zip package containing the static UI.
- A public pypi package of the built rAPId-sdk that provides an easy and pythonic way to interact with the API.
- A terraform module that can be used to create the AWS services necessary for running the various components of rAPId.
Performing a release involves tagging the repository with a new version number so that the API image, UI, SDK and Terraform all get versioned. Note that the API and SDK can be versioned and released independently, so you can conserve the API version number if just releasing an update to the SDK.
Prerequisites
- Download the GitHub CLI with
brew install gh
- Then run
gh auth login
and follow the steps
Steps
- Decide on the new version number for the API and the UI and/or the SDK following the semantic versioning approach.
- Update and commit the Changelog (you can follow the template). You'll need to separate SDK and API changes into their respective changelogs, under docs/changelog.
- Bundle API, UI and terraform changes as part of the API changelog.
- Insert SDK changes into the SDK changelog.
- Run
make release commit=<commit_hash> type=<sdk|api> version=vX.X.X
Ensure the version number follows the format
vX.X.X
with full-stops in the same places for both API and SDK changes.
Now the release pipeline will run automatically, build all images and packages off that version of the code and tag it within GitHub.