Skip to content

Latest commit

 

History

History
185 lines (131 loc) · 6.38 KB

README_DEV_SETUP.md

File metadata and controls

185 lines (131 loc) · 6.38 KB
Raw

LunaTrace Dev Setup

This is basically just going to be a collection of notes until we actually sit down to write this guide out better.

System Requirements

  • docker
  • docker-compose
  • tmux
  • tmuxp
  • golang (>=1.17)
  • node (>=16)
  • yarn (Just use npm i -g yarn if missing)
  • Bash and other GNU utils (BSD utils are untested, i.e. Mac OSX)

Dev Tools

We prefer IntelliJ by JetBrains and our dev configs are committed to the base of the repo under the .idea folder.

If you're using another IDE it'll probably work too because our workflows don't use the IDE a ton. We just use it for stuff like auto-linting and various other quality of life tasks like easily navigating the codebase.

AWS Credentials

In order to work with local dev, you'll have to have functional AWS credentials that the AWS CLI can find.

This exercise is left to you to sort out. Generally you'll need to put your credentials in ~/.aws/credentials though.

Repo setup

You'll have to clone the repo first and cd to it first.

Setup Node Dependencies

Just run yarn from anywhere in the repo and it'll go sort out building deps. Otherwise, deps are committed to Git as a part of the "Yarn PNP" flow. (You will not see a node_modules folder, only a .yarn one)

You'll also have to add smee to your path with:

npm install -g smee-client

Setup AWS Dependencies

From $REPO_ROOT/lunatrace/bsl/backend-cdk folder, you'll need to run the following. Replace YOUR_USERNAME with your user.

DEV_USER=YOUR_USERNAME yarn run cdk:deploy:dev

That will run a real AWS deployment of the "dev" resources required. Once it finished, you should see something like:

 ✅  lunatrace-EtlStorage

✨  Deployment time: 180.43s

...
Outputs:
lunatrace-alex-EtlStorage.CodeBucketName = lunatrace-alex-etlstorage-codebucketff4c7ad6-8c1wq1jf24k1
lunatrace-alex-EtlStorage.GrypeDatabaseBucketName = lunatrace-alex-etlstorag-grypedatabasebucketddd6c-z6d8vsv2rhah
lunatrace-alex-EtlStorage.LunaTraceDevelopmentGolangQueueName = lunatrace-alex-EtlStorage-LunaTraceDevelopmentGolangQueue6299EB49-ECpdzqPHhJvy
lunatrace-alex-EtlStorage.LunaTraceDevelopmentQueueName = lunatrace-alex-EtlStorage-LunaTraceDevelopmentQueue04E796C2-7RDss2Nc8gt0
lunatrace-alex-EtlStorage.ManifestBucketName = lunatrace-alex-etlstorage-manifestbucket46c412a5-1b0kc5cys81er
lunatrace-alex-EtlStorage.SbomBucketName = lunatrace-alex-etlstorage-sbombucket8550fee8-1drqtwb7yf2dg
Stack ARN:
arn:aws:cloudformation:us-west-2:134071937287:stack/lunatrace-alex-EtlStorage/4655a320-cb37-11ec-a2b1-02772921f86f


✨  Total time: 184.84s

You'll need to format those values into an env file at $REPO/lunatrace/backend/.env.dev that looks like:

S3_SBOM_BUCKET=xxx
S3_MANIFEST_BUCKET=xxx
S3_CODE_BUCKET=xxx
QUEUE_NAME=xxx


LUNATRACE_GRAPHQL_SERVER_URL=http://localhost:8080/v1/graphql
LUNATRACE_GRAPHQL_SERVER_SECRET=myadminsecretkey

Adding github app private key

The github app private key needs to exist at lunatrace/bsl/backend/github-app-dev.2022-03-09.private-key.pem. Ask for this file or set up your own github app.

Setting up ory kratos

Ory kratos needs the github app client id and secret at lunatrace/bsl/ory/kratos/config.dev.yaml. Ask for this file or set up your own with the format:

serve:
  public:
    base_url: http://localhost:4455/api/kratos/

selfservice:
  methods:
    password:
      enabled: false
    oidc:
      enabled: true
      config:
        providers:
          - id: github-app
            provider: github-app
            client_id: xxx
            client_secret: xxx
            mapper_url: file:///config/oidc.github.jsonnet

Running tmuxp

From the $REPO_ROOT/lunatrace/dev-cli folder run yarn run start and you should see:

Generating tmuxp config: /home/free/code/temp/csp-test/lunatrace/bsl/generated-lunatrace-tmuxp.yaml...
<GARBAGE CLIPPED>
Starting tmuxp...
cd /home/<USER>/<PATH_TO_REPO>/lunatrace/bsl && tmuxp load /home/<USER>/<PATH_TO_REPO>/lunatrace/bsl/generated-lunatrace-tmuxp.yaml

Now that you have that file, you can follow that bottom line and it'll set everything up for you to run. :)

NOTE: If you're on a Mac, you might have to swap all sudo docker calls with docker because of differences in how the OS deals with Docker.

Also, if you ever need to "kill" tmux, you must do Control-B and then type :kill-session followed by Enter to confirm.

That'll nuke the whole session and sometimes Docker Compose might get angry with you about ports the next time you start it up, so if you accidentally have problems after you can run sudo docker-compose down from the bsl folder.

Setting up the Database

We use Hasura to manage the database. Migrations for Kratos and Hasura will be applied automatically in dev.

Hasura

You should be able to pull up the Hasura console either by running hasura console or going to http://localhost:9695/. This will show you the GraphQL server and an admin dashboard for the database.

Frontend

To set up the frontend, pull the theme submodule and compile the CSS:

cd bsl/frontend
yarn run sass:pull
yarn run sass:build

Vulnerabilities

cd bsl/ingest-worker
LUNATRACE_GRAPHQL_SERVER_SECRET=myadminsecretkey LUNATRACE_GRAPHQL_SERVER_URL=http://localhost:8080/v1/graphql go run cmd/ingestworker/main.go sync --source ghsa --source-relative-path advisories/github-reviewed

Re-run everything

Either kill everything in tmuxp or just restart the front-end and back-end containers after Hasura comes up.

This is important because otherwise the GraphQL code gen stuff will probably not have run, and then everything might break.

Test flow

Open up http://localhost:4455/ and you should hit the front-end of the app. If it's busted... go hit us up on Slack and we'll help you sort it out. (Invite for the Discord)

Hopefully this works. Please improve this doc when you use it and something is broken or could be improved. Thanks!