Skip to content

Brendan/170/cookbook for s3 bucket cache setup #185

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/concepts/publishing.md
Original file line number Diff line number Diff line change
@@ -82,9 +82,10 @@ See the "Catalog Store" cookbook page to learn more about how to provision the s
Artifacts uploaded to a Catalog Store may be signed.
This key is provided to Flox via the `flox publish --signing-key` argument so that the key can be used to sign artifacts during the publish process.
Similarly, in order to install packages signed with this key, Nix must be configured to trust this key.
See the "Catalog Store" cookbook page to learn more about how to configure Nix to trust the signing key.
See the ["Catalog Store" cookbook page][catalog-store-cookbook] to learn more about how to configure Nix to trust the signing key.

[builds-concept]: ./manifest-builds.md
[early]: https://flox.dev/early/
[catalog-util]: https://github.com/flox/catalog-util
[catalog-concept]: ./packages-and-catalog.md
[catalog-store-cookbook]: ../cookbook/publish/flox-store.md
115 changes: 115 additions & 0 deletions docs/cookbook/publish/flox-store.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
---
title: Catalog Store
description: Create a Catalog Store for publishing your own Flox packages
---

# Setting up a Catalog Store

Publishing your own software to your organization's Flox Catalog requires some
initial setup, but the process is relatively straightforward. Flox supports
publishing packages to a Catalog Store, which can exist in an AWS S3 bucket
or in any S3 compatible service, like [MinIO][minio-s3-compatible]{:target="\_blank"}
or [Backblaze B2][backblaze-b2-cloud-storage]{:target="\_blank"}. (For the
sake of simplicity, this guide focuses on S3, but there are other providers
available if you prefer them to AWS.)

In order to use an S3 bucket to store artifacts built with Flox, you will need
to set ingress and egress URIs on the catalog using a utility published by Flox.
Then, all you need to do to publish your software is to call `flox publish`,
and Flox will take care of the rest.

[minio-s3-compatible]: https://min.io/product/s3-compatibility
[backblaze-b2-cloud-storage]: https://www.backblaze.com/cloud-storage

## Configure an AWS S3 Bucket

The first step in setting up your Catalog Store is creation and configuration of
an AWS S3 Bucket. There are numerous ways to accomplish this, including the AWS
Console, the AWS CLI, and Terraform (or another infrastructure-as-code tool),
to name a few. These processes are well documented, but to get started,
it's best to refer directly to AWS documentation.

- [What is Amazon S3?][amazon-s3]{:target="\_blank"}
- [AWS S3 CLI Reference][aws-cli-reference-s3]{:target="\_blank"}
- [Amazon Simple Storage Service API Reference][aws-s3-api-reference]{:target="\_blank"}

Once your S3 bucket is set up and configured with the access policies deemed
necessary by your organization's internal policies, you're ready to proceed to
the next step. Someone from Flox can help you if you run into trouble during
the setup process. Simply reach out to your designated point of contact,
and we'll work with you to get you up and running.

[amazon-s3]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html
[aws-cli-reference-s3]: https://docs.aws.amazon.com/cli/latest/reference/s3/
[aws-s3-api-reference]: https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html

## Ensure the Nix Daemon Has Access to the S3 Bucket

As you probably know by now, the underlying technology powering Flox is Nix.
Accordingly, we need to take a couple steps to ensure that the Nix daemon
has access to the S3 bucket you've just created. To do so,
you have a couple of options:

1. Set `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and, if applicable,
`AWS_SESSION_TOKEN` as environment variables, both for Flox and for
the daemon itself
1. Use the `aws configure` command
[as described in the CLI reference][aws-cli-configure-command]{:target="\_blank"}
to set those same values, and ensure that the AWS profile and region match those
configured for the S3 bucket

If you follow the second set of steps, you can confirm that everything is set
up correctly by inspecting the values stored in `$HOME/.aws/credentials`.

[aws-cli-configure-command]: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/configure/index.html#configure

## Set Catalog Store Ingress and Egress URIs

Once you have your S3 bucket configured, the next step is to set an ingress URI
and egress URI for your Catalog Store. Flox provides a utility for you
that does exactly what you need, within a Flox environment. To use this,
you'll need to run the following command:

```sh
flox activate -r flox/flox-catalog-util
```

When you run this command, you'll see the following output:

```console
✅ You are now using the environment 'flox/flox-catalog-util (remote)'.
To stop using this environment, type 'exit'
```

Within the active Flox environment, you can simply run the following command:

```sh
catalog-util store --catalog "<my-catalog-name>" set --store-config '{ "store-type": "nix-copy", "ingress_uri": "s3://<my-bucket>", "egress_uri": "s3://<my-bucket>" }'
```

You'll note that it's possible to set the ingress and egress URIs to the same
value, if you wish to do so.

## Create and Set a Signing Key

At this point, you should have an appropriately configured Catalog Store
to which you can publish your own software via the `flox publish` command.
The last thing you need to worry about is configuring a signing key
for publishing packages.

The first step in this process is generating a key. This example illustrates
how you would do so for a key called "my-key":

sh```
nix key generate-secret --key-name my-key > my-key.key
nix key convert-secret-to-public < my-key.key
```
Once you've generated the key, you can configure Flox to sign the packages
you publish with that key:
```sh
flox config --set publish.signing_key "</path/to/my-key.key"
```

Now you're ready to use Flox publish your own software to the Catalog Store
that you just set up.
2 changes: 1 addition & 1 deletion docs/tutorials/build-and-publish.md
Original file line number Diff line number Diff line change
@@ -10,7 +10,7 @@ Flox aims to be the one tool that you need for the entire software development l

## Prepare a project

Let's start by creating a simle Go project.
Let's start by creating a simple Go project.
We'll create a directory called `myproject` and create a Flox environment inside of it so we can install our tools.

```text
2 changes: 2 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
@@ -62,6 +62,8 @@ nav:
- ... | flat | cookbook/languages/*
- Builds:
- ... | flat | cookbook/builds/*
- Publish:
- ... | flat | cookbook/publish/*
- Reference:
- Flox manual:
- reference/command-reference/flox.md