Skip to content

Commit 409ceb8

Browse files
LesnyRumcajsDavid Himmelstrup
LesnyRumcajs
and
David Himmelstrup
authored
Forest & Lotus devnet (#2910)
* debug statements in forest, improved devnet notes * Use given chain name if everything else fail * Use KeepAlive to stay connected to Lotus peers * Bypass concurrency bug in the chain exchange * use devnet cids fil-actor-states branch * temp fix for null blocks * invert drand schedule logic * compute tipset null blocks * add special case for epoch 0 * add car-explorer tool * fix epochs in a day hardcode * devnet first-class citizen * no need for an own car explorer * actor debugging flag for devnet * cargo update * temporary 2 atto for pledge * solidify devnet configuration handling * remove genesis pledge workaround * fix merge conflicts * use lockfile from main * fmt * fix compilation * wip : compose for devnet * working compose devnet * initial cleanup * fix parameters typo * more cleanup * cleanup forest after merge * ignore snapshots in Docker context * backport special case for genesis block * add some docs * bump * add network name workaround explanation * add changelog entry --------- Co-authored-by: David Himmelstrup <david.himmelstrup@chainsafe.io>
1 parent 6445bc1 commit 409ceb8

File tree

22 files changed

+561
-32
lines changed

22 files changed

+561
-32
lines changed

‎.dockerignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,8 @@ Dockerfile
55
*.md
66
target/
77
scripts/
8+
9+
# Ignore CAR files fetched to the project directory which tends to happen during development.
10+
# Without it, the Docker context may bloat to hundreds of gigabytes of data.
11+
/*.car
12+
/*.car.zst

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,8 @@
2929

3030
### Added
3131

32+
- [#2816](https://github.com/ChainSafe/forest/issues/2816): Support `2k` devnet.
33+
3234
- [#3026](https://github.com/ChainSafe/forest/pull/3026): Expose
3335
`forest-cli state diff ...`
3436

‎documentation/src/developer_documentation/devnet_notes.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
export LOTUS_SKIP_GENESIS_CHECK=_yes_
1010
export CGO_CFLAGS_ALLOW="-D__BLST_PORTABLE__"
1111
export CGO_CFLAGS="-D__BLST_PORTABLE__"
12+
# For MacOS: LIBRARY_PATH=/opt/homebrew/Cellar/hwloc/2.9.1/lib
1213
make 2k
1314
./lotus fetch-params 2048
1415
./lotus-seed pre-seal --sector-size 2KiB --num-sectors 2

‎scripts/devnet/.env‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
FOREST_DATA_DIR=/forest_data
2+
LOTUS_DATA_DIR=/lotus_data
3+
FILECOIN_PROOFS_DIR=/var/tmp/filecoin-proof-parameters
4+
LOTUS_RPC_PORT=1234
5+
LOTUS_P2P_PORT=1235
6+
MINER_RPC_PORT=2345
7+
FOREST_RPC_PORT=3456

‎scripts/devnet/README.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Local devnet setup
2+
3+
The devnet consists of a:
4+
5+
- Lotus miner,
6+
- Lotus node,
7+
- Forest node.
8+
9+
It's packed in a docker-compose setup for convenience and ease of usage. By
10+
default, running it will expose relevant RPC and P2P ports to the host:
11+
12+
- 1234 - Lotus RPC,
13+
- 9090 - Lotus P2P port,
14+
- 2345 - Miner RPC,
15+
- 3456 - Forest RPC.
16+
17+
## Running the devnet
18+
19+
Run it with:
20+
21+
```shell
22+
docker-compose up --build
23+
```
24+
25+
This will build the local Forest (using the Dockerfile in the project's root)
26+
image, tagged Lotus and setup the devnet. Initial setup may be slow due to
27+
fetching params and setting everyting up. Consecutive starts will be quick.
28+
29+
Stop the devnet with:
30+
31+
```shell
32+
docker-compose down
33+
```
34+
35+
Remove the devnet:
36+
37+
```shell
38+
docker-compose rm
39+
```
40+
41+
## Interacting with the devnet via CLI
42+
43+
Exec into the `forest` container:
44+
45+
```shell
46+
docker exec -it forest /bin/bash
47+
```
48+
49+
and setup credentials. Then run any command:
50+
51+
```shell
52+
export TOKEN=$(cat /forest_data/token.jwt)
53+
export FULLNODE_API_INFO=$TOKEN:/dns/forest/tcp/1234/http
54+
55+
forest-cli net peers
56+
```
57+
58+
## Local devnet development
59+
60+
If you prefer to have Forest running directly on the host, you can comment it
61+
out and draw inspiration from the `docker-compose.yml` on how to connect it to
62+
Lotus. In short, you will need to obtain the peer id, network name and the
63+
genesis file.

‎scripts/devnet/docker-compose.yml‎

Lines changed: 192 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,192 @@
1+
# Docker compose file to run a local devnet.
2+
# The devnet consists of a:
3+
# - Lotus node (2k build),
4+
# - Lotus miner (2k build),
5+
# - Forest node.
6+
7+
version: "3.8"
8+
9+
services:
10+
# Basic devnet initialisation. This will populate Lotus data volume with necessary artifacts
11+
# to run a devnet. The initialisation is a lengthy process and occurs only at the first
12+
# `docker-compose up`.
13+
lotus_init:
14+
build:
15+
context: .
16+
dockerfile: lotus.dockerfile
17+
volumes:
18+
- filecoin-proofs:${FILECOIN_PROOFS_DIR}
19+
- lotus-data:${LOTUS_DATA_DIR}
20+
env_file:
21+
- lotus.env
22+
entrypoint: ["/bin/bash", "-c" ]
23+
command:
24+
- |
25+
if [ ! -f ${LOTUS_DATA_DIR}/NODE_INITIALISED ]; then
26+
lotus fetch-params 2048
27+
lotus-seed --sector-dir ${LOTUS_DATA_DIR}/genesis-sectors pre-seal --sector-size 2KiB --num-sectors 2
28+
lotus-seed --sector-dir ${LOTUS_DATA_DIR}/genesis-sectors genesis new ${LOTUS_DATA_DIR}/localnet.json
29+
lotus-seed --sector-dir ${LOTUS_DATA_DIR}/genesis-sectors genesis add-miner ${LOTUS_DATA_DIR}/localnet.json ${LOTUS_DATA_DIR}/genesis-sectors/pre-seal-t01000.json
30+
touch ${LOTUS_DATA_DIR}/NODE_INITIALISED
31+
fi
32+
33+
# Lotus node that communicates with the miner and the Forest node.
34+
# The RPC and P2P listening ports are exposed so Forest does not necessarily need to be running inside Docker.
35+
# For development purposes it might be practical to comment out Forest part altogether and use a local build.
36+
lotus_node:
37+
depends_on:
38+
lotus_init:
39+
condition: service_completed_successfully
40+
build:
41+
context: .
42+
dockerfile: lotus.dockerfile
43+
healthcheck:
44+
test: >-
45+
curl -s -x post -h "content-type: application/json"
46+
--data '{ "jsonrpc": "2.0", "method": "filecoin.chainhead", "params": [], "id": 1 }'
47+
http://lotus_node:${LOTUS_RPC_PORT}/rpc/v0 || exit 1
48+
interval: 10s
49+
retries: 10
50+
timeout: 5s
51+
container_name: lotus
52+
networks:
53+
- devnet
54+
volumes:
55+
- filecoin-proofs:${FILECOIN_PROOFS_DIR}
56+
- lotus-data:${LOTUS_DATA_DIR}
57+
ports:
58+
- ${LOTUS_RPC_PORT}:${LOTUS_RPC_PORT}
59+
- ${LOTUS_P2P_PORT}:${LOTUS_P2P_PORT}
60+
env_file:
61+
- lotus.env
62+
entrypoint: ["/bin/bash", "-c" ]
63+
command:
64+
- |
65+
lotus daemon --lotus-make-genesis=${LOTUS_DATA_DIR}/devgen.car --genesis-template=${LOTUS_DATA_DIR}/localnet.json --bootstrap=false
66+
67+
# Lotus miner container. It communicates only with Lotus and not with Forest.
68+
lotus_miner:
69+
depends_on:
70+
lotus_node:
71+
condition: service_healthy
72+
build:
73+
context: .
74+
dockerfile: lotus.dockerfile
75+
container_name: lotus-miner
76+
healthcheck:
77+
test: >-
78+
curl -s http://lotus-miner:${MINER_RPC_PORT} || exit 1
79+
interval: 20s
80+
retries: 5
81+
start_period: 6000s
82+
timeout: 10s
83+
networks:
84+
- devnet
85+
volumes:
86+
- filecoin-proofs:${FILECOIN_PROOFS_DIR}
87+
- lotus-data:${LOTUS_DATA_DIR}
88+
ports:
89+
- ${MINER_RPC_PORT}:${MINER_RPC_PORT}
90+
env_file:
91+
- lotus-miner.env
92+
restart: on-failure # lotus node might not be ready
93+
entrypoint: ["/bin/bash", "-c" ]
94+
command:
95+
- |
96+
if [ ! -f ${LOTUS_DATA_DIR}/MINER_INITIALISED ]; then
97+
lotus wallet import --as-default ${LOTUS_DATA_DIR}/genesis-sectors/pre-seal-t01000.key
98+
lotus-miner init --genesis-miner --actor=t01000 --sector-size=2KiB --pre-sealed-sectors=${LOTUS_DATA_DIR}/genesis-sectors --pre-sealed-metadata=${LOTUS_DATA_DIR}/genesis-sectors/pre-seal-t01000.json --nosync
99+
touch ${LOTUS_DATA_DIR}/MINER_INITIALISED
100+
fi
101+
lotus-miner run --nosync
102+
103+
# This container dumps relevant (for Forest) Lotus configuration to the shared volume.
104+
lotus_config:
105+
depends_on:
106+
lotus_node:
107+
condition: service_healthy
108+
build:
109+
context: .
110+
dockerfile: lotus.dockerfile
111+
volumes:
112+
- lotus-data:${LOTUS_DATA_DIR}
113+
entrypoint: ["/bin/bash", "-c" ]
114+
env_file:
115+
- lotus.env
116+
networks:
117+
- devnet
118+
command:
119+
- lotus net id > ${LOTUS_DATA_DIR}/PEER_ID
120+
121+
forest:
122+
user: root
123+
depends_on:
124+
lotus_config:
125+
condition: service_completed_successfully
126+
build:
127+
context: ../../.
128+
dockerfile: Dockerfile
129+
container_name: forest
130+
healthcheck:
131+
test: |
132+
export TOKEN=$$(cat ${FOREST_DATA_DIR}/token.jwt)
133+
export FULLNODE_API_INFO=$$TOKEN:/ip4/127.0.0.1/tcp/${FOREST_RPC_PORT}/http
134+
forest-cli sync status || exit 1
135+
interval: 10s
136+
retries: 10
137+
timeout: 5s
138+
start_period: 15s
139+
volumes:
140+
- lotus-data:${LOTUS_DATA_DIR}
141+
- filecoin-proofs:${FILECOIN_PROOFS_DIR}
142+
- forest-data:${FOREST_DATA_DIR}
143+
- ./forest_config.toml.tpl:/forest/forest_config.toml.tpl
144+
environment:
145+
- FIL_PROOFS_PARAMETER_CACHE=${FILECOIN_PROOFS_DIR}
146+
networks:
147+
- devnet
148+
ports:
149+
- ${FOREST_RPC_PORT}:${FOREST_RPC_PORT}
150+
entrypoint: ["/bin/bash", "-c" ]
151+
command:
152+
- |
153+
set -eu
154+
if [ ! -f ${FOREST_DATA_DIR}/forest_config.toml ]; then
155+
export NETWORK_NAME=$$(grep -o \"localnet.*\" ${LOTUS_DATA_DIR}/localnet.json)
156+
cp /forest/forest_config.toml.tpl ${FOREST_DATA_DIR}/forest_config.toml
157+
echo name = $$NETWORK_NAME >> ${FOREST_DATA_DIR}/forest_config.toml
158+
fi
159+
forest --genesis ${LOTUS_DATA_DIR}/devgen.car --config ${FOREST_DATA_DIR}/forest_config.toml --save-token ${FOREST_DATA_DIR}/token.jwt --rpc-address 0.0.0.0:${FOREST_RPC_PORT}
160+
161+
# At the moment of writing, Forest was not able to connect to a devnet node using its config.
162+
# This is a workaround to force the connection.
163+
forest_connecter:
164+
user: root
165+
depends_on:
166+
forest:
167+
condition: service_healthy
168+
build:
169+
context: ../../.
170+
dockerfile: Dockerfile
171+
volumes:
172+
- lotus-data:${LOTUS_DATA_DIR}
173+
- forest-data:${FOREST_DATA_DIR}
174+
networks:
175+
- devnet
176+
entrypoint: ["/bin/bash", "-c" ]
177+
command:
178+
- |
179+
set -eu
180+
export TOKEN=$$(cat ${FOREST_DATA_DIR}/token.jwt)
181+
export FULLNODE_API_INFO=$$TOKEN:/dns/forest/tcp/${FOREST_RPC_PORT}/http
182+
forest-cli net connect /dns/lotus/tcp/${LOTUS_P2P_PORT}/p2p/$$(cat ${LOTUS_DATA_DIR}/PEER_ID)
183+
184+
volumes:
185+
# Shared proof parameter files. It is re-used by both Lotus and Forest.
186+
filecoin-proofs:
187+
# Lotus node, miner and network data.
188+
lotus-data:
189+
forest-data:
190+
191+
networks:
192+
devnet:

‎scripts/devnet/forest_config.toml.tpl‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
[client]
2+
encrypt_keystore = false
3+
data_dir = "/forest_data"
4+
5+
[network]
6+
kademlia = false
7+
target_peer_count = 1
8+
9+
# Note that this has to come last. The actual TOML file will have
10+
# the chain name appended.
11+
[chain.network]
12+
type = "devnet"

‎scripts/devnet/lotus-miner.env‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
LOTUS_PATH=/lotus_data/lotus-local-net
2+
LOTUS_MINER_PATH=/lotus_data/lotus-miner-local-net
3+
LOTUS_SKIP_GENESIS_CHECK=_yes_
4+
LOTUS_API_LISTENADDRESS=/dns/lotus/tcp/2345/http
5+
LOTUS_API_LISTENADDRESS=/dns/lotus-miner/tcp/2345/http

‎scripts/devnet/lotus.dockerfile‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Lotus binaries image, to be used in the local devnet with Forest.
2+
FROM golang:1.19.7-buster AS lotus-builder
3+
4+
ARG LOTUS_TAG=v1.23.0
5+
6+
RUN apt-get update && apt-get install -y ca-certificates build-essential clang ocl-icd-opencl-dev ocl-icd-libopencl1 jq libhwloc-dev
7+
8+
WORKDIR /lotus
9+
RUN git clone --depth 1 --branch ${LOTUS_TAG} https://github.com/filecoin-project/lotus.git .
10+
RUN CGO_CFLAGS_ALLOW="-D__BLST_PORTABLE__" \
11+
CGO_CFLAGS="-D__BLST_PORTABLE__" \
12+
make 2k
13+
14+
FROM ubuntu:22.04
15+
16+
# Needed for the healthcheck
17+
RUN apt-get update && apt-get install -y curl
18+
19+
# Need to copy the relevant shared libraries from the builder image.
20+
# See https://github.com/filecoin-project/lotus/blob/master/Dockerfile
21+
COPY --from=lotus-builder /etc/ssl/certs /etc/ssl/certs
22+
COPY --from=lotus-builder /lib/*/libdl.so.2 /lib/
23+
COPY --from=lotus-builder /lib/*/librt.so.1 /lib/
24+
COPY --from=lotus-builder /lib/*/libgcc_s.so.1 /lib/
25+
COPY --from=lotus-builder /lib/*/libutil.so.1 /lib/
26+
COPY --from=lotus-builder /usr/lib/*/libltdl.so.7 /lib/
27+
COPY --from=lotus-builder /usr/lib/*/libnuma.so.1 /lib/
28+
COPY --from=lotus-builder /usr/lib/*/libhwloc.so.5 /lib/
29+
COPY --from=lotus-builder /usr/lib/*/libOpenCL.so.1 /lib/
30+
31+
# Copy only the binaries relevant for the devnet
32+
COPY --from=lotus-builder /lotus/lotus /lotus/lotus-miner /lotus/lotus-seed /usr/local/bin/
33+
34+
WORKDIR /lotus
35+
36+
CMD ["/bin/bash"]

‎scripts/devnet/lotus.env‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
LOTUS_PATH=/lotus_data/lotus-local-net
2+
LOTUS_MINER_PATH=/lotus_data/lotus-miner-local-net
3+
LOTUS_SKIP_GENESIS_CHECK=_yes_
4+
LOTUS_API_LISTENADDRESS=/dns/lotus/tcp/1234/http
5+
LOTUS_LIBP2P_LISTENADDRESSES=/ip4/0.0.0.0/tcp/1235

0 commit comments

Comments
 (0)