Skip to content

Commit 451a380

Browse files
jbaptperezjbaptperez
committed
Adds a new Compose form development
Compose standard changes are: - Removes the deprecated "version" field, - Adds a toplevel "name" field (prefix of container names nad network), - Adds a toplevel "network" field, with a common "timesketch-dev" network, - Removes container names (depends on and toplevel name and service names), - Do not bind to the 127.0.0.1 interface only (0.0.0.0), - Removes useless "links" (common network), - Refactors environment variables to don't use a YAML array, - Removes "restart" fields to detect undesired crashes in development, - Binds ports of other services to the host (opensearch, redis). General changes are: - Allows Docker image builds in a restricted company context (limited access to remote Ubuntu, Python or Node repositories) using variables, - Centralizes variables in a .env file (not versioned), - Adds a .env.template file as .env template with predefined variables, - Use a distinct directory for every service dependencies, - Use named volumes for portability and to avoid auto-creation of anonymous ones (PostgreSQL, Redis and Prometheus declare volumes in their Dockerfile; this leads to anonymous volume creations if they are not declared in Compose), - Use a per-service environment file, - Simplifies how development configuration files are transferred to Timesketch, - Simplifies manipulation of containers using Compose CLI instead of the Docker one, - Simplify and optimizes the Timesketch entrypoint, - Updates the Bash scripts to start frontend-ng, - Updates related documentation.
1 parent 657d8a4 commit 451a380

25 files changed

+1246
-0
lines changed

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,3 +53,6 @@ vagrant/*.log
5353

5454
# Exclude JetBrains IDE files
5555
.idea/
56+
57+
# Exclude Docker Compose .env file
58+
.env

‎contrib/docker/dev/.env.template‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
GIFT_PPA_TRACK="stable"
2+
GIFT_PPA_URL="https://ppa.launchpadcontent.net/gift/${GIFT_PPA_TRACK}/ubuntu"
3+
NODE_VERSION="18.x"
4+
NODE_PPA_URL="https://deb.nodesource.com/node_${NODE_VERSION}"
5+
NODE_NPMRC=""
6+
YARN_YARNRC=""
7+
PYTHON_PIP_CONF=""
8+
9+
TIMESKETCH_BASE_IMAGE="ubuntu:22.04"
10+
TIMESKETCH_CONF_DIR="/etc/timesketch"
11+
TIMESKETCH_SECRET_KEY="L4np0jV3yAdAFdbVzWRMaBqiFMV8FKYd+Je1WKE40o8="
12+
TIMESKETCH_USER="dev"
13+
TIMESKETCH_PASSWORD="dev"
14+
15+
POSTGRES_USER="timesketch"
16+
POSTGRES_PASSWORD="password"

‎contrib/docker/dev/README.md‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
## Docker for development
2+
3+
You can run Timesketch on Docker in development mode.
4+
Make sure to follow the docker [post-install](https://docs.docker.com/engine/install/linux-postinstall/) to run without superuser. If not then make sure to execute all `docker` commands here as *superuser*.
5+
6+
NOTE: It is not recommended to try to run on a system with less than 8 GB of RAM.
7+
8+
### Prepare a .env file
9+
10+
Compose requires a `.env` file with top level environment variables to be set.
11+
To create it, just copy the `.env.template` file as a base.
12+
13+
```bash
14+
cp .env.template .env
15+
```
16+
17+
Note the `.env` is ignored by Git: you can safely write sensitive data in it.
18+
19+
You can optionally edit the `.env` file.
20+
This is useful if you need to build images with some company restrictions (accessing
21+
remote Ubuntu, PyPI or Node repositories).
22+
23+
### Start a developer version of docker containers in this directory
24+
25+
```bash
26+
docker compose up -d
27+
```
28+
29+
The provided container definition runs Timesketch in development mode as a volume from your cloned repo. Any changes you make will appear in Timesketch automatically.
30+
31+
If you see the following message you can continue
32+
33+
```text
34+
Timesketch development server is ready!
35+
```
36+
37+
### Start a celery container shell
38+
39+
Start the container in foreground (add `-d` to run in background):
40+
41+
```bash
42+
docker compose exec timesketch \
43+
celery \
44+
-A timesketch.lib.tasks \
45+
worker \
46+
--loglevel info
47+
```
48+
49+
### Start development webserver (and metrics server)
50+
51+
Start the container in foreground (add `-d` to run in background):
52+
53+
```bash
54+
docker compose exec timesketch \
55+
gunicorn \
56+
--reload \
57+
-b 0.0.0.0:5000 \
58+
--log-file - \
59+
--timeout 600 \
60+
-c /usr/local/src/timesketch/data/gunicorn_config.py \
61+
timesketch.wsgi:application
62+
```
63+
64+
You now can access your development version at http://127.0.0.1:5000/
65+
66+
Log in with user: dev password: dev
67+
68+
You can also access a metrics dashboard at http://127.0.0.1:3000/
69+
70+
### Non-interactive
71+
72+
A script applies the previous commands in background for you.
73+
74+
```bash
75+
docker compose up -d
76+
./start-frontend-ng-no-dev.sh
77+
```
78+
79+
A second script starts an additional development server for the frontend
80+
(http://127.0.0.1:5001/).
81+
You need to wait a few seconds before accessing it.
82+
83+
```bash
84+
docker compose up -d
85+
./start-frontend-ng-dev.sh
86+
```
87+
88+
### Run tests
89+
90+
```bash
91+
docker compose exec \
92+
-w /usr/local/src/timesketch \
93+
-it \
94+
timesketch \
95+
python3 run_tests.py --coverage
96+
```
97+
98+
That will run all tests in your docker container. It is recommended to run all tests at least before creating a pull request.
99+
100+
### Jupyter Notebook
101+
102+
To access a Jupyter notebook that has access to the Timesketch development
103+
environment start a browser and visit http://localhost:8844/ . The password to
104+
gain access is "timesketch".
105+
106+
By default, the /tmp directory is mapped as the data directory to store all
107+
notebooks. To change that, modify the line:
108+
109+
```yaml
110+
- /tmp/:/usr/local/src/picadata/
111+
```
112+
113+
in the _compose.yaml_ file to point to a directory of your choosing.
114+
In order for the jupyter notebook to be able to make use of that folder it has
115+
to have read and write permission for the user with the UID 1000.
116+
117+
By default, the latest checked in code of the timesketch API client and
118+
timesketch import client are installed. In order to install a new version, if
119+
you are modifying the clients you'll need to make sure that the timesketch
120+
source code on your machine is readable by the user with the UID 1000 and
121+
gid 1000.
122+
If that is done, then the code is mapped into the `/usr/local/src/timesketch`
123+
folder on the docker container.
124+
125+
New versions of timesketch api client can then be installed using:
126+
127+
```bash
128+
!pip install -e /usr/local/src/timesketch/api_client/python/
129+
```
130+
131+
And the importer client:
132+
133+
```bash
134+
!pip install -e /usr/local/src/timesketch/importer_client/python
135+
```
136+
137+
Just remember to restart the kernel runtime in order for the changes to be
138+
active.
139+
140+
To update the docker image run:
141+
142+
```bash
143+
$ sudo docker image pull us-docker.pkg.dev/osdfir-registry/timesketch/notebook:latest
144+
```

‎contrib/docker/dev/compose.yaml‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
name: timesketch-dev
2+
3+
networks:
4+
timesketch-dev:
5+
6+
volumes:
7+
opensearch-data:
8+
postgres-data:
9+
redis-data:
10+
prometheus-data:
11+
12+
services:
13+
timesketch:
14+
image: us-docker.pkg.dev/osdfir-registry/timesketch/dev:latest
15+
build:
16+
context: ../../..
17+
dockerfile: contrib/docker/dev/timesketch/Dockerfile
18+
args:
19+
BASE_IMAGE: "${TIMESKETCH_BASE_IMAGE}"
20+
GIFT_PPA_TRACK: "${GIFT_PPA_TRACK}"
21+
GIFT_PPA_URL: "${GIFT_PPA_URL}"
22+
NODE_VERSION: "${NODE_VERSION}"
23+
NODE_PPA_URL: "${NODE_PPA_URL}"
24+
NODE_NPMRC: "${NODE_NPMRC}"
25+
YARN_YARNRC: "${YARN_YARNRC}"
26+
PYTHON_PIP_CONF: "${PYTHON_PIP_CONF}"
27+
command: timesketch
28+
ports:
29+
- "5000:5000"
30+
- "5001:5001"
31+
- "8080:8080"
32+
env_file:
33+
- timesketch/timesketch.env
34+
volumes:
35+
- "../../../:/usr/local/src/timesketch/"
36+
- "./timesketch/timesketch.conf:${TIMESKETCH_CONF_DIR}/timesketch.conf:ro"
37+
- "./timesketch/sigma_rules.txt:${TIMESKETCH_CONF_DIR}/sigma_rules.txt:ro"
38+
- "../../../data/regex_features.yaml:${TIMESKETCH_CONF_DIR}/regex_features.yaml:ro"
39+
- "../../../data/winevt_features.yaml:${TIMESKETCH_CONF_DIR}/winevt_features.yaml:ro"
40+
- "../../../data/tags.yaml:${TIMESKETCH_CONF_DIR}/tags.yaml:ro"
41+
- "../../../data/intelligence_tag_metadata.yaml:${TIMESKETCH_CONF_DIR}/intelligence_tag_metadata.yaml:ro"
42+
- "../../../data/plaso.mappings:${TIMESKETCH_CONF_DIR}/plaso.mappings:ro"
43+
- "../../../data/generic.mappings:${TIMESKETCH_CONF_DIR}/generic.mappings:ro"
44+
- "../../../data/ontology.yaml:${TIMESKETCH_CONF_DIR}/ontology.yaml:ro"
45+
- "../../../data/data_finder.yaml:${TIMESKETCH_CONF_DIR}/data_finder.yaml:ro"
46+
- "../../../data/bigquery_matcher.yaml:${TIMESKETCH_CONF_DIR}/bigquery_matcher.yaml:ro"
47+
- "../../../data/sigma_config.yaml:${TIMESKETCH_CONF_DIR}/sigma_config.yaml:ro"
48+
- "../../../data/sigma:${TIMESKETCH_CONF_DIR}/sigma:ro"
49+
- "../../../data/dfiq:${TIMESKETCH_CONF_DIR}/dfiq:ro"
50+
- "../../../data/context_links.yaml:${TIMESKETCH_CONF_DIR}/context_links.yaml:ro"
51+
- "../../../data/plaso_formatters.yaml:${TIMESKETCH_CONF_DIR}/plaso_formatters.yaml:ro"
52+
- "../../../data/nl2q:${TIMESKETCH_CONF_DIR}/nl2q:ro"
53+
- "../../../data/llm_summarize:${TIMESKETCH_CONF_DIR}/llm_summarize:ro"
54+
depends_on:
55+
- opensearch
56+
- postgres
57+
- redis
58+
networks:
59+
- timesketch-dev
60+
61+
opensearch:
62+
image: opensearchproject/opensearch:2.15.0
63+
env_file:
64+
- opensearch/opensearch.env
65+
ports:
66+
- "9200:9200"
67+
volumes:
68+
- "opensearch-data:/usr/share/opensearch/data"
69+
networks:
70+
- timesketch-dev
71+
ulimits:
72+
memlock:
73+
soft: -1
74+
hard: -1
75+
nofile:
76+
soft: 65536
77+
hard: 65536
78+
79+
postgres:
80+
image: postgres:13.1-alpine
81+
env_file:
82+
- postgresql/postgresql.env
83+
ports:
84+
- "5432:5432"
85+
volumes:
86+
- "postgres-data:/var/lib/postgresql/data"
87+
networks:
88+
- timesketch-dev
89+
90+
redis:
91+
image: redis:6.0.10-alpine
92+
ports:
93+
- "6379:6379"
94+
volumes:
95+
- "redis-data:/data"
96+
networks:
97+
- timesketch-dev
98+
99+
notebook:
100+
image: us-docker.pkg.dev/osdfir-registry/timesketch/notebook:latest
101+
build:
102+
context: ../../..
103+
dockerfile: contrib/docker/dev/notebook/Dockerfile
104+
args:
105+
PYTHON_PIP_CONF: "${PYTHON_PIP_CONF}"
106+
ports:
107+
- "8844:8844"
108+
volumes:
109+
- "../../../:/usr/local/src/timesketch/:ro"
110+
- "/tmp/:/usr/local/src/picadata/"
111+
depends_on:
112+
- opensearch
113+
networks:
114+
- timesketch-dev
115+
116+
prometheus:
117+
image: prom/prometheus:v2.24.1
118+
volumes:
119+
- "./prometheus:/etc/prometheus:ro"
120+
- "prometheus-data:/prometheus"
121+
ports:
122+
- "9090:9090"
123+
command: --config.file=/etc/prometheus/prometheus.yml
124+
depends_on:
125+
- timesketch
126+
networks:
127+
- timesketch-dev

0 commit comments

Comments
 (0)