Containerize

[victor-rays]

Containerize NCP

A project that started from a brainstorming in the Matrix Wiki chat room.
First post & preliminary information and all that sort of stuff 🙏

A very much Work In Progress, confirming and testing if a design idea could indeed work as thought-out theoretically

Project idea

Convert NCP and it's tool into something like a "binary" application container (or containers that only do "one thing/task") and services capable of being integrated with others, also making it possible to update/upgrade parts of the whole instead of everything.

Where ncp-config is the master container over the others, and this image can then be used as a service.

End goal

Containerize NCP completely

Starting point & proof-of-concept

Convert ncp-config's various scripts into individual containers & ncp-config to a container as well, being used as the master container, to control the others

Edit: To use one container, a bash control script (maybe?) called ncp-tools, or something, is entrypoint, possibly install it as plugin or only nc-encrypt which needs admin permissions. And put all the ncp script tools into one container directly using a bash script as a controller with case checking (?) for the different parts inside the container. Which right now seems to be the better option, but I don't know 🙏

Then combine that with nextcloud-aio, Nextcloud, PHP, mariaDB or a database and Caddy as front-end or reverse proxy, which is how I've used Caddy the most (reverse-proxy). Anyone have any other ideas?

• Category re-design/re-structuring (?)

• New category suggestion

  • BACKUP
  • NETWORK
  • SYSTEM
  • UPDATE

Status

• Stopped
• Not started
• Not continuing
• Researching
• Testing
• Ongoing
• Paused °
• Completed

° As I'm finishing my undergraduate degree at the moment this is currently paused

TODO

1. • Added a few relevant help articles, for basic understanding around the subject of the project.
2. • Added some more relevant help articles from the Docker documentation, can be really hard to find otherwise.
3. • Add links and script names to the categories for ncp-config until completed
4. • Expand explanations (partly done)
5. • Begin research
6. • Begin testing
7. • What else? ..

Related Help articles & Documentation information

Google - Best practice, Building containers
Google - Best practice, Operating containers
Docker - Best practice, Dockerfile
Docker - Best practice, Development
Docker - Best practice, Image-building
Docker - Build enhancements
Docker - Choosing a build driver
Docker - Manage images
Docker - Create a base image

Docker - Multi-container apps
Docker - Update the application
Docker - Packaging your software
Docker - Multi-stage builds
Docker - Compose, Overview
Docker - Reference, run command
Docker - Specify a Dockerfile

Docker - Announcement, Compose V2

Red Hat Dev - Blog Post, Systemd in Containers

Docker docs, Deprecated Features

Notes

Docker Hub, Nextcloudpi

Docker docs, IPv6 Support

A Nextcloud instance's directories to restore settings.

1. Config
2. Database
3. Data (User files & App data (?))

Commands to get IP-addresses in the terminal

# INTERNAL IP-ADDRESS
# IPv4 - String manipulation
"$(ip addr | grep 192 | awk '{print $2}' | cut -b 1-14)"

# IPv4 & IPv6 - String manipulation
ip a | grep "scope global" | awk '{print $2}' | head -2 | sed 's|/.*||g'

# IPv4, IPv6 & Link-local - JSON
ip -j address | jq '.[2].addr_info' | jq '.[].local'

# Without quotes - JSON
ip -j address | jq '.[2].addr_info' | jq -r '.[].local'

# IPv4 - JSON
ip -j address | jq '.[2].addr_info' | jq -r '.[0].local'

# IPv6 - JSON
ip -j address | jq '.[2].addr_info' | jq -r '.[1].local'

# Link-local - JSON
ip -j address | jq '.[2].addr_info' | jq -r '.[2].local'

# PUBLIC IP ADDRESS
# IPv4
curl -sL -m4 -4 https://icanhazip.com
# IPv6
curl -sL -m4 -6 https://icanhazip.com

Docker Context

Docker docs, Manage contexts

Docker Buildx

docker buildx build . \
--file /path/Dockerfile \
--tag ${OWNER}/${REPO}:${TAG}
# In this context it's regarding the docker hub
# Owner, Repo & Tag @DockerHub

Options

• --platform
  • Architecture(s) for the image
• --builder
• --push
• --build-arg

Create builder

docker buildx create --use \
--name container \
--driver docker-container \
--platform linux/arm64,linux/amd64,linux/armhf

Docker Driver

• docker
• docker-container Recommended for multiple architecture compatibility
• kubernetes

Orchestration

• Docker Swarm Default
• Kubernetes Deprecated in stack & context @v20.10 Source

Docker Compose

Docker docs, Compose extend services
Docker docs, Compose networking
Docker docs, Compose in production
Docker docs, Compose V2 compatibility
Docker docs, Compose FAQ

Old syntax - V1

• docker-compose

New syntax - V2

• docker compose

Ex. docker-compose.yml

services:
  nextcloudpi:
    command: "$(ip addr | grep 192 | awk '{print $2}' | cut -b 1-14)"
    container_name: nextcloudpi
    image: ownyourbits/nextcloudpi:latest
    ports:
    - published: 80
      target: 80
    - published: 443
      target: 443
    - published: 4443
      target: 4443
    restart: unless-stopped
    volumes:
    - ncdata:/data:ro
    - /etc/localtime:/etc/localtime:ro
version: '3.3'
volumes:
  ncdata:
    external: true

Docker Run

A working docker run command with the --init flag for PID 1 management and reaping of zombie processes.

docker run --init \
--publish 4443:4443 \
--publish 443:443 \
--publish 80:80 \
--volume ncdata:/data \
--name nextcloudpi \
--detach ownyourbits/nextcloudpi:latest \
"$(ip addr | grep 192 | awk '{print $2}' | cut -b 1-14)"

• "$(ip addr | grep 192 | awk '{print $2}' | cut -b 1-14)"

Greps an IP-address beginning with 192, modify to fit your system, test in terminal.

See "Commands to get IP-addresses in the terminal" above for other examples.

Nextcloud AIO

Used as example and reference

Docker Run AIO arm64

sudo docker run \
--sig-proxy=false \
--name nextcloud-aio-mastercontainer \
--restart always \
--publish 80:80 \
--publish 8080:8080 \
--publish 8443:8443 \
--volume nextcloud_aio_mastercontainer:/mnt/docker-aio-config \
--volume /var/run/docker.sock:/var/run/docker.sock:ro \
nextcloud/all-in-one:latest-arm64

Reverse proxy AIO

Dockerfile

Docker docs, Dockerfile reference

Naming scheme

• Dockerfile.name

Instead of using the ARG example below and getting each individual script.
Use alpine-git image instead and clone repo, alternatively clone it beforehand

ADD can be used in Dockerfile to import scripts

• ADD ${URL} ${PATH}

URL to fetch scripts in raw text

• https://raw.githubusercontent.com/
• Ex. https://raw.githubusercontent.com/${OWNER}/${REPO}/${BRANCH}/${PATH}

Ex. Docker ARG

ARG	DESCRIPTION
OWNER	Repository owner @ GitHub
REPO	Repository @ GitHub
BRANCH	Branch of repository @ GitHub
PATH	Path to the script directory
CATEGORY	Category in /bin/ncp (PATH)
PATH_BASH	Path to bash binary
URL	GH URL to get scripts in raw text

ARG Example

ARG OWNER ["nextcloud"]
ARG REPO ["nextcloudpi"]
ARG BRANCH ["master"]
ARG PATH ["bin/ncp"]
ARG CATEGORY ["BACKUPS"]
ARG SCRIPT ["nc-backup-auto.sh"]
ARG URL ["https://raw.githubusercontent.com"]
ARG PATH_BASH ["/usr/local/bin/bash"]

ADD ["${URL}/${OWNER}/${REPO}/${BRANCH}/${PATH}/${CATEGORY}/${SCRIPT}","${PATH}/${CATEGORY}/${SCRIPT}"]
COPY --from=bash ["$PATH_BASH", "$PATH_BASH"]
RUN ["$PATH_BASH","-c","chmod +x ${PATH}/${CATEGORY}/${SCRIPT}"]
SHELL ["$PATH_BASH"]
ENTRYPOINT ["$PATH_BASH","-c","${PATH}/${CATEGORY}/${SCRIPT}"]

Existing Containers

• Nextcloud
• Nextcloud AIO
• Linuxserver.io/Nextcloud
• MariaDB
• MySQL
• PHP
• Debian
• Alpine
• Bash
• Curl
• Apache2
• Caddy

Dockerized Bash Scripts - Examples

1. Transforming Bash Script to Docker Compose
2. Automatic Docker Container creation w/bash script
3. Docker w/Shell script or Makefile
4. Run scripts, Docker arguments
5. Run a scripts inside Docker container using Shell script
6. Run Script, with dev docker image

---

Scripts, Dependencies & Packages

IMPORTANT

Script shebang must be #!/usr/bin/env bash and not #!/bin/bash, to be compatible with the bash docker image natively.

Notes
There are a few main things that are important to note regarding this image:

Bash itself is installed at /usr/local/bin/bash, not /bin/bash, so the recommended shebang is #!/usr/bin/env bash, not #!/bin/bash (or explicitly running your script via bash /.../script.sh instead of letting the shebang invoke Bash automatically). The image does not include /bin/bash, but if it is installed via the package manager included in the image, that package will install to /bin/bash and might cause confusion (although /usr/local/bin is ahead of /bin in $PATH, so as long as plain bash or /usr/bin/env are used consistently, the image-provided Bash will be preferred).

Bash is the only thing included, so if your scripts rely on external tools (such as jq, for example), those will need to be added manually (via apk add --no-cache jq, for example).

Nestybox & Sysbox

Sysbox, an open-source runc, it's project was acquired by Docker, Inc. and they help solve the user permissions issue (mapping of user id) inside the docker containers

Nestybox website

Quote from Sysbox GitHub page

Sysbox solves problems such

Enhancing the isolation of containerized microservices (root in the container maps to an unprivileged user on the host).

Enabling a highly capable root user inside the container without compromising host security.

Securing CI/CD pipelines by enabling Docker-in-Docker (DinD) or Kubernetes-in-Docker (KinD) without insecure privileged containers or host Docker socket mounts.

Enabling the use of containers as "VM-like" environments for development, local testing, learning, etc., with strong isolation and the ability to run systemd, Docker, IDEs, and more inside the container.

Running legacy apps inside containers (instead of less efficient VMs).

Replacing VMs with an easier, faster, more efficient, and more portable container-based alternative, one that can be deployed across cloud environments easily.

Partitioning bare-metal hosts into multiple isolated compute environments with 2X the density of VMs (i.e., deploy twice as many VM-like containers as VMs on the same hardware at the same performance).

Partitioning cloud instances (e.g., EC2, GCP, etc.) into multiple isolated compute environments without resorting to expensive nested virtualization.

---

Packages in Docker environment/build

Docker Packages

DOCKER PACKAGES
git	bash

Extraction of the different environment variables, dependencies on/in other scripts & their dependencies in turn and which packages are required together with their location.

File & location

File	Repository	Installed	Dependencies
library.sh	/etc/library.sh	/usr/local/etc/library.sh	$ncc,$ARCH,$NCPCFG,$CFGDIR,$BINDIR,$NCDIR
ncc	/bin/ncc	/usr/local/bin/ncc	occ,$NCDIR
ncp.cfg	/etc/ncp.cfg	/usr/local/etc/ncp.cfg	-
occ	-	/var/www/nextcloud/	$NCDIR

Environment variables

ENVIRONMENT VARIABLE	VALUE
$ncc	/usr/local/bin/ncc
$CFGDIR	/usr/local/etc/ncp-config.d/
$BINDIR	/usr/local/bin/ncp/
$NCDIR	/var/www/nextcloud/
$NCPCFG	"${NCPCFG:-etc/ncp.cfg}"
$ARCH	"$(dpkg --print-architecture)"
$DESTDIR	``
$INCLUDEDATA	``
$COMPRESS	``
ncp-tools:
$BACKUPLIMIT	``
$BACKUPDAYS	``
$NCLATESTVER	$(jq -r .nextcloud_version < "$NCPCFG")
$PHPVER	$(jq -r .php_version < "$NCPCFG")
$RELEASE	$(jq -r .release < "$NCPCFG")
$NEXTCLOUD_URL	https://localhost sudo -E -u www-data "/var/www/nextcloud/apps/notify_push/bin/${ARCH}/notify_push" --allow-self-signed /var/www/nextcloud/config/config.php &>/dev/null &

Packages

	PACKAGES
dpkg	bash	jq
apt	dialog	cat
awk	mktemp	sudo

Users

USERS
www-data

Permissions

PERMISSIONS
sudo

---

[sunjam]

Do you want to also make a linked copy of this on the forum for wiki collab and further discussion?

[victor-rays]

Hmm ... I don't know 🤔

I don't mind if you want to link to it in the forum to start a discussion as such, I just hope it won't be regarded as a promise or anything like that :D it's still very early as I just started writing this up and I don't know how far it'll go, how long it'll take or even if it'll all work out in the end fully yet. But if that is understood (or those expectations are set, to say it in another way), then I don't mind at all, you can go ahead and link it in the forum 🙏

[sunjam]

Great, in that case we'll discuss things on the forum if it makes sense. Just thinking of ways to keep collaborating on this together. Thanks for writing this up!

[victor-rays]

Moving this to a readme instead within the branch

https://github.com/nextcloud/nextcloudpi/tree/containerize/build/docker/containerize

[szaimen]

Hello, just had a look and the idea sounds interesting!

A possible approach that I could imagine would be building Nextcloudpi in the future around AIO which would allow to split-up the maintenance work. You could simply run AIO in reverse proxy mode and implement further features then that NextlcoudPi currently offers, but AIO can't. Going the same path like AIO and introducing a mastercontainer that manages other docker containers would make NextcloudPi easier to maintain but would introduce some limits regarding what NextcloudPis mastercontainer can do so it should be carefully discussed.

Also as a sidenote: modularization using docker containers is a good approach for maintainability but creating ~50 docker containers that need to get mounted and run in a specific way does not make it more maintainable.

[theCalcaholic]

@szaimen I did not have time yet to do a write up on my ideas and criteria regarding this topic, but I agree - my vision encompasses just one container that runs ncp tasks and overwrites the entry point or command for the various scripts and one or two othwr containers for the web interface and maybe a cron-like service (we'll have to see what's a good approach here). And then of course we need the kind of services that AIO contains (or maybe just integrate with AIO). :)

Some thoughts I have about this (a complete writeup will follow):

Generally, I want to explore how well we can abstract script environments that way to generalize them for testing.
E. g. a script can always operate on fixed paths if running and docker and selecting the correct part could be implemented by how the container is run - effectively making scripts more agnostic to the system environment, which could make automated tests a lot easier and more powerful.

A big challenge will be to provide our users with working upgrade paths (seemless upgrades from both, the current ncp docker container as well as bare metal installations to a multi-container setup) - I'm actually not sure yet, if it will be feasible at all, but I will at least explore that

[szaimen]

A big challenge will be to provide our users with working upgrade paths (seemless upgrades from both, the current ncp docker container as well as bare metal installations to a multi-container setup) - I'm actually not sure yet, if it will be feasible at all, but I will at least explore that

Just discussed this with @st3iny and came to the conclusion that this should indeed be carefully considered 👍