K8s Deployment Configuration

[trisongz]

Hi! I've been playing around with deploying fief onto a kubernetes cluster today after running it on docker and have been trying to figure out a few things, specifically around the ingress for things to work.

I had made a few modifications to the boot-up process to remove requiring the user to interact with the container.

---

# run_init.py

...
_donefile = Path('/data/db/.init_run')

async def run_app_startup():
    if _donefile.exists():
        typer.secho('Init already completed!', fg = 'green')
        return
    # allow for migration to take place
    time.sleep(15)
    typer.secho('Running Init!', fg = 'green')
    try:
        await create_main_fief_workspace()
        typer.echo("Main Fief workspace created")
    except MainWorkspaceAlreadyExists as e:
        typer.echo("Main Fief workspace already exists")

    main_user = os.getenv('MAIN_USER_EMAIL', 'ts@....')
    main_pass = os.getenv('MAIN_USER_PASS')

    if not main_pass:
        main_pass = str(uuid4())
        typer.echo(f'Generating User Pass: {main_pass}')
    try:
        await create_main_fief_user(main_user, main_pass)
        typer.echo(f"Main Fief user {main_user} created")
    except CreateMainFiefUserError as e:
        typer.echo("An error occured")

    typer.echo('Completed Fief Startup')
    _donefile.write_text('completed')
if __name__ == '__main__':
    anyio.run(run_app_startup)

# start.sh

#!/bin/sh

# this part was later added after failing to set proper vars.

if [ "$RUN_K8S" == "true" ]; then
    echo "Setting Env Vals"
    export FIEF_BASE_URL="http://$HOSTNAME:$PORT"
    export FIEF_DOMAIN="$HOSTNAME:$PORT"
    echo "FIEF_BASE_URL = $FIEF_BASE_URL"
    echo "FIEF_DOMAIN = $FIEF_DOMAIN"
fi

nohup python /run_init.py &
/usr/bin/supervisord -c /etc/supervisord.conf

---

Like this issue originally mentioned, whenever accessing /admin/ it returns a 500 error. My hunch is that it has something to do with the ingress-controller not passing through the proper headers.

Settings that have been attempted:

ROOT_DOMAIN=v1.domain.com
# ROOT_DOMAIN=fief-server.kops.svc:8000
# ROOT_DOMAIN=auth.v1.domain.com:80
# ROOT_DOMAIN=auth.v1.domain.com:443

FIEF_DOMAIN=auth.v1.domain.com
# FIEF_DOMAIN=fief-server.kops.svc:8000
# FIEF_DOMAIN=auth.v1.domain.com
# FIEF_DOMAIN=auth.v1.domain.com:80

# FIEF_BASE_URL=https://auth.v1.domain.com
# FIEF_BASE_URL=http://fief-server.kops.svc:8000
# FIEF_BASE_URL=http://auth.v1.domain.com:800

ALLOW_ORIGIN_REGEX=https://*
# all _COOKIE_SECURE both False/True

Some permutations look like they're working when they're set to fief-server.kops.svc:8000 as the 500 doesn't appear, and the url has the client_id and secret, but when the redirect kicks in, it obvs doesn't work since the url is in-cluster only.

---

In addition, I've also played with trying to add hostAliases to match the add-host requirement from docker, and it doesn't seem to have any effect.

# deployment.yaml

spec:
      hostAliases:
      - ip: "127.0.0.1"
        hostnames:
        - "auth.v1.domain.com"
        - "fief-server.kops.svc"
      # IP Address of Ingress Controller that is forwarding requests
      - ip: "10.0.45.151" 
        hostnames:
        - "localhost"
        - "auth.v1.domain.com"
        - "fief-server.kops.svc"

---

As for ingress-controllers, I've tried using both traefik and nginx with and without TLS.

Any tips would be appreciated!

[frankie567]

Hi @trisongz 👋 Welcome to Fief!

First of all, I would like to say that I'm not very versed in the arts of Kubernetes... But I'll try to help as much as I can!

The key thing to understand is that your main workspace should be accessible both from the inside and the outside with the same host. Fief determines the right workspace to use following the Host header.

I think your best bet is to stick with the "public domain":

ROOT_DOMAIN=v1.domain.com
FIEF_DOMAIN=auth.v1.domain.com
FIEF_BASE_URL=https://auth.v1.domain.com

1. Make sure that your main workspace is properly created in database with the right host in the column named domain: it should be the value of FIEF_DOMAIN, i.e. auth.v1.domain.com).
2. Make sure that the Fief container is able to make requests to https://auth.v1.domain.com
3. Make sure that your reverse proxy (nginx or traefik) does forward the correct Host header: requests should come to the Fief container with the external host, like auth.v1.domain.com.

I'll let you explore this and let us know what happens :)

[trisongz]

Last note on something I noticed while doing docker builds on a separate project, I noticed that the dockerfile installs supervisor with the pkg manager, rather than with pip. With ubuntu distros, supervisor installs with python2 which may add more dependencies. I can't say for certain about alpine but thought it's worth bringing up.

[frankie567]

Indeed, I wasn't aware of that. After checking, it seems that alpine installs a separate Python distro for this 🙄 I'll try to improve this!

That said, the purpose of supervisor is solely to be able to run the Fief server, the Fief worker and Redis in the same container. In a K8S context, we could probably think of a more "pure" way where we have a container for each one, removing the need for a process manager.

In this case, the Dockerfile could be something like this:

FROM python:3.10-alpine

ARG FIEF_VERSION

WORKDIR /app

RUN apk update \
    && apk add --virtual build-deps build-base \
    && apk add --no-cache libffi-dev postgresql-dev mariadb-dev \
    && pip install --upgrade pip \
    && python --version \
    && pip install fief-server==${FIEF_VERSION} \
    && apk del build-deps && \
    mkdir -p /data/db

ENV DATABASE_LOCATION=/data/db

ENV PORT=8000
EXPOSE ${PORT}

CMD ["fief", "run-server"]
# CMD ["fief", "run-worker"]

[trisongz]

I think the easiest way to account for k8s without having to do too much refactoring would be to have a bootstrap bash script, that would route the container init based on env params.

#!/bin/sh


# If user provides Redis URI, then use the provided redis instance
FIEF_REDIS_URI=${FIEF_REDIS_URI:-}

# Set both to true for docker based deployments
FIEF_SERVER_ENABLED=${FIEF_SERVER_ENABLED:-"true"}
FIEF_WORKER_ENABLED=${FIEF_WORKER_ENABLED:-"true"}

# If user defined, and exists, then use it, otherwise create dynamically.
FIEF_SUPERVISORD_CONF=${FIEF_SUPERVISORD_CONF:-"/etc/supervisord.conf"}

if [ -f $FIEF_SUPERVISORD_CONF ]; then
    echo "Running with Supervisor using Custom Config: ${FIEF_SUPERVISORD_CONF}"
    supervisord -c $FIEF_SUPERVISORD_CONF
fi

cat <<EOF > $FIEF_SUPERVISORD_CONF
[supervisord]
nodaemon=true

EOF

# Handle launching fief server
if [ "$FIEF_SERVER_ENABLED" == "true" ]; then
    echo "Fief Server Enabled"
    cat <<EOF >> $FIEF_SUPERVISORD_CONF
[program:fief-server]
command=fief run-server --port %(ENV_PORT)s
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
redirect_stderr=true

EOF
fi

# Handle launching fief worker
if [ "$FIEF_WORKER_ENABLED" == "true" ]; then
    echo "Fief Worker Enabled"
    if [ "$FIEF_REDIS_URI" == "" ]; then
        echo "Will launch Redis Server"
        cat <<EOF >> $FIEF_SUPERVISORD_CONF
[program:redis]
command=redis-server

EOF

    else
        echo "Using Provided Redis Server"
    fi
    cat <<EOF >> $FIEF_SUPERVISORD_CONF
[program:fief-worker]
command=fief run-worker -p 1 -t 1
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
redirect_stderr=true

EOF

echo "Starting Supervisor"
supervisord -c $FIEF_SUPERVISORD_CONF

FROM python:3.10-alpine

ARG FIEF_VERSION

WORKDIR /app

RUN apk update \
    && apk add --virtual build-deps build-base \
    && apk add --no-cache libffi-dev postgresql-dev mariadb-dev \
    && pip install --upgrade pip \
    && python --version \
    && pip install fief-server==${FIEF_VERSION} \
    && apk del build-deps && \
    mkdir -p /data/db

COPY bootstrap.sh /fief-bootstrap.sh
RUN chmod +x /fief-bootstrap.sh

ENV DATABASE_LOCATION=/data/db

ENV PORT=8000
EXPOSE ${PORT}

CMD ["/fief-bootstrap.sh"]

That way, deployment can be done like

containers:
- name: fief-server
  image: ghcr.io/fief-dev/fief:latest
  imagePullPolicy: Always
  env:
  - name: FIEF_SERVER_ENABLED
    value: 'true'
  - name: FIEF_WORKER_ENABLED
    value: 'false'
  envFrom:
  - secretRef:
      name: fief-server-secrets

- name: fief-worker
  image: ghcr.io/fief-dev/fief:latest
  imagePullPolicy: Always
  env:
  - name: FIEF_SERVER_ENABLED
    value: 'false'
  - name: FIEF_WORKER_ENABLED
    value: 'true'
  envFrom:
  - secretRef:
      name: fief-server-secrets

[frankie567]

That's interesting but isn't just setting the command in the K8S config file sufficient?

[trisongz]

It certainly is. But personally, I like to try to prevent user error as much as possible. Defining env values when templating with helm in my opinion is easier, and gives you more control over the startup process as the project evolves over time for future compatibility.

[AlissonMachadoEmbitel]

Hi Guys,

I am running it on k8s as well, however the application is being killed for no reason.

The only message i see on the logs is this one:

2023-12-05 12:06:12,084 WARN exited: fief-server (terminated by SIGKILL; not expected)

The server is stopping, but i dont have any clue why it is happening.

https://docs.fief.dev/self-hosting/environment-variables/

on the documentation is says i can add LOG_LEVEL env var, but i tried to add debug or 1, as a value, but it doesn't accept.

Any ideas?

[kielerrr]

For anyone interested, I was able to get running on k8s with this helm chart.
I already had traefik running so it doesn't include the traefik config. It also uses a digital ocean volume to persist the db.

values.yaml

replicaCount: 1

image:
  repository: ghcr.io/fief-dev/fief
  tag: "0.27.0"
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 8000

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: traefik
  paths:
    - "/"
  hosts:
    - auth.yourdomain.com
  tls: []

postgresql:
  enabled: true
  postgresqlUsername: fief
  postgresqlPassword: SOMEPASSWORD
  postgresqlDatabase: fief

redis:
  enabled: true
  password: SOMEPASSWORD

env:
  FIEF_DOMAIN: auth.yourdomain.com
  ROOT_DOMAIN: auth.yourdomain.com
  FIEF_BASE_URL: https://auth.yourdomain.com
  CSRF_COOKIE_SECURE: "True"
  LOGIN_SESSION_COOKIE_SECURE: "True"
  SESSION_COOKIE_SECURE: "True"
  FIEF_ADMIN_SESSION_COOKIE_SECURE: "True"
  FORWARDED_ALLOW_IPS: "*"
  LOG_LEVEL: DEBUG
  DATABASE_TYPE: POSTGRESQL


  DATABASE_HOST: localhost
  DATABASE_PORT: 5432
  DATABASE_USERNAME: fief
  DATABASE_PASSWORD: SOMEPASSWORD
  DATABASE_NAME: fief

  REDIS_URL: redis://localhost:6379

  SECRET: SOMEPASSWORD
  ENCRYPTION_KEY: SOMEKEY-4Zo5p3WFFFFFFFFc7hzyo=
  FIEF_CLIENT_ID: someclientid
  FIEF_CLIENT_SECRET: somesecret

  FIEF_ADMIN_API_KEY: SOMEPASSWORD

  FIEF_MAIN_USER_EMAIL: [email protected]
  FIEF_MAIN_USER_PASSWORD: SOMEPASSWORD
  FIEF_MAIN_ADMIN_API_KEY: SOMEPASSWORD

  TELEMETRY_ENABLED: false

  EMAIL_PROVIDER: SENDGRID
  DEFAULT_FROM_EMAIL: [email protected]
  DEFAULT_FROM_NAME: YourName


resources: {}

nodeSelector: {}

tolerations: []

affinity: {}

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fief
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: fief
  template:
    metadata:
      labels:
        app: fief
    spec:
      containers:
        - name: fief
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          env:
          {{- range $key, $value := .Values.env }}
            - name: {{ $key }}
              value: "{{ $value }}"
          {{- end }}
            - name: EMAIL_PROVIDER_PARAMS
              valueFrom:
                configMapKeyRef:
                  name: email-config
                  key: EMAIL_PROVIDER_PARAMS
          ports:
            - containerPort: 8000
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
        - name: postgres
          image: postgres:latest
          env:
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
            - name: POSTGRES_USER
              value: "{{ .Values.postgresql.postgresqlUsername }}"
            - name: POSTGRES_PASSWORD
              value: "{{ .Values.postgresql.postgresqlPassword }}"
            - name: POSTGRES_DB
              value: "{{ .Values.postgresql.postgresqlDatabase }}"
          ports:
            - containerPort: 5432
          volumeMounts:
            - name: postgresql-storage
              mountPath: /var/lib/postgresql/data
        - name: redis
          image: redis:latest
          env:
            - name: REDIS_PASSWORD
              value: "{{ .Values.redis.password }}"
          ports:
            - containerPort: 6379
      volumes:
        - name: postgresql-storage
          persistentVolumeClaim:
            claimName: postgresql-pvc

email-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: email-config
data:
  EMAIL_PROVIDER: 'SENDGRID'
  EMAIL_PROVIDER_PARAMS: '{ "api_key": "SOMESENDGRIDAPIKEY" }'

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: fief
  annotations:
    {{- range $key, $value := .Values.ingress.annotations }}
      {{ $key }}: {{ $value | quote }}
    {{- end }}
spec:
  rules:
    - host: "{{ index .Values.ingress.hosts 0 }}"
      http:
        paths:
          - path: "{{ index .Values.ingress.paths 0 }}"
            pathType: Prefix
            backend:
              service:
                name: fief
                port:
                  number: 8000
  {{- if .Values.ingress.tls }}
  tls:
    - hosts:
        - "{{ index .Values.ingress.hosts 0 }}"
      secretName: fief-tls
  {{- end }}

postgresql-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgresql-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: do-block-storage

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: fief
spec:
  type: {{ .Values.service.type }}
  ports:
    - port: {{ .Values.service.port }}
      targetPort: 8000
  selector:
    app: fief