Is it possible to scale horizontally?

[marcelomanchester]

I've deployed a WAHA plus server on AWS ECS fargate, i'm using DocumentDB (Mongo) for store my client's sessions with an EFS volume mounted to the /app/.media container path, which looks like is working as well. We've restarted the server several times and could retrieve the sessions and media files. All of this behind a NLB load balancer.

I've set horizontal autoscaling policies on the ECS service for scaling up and down instances based on CPU/Memory spikes, but when we're with more than one replicas we can't see all the sessions unified, as the load balancer round robin the server we see the sessions that handled by the specific fargate instance, so my client's app has been losing the track of the other's sessions.

So, how can we scale it horizontally? Is it possible? Am I missing something?

Docker image:
waha-plus:chrome-2024.7.7

ECS fargate task defintion:

{
  "taskDefinitionArn": "arn:aws:ecs:us-east-1:${CLIENT_AWS_ACCOUNT}:task-definition/whatsapp-http-api-plus:21",
  "containerDefinitions": [
      {
          "name": "whatsapp-http-api-plus",
          "image": "${CLIENT_AWS_ACCOUNT}.dkr.ecr.us-east-1.amazonaws.com/waha-plus:chrome-2024.7.7",
          "cpu": 1024,
          "memory": 2048,
          "memoryReservation": 100,
          "portMappings": [
              {
                  "name": "whatsapp-http-api-plus",
                  "containerPort": 3000,
                  "hostPort": 3000,
                  "protocol": "tcp"
              }
          ],
          "essential": true,
          "environment": [
              {
                  "name": "WAHA_WEBJS_WEB_VERSION",
                  "value": "2.2412.54-videofix"
              },
              {
                  "name": "WHATSAPP_RESTART_ALL_SESSIONS",
                  "value": "True"
              },
              {
                  "name": "WHATSAPP_FILES_FOLDER",
                  "value": "/app/.media"
              },
              {
                  "name": "WHATSAPP_HOOK_EVENTS",
                  "value": "*"
              },
              {
                  "name": "WHATSAPP_FILES_LIFETIME",
                  "value": "0"
              },
              {
                  "name": "WHATSAPP_API_HOSTNAME",
                  "value": "${NLB_HOST}"
              },
              {
                  "name": "WHATSAPP_SESSIONS_MONGO_URL",
                  "value": "mongodb://${USER}:${PASSWORD}@${DOCUMENT_DB_HOST}:27017/?replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false"
              }
          ],
          "mountPoints": [
              {
                  "sourceVolume": "efs-media-volume",
                  "containerPath": "/app/.media"
              }
          ],
          "volumesFrom": [],
          "linuxParameters": {
              "initProcessEnabled": false
          },
          "startTimeout": 30,
          "stopTimeout": 120,
          "user": "0",
          "privileged": false,
          "readonlyRootFilesystem": false,
          "interactive": false,
          "pseudoTerminal": false,
          "logConfiguration": {
              "logDriver": "awslogs",
              "options": {
                  "awslogs-group": "/aws/ecs/whatsapp-http-api-plus/whatsapp-http-api-plus",
                  "awslogs-region": "us-east-1",
                  "awslogs-stream-prefix": "ecs"
              }
          },
          "systemControls": []
      }
  ],
  "family": "whatsapp-http-api-plus",
  "taskRoleArn": "arn:aws:iam::${CLIENT_AWS_ACCOUNT}:role/Production-tasks-20240706022332889100000004",
  "executionRoleArn": "arn:aws:iam::${CLIENT_AWS_ACCOUNT}:role/whatsapp-http-api-plus-20240706022332889100000002",
  "networkMode": "awsvpc",
  "revision": 21,
  "volumes": [
      {
          "name": "efs-media-volume",
          "efsVolumeConfiguration": {
              "fileSystemId": "fs-aaaaaaaaaa",
              "rootDirectory": "/"
          }
      }
  ],
  "status": "ACTIVE",
  "requiresAttributes": [
      {
          "name": "ecs.capability.execution-role-awslogs"
      },
      {
          "name": "com.amazonaws.ecs.capability.ecr-auth"
      },
      {
          "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
      },
      {
          "name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
      },
      {
          "name": "com.amazonaws.ecs.capability.task-iam-role"
      },
      {
          "name": "ecs.capability.execution-role-ecr-pull"
      },
      {
          "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
      },
      {
          "name": "ecs.capability.task-eni"
      },
      {
          "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
      },
      {
          "name": "ecs.capability.efsAuth"
      },
      {
          "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
      },
      {
          "name": "ecs.capability.efs"
      },
      {
          "name": "ecs.capability.container-ordering"
      },
      {
          "name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
      }
  ],
  "placementConstraints": [],
  "compatibilities": [
      "EC2",
      "FARGATE"
  ],
  "requiresCompatibilities": [
      "FARGATE"
  ],
  "cpu": "1024",
  "memory": "4096",
  "runtimePlatform": {
      "cpuArchitecture": "X86_64",
      "operatingSystemFamily": "LINUX"
  },
  "registeredAt": "2024-08-02T22:12:01.604Z",
  "registeredBy": "arn:aws:sts::${CLIENT_AWS_ACCOUNT}:assumed-role/AWSReservedSSO_AdministratorAccess_6577028d392518dd/marcelomanchester",
  "tags": [
      {
          "key": "Repository",
          "value": "https://github.com/terraform-aws-modules/terraform-aws-ecs"
      },
      {
          "key": "Name",
          "value": "Production"
      }
  ]
}

[devlikepro]

Am I missing something?

You're missing that WAHA is not stateless application, it has a runtime state (not technically a state as in database, but still a state) - the connection to WhatsApp (either browser or websocket connection) which can not be moved automatically, so all HTTP request must be "sticky", meaning it MUST go only to the certain "worker" - one with "running" session.
So it's not quite easy to manage with current clouds...

General advice for WAHA Horizontal Scaling is to use Sharding (workers) approach - meaning that you need to run predefined numbers of workers and distribute in your application code sessions across it based on some fact - usually # of session on that container.
So you'll have like waha1.example.com with 8CPU and 32RAM memory which can handle 500 sessions (we don't recommend running more in a single container) and waha2.example.com with 4CPU and 16RAM can handle 250 sessions, both connected to different MongoDB (not the database, but the connection URL), otherwise they can interfere each other.
You create a record for that in your database, that customer.id=123 has session_1123123 in waha1.example.com so all requests goes it it.
If the server crashes - just start a new one (it can be done by using ECS, it doesn't have to be EC2 instances).
We're writing an article about WAHA sharding, will share it with you a bit later 🤞

We're working right now on built-in workers support, as well as built-in load balancing, but no ETA (estimated time of arrival) yet :(

Right now Sharding is the best way how you can handle >1000 sessions.

Btw, what number of session are we talking about?

---

btw, add +1 if you need AWS S3 support instead of EFS! #353

[marcelomanchester]

Thanks for the complete answer!

I'm moving right now for the NOWEB version since the WEBJS handle less sessions, are you planning to support the chat logs on the NOWEB at some point? That would be amazing for my client.

My client wants an undefined number of sessions, that's why i'm trying to make this scalable.
Is there a way to get the container's current number of sessions via API? I would like to monitor that to get notified when we get close to 500.

Regarding S3 support, I feel like EFS is working fine for us atm, but if you need some thoughts or help on this implementation I would be glad to help you.

[devlikepro]

Are you planning to support the chat logs on the NOWEB at some point

What is it, could you explain a bit?

NOWEB now has the store (you just need to enable it), so you can get the history from it as well
https://waha.devlike.pro/docs/engines/noweb/#store

Is there a way to get the container's current number of sessions via API?

Yes, you can use GET /api/sessions?all=True and count the number of sessions

[devlikepro]

Here your go!
WAHA Scaling - How To Handle 500+ Sessions

[marcelomanchester]

Amazing!
I'm going over the vertical scaling approach.

[devlikepro]

I hope we'll make out-of-the-box scaling when you grow up to 500 sessions 😉 🤞