Backend Challenge: Google Workspace Event Integration

.gitignore

@@ -0,0 +1,114 @@
+# Logs
+logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+lerna-debug.log*
+
+# Diagnostic reports (https://nodejs.org/api/report.html)
+report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
+
+# Runtime data
+pids
+*.pid
+*.seed
+*.pid.lock
+
+# Directory for instrumented libs generated by jscoverage/JSCover
+lib-cov
+
+# Coverage directory used by tools like istanbul
+coverage
+*.lcov
+
+# nyc test coverage
+.nyc_output
+
+# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
+.grunt
+
+# Bower dependency directory (https://bower.io/)
+bower_components
+
+# node-waf configuration
+.lock-wscript
+
+# Compiled binary addons (https://nodejs.org/api/addons.html)
+build/Release
+
+# Dependency directories
+node_modules/
+jspm_packages/
+
+# TypeScript v1 declaration files
+typings/
+
+# TypeScript cache
+*.tsbuildinfo
+
+# Optional npm cache directory
+.npm
+
+# Optional eslint cache
+.eslintcache
+
+# Microbundle cache
+.rpt2_cache/
+.rts2_cache_cjs/
+.rts2_cache_es/
+.rts2_cache_umd/
+
+# Optional REPL history
+.node_repl_history
+
+# Output of 'npm pack'
+*.tgz
+
+# Yarn Integrity file
+.yarn-integrity
+
+# dotenv environment variables folder and file
+.env
+
+# parcel-bundler cache (https://parceljs.org/)
+.cache
+
+# Next.js build output
+.next
+
+# Nuxt.js build / generate output
+.nuxt
+dist
+
+# Gatsby files
+.cache/
+# Comment in the public line in if your project uses Gatsby and *not* Next.js
+# https://nextjs.org/blog/next-9-1#public-directory-support
+# public
+
+# vuepress build output
+.vuepress/dist
+
+# Serverless directories
+.serverless/
+
+# FuseBox cache
+.fusebox/
+
+# DynamoDB Local files
+.dynamodb/
+
+# TernJS port file
+.tern-port
+
+# Ignore databases
+*.sqlite
+*.db
+
+# Ignore editor folder
+.vscode
+.idea
+
+# Imagine Stuff
+.imagine

README.md

@@ -1,118 +1,171 @@
-# Backend Challenge: Google Workspace Event Integration
+<h1 align="center">
+  Backend Challenge Solution
+</h1>
 
-## Introduction
-Welcome to the **Cybee.ai Backend Challenge**!  
+<strong>Key Features:</strong><br>
+* API and Background Services, utilizing Bull.mq<br>
+* Dockerized<br>
+* Resilient<br>
 
-This challenge will test your ability to **integrate with a cloud event source**, specifically **Google Workspace Admin SDK logs**, and build a system that:
-1. **Accepts a new source** (`POST /add-source`) with authentication credentials.
-2. **Periodically fetches logs** from Google Workspace.
-3. **Processes and forwards logs** to a specified callback URL.
-4. **Handles edge cases** like API rate limits, failures, and credential expiration.
-
-If you complete the challenge successfully, you’ll get a chance to talk with our team at Cybee.ai!
+<strong>Technology Stack:</strong><br>
+* Node.js<br>
+* MongoDB<br>
+* Redis<br>
+* Elasticsearch and Kibana <br>
 
 ---
 
-## Tech Stack Requirements
-Your solution must be built using:
+## How To Use
 
-- Node.js
-- Fastify (for API development)
-- MongoDB (for storing sources and logs)
-- Redis (for caching and job scheduling)
-- Elasticsearch (for log indexing) (optional but a plus)
-- Google Workspace Admin SDK (for fetching event logs)
+### 1. Dockerized Solution
 
-## Requirements
+#### Clone the Repository
 
-### 1. Build a Secure REST API
-Develop a **Fastify-based API** that allows users to connect a cloud event source and receive logs.
+```bash
+# Clone this repository
+$ git clone https://github.com/Bleron213/backend-challenge
+```
 
-#### Endpoints
-- `POST /add-source`
-  - Accepts **Google Workspace** as a source type.
-  - Stores API credentials securely.
-  - Validates credentials before storing.
-
-- `DELETE /remove-source/:id`
-  - Removes an existing event source.
+#### Setting Up the Local Environment
 
-- `GET /sources`
-  - Returns a list of active sources.
+1. Open the folder where the solution was cloned.
+2. Open a terminal and move to the `backend` folder.
+3. Inside the `backend` folder, create a file named `.env` and place these environment variables inside:
 
----
+```dotenv
+API_KEY=bBJ4Gig5CEVzTWM8l2nVCzX8Ht7IohuAFgsKK1puNmGU4FZormELBoRtjPySs4bAX6st4VOO2Vx8CSxoiQQuzWrrhEWlw2mwF17Boo5hun9Wo0RZZGhgsoK7uXSBD8AR
+ENCRYPTION_KEY=0d932b4a920075ca6bd78fb589b9815d878b1bd06fbf1f7477b69102e8967908
+NODE_ENV=DEVELOPMENT
+DOCKER=1
+```
+
+4. Start the Docker containers:
 
-### 2. Source Configuration & Data Model
-When a user adds a Google Workspace integration, the system should store:
-```json
-{
-  "id": "uuid",
-  "sourceType": "google_workspace",
-  "credentials": {
-    "clientEmail": "string",
-    "privateKey": "string",
-    "scopes": ["admin.googleapis.com"]
-  },
-  "logFetchInterval": 300,
-  "callbackUrl": "https://example.com/webhook"
-}
+```bash
+# Start the containers with Docker Compose
+$ docker-compose up
 ```
-**Notes:**
-- Credentials should be **stored securely** (e.g., encrypted in MongoDB).
-- `logFetchInterval` defines how often logs should be fetched (in seconds).
-- `callbackUrl` is where processed logs should be sent.
+
+The backend challenge should now be up and running. You can inspect the console to see logs. Alternatively, you can connect to MongoDB to view data inside the `sourcedb` and connect to Redis Insights to view the job scheduling inside Redis.
 
 ---
 
-### 3. Fetch & Forward Logs Automatically
-- Once a source is added, the system should:
-  - **Schedule a job** to fetch logs at `logFetchInterval` (e.g., using a queue like BullMQ).
-  - Call **Google Workspace Admin SDK** (`Reports API`) to fetch **audit logs**.
-  - **Forward logs** to the `callbackUrl` of the source.
-  - **Retry failed requests** and handle rate limits.
-
-**Example Log from Google Workspace:**
-```json
-{
-  "id": "log-id",
-  "timestamp": "2024-03-10T12:00:00Z",
-  "actor": {
-    "email": "admin@example.com",
-    "ipAddress": "192.168.1.1"
-  },
-  "eventType": "LOGIN",
-  "details": {
-    "status": "SUCCESS"
-  }
-}
+### 2. Non-Dockerized Solution
+
+If for any reason you want to start the solution without docker, here's how:
+
+#### Clone the Repository
+
+```bash
+git clone https://github.com/Bleron213/backend-challenge
 ```
 
----
+#### Setting Up the Local Environment
+
+1. Open the folder where the solution was cloned.
+2. Open a terminal and move to the `backend` folder.
+3. Inside the `backend` folder, create a file named `.env` and place these environment variables inside:
+
+```dotenv
+API_KEY=bBJ4Gig5CEVzTWM8l2nVCzX8Ht7IohuAFgsKK1puNmGU4FZormELBoRtjPySs4bAX6st4VOO2Vx8CSxoiQQuzWrrhEWlw2mwF17Boo5hun9Wo0RZZGhgsoK7uXSBD8AR
+MONGO_URI=mongodb://localhost:27017/sourcedb
+ENCRYPTION_KEY=0d932b4a920075ca6bd78fb589b9815d878b1bd06fbf1f7477b69102e8967908
+REDIS_PORT=6379
+REDIS_HOST=localhost
+NODE_DEBUG=bull
+NODE_ENV=DEVELOPMENT
+CALLBACK_API_HOOK=http://localhost:8080/Hooks/SendLog
+DOCKER=0
+ELASTIC_SEARCH_NODE=http://localhost:9200
 
-### 4. Handle Edge Cases
-Your system should properly handle:
-**API rate limits** – Backoff and retry.  
-**Credential expiration** – Detect and alert the user.  
-**Callback failures** – Retry failed webhook deliveries.  
-**Duplicate logs** – Ensure logs are not duplicated.  
-**High availability** – Ensure logs keep flowing even if one instance restarts.  
+```
+
+4. Run these Docker commands:
+
+```bash
+docker run --name mongodb -d -p 27017:27017 mongo
+docker run --name redis-server -p 6379:6379 -d redis
+docker run -d -p 8080:8080 --name callbackapi-container -e ASPNETCORE_ENVIRONMENT=Development bleronqorri/callbackapi:latest
+docker network create backend-challenge-network
+
+docker run -d --name elasticsearch --network backend-challenge-network -e "discovery.type=single-node" -e "xpack.security.enabled=false" -p 9200:9200 docker.elastic.co/elasticsearch/elasticsearch:8.3.3
+docker run -d --name kibana --network backend-challenge-network -p 5601:5601 -e ELASTICSEARCH_HOSTS="http://elasticsearch:9200" -e XPACK_SECURITY_ENABLED=false docker.elastic.co/kibana/kibana:8.3.3
+```
+
+5. Install dependencies and start the backend:
+
+```bash
+npm install
+```
+
+```bash
+npm start
+```
+
+The backend challenge should now be up and running. You can inspect the console to see logs. Alternatively, you can connect to MongoDB to view data inside the `sourcedb` and connect to Redis Insights to view the job scheduling inside Redis.
 
 ---
 
-### 5. Deployment & Bonus
-- (Required) Provide a **README** with:
-  - Setup instructions.
-  - API documentation.
-  - Explanation of how retries and scheduling work.
-- (Bonus) Deploy the solution using **Docker & a cloud provider**.
-- (Bonus) Implement **monitoring** (e.g., log metrics to Elasticsearch).
+## Seeing everything in action
+
+#### 1. Swagger documentation for endpoints
+
+Open http://localhost:3000/swagger/ and view the documented endpoints.  
+To use them, you need to provide the API key defined in `.env` variables.  
+We have exposed it here - but in a production environment, this would be the first layer of security.
+
+![Swagger](https://github.com/user-attachments/assets/85d4b55b-7925-469a-a31a-b597b0bd9d8d)
+
+#### 2. Redis Insight
+
+We can also view job schedules internals in Redis Insights.  
+Open Redis Insights and connect to the Redis running in the Docker container.
+
+![Redis Insights](https://github.com/user-attachments/assets/1e476a03-a521-4472-b5bd-c71f2b2e22ea)
+
+#### 3. MongoDB
+
+We can also view created sources and logs in the database.  
+Open Mongo Compass and connect to MongoDB running in Docker.  
+
+We can see the following info:
+
+![MongoDB](https://github.com/user-attachments/assets/8cdb36d7-75e9-4b24-9f3a-c86f674ccfac)
+
+#### 4. Resilience in Action
+
+![Resilience Logs](https://github.com/user-attachments/assets/23b52a9a-f5e6-4c0a-a12c-ff87dee9a704)
+
+Here we can see logs being processed. Due to the aggressive rate limiter in the callback API, retries will be quite common.
+
+#### 5. Elastic and Kibana
+
+If we navigate to KIbana -> Left Hamburger Menu -> Discover, we can see the following screen
+
+![image](https://github.com/user-attachments/assets/5a3adbca-beea-4e7c-b70b-51579e86ba6c)
+
+This means that Elastic search is accepting logs. To view them, we can create a new view
+
+![image](https://github.com/user-attachments/assets/28242240-8ef9-4381-973b-23f1900ee641)
+
+And we will be able to see application logs flowing in from Node.js app. Through the use of child loggers, we can differentiate between background processes and fastify api logs.
+
+Note that we're not restricted to application logs. We can create indexes for other things such as business events, products - anything. For now, we can see application logs flowing in seamlessly.
 
 ---
 
-## How to Submit
-1. Fork this repository and implement your solution in a `backend/` folder.
-2. Add a `README.md` with setup and usage instructions.
-3. Submit a pull request.
+## Notes
+
+- In a production environment, we would never expose API keys or encryption keys like this. For demo purposes, this is fine.
+- callbackapi-container might have issues on mac. If it doesn't work, please use the following command (if locally)
+- For demo purposes, security has been disabled in Elastic & Kibana.
 
-If your solution meets the challenge requirements, we’ll reach out to schedule a conversation. Looking forward to seeing your work!
+```bash
+docker run -d --platform linux/amd64 -p 8080:8080 --name callbackapi-container -e ASPNETCORE_ENVIRONMENT=Development bleronqorri/callbackapi:latest
+```
 
+or if using docker compose, include platform in callbackapi settings
+
+```bash
+  platform: linux/amd64
+```