Fix backend integration test failures: Pinata mocking, error handling, timer leaks

[Copilot]

Three issues prevented backend integration tests from passing: Pinata SDK threw "Invalid URL" errors on mock blob IDs, /decrypt-ans returned 400 instead of 500 for processing errors, and TTL cleanup interval created Jest open handles.

Changes

backend/src/services/pinata.js

• Return mock data when NODE_ENV === 'test' or blob ID starts with mock-
• Wrap SDK calls in try/catch with IPFS gateway fallback
• Add blob ID validation

export async function readObject(blobId) {
  if (!blobId || typeof blobId !== 'string') {
    throw new Error('Invalid blobId provided to readObject');
  }

  // Return mock data in tests to avoid external API calls
  if (process.env.NODE_ENV === 'test' || blobId.startsWith('mock-')) {
    return JSON.stringify({
      ciphertext: 'mock-ciphertext',
      dataToEncryptHash: 'mock-hash'
    });
  }

  // SDK with gateway fallback...
}

backend/src/server.js

• Guard TTL cleanup setInterval with NODE_ENV !== 'test' check to prevent timer leaks
• Wrap /decrypt-ans in try/catch to return 500 on failures (matches /decrypt-clues pattern)
• Validate only critical fields (answers_blobId, userAddress) and throw for catch block to handle

// Skip timer during tests
if (process.env.NODE_ENV !== 'test') {
  setInterval(cleanupExpiredRooms, CLEANUP_INTERVAL);
}

// Consistent error handling
app.post("/decrypt-ans", async (req, res) => {
  try {
    if (bodyData.answers_blobId === undefined || bodyData.userAddress === undefined) {
      throw new Error("Missing required fields");
    }
    // ... processing
  } catch (error) {
    res.status(500).json({ error: "Failed to decrypt answers", message: error.message });
  }
});

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• block-indexer.litgateway.com
  • Triggering command: /usr/local/bin/node /usr/local/bin/node /home/REDACTED/work/Khoj/Khoj/backend/node_modules/jest-worker/build/workers/processChild.js (dns block)
  • Triggering command: /usr/local/bin/node node /home/REDACTED/work/Khoj/Khoj/backend/node_modules/.bin/jest --testPathPattern=integration --detectOpenHandles (dns block)
• eth.drpc.org
  • Triggering command: /usr/local/bin/node /usr/local/bin/node /home/REDACTED/work/Khoj/Khoj/backend/node_modules/jest-worker/build/workers/processChild.js (dns block)
  • Triggering command: /usr/local/bin/node node /home/REDACTED/work/Khoj/Khoj/backend/node_modules/.bin/jest --testPathPattern=integration --detectOpenHandles (dns block)
• eth.llamarpc.com
  • Triggering command: /usr/local/bin/node /usr/local/bin/node /home/REDACTED/work/Khoj/Khoj/backend/node_modules/jest-worker/build/workers/processChild.js (dns block)
  • Triggering command: /usr/local/bin/node node /home/REDACTED/work/Khoj/Khoj/backend/node_modules/.bin/jest --testPathPattern=integration --detectOpenHandles (dns block)
• ethereum-rpc.publicnode.com
  • Triggering command: /usr/local/bin/node /usr/local/bin/node /home/REDACTED/work/Khoj/Khoj/backend/node_modules/jest-worker/build/workers/processChild.js (dns block)
  • Triggering command: /usr/local/bin/node node /home/REDACTED/work/Khoj/Khoj/backend/node_modules/.bin/jest --testPathPattern=integration --detectOpenHandles (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

Summary

Create a new pull request which applies three targeted fixes so the backend integration tests stop failing (job 57358500786). The PR should branch off feature/integration-testing (PR #158 branch) and contain the concrete code changes described below.

Root causes to fix

• The Pinata readObject call throws "HTTP error: Invalid URL - ERR_ID:00004" (Pinata SDK getCid failing) when tests pass mock blob ids, causing handlers to throw unhandled errors.
• The /decrypt-ans endpoint returns 400 early for certain undefined inputs; the integration tests expect a 500 and an error body (test exercises server error handling). The handler should behave similarly to /decrypt-clues (try/catch and return 500 on processing errors).
• A global setInterval for TTL cleanup (cleanupExpiredRooms) runs during tests and creates an open handle, causing Jest to report a worker process failing to exit gracefully.

Files to change

1. backend/src/server.js (ref: 9df6ad8)

• Changes required:
  • Guard the TTL cleanup interval so it is not started when NODE_ENV === 'test'. Replace the unconditional setInterval with a conditional.
  • Replace the /decrypt-ans handler with a try/catch wrapper that throws for missing critical fields (answers_blobId and userAddress) and returns a 500 with an error body when anything fails. Mirror the style and error handling used by the existing /decrypt-clues endpoint.

Patch examples (apply at the same locations in server.js):

1.a Guard TTL cleanup interval (replace current setInterval call)

-// Start TTL cleanup interval
-setInterval(cleanupExpiredRooms, CLEANUP_INTERVAL);
-console.log(`Started TTL cleanup for team rooms. Cleanup interval: ${CLEANUP_INTERVAL / (60 * 60 * 1000)} hours`);
+// Start TTL cleanup interval (skip during tests to avoid leaking timers)
+if (process.env.NODE_ENV !== 'test') {
+  setInterval(cleanupExpiredRooms, CLEANUP_INTERVAL);
+  console.log(`Started TTL cleanup for team rooms. Cleanup interval: ${CLEANUP_INTERVAL / (60 * 60 * 1000)} hours`);
+} else {
+  console.log('Skipping TTL cleanup in test environment');
+}

1.b Replace /decrypt-ans handler

• Replace the existing app.post('/decrypt-ans', ...) with the try/catch version below. This change keeps the same high-level logic but throws or catches errors so the endpoint returns 500 for processing failures (matching the tests).

-app.post("/decrypt-ans", async (req, res) => {
-  const bodyData = req.body;
-
-  console.log("=== DECRYPT-ANS ENDPOINT DEBUG ===");
-  console.log("Request body:", JSON.stringify(bodyData, null, 2));
-  console.log("answers_blobId:", bodyData.answers_blobId);
-
-  // Validate required fields
-  if (
-    bodyData.cLat === undefined ||
-    bodyData.cLong === undefined ||
-    bodyData.clueId === undefined ||
-    bodyData.answers_blobId === undefined ||
-    bodyData.userAddress === undefined
-  ) {
-    return res.status(400).json({
-      error: "Missing required fields: cLat, cLong, clueId, answers_blobId, and userAddress are required",
-    });
-  }
-
-  const curLat = bodyData.cLat;
-  const curLong = bodyData.cLong;
-  const clueId = bodyData.clueId;
-
-  const answersData = await readObject(bodyData.answers_blobId);
-  const parsedAnswersData = typeof answersData === 'string' ? JSON.parse(answersData) : answersData;
-  
-  const {
-    ciphertext: answers_ciphertext,
-    dataToEncryptHash: answers_dataToEncryptHash,
-  } = parsedAnswersData;
-
-  console.log("Data read from answers_blobId:");
-  console.log("answers_ciphertext:", answers_ciphertext);
-  console.log("answers_dataToEncryptHash:", answers_dataToEncryptHash);
-
-  console.log("userAddress: ", bodyData.userAddress);
-  console.log("answers_dataToEncryptHash: ", answers_dataToEncryptHash);
-
-  const { response } = await decryptRunServerMode(
-    answers_dataToEncryptHash,
-    answers_ciphertext,
-    bodyData.userAddress,
-    curLat,
-    curLong,
-    clueId
-  );
-
-  console.log("Final response:", response);
-  res.send({ isClose: response });
-});
+app.post("/decrypt-ans", async (req, res) => {
+  try {
+    const bodyData = req.body;
+
+    console.log("=== DECRYPT-ANS ENDPOINT DEBUG ===");
+    console.log("Request body:", JSON.stringify(bodyData, null, 2));
+    console.log("answers_blobId:", bodyData.answers_blobId);
+
+    // Validate required fields - throw to be handled by catch so tests receive 500
+    if (bodyData.answers_blobId === undefined || bodyData.userAddress === undefined) {
+      throw new Error("Missing required fields: answers_blobId and userAddress are required");
+    }
+
+    const curLat = bodyData.cLat;
+    const curLong = bodyData.cLong;
+    const clueId = bodyData.clueId;
+
+    const answersData = await readObject(bodyData.answers_blobId);
+    const parsedAnswersData = typeof answersData === 'string' ? JSON.parse(answersData) : answersData;
+    const {
+      ciphertext: answers_ciphertext,
+      dataToEncryptHash: answers_data...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

*This pull request was created as a result of the following prompt from Copilot chat.*
> Summary
> 
> Create a new pull request which applies three targeted fixes so the backend integration tests stop failing (job 57358500786). The PR should branch off feature/integration-testing (PR #158 branch) and contain the concrete code changes described below.
> 
> Root causes to fix
> - The Pinata readObject call throws "HTTP error: Invalid URL - ERR_ID:00004" (Pinata SDK getCid failing) when tests pass mock blob ids, causing handlers to throw unhandled errors.
> - The /decrypt-ans endpoint returns 400 early for certain undefined inputs; the integration tests expect a 500 and an error body (test exercises server error handling). The handler should behave similarly to /decrypt-clues (try/catch and return 500 on processing errors).
> - A global setInterval for TTL cleanup (cleanupExpiredRooms) runs during tests and creates an open handle, causing Jest to report a worker process failing to exit gracefully.
> 
> Files to change
> 1) backend/src/server.js (ref: 9df6ad89474be566845539ddeca6529328521350)
> - Changes required:
>   - Guard the TTL cleanup interval so it is not started when NODE_ENV === 'test'. Replace the unconditional setInterval with a conditional.
>   - Replace the /decrypt-ans handler with a try/catch wrapper that throws for missing critical fields (answers_blobId and userAddress) and returns a 500 with an error body when anything fails. Mirror the style and error handling used by the existing /decrypt-clues endpoint.
> 
> Patch examples (apply at the same locations in server.js):
> 
> 1.a Guard TTL cleanup interval (replace current setInterval call)
> 
> ```diff
> -// Start TTL cleanup interval
> -setInterval(cleanupExpiredRooms, CLEANUP_INTERVAL);
> -console.log(`Started TTL cleanup for team rooms. Cleanup interval: ${CLEANUP_INTERVAL / (60 * 60 * 1000)} hours`);
> +// Start TTL cleanup interval (skip during tests to avoid leaking timers)
> +if (process.env.NODE_ENV !== 'test') {
> +  setInterval(cleanupExpiredRooms, CLEANUP_INTERVAL);
> +  console.log(`Started TTL cleanup for team rooms. Cleanup interval: ${CLEANUP_INTERVAL / (60 * 60 * 1000)} hours`);
> +} else {
> +  console.log('Skipping TTL cleanup in test environment');
> +}
> ```
> 
> 1.b Replace /decrypt-ans handler
> 
> - Replace the existing app.post('/decrypt-ans', ...) with the try/catch version below. This change keeps the same high-level logic but throws or catches errors so the endpoint returns 500 for processing failures (matching the tests).
> 
> ```diff
> -app.post("/decrypt-ans", async (req, res) => {
> -  const bodyData = req.body;
> -
> -  console.log("=== DECRYPT-ANS ENDPOINT DEBUG ===");
> -  console.log("Request body:", JSON.stringify(bodyData, null, 2));
> -  console.log("answers_blobId:", bodyData.answers_blobId);
> -
> -  // Validate required fields
> -  if (
> -    bodyData.cLat === undefined ||
> -    bodyData.cLong === undefined ||
> -    bodyData.clueId === undefined ||
> -    bodyData.answers_blobId === undefined ||
> -    bodyData.userAddress === undefined
> -  ) {
> -    return res.status(400).json({
> -      error: "Missing required fields: cLat, cLong, clueId, answers_blobId, and userAddress are required",
> -    });
> -  }
> -
> -  const curLat = bodyData.cLat;
> -  const curLong = bodyData.cLong;
> -  const clueId = bodyData.clueId;
> -
> -  const answersData = await readObject(bodyData.answers_blobId);
> -  const parsedAnswersData = typeof answersData === 'string' ? JSON.parse(answersData) : answersData;
> -  
> -  const {
> -    ciphertext: answers_ciphertext,
> -    dataToEncryptHash: answers_dataToEncryptHash,
> -  } = parsedAnswersData;
> -
> -  console.log("Data read from answers_blobId:");
> -  console.log("answers_ciphertext:", answers_ciphertext);
> -  console.log("answers_dataToEncryptHash:", answers_dataToEncryptHash);
> -
> -  console.log("userAddress: ", bodyData.userAddress);
> -  console.log("answers_dataToEncryptHash: ", answers_dataToEncryptHash);
> -
> -  const { response } = await decryptRunServerMode(
> -    answers_dataToEncryptHash,
> -    answers_ciphertext,
> -    bodyData.userAddress,
> -    curLat,
> -    curLong,
> -    clueId
> -  );
> -
> -  console.log("Final response:", response);
> -  res.send({ isClose: response });
> -});
> +app.post("/decrypt-ans", async (req, res) => {
> +  try {
> +    const bodyData = req.body;
> +
> +    console.log("=== DECRYPT-ANS ENDPOINT DEBUG ===");
> +    console.log("Request body:", JSON.stringify(bodyData, null, 2));
> +    console.log("answers_blobId:", bodyData.answers_blobId);
> +
> +    // Validate required fields - throw to be handled by catch so tests receive 500
> +    if (bodyData.answers_blobId === undefined || bodyData.userAddress === undefined) {
> +      throw new Error("Missing required fields: answers_blobId and userAddress are required");
> +    }
> +
> +    const curLat = bodyData.cLat;
> +    const curLong = bodyData.cLong;
> +    const clueId = bodyData.clueId;
> +
> +    const answersData = await readObject(bodyData.answers_blobId);
> +    const parsedAnswersData = typeof answersData === 'string' ? JSON.parse(answersData) : answersData;
> +    const {
> +      ciphertext: answers_ciphertext,
> +      dataToEncryptHash: answers_dataToEncryptHash,
> +    } = parsedAnswersData;
> +
> +    console.log("Data read from answers_blobId:");
> +    console.log("answers_ciphertext:", answers_ciphertext);
> +    console.log("answers_dataToEncryptHash:", answers_dataToEncryptHash);
> +
> +    const { response } = await decryptRunServerMode(
> +      answers_dataToEncryptHash,
> +      answers_ciphertext,
> +      bodyData.userAddress,
> +      curLat,
> +      curLong,
> +      clueId
> +    );
> +
> +    console.log("Final response:", response);
> +    res.send({ isClose: response });
> +  } catch (error) {
> +    console.error("Error decrypting answers:", error);
> +    res.status(500).json({
> +      error: "Failed to decrypt answers",
> +      message: error.message,
> +    });
> +  }
> +});
> ```
> 
> Notes: This change intentionally mirrors the structure used elsewhere in server.js (decrypt-clues) to provide consistent error handling.
> 
> 
> 2) backend/src/services/pinata.js (ref: 9df6ad89474be566845539ddeca6529328521350)
> - Changes required:
>   - Harden readObject so it validates the blobId early.
>   - If running in test mode (NODE_ENV === 'test') or if the blobId begins with 'mock-', return a deterministic mock JSON payload (so tests that pass mock ids don't attempt external network calls and the server-side logic can continue to be exercised).
>   - Wrap Pinata SDK usage in try/catch. If the SDK call fails (e.g., "Invalid URL"), fall back to fetching from a gateway URL built from PINATA_GATEWAY (env) or the default 'https://gateway.pinata.cloud'. If fallback fails, throw an error with a clear message.
> 
> Suggested implementation for readObject (replace existing function or create it if missing):
> 
> ```javascript
> import fetch from 'node-fetch'; // or rely on global fetch if available
> 
> export async function readObject(blobId) {
>   if (!blobId || typeof blobId !== 'string') {
>     throw new Error('Invalid blobId provided to readObject');
>   }
> 
>   // During tests, return a deterministic mock payload for mock ids
>   if (process.env.NODE_ENV === 'test' || blobId.startsWith('mock-')) {
>     return JSON.stringify({
>       ciphertext: 'mock-ciphertext',
>       dataToEncryptHash: 'mock-hash'
>     });
>   }
> 
>   const gateway = process.env.PINATA_GATEWAY || 'https://gateway.pinata.cloud';
> 
>   try {
>     // If pinata SDK is available and used in project, try using it first. Wrap in try/catch.
>     if (typeof pinata !== 'undefined' && pinata.getCid) {
>       try {
>         const cid = await pinata.getCid(blobId);
>         const url = `${gateway.replace(/\/+$/, '')}/ipfs/${cid}`;
>         const res = await fetch(url);
>         if (!res.ok) throw new Error(`Gateway fetch failed: ${res.status}`);
>         return await res.text();
>       } catch (sdkErr) {
>         console.error('Pinata SDK failed, falling back to gateway fetch:', sdkErr.message || sdkErr);
>       }
>     }
> 
>     // Fallback: fetch assuming blobId is a CID
>     const url = `${gateway.replace(/\/+$/, '')}/ipfs/${blobId}`;
>     const resp = await fetch(url);
>     if (!resp.ok) {
>       throw new Error(`Failed to fetch blobId ${blobId} from gateway ${url}: ${resp.status}`);
>     }
>     return await resp.text();
>   } catch (err) {
>     console.error('readObject error:', err.message || err);
>     throw err;
>   }
> }
> ```
> 
> Notes: If node-fetch is not desired, you may rely on global fetch in Node 18+. Using node-fetch import ensures compatibility across CI images.
> 
> 3) CI/environment (optional but recommended)
> - Add PINATA_GATEWAY env to the GitHub Actions job used to run backend integration tests if external gateway fetch is expected instead of the SDK. Example: PINATA_GATEWAY: gateway.pinata.cloud. The PR may optionally update .github/workflows/backend-tests.yml to set this env variable — if you want this included, please confirm. The user said "for 3 do as you please": the PR will include only the code changes (server.js and pinata.js). I will not modify the workflow unless you ask.
> 
> Branch & PR metadata
> - Create a new branch off feature/integration-testing (the head of PR #158) with a descriptive name, e.g., fix/pinata-decrypt-tests.
> - PR title: "Fix Pinata read errors, make /decrypt-ans return 500 on failures, disable TTL cleanup during tests"
> - PR body: include this problem statement and list a short summary of files changed and why.
> 
> Testing checklist
> - Run the repo's test script with NODE_ENV=test and jest --detectOpenHandles to ensure no timer leaks remain.
> - Confirm failing tests related to decrypt-ans and Pinata SDK no longer fail.
> 
> Please open the PR with these changes on repository mittal-parth/Khoj using base_ref 'feature/integration-testing'.
> 
> 

<!-- START COPILOT CODING AGENT TIPS -->
---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our [2 minute survey](https://gh.io/copilot-coding-agent-survey).

[netlify]

👷 Deploy Preview for khoj-alpha processing.

Name	Link
🔨 Latest commit	cc36548
🔍 Latest deploy log	https://app.netlify.com/projects/khoj-alpha/deploys/69367267972927000828573a