Merge pull request #473 from shamanic-technologies/main

chore: sync hotfix v0.39.89 to staging

Author: KevinLourd

CLAUDE.md

@@ -442,6 +442,14 @@ window. `src/lib/linkedin-engagement-spec.ts` (pure) + `src/lib/linkedin-engagem
   ~1,000 reads/h from lead-service's serve loop) is skipped WITHOUT a call for
   `POSTS_FAILED_RETRY_MS` (1h), same answer; an unreadable 200/4xx stops the chain.
   No provider resolves `showcase/eimmigration` (all three probed).
+  **A provider 4xx (other than its own "not found" / treg "no tool") is a REFUSAL: next
+  provider, not transient** (LI-ENGAGEMENT-NEVER-DONE-1029: tikhub 400s "Request failed.
+  Please retry" on EVERY page since 2026-10-28 and used to stop the chain). One says not
+  found + the rest refused, none throttled/down/slow = `posts_status = unreadable`:
+  permanent like not_found (re-read after 24h, ~$0.002/day/page), NEVER keeps the serve
+  not-done, and listed in the response's additive `unreadablePages`. Only a TRANSIENT
+  failure keeps `done=false` (diffco-us kept Beyond Logic's audience not-done to ~18k
+  serves/hour).
 - **`doneReason` says WHY a serve is done** (`/search/next`, engagement only, null
   while `done=false`): `no_recent_posts` = no competitor page has a post published
   in the window (nothing to read engagers from; NOT exhaustion, new posts may

openapi.json

@@ -1157,6 +1157,33 @@
             ],
             "description": "source=linkedin_engagement only (absent everywhere else). WHY done is what it is: null while done=false; when done=true, no_recent_posts = no competitor page has a post published inside the audience's window (nothing to read engagers from: NOT exhaustion of an engaged pool, new posts may come), no_engagers = posts exist inside the window but nobody reacted or commented, all_served = there were engagers and every one was already considered for this audience (served, competitor employee or profile unreadable)."
           },
+          "unreadablePages": {
+            "type": "array",
+            "items": {
+              "type": "object",
+              "properties": {
+                "page": {
+                  "type": "string"
+                },
+                "state": {
+                  "type": "string",
+                  "enum": [
+                    "not_found",
+                    "unreadable"
+                  ]
+                },
+                "reason": {
+                  "type": "string"
+                }
+              },
+              "required": [
+                "page",
+                "state",
+                "reason"
+              ]
+            },
+            "description": "source=linkedin_engagement only. Competitor pages no posts provider can read (not_found = every provider says the page does not exist; unreadable = one says so and the others refuse it). Permanent: re-checked once a day, and they never keep done=false, so done/doneReason describe the READABLE pages. A page that failed transiently (rate limit, outage, timeout) is not listed and keeps done=false."
+          },
           "source": {
             "type": "string",
             "enum": [

src/lib/linkedin-company-posts.ts

@@ -27,7 +27,16 @@
  * provider is asked. "Page not found" passes to the next provider too: it is a
  * fact about that provider's coverage, not about the page (scrapecreators did
  * not know oxblue-corporation, tikhub listed 50 of its posts). A page is dead
- * only when every provider says so. Any other 4xx fails loud.
+ * only when every provider says so.
+ *
+ * Any other 4xx is the provider REFUSING the call: it says nothing about the
+ * page either, so the next provider is asked, but it is not transient. tikhub
+ * answers 400 "Request failed. Please retry ... You won't be charged" for EVERY
+ * page since 2026-10-28 (docketwise-software too), and that answer used to stop
+ * the chain, so harvestapi was never asked. A page no provider reads while at
+ * least one says "not found" and the rest refused (none throttled, down or slow)
+ * is UNREADABLE: permanent, re-checked daily like a not_found
+ * (LI-ENGAGEMENT-NEVER-DONE-1029). An unreadable 200 body still fails loud.
  */
 import type { WirePost } from "./linkedin-engagement-spec.js";
 import type { CompetitorPage } from "./linkedin-engagement-spec.js";
@@ -105,8 +114,10 @@ export const POSTS_PROVIDERS: PostsProvider[] = [
 export type PostsVerdict =
   | { kind: "posts"; posts: WirePost[] }
   | { kind: "not_found"; reason: string }
-  /** Says nothing about the page: ask the next provider. */
+  /** Says nothing about the page, transiently (rate limit, outage, withdrawn tool): ask the next provider. */
   | { kind: "next"; reason: string }
+  /** The provider refused the call (a 4xx): says nothing about the page, not transient. Ask the next provider. */
+  | { kind: "refused"; reason: string }
   /** An answer we do not understand: fail loud, do not guess. */
   | { kind: "fail"; reason: string };
 
@@ -131,5 +142,24 @@ export function postsVerdict(provider: PostsProvider, status: number, body: Reco
   }
   // The provider answered that the page does not exist (scrapecreators: 404 {"error":"not_found","message":"Company not found"}).
   if (status === 404 && /not.?found/i.test(`${body?.error ?? ""} ${body?.message ?? ""}`)) return { kind: "not_found", reason: why };
+  if (status >= 400 && status < 500) return { kind: "refused", reason: why };
   return { kind: "fail", reason: why };
 }
+
+/** What a page is once every provider was asked and none returned posts. */
+export type PageReadVerdict = { kind: "not_found"; reason: string } | { kind: "unreadable"; reason: string } | { kind: "failed"; reason: string };
+
+/**
+ * not_found = every provider said the page does not exist. unreadable = at least
+ * one said so and every other one REFUSED (4xx): nobody can read it and waiting
+ * an hour will not change that (permanent, re-checked daily). failed = any
+ * provider was throttled, down or slow, or none said "not found": retried soon.
+ */
+export function settlePageRead(answers: { notFound: string[]; refused: string[]; transient: string[] }, providerCount: number): PageReadVerdict {
+  const { notFound, refused, transient } = answers;
+  if (notFound.length === providerCount) return { kind: "not_found", reason: `no posts provider knows the page: ${notFound.join(" | ")}` };
+  if (transient.length === 0 && notFound.length > 0 && notFound.length + refused.length === providerCount) {
+    return { kind: "unreadable", reason: `no posts provider can read the page: ${[...notFound, ...refused].join(" | ")}` };
+  }
+  return { kind: "failed", reason: `no posts provider could read the page: ${[...notFound, ...refused, ...transient].join(" | ")}` };
+}

src/lib/linkedin-engagement.ts

@@ -69,7 +69,7 @@ import {
   type WireEngagement,
   type WirePost,
 } from "./linkedin-engagement-spec.js";
-import { POSTS_PROVIDERS, postsVerdict, type PostsProvider } from "./linkedin-company-posts.js";
+import { POSTS_PROVIDERS, postsVerdict, settlePageRead, type PageReadVerdict, type PostsProvider } from "./linkedin-company-posts.js";
 
 const TREG_BASE = "https://treg.to/call/";
 export const ENGAGEMENT_ENDPOINT = "fetchinio.linkedin.post.engagement";
@@ -430,7 +430,7 @@ function failure(endpoint: string, answer: TregAnswer): LinkedinTregError {
 // ─── Harvest (posts + engagement → silver) ───────────────────────────────────
 
 /** A page's state after the posts step of one serve. */
-export type PageOutcome = { page: CompetitorPage; state: "readable" } | { page: CompetitorPage; state: "not_found" | "failed"; reason: string };
+export type PageOutcome = { page: CompetitorPage; state: "readable" } | { page: CompetitorPage; state: "not_found" | "unreadable" | "failed"; reason: string };
 
 /**
  * Ask the providers in order until one returns the page's posts. A provider
@@ -440,41 +440,40 @@ export type PageOutcome = { page: CompetitorPage; state: "readable" } | { page:
  * posts, 2026-10-04). The page is not_found only when EVERY provider said so; a
  * mix of not-found and failures is a failure (retried, never remembered).
  */
-async function readPosts(meter: EngagementMeter, page: CompetitorPage, providers: PostsProvider[]): Promise<{ kind: "posts"; posts: WirePost[] } | { kind: "not_found"; reason: string } | { kind: "failed"; reason: string }> {
-  const skipped: string[] = [];
-  const notFound: string[] = [];
+async function readPosts(meter: EngagementMeter, page: CompetitorPage, providers: PostsProvider[]): Promise<{ kind: "posts"; posts: WirePost[] } | PageReadVerdict> {
+  const answers = { notFound: [] as string[], refused: [] as string[], transient: [] as string[] };
   for (const provider of providers) {
     let answer: TregAnswer;
     try {
       answer = await meter.call(provider.endpoint, { method: "GET", query: provider.query(page), maxMicro: provider.maxMicro, routed: false });
     } catch (err) {
       if (!(err instanceof LinkedinTregTransientError)) throw err;
-      skipped.push(messageOf(err));
+      answers.transient.push(messageOf(err));
       continue;
     }
     const verdict = postsVerdict(provider, answer.status, answer.body);
-    if (verdict.kind === "next") {
-      skipped.push(verdict.reason);
+    if (verdict.kind === "next" || verdict.kind === "refused") {
+      (verdict.kind === "next" ? answers.transient : answers.refused).push(verdict.reason);
       console.warn(`[Apollo Service][linkedin-engagement] posts provider skipped for ${page.url}: ${verdict.reason}`);
       continue;
     }
     if (verdict.kind === "not_found") {
-      notFound.push(verdict.reason);
+      answers.notFound.push(verdict.reason);
       continue;
     }
     // An answer we do not understand stops the chain (no other provider is paid) and
     // fails the page for this serve; it is remembered like any failure.
     if (verdict.kind === "fail") return { kind: "failed", reason: verdict.reason };
     return verdict;
   }
-  if (notFound.length === providers.length) return { kind: "not_found", reason: `no posts provider knows the page: ${notFound.join(" | ")}` };
-  return { kind: "failed", reason: `no posts provider could read the page: ${[...notFound, ...skipped].join(" | ")}` };
+  return settlePageRead(answers, providers.length);
 }
 
 /** The page's remembered posts answer while it is fresh; null = due for a read. */
 export function freshPostsOutcome(row: typeof linkedinCompanyPages.$inferSelect | undefined, page: CompetitorPage, now: Date): PageOutcome | null {
   if (row?.postsFetchedAt && now.getTime() - row.postsFetchedAt.getTime() < POSTS_REFRESH_MS) {
-    return row.postsStatus === "not_found" ? { page, state: "not_found", reason: row.postsError ?? "not found" } : { page, state: "readable" };
+    if (row.postsStatus === "not_found" || row.postsStatus === "unreadable") return { page, state: row.postsStatus, reason: row.postsError ?? row.postsStatus };
+    return { page, state: "readable" };
   }
   // Due, but every provider failed on it a moment ago: skipped, no provider call.
   const failedAt = row?.postsFailedAt;
@@ -511,11 +510,11 @@ async function readPageUnderLock(meter: EngagementMeter, page: CompetitorPage, n
       await tx.insert(linkedinCompanyPages).values({ slug: page.slug, url: page.url, ...failed }).onConflictDoUpdate({ target: linkedinCompanyPages.slug, set: failed });
       return { page, state: "failed", reason: read.reason };
     }
-    if (read.kind === "not_found") {
+    if (read.kind === "not_found" || read.kind === "unreadable") {
       // A permanent answer: remembered for the refresh period so no serve pays to re-learn it.
-      const set = { postsFetchedAt: now, postsCount: 0, postsStatus: "not_found", postsError: read.reason.slice(0, 1000), postsFailedAt: null, postsFailure: null };
+      const set = { postsFetchedAt: now, postsCount: 0, postsStatus: read.kind, postsError: read.reason.slice(0, 1000), postsFailedAt: null, postsFailure: null };
       await tx.insert(linkedinCompanyPages).values({ slug: page.slug, url: page.url, ...set }).onConflictDoUpdate({ target: linkedinCompanyPages.slug, set });
-      return { page, state: "not_found", reason: read.reason };
+      return { page, state: read.kind, reason: read.reason };
     }
 
     for (const p of read.posts) {
@@ -602,7 +601,7 @@ export async function harvest(meter: EngagementMeter, pages: CompetitorPage[], s
   const skipped = outcomes.flatMap((o) => (o.state === "readable" ? [] : [{ page: o.page.url, state: o.state, reason: o.reason }]));
   for (const s of skipped) console.warn(`[Apollo Service][linkedin-engagement] competitor page SKIPPED (${s.state}) ${s.page}: ${s.reason}`);
   if (skipped.length === pages.length) {
-    throw new LinkedinCompetitorPagesUnreadableError(skipped.map((s) => ({ page: s.page, reason: s.reason })), skipped.every((s) => s.state === "not_found"));
+    throw new LinkedinCompetitorPagesUnreadableError(skipped.map((s) => ({ page: s.page, reason: s.reason })), !outcomes.some(blocksCompletion));
   }
   const stale = new Date(now.getTime() - ENGAGEMENT_REFRESH_MS);
   const due = await db
@@ -626,8 +625,19 @@ export async function harvest(meter: EngagementMeter, pages: CompetitorPage[], s
   });
   const failed = read.find((r) => r?.status === "rejected") as PromiseRejectedResult | undefined;
   if (failed) throw failed.reason;
-  // A page that failed transiently may still hold engagers: not exhausted yet.
-  return { complete: complete && !skipped.some((s) => s.state === "failed"), skipped };
+  // A page that failed transiently may still hold engagers: not exhausted yet. A
+  // not_found or unreadable page is permanent (re-checked daily): it holds nobody
+  // we can reach, so it never keeps the serve from ending.
+  return { complete: complete && !outcomes.some(blocksCompletion), skipped };
+}
+
+/**
+ * Only a TRANSIENT failure keeps a harvest incomplete. Before (LI-ENGAGEMENT-NEVER-DONE-1029)
+ * a page every provider refused counted as failed forever: Beyond Logic's diffco-us
+ * kept its audience answering "nothing yet, not done" to ~18k serves an hour.
+ */
+export function blocksCompletion(outcome: PageOutcome): boolean {
+  return outcome.state === "failed";
 }
 
 // ─── Serve ───────────────────────────────────────────────────────────────────
@@ -783,7 +793,7 @@ export interface EngagementServeResult {
   unresolvable: number;
   /** Lookups that failed transiently: released, retried by a later serve. */
   deferred: number;
-  /** Competitor pages not read this serve: `not_found` (no provider knows the page) or `failed` (every provider failed). */
+  /** Competitor pages not read this serve: `not_found` (no provider knows the page), `unreadable` (one says not found, the others refuse) or `failed` (transient: retried). */
   skippedPages: Array<{ page: string; state: string; reason: string }>;
   calls: number;
   chargedMicro: number;

src/routes/search.ts

@@ -723,6 +723,7 @@ router.post("/search/next", serviceAuth, async (req: AuthenticatedRequest, res)
         people: served.people,
         done: served.done,
         doneReason: served.doneReason,
+        unreadablePages: served.skippedPages.filter((s) => s.state !== "failed").map((s) => ({ page: s.page, state: s.state, reason: s.reason })),
         totalEntries: served.poolSize,
         page: 1,
         totalPages: 1,

src/schemas.ts

@@ -758,6 +758,12 @@ const SearchNextResponseSchema = z
     doneReason: z.enum(["no_recent_posts", "no_engagers", "all_served"]).nullable().optional().openapi({
       description: "source=linkedin_engagement only (absent everywhere else). WHY done is what it is: null while done=false; when done=true, no_recent_posts = no competitor page has a post published inside the audience's window (nothing to read engagers from: NOT exhaustion of an engaged pool, new posts may come), no_engagers = posts exist inside the window but nobody reacted or commented, all_served = there were engagers and every one was already considered for this audience (served, competitor employee or profile unreadable).",
     }),
+    unreadablePages: z
+      .array(z.object({ page: z.string(), state: z.enum(["not_found", "unreadable"]), reason: z.string() }))
+      .optional()
+      .openapi({
+        description: "source=linkedin_engagement only. Competitor pages no posts provider can read (not_found = every provider says the page does not exist; unreadable = one says so and the others refuse it). Permanent: re-checked once a day, and they never keep done=false, so done/doneReason describe the READABLE pages. A page that failed transiently (rate limit, outage, timeout) is not listed and keeps done=false.",
+      }),
     source: z.enum(["quickenrich", "linkedin_engagement"]).optional().openapi({
       description: "quickenrich: this page came from the FREE QuickEnrich search (the audience is switched to quickenrich); those people carry an id `qe:<emp_id>`, their full name, LinkedIn URL and company domain. linkedin_engagement: the filters carry a linkedin_engagement buying_signal; up to 5 engagers of the competitor pages per call, each with an id `li:<profileId>`, full name, title, headline, employer and LinkedIn URL, never the same person twice for the audience (x-audience-id, else the campaign), competitor employees excluded; done=true only once nobody is left. For both, POST /enrich with that id finds the email with treg and verifies it. Absent = the Apollo walk, unchanged.",
     }),