webex_meetings: one host's API error stalls the global checkpoint, causing every meeting to be re-ingested each interval

[sideview-nick]

Summary

The webex_meetings input can get into a state where it re-ingests every meeting on every interval, producing unbounded duplicates. The trigger is a single host in the org whose GET /v1/meetings call fails (we observed a persistent 502 Bad Gateway for one hostEmail).

This issue documents the root design flaw and proposes the proper fix. A separate PR offers an interim stop-the-bleeding fix and links here.

Root cause

In input_module_webex_meetings.py, collect_events():

1. Computes a single, global window from/to for the whole org.
2. Loops over every user/hostEmail, fetching and writing that host's meetings.
3. Calls save_check_point() exactly once, after the entire loop, using one global checkpoint key ({stanza}_meeting_report_last_timestamp).

Two things make a single host's failure fatal to the whole run:

• The per-host fetch paging_get_request_to_webex(...) is outside any try/except.
• The write path catches, logs, then raise e.

So if any host errors, the run aborts before save_check_point().

The per-meeting dedup gate is:

checkpoint_value = helper.get_check_point(last_timestamp_checkpoint_key) or opt_start_time
last_checkpoint_time = datetime.strptime(checkpoint_value, "%Y-%m-%dT%H:%M:%SZ")
...
if meeting_start_time > last_checkpoint_time:
    ew.write_event(...)

With the checkpoint never saved, get_check_point(...) stays None forever and the comparison permanently falls back to opt_start_time. Every scheduled run re-requests start_time → now and re-writes every meeting. There is no event-level idempotency, so these are real duplicate events.

Observed symptoms

• The modinput log shows one hostEmail's request returning 502, a traceback, and Get error when collecting events.
• "Saved checkpoint" never appears in the log; the input's checkpoint directory stays empty.
• Duplicate cisco:webex:meetings events accumulate on every interval.

Proposed proper fix: per-host checkpointing

Track progress per host instead of one global checkpoint:

• Checkpoint key per host, e.g. {stanza}_{hostEmail}_meeting_report_last_timestamp.
• Compute the from/to window per host from that host's own checkpoint.
• On success, advance only that host's checkpoint; on failure, log and skip — that host simply retries its unchanged window next run, while healthy hosts advance independently.

This makes one bad host cost nothing: no stall, no duplicates, and no data loss for the failing host (it's retried, not skipped-past).

Interim fix (separate PR)

Because per-host checkpointing changes the window/dedup semantics, a smaller PR will first make the per-host fetch/write log-and-continue so the global checkpoint can advance and the duplicate flood stops. That interim fix has a known trade-off — a persistently-failing host's meetings for the current window are not captured — which this issue's per-host approach would eliminate. Happy to implement the per-host version if the maintainers agree on the direction.

[sideview-nick]

Update: this is more systemic than the original report — other inputs are susceptible to
unexpected 4xx and 5xx errors causing indefinite reindexing loops. Notably also the codebase
already contains the correct pattern in its newer inputs, which seems to confirm that our
proposed fix here aligns with the team's thinking, and it should be extended across all
affected inputs.

Confirmed second instance — webex_meeting_qualities (PR #31 now covers both). Same
structure, but worse since it seems like everyone will hit it inevitably:
GET /v1/meeting/qualities returns 425 "…expired after 14 days" for any meeting older than
14 days, so any window spanning >14 days contains a guaranteed failure and the input will
from there re-ingest every meeting every interval indefinitely. Trivially reproducible —
point the input at a start_time >14 days in the past.

Audit of all 8 inputs:

Input	Checkpoint saved	Vulnerable?
webex_meetings	after loop	yes (this issue / PR #31)
webex_meeting_qualities	after loop	yes (PR #31)
webex_detailed_call_history	after loop, no per-item dedup gate	yes — re-writes the entire window
webex_meetings_summary_report	after loop	yes (trigger: per-meeting attendees sub-fetch)
webex_contact_center_search	after loop, no gate	yes
webex_admin_audit_events	per-event, inside loop	no
webex_security_audit_events	per-event, inside loop	no
webex_generic_endpoint	per-item, inside loop	no

The three safe inputs (admin_audit, security_audit, generic_endpoint) already persist
the checkpoint incrementally as each item is written, so a mid-loop failure never resets
progress. It would seem reasonable to continue this line of thinking and update the five
older list-then-loop inputs in line with the pattern these three already use — persist the
max processed timestamp per item (and/or a dead-letter for permanently-failing items) —
rather than gating one end-of-loop save behind a fully successful loop. PR #31 is the
interim log-and-continue stopgap for the two inputs that we've seen actively duplicating
data in our lab.

For webex_meetings specifically, the log-and-continue stop-gap does drop a persistently
failing host's meetings for that window; the more comprehensive fix is a per-host checkpoint
keyed by hostEmail, so that host catches up on its own next run rather than being skipped.