Persist conversation events to datadog

[simonrosenberg]

Description

• Add benchmarks/utils/conversation.py with a reusable build_event_persistence_callback that persists conversation events: logs full JSON for small events, logs trimmed metadata for large ones to avoid size limits, and gracefully tolerates
  failures.
• Wire the persistence callback into every benchmark run_infer.py so Conversation instances now emit structured events (with run_id, instance_id, and attempt) instead of ad-hoc debug logs; EvalOutput now records the attempt alongside
  instance_id.
• Track the current attempt on Evaluation (current_attempt), update it per retry, and surface it in logged events.
• Extend reset_logger_for_multiprocessing to add a dedicated JSON logger/handler for conversation events, filtering only the conversation_event and conversation_event_metadata records and bypassing stdout redirection for reliable capture
  (e.g., Datadog).

[simonrosenberg]

Updated to persist events via logging instead of bash commands.

Changes:

• Events are now logged to centralized logging (Datadog) instead of writing to container filesystem
• Added run_id parameter to distinguish parallel evaluation runs
• Large events (>64KB) log metadata only to avoid size limits
• Events persist beyond pod lifetime and are queryable by run_id + instance_id

Query example:

run_id:eval_out/swebench-lite-test/... instance_id:django__django-11490

Benefits:

1. Fixes Argument list too long errors when persisting large events via bash
2. Events survive pod termination (no more lost data when runtimes are killed)
3. Searchable/queryable in Datadog UI

[openhands-ai]

Looks like there are a few issues preventing this PR from being merged!

• GitHub Actions are failing:
  • Pre-commit checks

If you'd like me to help, just leave a comment, like

@OpenHands please fix the failing actions on PR #285 at branch `feature/issue-284-conversation-events`

Feel free to include any additional details that might help me get this PR into a better state.

_{^{You can manage your notification settings}}

[neubig]

This looks good, but maybe we can make this configurable, only on when we set an environment variable?