[history server][collector] Fix getJobID for job event collection #4342

Future-Outlier · 2026-01-05T13:11:55Z

Summary:
when developing historyserver, please

use this branch to rebulid your collector image
use this branch's yaml file for raycluster and rayjob

Why are these changes needed?

Before

Job events were not collected because getJobID only checked top-level jobId.
Ray's export API returns job-related events with jobId nested inside event type objects.

After

Fix getJobID function to correctly extract jobId from nested event structures, enabling job event collection in history server.

What did I do?

Changes

code: Check jobId in nested event types: driverJobDefinitionEvent, driverJobLifecycleEvent, taskDefinitionEvent, taskLifecycleEvent, actorTaskDefinitionEvent, actorDefinitionEvent, taskProfileEvents [1]
raycluster CR: Add RAY_enable_ray_event=true in the example YAML to enable all events [2]
raycluster CR: Add RAY_DASHBOARD_AGGREGATOR_AGENT_EXPOSABLE_EVENT_TYPES in the example YAML to enable all events [2]

How I tested it

Deploy history server collector with a RayCluster
Submit a Ray job to the cluster (including task and actor)
Verify job events are collected and stored in the storage backend (S3)
Download all files from dir job_events and node_events files from S3 and verify all 9 event types are present [3]

References

[1] Ray event proto definitions:

[2]

Ray config to enable export API: ray_config_def.h#L542
2026.01.06 dialogue from @sampan-s-nayak

RAY_DASHBOARD_AGGREGATOR_AGENT_EXPOSABLE_EVENT_TYPES is used in older releases (upto 2.52)
RAY_DASHBOARD_AGGREGATOR_AGENT_PUBLISHER_HTTP_ENDPOINT_EXPOSABLE_EVENT_TYPES will be used going forward (need to see if this is part of 2.53)

[3]
those files are here:
session_2026-01-05_19-49-34_480793_1.zip

#	eventType	nestedKey	Has jobId?	AQAAAA==	AgAAAA==	node_events	Total
1	DRIVER_JOB_DEFINITION_EVENT	`driverJobDefinitionEvent`	✅ Yes	0	1	0	1
2	DRIVER_JOB_LIFECYCLE_EVENT	`driverJobLifecycleEvent`	✅ Yes	0	2	0	2
3	TASK_DEFINITION_EVENT	`taskDefinitionEvent`	✅ Yes	1	3	0	4
4	TASK_LIFECYCLE_EVENT	`taskLifecycleEvent`	⚠️ Sometimes empty	0	4	2	6
5	TASK_PROFILE_EVENT	`taskProfileEvents`	✅ Yes	3	1	0	4
6	ACTOR_DEFINITION_EVENT	`actorDefinitionEvent`	✅ Yes	1	1	0	2
7	ACTOR_TASK_DEFINITION_EVENT	`actorTaskDefinitionEvent`	✅ Yes	0	2	0	2
8	NODE_DEFINITION_EVENT	`nodeDefinitionEvent`	❌ No	0	0	2	2
9	NODE_LIFECYCLE_EVENT	`nodeLifecycleEvent`	❌ No	0	0	3	3
10	ACTOR_LIFECYCLE_EVENT	`actorLifecycleEvent`	❌ No	0	0	4	4
	Total			5	14	11	30

Related issue number

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

Signed-off-by: Future-Outlier <[email protected]>

Signed-off-by: Future-Outlier <[email protected]> Co-authored-by: Jia-Wei Jiang <[email protected]>

Future-Outlier

this is super urgent, we need to merge this ASAP to unblock others to develop history server, and this is related to @chiayi 's event processor.

cc @rueian @andrewsykim plz merge, thank you!

Signed-off-by: Future-Outlier <[email protected]>

Future-Outlier · 2026-01-05T13:55:19Z

historyserver/pkg/collector/eventserver/eventserver.go

-	if jobID, hasJob := eventData["jobId"]; hasJob && jobID != "" {
-		return fmt.Sprintf("%v", jobID)


Ray event structure never contains jobId at the top level of the event data.

Looking at actual Ray events, the structure is:

{ "eventId": "...", "eventType": "TASK_DEFINITION_EVENT", "message": "", "sessionName": "...", "severity": "INFO", "sourceType": "CORE_WORKER", "timestamp": "...", "taskDefinitionEvent": { // <-- Nested event object "jobId": "AgAAAA==", // <-- jobId is ALWAYS here (nested) "language": "PYTHON", ... } }

The jobId field is always nested inside the specific event type object (e.g., taskDefinitionEvent, actorDefinitionEvent, driverJobDefinitionEvent), never at the top level of eventData.

Future-Outlier · 2026-01-05T13:55:43Z

historyserver/config/raycluster.yaml

+          - name: RAY_enable_ray_event
+            value: "true"


to enable all types of event, we should always enable it.

this is for enabling gcs level event

I think this should also be "1" instead of bool string

nevermind, was not aware of https://github.com/ray-project/kuberay/pull/4342/changes#r2663725854

rueian · 2026-01-05T14:02:51Z

historyserver/pkg/collector/eventserver/eventserver.go

 	}
+
+	for _, eventType := range eventTypesWithJobID {
+		if nestedEvent, ok := eventData[eventType].(map[string]interface{}); ok {


Does the eventData come from the oneof protobuf? Can we just iterate it without using eventTypesWithJobID?

for _, nestedEvent := range eventData { .... }

it doesn't come from the oneof protobuf
https://github.com/ray-project/ray/blob/188d08743fff3baaf7e1baf076dc41e273f8635b/src/ray/protobuf/public/events_base_event.proto#L101

I updated the code to

for _, value := range eventData { if nestedEvent, ok := value.(map[string]interface{}); ok { if jobID, hasJob := nestedEvent["jobId"]; hasJob && jobID != "" { return fmt.Sprintf("%v", jobID) } } }

since this can support other case, for example in the future ray's proto change.

hmm, if it doesn't come from the oneof, then using this approach is probably not a good idea. we better define eventTypesWithJobID globally.

ah ok, will revert it later.

fixed here
0177160
420ce9a

Signed-off-by: Future-Outlier <[email protected]>

Future-Outlier · 2026-01-05T14:53:38Z

historyserver/config/raycluster.yaml

+          - name: RAY_DASHBOARD_AGGREGATOR_AGENT_PUBLISHER_HTTP_ENDPOINT_EXPOSABLE_EVENT_TYPES
+            value: "TASK_DEFINITION_EVENT,TASK_LIFECYCLE_EVENT,ACTOR_TASK_DEFINITION_EVENT,
+                    TASK_PROFILE_EVENT,DRIVER_JOB_DEFINITION_EVENT,DRIVER_JOB_LIFECYCLE_EVENT,
+                    ACTOR_DEFINITION_EVENT,ACTOR_LIFECYCLE_EVENT,NODE_DEFINITION_EVENT,NODE_LIFECYCLE_EVENT"


will 1 on 1 with @sampan-s-nayak to figure out

why I can't get task profile event now

why default disable the task profile event

this is fixed 8f87572

I updated the PR description above with proof.

cc @rueian to merge, thank you!

historyserver/pkg/collector/eventserver/eventserver.go

Signed-off-by: Future-Outlier <[email protected]>

sampan-s-nayak · 2026-01-06T05:54:46Z

historyserver/config/raycluster.yaml

        - env:
+          - name: RAY_enable_ray_event
+            value: "true"
          - name: RAY_enable_core_worker_ray_event_to_aggregator


this should be "1", I dont think it accepts bool strings

it actually accepts bool strings
https://github.com/ray-project/ray/blob/9d0ad57ceee03fd133fafdd0c52199085027e9a2/src/ray/common/ray_config.h#L45-L48

also note that even when this config is enabled, events will still go to GCS. to disable that we need to set RAY_enable_core_worker_task_event_to_gcs="0". but this will break cli and state API.

thank you, will keep in mind, curretly we might need to enable it until we can fully support removed the GCS, since there are some data not provided from base event.

for example:

/api/data/datasets/{job_id} /api/serve/applications/

sampan-s-nayak

LGTM from ray One-Event perspective

[historyserver] Fix getJobID for job event collection

b0172bc

Signed-off-by: Future-Outlier <[email protected]>

Future-Outlier added the P0 Critical issue that should be fixed ASAP label Jan 5, 2026

add jia-wei as co-author, since he debug with me together

eeb7a7f

Signed-off-by: Future-Outlier <[email protected]> Co-authored-by: Jia-Wei Jiang <[email protected]>

Future-Outlier added this to @Future-Outlier's kuberay project Jan 5, 2026

Future-Outlier moved this to can be merged in @Future-Outlier's kuberay project Jan 5, 2026

Future-Outlier commented Jan 5, 2026

View reviewed changes

remove unused code

0ff5bcf

Signed-off-by: Future-Outlier <[email protected]>

Future-Outlier commented Jan 5, 2026

View reviewed changes

rueian reviewed Jan 5, 2026

View reviewed changes

Future-Outlier added 4 commits January 5, 2026 22:11

update rueian's advice

5aea305

Signed-off-by: Future-Outlier <[email protected]>

add task profile event example

5f547a2

Signed-off-by: Future-Outlier <[email protected]>

revert back oneof solution

0177160

Signed-off-by: Future-Outlier <[email protected]>

add task profile event

420ce9a

Signed-off-by: Future-Outlier <[email protected]>

Future-Outlier commented Jan 5, 2026

View reviewed changes

Future-Outlier requested a review from rueian January 5, 2026 15:38

JiangJiaWei1103 mentioned this pull request Jan 5, 2026

[Test] [history server] [collector] Ensure event type coverage #4343

Open

4 tasks

rueian reviewed Jan 6, 2026

View reviewed changes

historyserver/pkg/collector/eventserver/eventserver.go Outdated Show resolved Hide resolved

Future-Outlier added 2 commits January 6, 2026 09:59

update rueian's advice

1b1d004

Signed-off-by: Future-Outlier <[email protected]>

a worked version in ray 2.52.0

8f87572

Signed-off-by: Future-Outlier <[email protected]>

Future-Outlier requested a review from rueian January 6, 2026 04:02

sampan-s-nayak reviewed Jan 6, 2026

View reviewed changes

sampan-s-nayak approved these changes Jan 6, 2026

View reviewed changes

rueian approved these changes Jan 6, 2026

View reviewed changes

rueian merged commit 4540f18 into ray-project:master Jan 6, 2026
28 checks passed

github-project-automation bot moved this from can be merged to Done in @Future-Outlier's kuberay project Jan 6, 2026

Future-Outlier mentioned this pull request Jan 14, 2026

[Test] [historyserver] [collector] Enhance log file coverage test for both head and worker pods #4351

Open

4 tasks

		if jobID, hasJob := eventData["jobId"]; hasJob && jobID != "" {
		return fmt.Sprintf("%v", jobID)

[history server][collector] Fix getJobID for job event collection #4342

[history server][collector] Fix getJobID for job event collection #4342

Uh oh!

Conversation

Future-Outlier commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Before

After

What did I do?

Changes

How I tested it

References

Related issue number

Checks

Uh oh!

Future-Outlier left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Future-Outlier Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Future-Outlier Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Future-Outlier Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Future-Outlier Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sampan-s-nayak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Future-Outlier commented Jan 5, 2026 •

edited

Loading

Future-Outlier Jan 5, 2026 •

edited

Loading

Future-Outlier Jan 5, 2026 •

edited

Loading

Future-Outlier Jan 6, 2026 •

edited

Loading

Future-Outlier Jan 6, 2026 •

edited

Loading