Skip to content

[Feature][history server] support endpoint /timezone#4510

Open
CheyuWu wants to merge 4 commits intoray-project:masterfrom
CheyuWu:feat/historyserver/timezone
Open

[Feature][history server] support endpoint /timezone#4510
CheyuWu wants to merge 4 commits intoray-project:masterfrom
CheyuWu:feat/historyserver/timezone

Conversation

@CheyuWu
Copy link
Collaborator

@CheyuWu CheyuWu commented Feb 14, 2026

Why are these changes needed?

we need to support something like this for dead cluster using golang.

https://github.com/ray-project/ray/blob/9cbcdb468904f8aa8122b817be216e0da237c086/python/ray/dashboard/timezone_utils.py#L1-L56

Summary

This PR adds timezone metadata collection and API support to the KubeRay history server, enabling proper timestamp interpretation for historical Ray cluster sessions.

Changes Overview

Commit 1: Collector-side timezone metadata collection (5f7600f)

Added timezone information collection in the log collector component:

Data Structure

  • Introduced timezoneInfo struct with two fields:
    • Offset: UTC offset string (e.g., "+08:00")
    • Value: IANA timezone name (e.g., "Asia/Shanghai")

Timezone Detection Strategy

  • Fetches timezone from the Ray Dashboard endpoint (GET {DashboardAddress}/timezone)
  • If the dashboard is unavailable, the write is skipped (no local fallback)

Storage

  • Writes timezone metadata to {clusterDir}/{session}/fetched_endpoints/restful__timezone following the same storage path convention as FetchAndStoreClusterMetadata
  • Called in three locations:
    • processSessionLatestLogs
    • processSessionPrevLogs
    • WatchSessionLatestLoops

Testing

Added unit tests including:

  • Mock HTTP client to simulate dashboard responses
  • Test case verifying that when the dashboard is unavailable, no file is written
  • Test case for dashboard-based timezone fetching (Asia/Seoul)

Commit 2: History server API endpoint (6551317)

Exposed timezone metadata through a REST API:

New Endpoint: GET /timezone

  • Returns session-specific timezone metadata for historical sessions
  • Redirects to live cluster if session is "live"
  • Returns {"offset":"","value":""} if metadata file doesn't exist
  • Returns actual timezone data if metadata is found

Implementation

  • Reads from storage path {clusterNameID}/{sessionName}/fetched_endpoints/restful__timezone
  • Returns the JSON content directly

Manual Testing

Follow the instruction to create the live cluster

https://github.com/ray-project/kuberay/blob/master/historyserver/docs/set_up_historyserver.md

Get live cluster timezone

$ curl -c ~/cookies.txt "http://localhost:8080/enter_cluster/default/raycluster-historyserver/live"
{
 "name": "raycluster-historyserver",
 "namespace": "default",
 "result": "success",
 "session": "live"
}
$ curl -b ~/cookies.txt "http://localhost:8080/timezone"
{"offset": "-08:00", "value": "America/Los_Angeles"}
image

File in MinIO

  • image
image

Get Dead cluster timezone

Delete the raycluster

$ kubectl delete -f historyserver/config/raycluster.yaml

Clean the cookies

$ rm ~/cookies.txt 

Read dead cluster

$ curl "http://localhost:8080/clusters"
[
 {
  "name": "raycluster-historyserver",
  "namespace": "default",
  "sessionName": "session_2026-02-14_08-35-24_859677_1",
  "createTime": "2026-02-14T08:35:24Z",
  "createTimeStamp": 1771058124
 }
]

Get cookies

$ curl -c ~/cookies.txt "http://localhost:8080/enter_cluster/default/raycluster-historyserver/session_2026-02-14_03-06-40_606126_1"
{
 "name": "raycluster-historyserver",
 "namespace": "default",
 "result": "success",
 "session": "session_2026-02-14_08-35-24_859677_1"
}

Get dead cluster timezone

$ curl -b ~/cookies.txt "http://localhost:8080/timezone"
{"offset":"-08:00","value":"America/Los_Angeles"}
image

Log in historyserver

$ k get pods
NAME                                  READY   STATUS      RESTARTS      AGE
historyserver-demo-64bd94bc79-h66wk   1/1     Running     0             2m19s
kuberay-operator-94fcdcd69-h8hp4      1/1     Running     4 (22m ago)   7d8h
rayjob-ns5r8                          0/1     Completed   0             101s

$ k logs historyserver-demo-64bd94bc79-h66wk
image

Related issue number

Closes #4503

Checks

  • I've made sure the tests are passing.
  • Testing Strategy
    • Unit tests
    • Manual tests
    • This PR is not tested :(

@CheyuWu CheyuWu force-pushed the feat/historyserver/timezone branch from 63769e2 to cf06745 Compare February 14, 2026 16:31
Copy link
Member

@Future-Outlier Future-Outlier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @CheyuWu can you help me follow this PR's pattern?
#4519

@CheyuWu
Copy link
Collaborator Author

CheyuWu commented Feb 24, 2026

Hi, @CheyuWu can you help me follow this PR's pattern? #4519

OK no problem

Signed-off-by: Cheyu Wu <cheyu1220@gmail.com>
Signed-off-by: Cheyu Wu <cheyu1220@gmail.com>
@CheyuWu CheyuWu force-pushed the feat/historyserver/timezone branch from cf06745 to 6551317 Compare February 26, 2026 17:39
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Signed-off-by: Cheyu Wu <cheyu1220@gmail.com>
@CheyuWu
Copy link
Collaborator Author

CheyuWu commented Feb 27, 2026

Hi @Future-Outlier, I have address the problem you have mentioned, and rewrite the code PTAL

@Future-Outlier Future-Outlier self-assigned this Mar 2, 2026
storageKey := utils.EndpointPathToStorageKey(timezoneEndpoint)
objectKey := path.Join(
r.ClusterDir,
path.Base(sessionID),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
path.Base(sessionID),
sessionID,

do we really need path.Base here?

Signed-off-by: Future-Outlier <eric901201@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature][history server] support endpoint /timezone

3 participants