[WIP] Address review comments

[JiangJiaWei1103]

Why are these changes needed?

In this PR, we address review comments in the log collector implementation to allow development to continue in parallel.

Change Summary

We focus mainly on cleanup and refactoring. The following summarize the tasks which need further clarification, discussion, and survey.

Follow-Ups

• Unused fields in RayCollectorConfig
• Add var usage description
• Context shutdown
• Refactor complete and completeHSConfig
• Consider adding GCS
• Survey AWS Go lib for auth checks
• Naming convention of env vars
• Make sure background upload goroutine finishes
• Error handling and security
  • Check s3 dir existence
  • Graceful data retention
• Dockerfile (I'll refactor the Dockerfile with respect to this issue to mitigate image building overhead.)
  • Dashboard
  • Explicit COPY of necessary source code

Related issue number

• [Epic][Feature] Support History Server ray-project/kuberay#3966
• [Epic][Feature][history server] collector ray-project/kuberay#4274

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[JiangJiaWei1103]

Hi @Future-Outlier,

Would you mind taking a quick look and determine if we need to address all review comments in this single PR cause we might need further discussion and survey on some specific sub-topics ? Thanks.

Proposal

• Finish the "Error handling and security" part in this PR and merge first
• Refactor Dockerfile with log collector setup doc here at the same time
  • For building the image, we'd like to mitigate the overhead and focus only on the log collector
• Fix and integrate the remaining follow-ups on the fly without blocking others' development
  • For all these tasks, we might need further discussion or survey before making changes