Enable conf to read from cloud path, needed for spark cluster mode, removed logging levels

[krisnaru]

Summary

This PR introduces support for reading configuration files from various cloud storage providers, including Google Cloud Storage (GCS), Amazon S3, and Azure Blob Storage, in addition to local file systems. The implementation uses Hadoop's FileSystem API to handle paths dynamically based on their prefixes, allowing seamless integration of cloud-based file management into existing workflows.

Why / Goal

The goal of this change is to enhance the flexibility of our configuration management by enabling the application to read configuration files directly from cloud storage. This improvement supports various use cases, such as:

Simplifying configuration management for distributed applications.
Enhancing portability and scalability by allowing configurations to be stored in a centralized cloud location.
Unlocking opportunities for easier deployment and integration with cloud-native architectures.

Test Plan

Added Unit Tests: Implemented tests to verify cloud path resolution and file accessibility for GCS, S3, and Azure.
Covered by existing CI: Updated CI configurations to include cloud-related tests.
Integration tested: Validated functionality through end-to-end tests with real cloud path

Reviewers

@nikhil-zlai @caiocamatta-stripe @haotizhong

[hzding621]

cc @pallavia7 to tal

[pengyu-hou]

could you undo this? Otherwise, the logs will be too spammy