Distributed Data Processing updated with Ray Data & Ray Job

[karajendran]

This PR transitions the preprocessing pipeline to use Ray Data for distributed, streaming data processing and adds support for running it as a Ray Job. It also fixes associated CI/CD failures (unit tests, coverage, and formatting).

Major Changes:

1. Introduced Ray Data & Streaming Architecture:

• Transitioned from standard Python/Pandas processing to Ray Data for distributed, streaming data preprocessing.
• Implemented RayDataPipelineOrchestrator in ray_data_pipeline.py to manage the streaming pipeline.
• Retired and removed legacy preprocessing utilities: DataLoader, DataPrep, and RayUtils.

2. Added Ray Job Support:

• Added entry point for submitting and running the pipeline as a Ray Job (configured in preprocessing_finetuning.py).
• Configured runtime environment for shipping local modules and pip dependencies to the Ray cluster.

3. RAG Input Optimization:

• Added DataPrepForRag class in datacleaner.py to filter and structure pipeline inputs specifically for RAG vectorization formats.

4. Data Preprocessor Optimizations:

• Optimized DataPreprocessor in datacleaner.py for parallel Ray environments, including:
  • Single storage client initialization per worker process.
  • Dynamic ThreadPoolExecutor scaling based on Ray-allocated CPU resources.

CI/CD Fixes:

1. Cleaned up Obsolete Tests:

• Removed test_dataloader.py, test_dataprep.py, and test_ray_utils.py which were testing the deleted legacy helper classes.

2. Fixed Unit Tests in test_datacleaner.py:

• Mocked GCS Client: Added storage.Client patching in setUp/tearDown to prevent DefaultCredentialsError in environments without active Google Cloud credentials (like
  GitHub Actions runners).
• Fixed SpaCy Mocking: Initialized DataPreprocessor locally in test_prep_product_desc to ensure the spacy.load patch is correctly applied.
• Updated Assertions: Fixed test_get_product_image to use pd.isna for asserting missing images, handling pandas NaN conversion correctly.
• Refactored Decorators: Simplified @patch.object decorators to use the imported DataPreprocessor directly.

3. Increased Code Coverage:

• Added TestDataPrepForRag class to test the new DataPrepForRag helper class (including filter_low_value_count_rows and process_rag_input methods).
• This increased the coverage of datacleaner.py to 82% (total coverage to 89%), passing the 80% CI/CD threshold.

4. Code Formatting:

• Reformatted modified test files with black to satisfy formatting checks.

[gushob21]

Please find my high level comments. Next, I will run through the guide and provide the feedback. By the way , for later - we should move this to accelerated-platforms

[karajendran]

This PR also includes changes needed for #378

[gushob21]

duplicate, we have verified the dataset file two steps above

[gushob21]

None of the queries yielded any results. However, when i went to the logs of the job from GKE console, I could see the logs