[feat] Added sdcript to create fast_llm_config.yaml files for datasets prepared with older fast llm version

[oleksost]

✨ Description

Added a script that is supposed to create the fast_llm_config.yaml file for datasets prepared with older fast llm version.

I think this is effected by the same problem as #172

🔍 Type of change

Select all that apply:

• 🐛 Bug fix (non-breaking change that addresses a specific issue)
• 🚀 New feature (non-breaking change that adds functionality)
• ⚠️ Breaking change (a change that could affect existing functionality)
• 📈 Performance improvement/optimization (improves speed, memory usage, or efficiency)
• 🛠️ Code refactor (non-functional changes that improve code readability, structure, etc.)
• 📦 Dependency bump (updates dependencies, including Dockerfile or package changes)
• 📝 Documentation change (updates documentation, including new content or typo fixes)
• 🔧 Infrastructure/Build change (affects build process, CI/CD, or dependencies)

📝 Changes

List the key changes introduced in this PR:

1. created function generate_config_yaml_for_sharded_dst in prepare.py
2. added generate_config_yaml_for_sharded_dst.py script

✅ Checklist

General

• 📜 I have read and followed the contributing guidelines.
• 🏷️ I am using a clear and descriptive PR title that summarizes the key change or feature introduced.
• 🎉 The functionality is complete, and I have tested the changes.
• 📝 I have updated the documentation if needed.
• ⚠️ The change does not introduce any new issues (e.g., runtime warnings, type checker errors, linting problems, unhandled edge cases).
• 🧩 I have commented my code, especially in hard-to-understand areas.

Dependencies and Configuration

• 🐋 I have updated the Docker configuration or dependencies, if applicable.
• 🔄 I have ensured compatibility with the existing setup after dependency changes.

Testing

• 🧪 I have added or updated tests to cover my changes.
• ✔️ New and existing tests pass locally with my changes.
• 🚦 I have tested these changes on GPUs and verified training stability.
• 🏋️ I have tested the changes on realistic training workloads, if applicable.

[jlamypoirier]

Looks good, but please make sure the file is formatted properly with pre-commit.

[jlamypoirier]

Missing pre-commit?

[jlamypoirier]

I'd add a disclaimer docstring saying that this is only for older datasets, and not really intended for anything else

[oleksost]

@jlamypoirier I suspect this script is not really needed if we are backward compatible with the json files that were produced by older version of the prepare command. Could you give an example of how to use those older json files in in the current version?

[jlamypoirier]

@oleksost the only supported way to use the json format is with the old way of defining datasets, using a single json file as-is to define the whole dataset. You can't really use it to mix datasets or combine with the file format, so you do need this script for a proper solution to #25. (Or add more backward compatibility for json format, but I think it would need more work)

[jlamypoirier]

Is this a common convention? Also missing newline.

[oleksost]

its not common, I removed it.