Updated filtering for identity

[natolambert]

Note

Broaden identity-provider filtering patterns, add column-aware filtering, and speed up load/filter/upload with caching disabled and parallel processing.

• Filtering logic:
  • Expand PROVIDERS with "ChatGPT" and add new regexes (e.g., "I am called", direct "I'm/I am ") with bounded matches; broaden patterns for DeepSeek, Qwen, and Alibaba mentions.
  • Add column-aware filtering (example[column]) and treat None content as removable.
• CLI & usage:
  • Add --column arg; update usage path in script header.
• Performance & reliability:
  • Disable HF datasets caching (HF_DATASETS_DISABLE_CACHING, disable_caching()).
  • Parallelize dataset load/filter and push_to_hub with num_proc; add parquet fallback loader.
  • Replace inline debug collection with a lightweight post-filter sampling of filtered examples.

^{Written by Cursor Bugbot for commit b677b24. This will update automatically on new commits. Configure here.}

[gemini-code-assist]

Summary of Changes

Hello @natolambert, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly upgrades the dataset filtering script, filter_dataset_by_keywords.py, by introducing more comprehensive and flexible mechanisms to identify and remove unwanted content. The changes aim to improve the quality of filtered datasets by catching a broader range of model self-referential statements and specific model mentions, while also boosting filtering performance through multiprocessing and increasing the script's adaptability to various dataset structures.

Highlights

• Expanded Filtering Patterns: New regular expressions have been added to detect more instances of model self-identification (e.g., "I am ChatGPT") and specific model mentions (e.g., "DeepSeek-R1", "Qwen", "Alibaba Cloud").
• Configurable Message Column: The filtering script now supports a --column argument, allowing users to specify which dataset column contains the messages to be filtered, enhancing flexibility.
• Performance Improvement: Multiprocessing has been enabled for the dataset filtering step, utilizing num_proc=32 to speed up the process.
• Robustness Enhancements: An explicit check for None content in messages was added to prevent errors, and the PROVIDERS list was updated with "ChatGPT" (though "Qwen" appears to be duplicated).

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request updates the keyword filtering logic to better identify and remove model identity mentions from datasets. The changes include adding more providers and regex patterns, making the message column configurable, and improving performance by enabling parallel processing. My review identified a critical bug that prevents the correct display of filtered examples, a contradiction in a regex comment, and some opportunities to improve code clarity and remove redundancy. Addressing these points will improve the correctness and maintainability of the script.

[natolambert]

Suggested change

	print_within = min(10000, len(dataset))
	print_within = min(1000, len(dataset))

[saumyamalik]

lgtm!