Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add max_features and tokenizer to CountVectorizer #376

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

marco-cloudflare
Copy link

@marco-cloudflare marco-cloudflare commented Feb 6, 2025

add max_features and tokenizer to CountVectorizer (similar to what's available at sklearn). Note that tokenizer and regex as competing parameters, in case sklearn, it disables regex if you pass a tokenizer and gives you a warning, so here we could think of a single parameter that would encompass both.

another caveat is the serialization of the tokenizer function pointer, the workaround I made was not skip it, allow it to be reset after deserialization and keep a guard that will error if you try to use transform after deserialization without resetting a tokenizer

Copy link

codecov bot commented Feb 7, 2025

Codecov Report

Attention: Patch coverage is 54.54545% with 10 lines in your changes missing coverage. Please review.

Project coverage is 35.46%. Comparing base (a30e5f1) to head (31a78fb).
Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
...gorithms/linfa-preprocessing/src/countgrams/mod.rs 52.94% 8 Missing ⚠️
...ms/linfa-preprocessing/src/tf_idf_vectorization.rs 50.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #376      +/-   ##
==========================================
+ Coverage   34.93%   35.46%   +0.52%     
==========================================
  Files          96       96              
  Lines        6334     6384      +50     
==========================================
+ Hits         2213     2264      +51     
+ Misses       4121     4120       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@relf relf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution. As you mentioned, it would be great to have only one parameter with an appropriate enum type to unify regexp and tokenizer function.
Could you also add some tests and maybe an example of the tokenizer function you introduce?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants