PARSeq Model #2089
PARSeq Model #2089
Conversation
|
@sineeli - which parts of the PR are ready for review? Asking because it's still marked as draft |
|
Sure @abheesht17 First preprocessing and tokenizer these parts I think are good for reviewing, as they are the primary steps.
|
| "keras_hub.models.PARSeqTokenizer", | ||
| ] | ||
| ) | ||
| class PARSeqTokenizer(tokenizer.Tokenizer): |
There was a problem hiding this comment.
Please add a doc-string here, with examples. Makes it easier to review when we have examples :P
There was a problem hiding this comment.
Let's add unit tests as well
There was a problem hiding this comment.
Yes, will add them
| self.char_to_id = tf.lookup.StaticHashTable( | ||
| initializer=tf.lookup.KeyValueTensorInitializer( | ||
| keys=list(self._stoi.keys()), | ||
| values=list(self._stoi.values()), | ||
| key_dtype=tf.string, | ||
| value_dtype=tf.int32, | ||
| ), | ||
| default_value=0, | ||
| ) | ||
| self.id_to_char = tf.lookup.StaticHashTable( | ||
| initializer=tf.lookup.KeyValueTensorInitializer( | ||
| keys=list(self._stoi.values()), | ||
| values=list(self._stoi.keys()), | ||
| key_dtype=tf.int32, | ||
| value_dtype=tf.string, | ||
| ), | ||
| default_value=self.pad_token, | ||
| ) |
There was a problem hiding this comment.
The defaults don't match. EOS is the 0th token, and pad is the len(vocabulary) - 1th token
There was a problem hiding this comment.
I recognized the same in the original code, but seems they are using EOS -> 0, BOS->len(vocabulary), but while padding they are doing BOS first and then EOS at the end.
| ), | ||
| default_value=0, | ||
| ) | ||
| self.id_to_char = tf.lookup.StaticHashTable( |
There was a problem hiding this comment.
Do we need this? We aren't using it anywhere
There was a problem hiding this comment.
But in case if user wants to bulk change the token ids to characters it will be helpful
| label = tf.strings.upper(label) | ||
|
|
||
| label = tf.strings.regex_replace(label, self.unsupported_regex, "") | ||
| label = tf.strings.substr(label, 0, self.max_label_length) |
There was a problem hiding this comment.
Why are we truncating the input to 25 characters?
There was a problem hiding this comment.
While preparing the dataset in the preprocessing itself if the label is above 25 they jus ignore that datapoint itself. Instead I truncated and we can start and end tokens instead.
sachinprasadhs
added
the
WIP
No description provided.