Repository navigation
[docs] Output capturing - #50529
[docs] Output capturing#50529
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. Pages changed in this PR: |
vasqu
left a comment
vasqu
left a comment
There was a problem hiding this comment.
Just some nits but overall lgtm 🫡
| class LlavaModel(LlavaPreTrainedModel): | ||
|
|
||
| @merge_with_config_defaults | ||
| @can_return_tuple |
There was a problem hiding this comment.
hmm can we have one that does not need the merge config with defaults (because llava is a bit exotic tbh)
There was a problem hiding this comment.
yeah, probably better to use a generic example so readers can just focus on the important bits
zucchini-nlp
left a comment
zucchini-nlp
left a comment
There was a problem hiding this comment.
sgtm, tho i am not happy with a new decorator that overlaps hugely with existing one 🥹 Not on you, would be nice to merge them into one some time in the future
| > [!CAUTION] | ||
| > From v5.17 to v5.22, a list index `k` selected the output of layer `k` and left out the embedding output. It now selects `hidden_states[k]`, so add one to any positive index written for those versions. |
| Feature extraction methods like `get_image_features` often read one or a few hidden states from a submodel, such as the vision tower. Requesting `output_hidden_states=True` keeps every layer alive to use only a few of them. Decorate the method with `@select_hidden_states` instead. It sets `output_hidden_states` to the indices the method uses, and the `@capture_outputs` of the submodel that receives the kwargs captures only those. | ||
|
|
||
| Pass the name of the method argument that holds the indices. The decorator reads the argument's value, including its default, and forwards it as `output_hidden_states`. |
There was a problem hiding this comment.
wait what, we added select_hidden_states on top of filter_hidden_states 😭
nooo
There was a problem hiding this comment.
The difference:
select_hidden_statesonly materialises and returns the requested states eagerlyfilter_output_hidden_statesmaterialises them all and filters to the requested states lazily
Ideally, the mechanism in filter_output_hidden_states should never have existed because it performs worse
There was a problem hiding this comment.
yeah, if that filters afterwards - defi not the winning option. We should remove and replace it for next releases
There was a problem hiding this comment.
There is an additional difference tho, in select hidden states we can handle the hidden states after the fact
hidden_states = model(...).hidden_states
# do smthn with hidden states
...
while output hidden states is filtering at the end of the model output
hidden_states = model(...).hidden_states
return ModelOutput(..., hidden_states=hidden_states) # filters this hidden states out if not explicitly requested
Imo both could be fused into one but there is this difference that should be noted so it's not the same. In filter_output_hidden_states it's also mostly bound to backbones which need most hidden states anyways afaik
Adds docs for #49594:
output_hidden_statesto capture only the hidden states you need (this is for readers using a model)@select_hidden_statesto capture only the hidden states a method uses (this is for readers adding a model)