When using TrainState with ReactantDevice, single_train_step! handles compilation and caching of the training step internally via TrainState.cache. However, there is no public API to perform a compiled forward pass for validation that shares this compilation cache. Is that possible?
The reason I ask is that a typical training loop requires both:
- A training step — compute loss, compute gradients, and update parameters (
single_train_step! handles this)
- A validation step — compute loss without computing gradients.
However, with the current public API, I can't see how Step 2 can be done efficiently (compute_gradients uses the cache of TrainState but computes the gradients, which is wasteful). I know that @compile exists, but this doesn't share the cache with the TrainState object and doesn't include the user-friendly set-up of the public TrainState constructor.
Would it be hard to split compute_gradients into a function compute_objective, which does:
loss, stats, ts = compute_objective(objective_function, data, ts)
When using
TrainStatewithReactantDevice,single_train_step!handles compilation and caching of the training step internally viaTrainState.cache. However, there is no public API to perform a compiled forward pass for validation that shares this compilation cache. Is that possible?The reason I ask is that a typical training loop requires both:
single_train_step!handles this)However, with the current public API, I can't see how Step 2 can be done efficiently (
compute_gradientsuses the cache ofTrainStatebut computes the gradients, which is wasteful). I know that@compileexists, but this doesn't share the cache with theTrainStateobject and doesn't include the user-friendly set-up of the publicTrainStateconstructor.Would it be hard to split
compute_gradientsinto a functioncompute_objective, which does: