remove minibatch_size and minibatches_per_step #623
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: Pull Request resolved: facebookresearch#447 Reviewed By: czxttkl Differential Revision: D26627900 fbshipit-source-id: 7be325fada7819f011092726d1cd29fb5483d599
Summary: Change the Klotski training code to use the Lightning training API Reviewed By: alexzhangxx Differential Revision: D28018402 fbshipit-source-id: 8c3054da176f5e08a68f4b87cc522af1fcd4912b
facebookresearch#463) Summary: Pull Request resolved: facebookresearch#463 Reviewed By: czxttkl Differential Revision: D28114174 fbshipit-source-id: c6f9953b2b4922c4c1b0271f3243c14f7261e103
Summary: Pull Request resolved: facebookresearch#462 title Reviewed By: czxttkl Differential Revision: D28044160 fbshipit-source-id: ac3d3231a164208d27deb4a0ddd0ac3de8fe8948
Differential Revision: D28150387 fbshipit-source-id: b6409f37823e99027baec8cc349215c3fd799bb4
Summary: Add backbone of one particular model of synthetic reward attribution. This model uses an MLP to predict each step's reward. A single step synthetic reward model works as follows: 1. Suppose you have an MDP: s0, a0, r0, s1, a1, r1, ...st, at, rt. 2. However you only know the aggregated reward R=r0 + r1 +... + rt. To facilitate RL model learning, it is ideal to distribute the aggregated reward to individual steps. 3. So we create a neural network net. 4. Fit the neural network by: MSE(R, net(s0, a0) + net(s1, a1) + ... net(st, at)) Reviewed By: j-jiafei Differential Revision: D27934701 fbshipit-source-id: c57418459e9378c8d690596cab8a627784551a18
Differential Revision: D28190581 fbshipit-source-id: a976503c8ea44495350744f68c7306e686dc4c28
Summary: This applies the formatting changes from black v21.4b2 to all covered projects in fbsource. Most changes are to single line docstrings, as black will now remove leading and trailing whitespace to match PEP8. Any other formatting changes are likely due to files that landed without formatting, or files that previously triggered errors in black. Any changes to code should be AST identical. Any test failures are likely due to bad tests, or testing against the output of pyfmt. Reviewed By: thatch Differential Revision: D28204910 fbshipit-source-id: 804725bcd14f763e90c5ddff1d0418117c15809a
Summary: Pull Request resolved: facebookresearch#465 A recent change in PyTorch Lightning set the states of optimizers (https://fburl.com/code/5tpf2i0j), which contradicts the frozen dataclass we had for the Optimizer wrapper in ReAgent. This diff removes the frozen settings, and replaces `__getattr__` with the safer, more explicit property functions. Reviewed By: MisterTea Differential Revision: D28205046 fbshipit-source-id: 848e3a0f90565eb041c0e91ef27c2be9102c5a7d
Summary: Pull Request resolved: facebookresearch#466 See title Reviewed By: MisterTea Differential Revision: D28236105 fbshipit-source-id: 9fc750e4c73d40b42d25b5378af94e722d96f5c5
Summary: Pull Request resolved: facebookresearch#467 Reviewed By: alexnikulkov Differential Revision: D28237308 fbshipit-source-id: 0025540b11ffa7d4325147c4304728c644f65c5d
Summary: Pull Request resolved: facebookresearch#469 One test failure only happens in OSS: https://app.circleci.com/pipelines/github/facebookresearch/ReAgent/1655/workflows/cbf167ec-76b2-423a-91b2-d454ba8d41d2/jobs/10454. This diff fixes it. Reviewed By: gji1 Differential Revision: D28248488 fbshipit-source-id: efc777757d9bc18d6b573e394e81997404252fb7
…kresearch#473) Summary: Pull Request resolved: facebookresearch#473 To satisfy a client team's request Differential Revision: D28327536 fbshipit-source-id: d3b1f9ef0c6b6bc09b29930d59ed2834cdadd7df
…el (facebookresearch#471) Summary: Pull Request resolved: facebookresearch#471 As titled. See T83887308 & T83886520 for more details. Reviewed By: kaiwenw Differential Revision: D26498062 fbshipit-source-id: ea0242d16f7673cad25d018235abb31742ab7434
Summary: Pull Request resolved: facebookresearch#474 Reviewed By: czxttkl Differential Revision: D28312845 fbshipit-source-id: abb039d445a1228bb11ffb6103744854b209b3dc
Summary: Pull Request resolved: facebookresearch#476 Add a n-gram MLP for synthetic reward attribution. This model uses an MLP to predict each step's reward. Compared with single-step reward model, it uses n-gram with a context window centered around each step and zero padding. Reviewed By: czxttkl Differential Revision: D28362111 fbshipit-source-id: 624de95f14b7fedb79ccb0cd47cb811b651fab04
Summary: Pull Request resolved: facebookresearch#472 Distributed readers are not supported yet, as shown in the test plan below czxttkl. Reviewed By: czxttkl Differential Revision: D28292330 fbshipit-source-id: 0f03d27fdba75740ab9590747ae025c6da6ce9fa
Summary: Pull Request resolved: facebookresearch#478 Reviewed By: bankawas Differential Revision: D28427686 fbshipit-source-id: b53a9f974f9c2ee615fb453b5efe48b9de487dbf
…ch#479) Summary: Pull Request resolved: facebookresearch#479 Making these changes can finally get us distributed training for reward networks (hopefully. Still need to wait for the workflow to finish). Fix the error asked in https://fb.workplace.com/groups/pytorchLightning/permalink/455491295468768/. Reviewed By: gji1 Differential Revision: D28318470 fbshipit-source-id: fe3836ef49864a20af07511a10e25c0d1a20ba0d
Summary: Pull Request resolved: facebookresearch#480 Lower the number of training samples & threshold, use Adam instead of SGD. Reviewed By: j-jiafei Differential Revision: D28464831 fbshipit-source-id: 918329290be62bd846507e2bd3697af4c3e710db
…bookresearch#470) Summary: Pull Request resolved: facebookresearch#470 Reviewed By: czxttkl Differential Revision: D28093192 fbshipit-source-id: 6b260c3e8d49c8b302e40066e2be49a0bfe96688
Summary: Pull Request resolved: facebookresearch#477 Add ConvNet support to n-gram synthetic reward network. Reviewed By: czxttkl Differential Revision: D28402551 fbshipit-source-id: c2201be3d71c32977c2f19b69e5a0abcaf0a855d
Summary: Pull Request resolved: facebookresearch#481 Add LSTM synthetic reward net. Reviewed By: czxttkl Differential Revision: D28448615 fbshipit-source-id: e8c77ef8c7b4ad69fcda2fd432cc018cfb7495cd
Summary: Pull Request resolved: facebookresearch#482 as titled. Also support discrete action. Reviewed By: j-jiafei Differential Revision: D28248528 fbshipit-source-id: bf87afa18914e9331177b22f0c9a823ac2ba2337
…h#483) Summary: Pull Request resolved: facebookresearch#483 As title. Reviewed By: czxttkl Differential Revision: D28551285 fbshipit-source-id: 3cc14daa930399daa0880c8569f8f36b46c1ff94
Summary: Pull Request resolved: facebookresearch#484 Refactoring so that we can use spark transform to bulk eval synthetic reward models. Things changed: 1. Improve API for defining models. In `reagent/models/synthetic_reward.py`, we create `SyntheticRewardNet`, which takes in different architecture implementations with standardized input/output shapes. 2. Net builders will build different architectures to construct `SyntheticRewardNet`. So we follow a composite pattern in net builders. 3. All net builders now share the same `build_serving_module` method. 4. Improve test methods so they share as much code as possible between different architectures. Reviewed By: j-jiafei Differential Revision: D28549704 fbshipit-source-id: 535a6191b6cfc4c55ed8b4f8c366af77ceac5c79
Summary: Added binary_difference_scorer to discrete_dqn.py Reviewed By: czxttkl Differential Revision: D28691568 fbshipit-source-id: dd9fe5518b13aea2acb94dae10823cdfd9253926
…cebookresearch#485) Summary: Pull Request resolved: facebookresearch#485 As title. Reviewed By: czxttkl Differential Revision: D28790947 fbshipit-source-id: 26405326402a0b913731c2a9ccb4badde4b47a9b
…s set up, required in lightning 1.3.3 Summary: with move to lightning 1.3 (D28792413), MDNRNNTrainer cannot call self.log() without setting up a LoggerConnector Reviewed By: kandluis Differential Revision: D28825504 fbshipit-source-id: 145028b62647f7466d44833bde0c0d4fb4c6d729
Summary: Data module for CFEval Reviewed By: gji1 Differential Revision: D28661138 fbshipit-source-id: c248600105bad5e66c717deb1fc0dee44d415005
Summary: As a showcase for how to add sparse features to ReAgent See "Model Training" section in quip https://fb.quip.com/1RdkAeTsSjgh Reviewed By: alexnikulkov Differential Revision: D34082046 fbshipit-source-id: 82a7294f0d9dd36c0f63d85c6366b9b2e0114dc4
Summary: Necessary changes in model managers to accommodate previous changes in the stack. Reviewed By: alexnikulkov Differential Revision: D34082048 fbshipit-source-id: 638554012aefaf71acc058b8add679dfb4382703
Summary: as titled Reviewed By: alexnikulkov Differential Revision: D34082045 fbshipit-source-id: 2f71e1b735512f01b65778d7b83a283832aa4ffe
…arch#604) Summary: Pull Request resolved: facebookresearch#604 All tests accompanied with D33850915 Reviewed By: alexnikulkov Differential Revision: D33971614 fbshipit-source-id: 215ce0f609ab0d0a47cc1e6f88806444ef900ae0
Summary: Pull Request resolved: facebookresearch#605 as titled Reviewed By: gji1 Differential Revision: D34114567 fbshipit-source-id: e5a792c36c55fe047ef7bdd1620ee56c76104f58
Summary: Pull Request resolved: facebookresearch#606 A new foreach flag is being added to the optimizers to indicate whether foreach logic or single tensor logic is used (see D33767870 and the associated stack). This causes reagent tests to fail such as https://www.internalfb.com/intern/testinfra/diagnostics/7318349469673867.281475021413633.1644559942/ The issue arises from this line https://fburl.com/code/lroy3a2p where the value for foreach cannot be found in `getattr(self, k)`. This PR adds the foreach flag to `uninferrable_optimizers.py` to address this (Note that we do not add this flag to `LBFGS` and `SparseAdam` as they do not support this option) Reviewed By: alexnikulkov Differential Revision: D34216723 fbshipit-source-id: fac4e6095157c7cd33184bfa5b7042bdd151688e
Reviewed By: shannonzhu Differential Revision: D34226909 fbshipit-source-id: 4045a574efe46205ddf87ff839f52e2aac454fc5
Reviewed By: shannonzhu Differential Revision: D34333122 fbshipit-source-id: 896c3306d85863ee8831ed08023bcd87e36f1657
Summary: Pull Request resolved: facebookresearch#607 1. add log performance of each episode 2. crease usecase specific episode post callback 3. create step post callback Reviewed By: vgup0, alexnikulkov Differential Revision: D34295015 fbshipit-source-id: 2a72c9d291421707fb3192c34b74f5bcbd788a53
Summary: Pull Request resolved: facebookresearch#608 update READEME and add ForkedPdb in reagent Reviewed By: alexnikulkov Differential Revision: D34425175 fbshipit-source-id: c59ee44b8ff89cf87a13794f23d85f0890f52cb2
Summary: Pull Request resolved: facebookresearch#609 X-link: meta-pytorch/torchrec#112 As discussed in D33960410, we want the responsibility of processing KeyedTensor into sparse features the responsibility of SparseArch. A motivation for this is that we want to have an extension EsuhmDLRM, where all we would need to do is replace the sparse arch component. However, the esuhm sparse arch's output doesn't adhere to the current KeyedTensor output. Reviewed By: bigning Differential Revision: D34482853 fbshipit-source-id: 90048cc1d36327593422d459b49cb8d3783226e2
Summary: - Model Manager for BehaviorCloning - UnitTest of the ModelManager - DataModule for UnitTest Reviewed By: czxttkl Differential Revision: D33829752 fbshipit-source-id: 9d1d6af293f652e095b914608108fc0d215ff257
Summary: Our mission at [Meta Open Source](https://opensource.facebook.com/) is to empower communities through open source, and we believe that it means building a welcoming and safe environment for all. As a part of this work, we are adding this banner in support for Ukraine during this crisis. Pull Request resolved: facebookresearch#613 Reviewed By: alexnikulkov Differential Revision: D34630775 Pulled By: dmitryvinn-fb fbshipit-source-id: 7108199313663725759377fe0972e59e9ae2cb22
Summary: ### New commit log messages - [a52a6ea03 Add support for pluggable Accelerators (#12030)](Lightning-AI/pytorch-lightning#12030) Reviewed By: edward-io Differential Revision: D34608197 fbshipit-source-id: ee87d0ce693659a4e689290a079f8c5a4772faf2
Differential Revision: D34666657 fbshipit-source-id: 02546bd9ce2d328ad1210eb18499d8db86267e65
Summary: Pull Request resolved: facebookresearch#614 Reviewed By: czxttkl Differential Revision: D34657092 fbshipit-source-id: 47e0af9b751dffaeafbf9019b7bb5967c0ff84c1
…#189) Summary: X-link: facebookresearch/d2go#189 X-link: facebookresearch/recipes#14 Pull Request resolved: facebookresearch#616 ### New commit log messages - [9b011606f Add callout items to the Docs landing page (#12196)](Lightning-AI/pytorch-lightning#12196) Reviewed By: edward-io Differential Revision: D34687261 fbshipit-source-id: 3ef6be5169a855582384f9097a962d2261625882
Summary: Pull Request resolved: facebookresearch#617 Improve the reinforce trainer by 1. Allowing reward mean subtraction without normalization, 2. Providing the option to log training loss and ips ratio mean per epoch. Reviewed By: alexnikulkov Differential Revision: D34688279 fbshipit-source-id: 50e94140fbf2182523e03c350f7bbe6812cb6e74
Summary: Pull Request resolved: facebookresearch#618 as titled Reviewed By: sinannasir Differential Revision: D34587407 fbshipit-source-id: 738aa3fb580716628330efa65a8c5ca7596aff14
Summary: Pull Request resolved: facebookresearch#615 as titled Reviewed By: PavlosApo Differential Revision: D34677139 fbshipit-source-id: 9fa8a0884d8f4abf0c7ca47fa669932d739a2d4c
Summary: Pull Request resolved: facebookresearch#619 as titled Reviewed By: alexnikulkov Differential Revision: D34940029 fbshipit-source-id: 9f6add38bd7f03f6811b6f4c51db431a1412660c
Summary: Pull Request resolved: facebookresearch#620 Officially import torchrec Reviewed By: alexnikulkov Differential Revision: D34942469 fbshipit-source-id: d4d47f4e90ff99f738f27c0720fd5462f40abe86
Summary: Pull Request resolved: facebookresearch#621 As the creative ranking project runs only 1 epoch, enable per-batch logging to TensorBoard, as did in the SAC trainer in ReAgent. Reviewed By: czxttkl Differential Revision: D35100625 fbshipit-source-id: 37bf361a4f668665de7691731467755c37b31067
Differential Revision: D35275827 fbshipit-source-id: e1e402f8a07f97e3243318bb0101e2943a40c48c
Differential Revision: D35313194 fbshipit-source-id: 30b3f317f90b2e736453ae5162caad765fbfa414
Summary: minibatch_size and minibatches_per_step are unused in dqn trainers. Differential Revision: D35338905 fbshipit-source-id: 9c89997442dab0418bf142f18db4616e0dccb8fb
|
This pull request was exported from Phabricator. Differential Revision: D35338905 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: minibatch_size and minibatches_per_step are unused in dqn trainers.
Differential Revision: D35338905