docs: improve readability #667
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: Pull Request resolved: facebookresearch#491 Reviewed By: czxttkl, bankawas Differential Revision: D29251412 fbshipit-source-id: 0a6cbcf59956ecc113e9425079f91a6b3098c2de
Summary: Pull Request resolved: facebookresearch#492 Reviewed By: bankawas Differential Revision: D29252722 fbshipit-source-id: d855c6688199d2c3a09fab200e9b8d66c52d7273
Summary: We've implemented data modules; this method is redundant Reviewed By: bankawas Differential Revision: D29252903 fbshipit-source-id: 044cde768b481d4a12d4a17cca42180b4bd989cb
Summary: redundant Reviewed By: bankawas Differential Revision: D29252914 fbshipit-source-id: 536982d3b7886bda68fc14c5c933343167213224
Summary: redundant Reviewed By: bankawas Differential Revision: D29253003 fbshipit-source-id: cd05c62a0840b4f2d10c8bf4d9fe9ea057b6a13f
Summary: redundant Reviewed By: bankawas Differential Revision: D29253030 fbshipit-source-id: 969d03b6428aead6c6982a26b2e2c4a9a940273f
Summary: This is the start of making model manager stateless to reduce complexity Reviewed By: czxttkl Differential Revision: D29253248 fbshipit-source-id: 681d141cb46784e40c8802f2325c1636044c61de
…less Summary: Removing state from model managers Reviewed By: czxttkl Differential Revision: D29253249 fbshipit-source-id: 93ecb090cd2e2b66f86480679ae6145519227360
Summary: Prereq for making model managers stateless Reviewed By: czxttkl Differential Revision: D29253385 fbshipit-source-id: 9db747f46a84f26bce079efe8c4394efd3c8adc7
…h#493) Summary: Pull Request resolved: facebookresearch#493 Finally removed normalization data from model manager state Reviewed By: czxttkl Differential Revision: D29253429 fbshipit-source-id: 619b93b473e49b07fe74d0b525d6fc5f30f52550
Summary: Give it to `build_trainer()` directly so that we can remove state in model managers Reviewed By: czxttkl Differential Revision: D29258017 fbshipit-source-id: 39f4a7e8ad9a92499ffeb3c04e2e1c61c10769c0
Summary: Pull Request resolved: facebookresearch#494 - Remove `initialize_trainer()` - Implement `train()` on ModelManager base class; remove all the duplicates - Make `build_serving_module[s]()` takes the trainer module so it can extract whatever nets in the trainer module - `ModelManager.train()` now returns `Tuple[RLTrainingOutput, pl.Trainer]` so that `_lightning_trainer` member can be deleted Reviewed By: czxttkl Differential Revision: D29258016 fbshipit-source-id: 71545dc77c386b532bb48fe4c8ee94c79c20f5c6
Summary: Implement multi-stage trainer module so that multi-stage training looks the same as other training. Internally, the multi-stage trainer forward calls to internal trainers. Reviewed By: czxttkl Differential Revision: D29273266 fbshipit-source-id: b51e91e5670362fc8ed85d9eeb05bd685fc7cbfd
Differential Revision: D29398026 fbshipit-source-id: 76923009da0f6fbc82a9fa8ae96c9417422c2577
Summary: Pull Request resolved: facebookresearch#497 Reviewed By: czxttkl Differential Revision: D29405221 fbshipit-source-id: 3e3524d92fb8d243b7fe62a04830b8f2b80df6ce
Differential Revision: D29458224 fbshipit-source-id: dcef29cd83ee7aecc94100ed579d023072ab581e
Summary: Pull Request resolved: facebookresearch#498 Add some assertions to make sure end users can use algorithms correctly. Reviewed By: bankawas Differential Revision: D29481662 fbshipit-source-id: 0332d990df7d3eca61e1f7bd205136d32f04a7b2
Summary: Pull Request resolved: facebookresearch#499 Remove Seq2SlateDifferentiableRewardTrainer because it's not tested and wouldn't be used. Reviewed By: kittipatv Differential Revision: D29522083 fbshipit-source-id: 9cd7e0d6d1d10c17cc174a54d77a4b37b0f279b7
Summary: Pull Request resolved: facebookresearch#500 Migrate the regular seq2slate to PyTorch Lightning, which includes one model manager `Seq2SlateTransformer` and three trainers `Seq2SlateTrainer`, `Seq2SlateSimulationTrainer` and `Seq2SlateTeacherForcingTrainer`. Manual optimization (https://pytorch-lightning.readthedocs.io/en/latest/common/optimizers.html#manual-optimization) is used to handle the sophisticated usage of optimizers during training. Model manager `Seq2SlatePairwiseAttn` and trainer `Seq2SlatePairwiseAttnTrainer` are not migrated in this diff. But to make them compatible with the changes, the setting of `minibatch_size` is also moved from `trainer_params` to `reader_options`. Reviewed By: czxttkl Differential Revision: D29436608 fbshipit-source-id: 612a1de4923eb7d138fcb6cb4715be6e4d05b424
Summary: AutoDataModule yields dictionary of tensors. Therefore, we need to manually type the input Reviewed By: czxttkl Differential Revision: D29479986 fbshipit-source-id: ab135bb869d8f0eb1fba1813aebf5af6d5ca3401
Differential Revision: D29573192 fbshipit-source-id: 65dc670d1777dd1d6b86c9228a198cd16f504c6e
…h#489) Summary: Pull Request resolved: facebookresearch#489 Reviewed By: czxttkl Differential Revision: D29144000 fbshipit-source-id: b72401ee3bb69f4973c32914a440e571d56241f6
…ebookresearch#502) Summary: Pull Request resolved: facebookresearch#502 Use transformers to learn the return decomposition model. 1) customized attention layers that feed positional encoding to Key & Query but not V. 2) residual connections that learn meaningful embeddings. Reviewed By: czxttkl Differential Revision: D29346526 fbshipit-source-id: c6e642548d4d2b0bcc7f089c08d9144c6f96f8e0
Reviewed By: zertosh Differential Revision: D29656934 fbshipit-source-id: c40bbc8e4512b145050ee47db2c8dc781f3c36e9
…search#501) Summary: Pull Request resolved: facebookresearch#501 Migrate model manager `Seq2SlatePairwiseAttn` and trainer `Seq2SlatePairwiseAttnTrainer` to PyTorch Lightning. This diff marks the completeness of the migration to PyTorch Lightning for the entire reagent codebase. `train_and_evaluate_generic` is removed. Only `train_eval_lightning` from now on! Reviewed By: kittipatv, czxttkl Differential Revision: D29545053 fbshipit-source-id: 71d115c07354b297d3b56d9bfcd13854cd60cb34
Summary: Pull Request resolved: facebookresearch#503 (1) Entropy regularization is added in the CRR to test whether it can help improve the stability of the training or not. (2) Modification in rl_offline_analysis: extract `dqn` manifold path from CRR outputs. Reviewed By: czxttkl Differential Revision: D29469826 fbshipit-source-id: 705ee9069edff9a2b2ff5362d3c4ff464b5a27bd
Summary: There are several modules in the ReAgent library where the logger level is set in the library code thus overriding the level set by the library client resulting in very verbose stdout. This diff removes places in the library where the logger level is set so that the client's setting is always maintained. Reviewed By: bankawas Differential Revision: D29673661 fbshipit-source-id: 8f6db342571d4524768f75d6d6bf4416bad8ad1c
Summary: Delete old style trainer classes Reviewed By: czxttkl Differential Revision: D29700788 fbshipit-source-id: 2f4448d9a7cb8d31d11b25bf35184e1f8c1ce9f6
Differential Revision: D29738340 fbshipit-source-id: 97c83cea89c46c469cdc967cce2ac7ce281c55fc
Summary: Pull Request resolved: facebookresearch#508 Reviewed By: czxttkl Differential Revision: D29805519 fbshipit-source-id: dbcde11f8292eb167a0b7a66384e0d1d723b38e4
Summary: Applies new import merging and sorting from µsort v1.0. When merging imports, µsort will make a best-effort to move associated comments to match merged elements, but there are known limitations due to the diynamic nature of Python and developer tooling. These changes should not produce any dangerous runtime changes, but may require touch-ups to satisfy linters and other tooling. Note that µsort uses case-insensitive, lexicographical sorting, which results in a different ordering compared to isort. This provides a more consistent sorting order, matching the case-insensitive order used when sorting import statements by module name, and ensures that "frog", "FROG", and "Frog" always sort next to each other. For details on µsort's sorting and merging semantics, see the user guide: https://usort.readthedocs.io/en/stable/guide.html#sorting Reviewed By: lisroach Differential Revision: D36402214 fbshipit-source-id: b641bfa9d46242188524d4ae2c44998922a62b4c
Summary: Pull Request resolved: facebookresearch#635 as titled Reviewed By: alexnikulkov Differential Revision: D36021439 fbshipit-source-id: ce008f941caf2d2b137851662a0b7926bd8520f8
Summary: Pull Request resolved: facebookresearch#641 as titled Reviewed By: alexnikulkov Differential Revision: D36039334 fbshipit-source-id: 863027d8ad1a65cd5510853ccca8e947f88ef6e0
Summary: Pull Request resolved: facebookresearch#643 1. Add new sections to YAML for model and optimizer configs 2. Add support for weights in Parametric DQN input 3. Expose FC hidden layer dims in config 4. Sort data in the batch by separable_id, timestamp, position. 5. Zero-out the weight for observations for which we don't know the next state ("terminal", but they are actually not terminal, we just don't know their next state), the time_diff is negative or the position feature is missing, preventing us from sorting properly. 6. Read and pass in the batch time gap to next state 7. Clip reward (paced bid) To launch MC LTV training: - local run: `starlight app run -j 1 free.reagent.train_ltv:train` - submit to MAST: `starlight app submit reagent/submit_config.py:get_config_ltv` To launch SARSA LTV training: - local run: `starlight app run -j 1 free.reagent.train_ltv:train_sarsa` - submit to MAST: `starlight app submit reagent/submit_config.py:get_config_ltv -- --model_type SARSA` Reviewed By: czxttkl Differential Revision: D36360500 fbshipit-source-id: c07f0b2ea297844970389b2059a7c42d63d16a8d
Summary: Pull Request resolved: facebookresearch#644 Add extra optional columns of mdp_id and arms in the ReAgent codebase. These are used in eval workflow for linucb. Reviewed By: alexnikulkov Differential Revision: D36493574 fbshipit-source-id: 509a5b17617381b244202d7a857a7d7b1eb8bcc9
…ch#645) Summary: Pull Request resolved: facebookresearch#645 Instead of always using a linear output activation, I want to specify which activation to use. I want to try this with positive activation functions (e.g. relu) because I know that my Q-values have to be positive (all rewards are non-negative) Reviewed By: czxttkl Differential Revision: D36744360 fbshipit-source-id: 81296d2dfebb0ec77917d6024c0216d0e3fed4d1
Summary: pyfmt now specifies a target Python version of 3.8 when formatting with black. With this new config, black adds trailing commas to all multiline function calls. This applies the new formatting as part of rolling out the linttool-integration for pyfmt. paintitblack Reviewed By: zertosh, lisroach Differential Revision: D37084377 fbshipit-source-id: 781a1b883a381a172e54d6e447137657977876b4
Differential Revision: D37172467 fbshipit-source-id: c5b8f6fceb327eb61a013836f57d315fd6b17211
Summary: Pull Request resolved: facebookresearch#646 Fixing this problem: https://fb.workplace.com/groups/horizon.users/posts/1105605083328644/ Differential Revision: D37244190 fbshipit-source-id: 498d6f0790d3954f83758c4d46a6203c672fff67
Differential Revision: D37305159 fbshipit-source-id: 3532b6de87431137832c546d6544ba8d419cd726
Summary: Pull Request resolved: facebookresearch#647 X-link: meta-pytorch/torchrec#447 Add a static function to concat a list of KJTs Reviewed By: dstaay-fb Differential Revision: D36944002 fbshipit-source-id: 1b6865f60dcea91ee250b69360e4606184ffad53
Summary: Made LinUCB neural-based to enable distributed training. Differential Revision: D37009962 fbshipit-source-id: 4bb3e68ea60a264d26e13e4ce19832bca67a2c7e
Differential Revision: D37353746 fbshipit-source-id: b0dc7a8b59f4c6a1e39daa67fbae1e519488b6ef
Differential Revision: D37371791 fbshipit-source-id: 8247af28ab27242782ab8d317e7d9763121bcc74
Summary: Pull Request resolved: facebookresearch#649 fix world model reporter so that we can read losses per epoch make the logic needed to perform at the end of a train/test/validation epoch more explicit in reagent_lightning_module Differential Revision: D37305377 fbshipit-source-id: 2204cfe94269cfba839b72c77bfea341ab63637d
Summary: We want to make the EB/EBC scriptable by default w/o the need of running torch.fx first. Not able to script EB/EBC modules by default (especially when needed for inference) is very non-intuitive and inconvenient. In the same time we don't plan to make ShardedEB/EBC scriptable. 1. Do not use property, use methods instead. This is consistent w/ KJT. 1. _embedding_bag_configs has a complex type List[EmbeddingBagConfig] which does not script. Use List[str] to store features instead. Pull Request resolved: facebookresearch#648 X-link: meta-pytorch/torchrec#467 Reviewed By: colin2328 Differential Revision: D37389962 fbshipit-source-id: 5ce079a946b9458ee63658cae2fd731cfc1c7958
Summary: Pull Request resolved: facebookresearch#650 In the base LightningModule class we define an optional `logger` property which may be initialized to None, while in DQNtrainer._log_dqn method we try to access the `logger` object without checking first if it was initialized. The issue surfaced when trying to run unit tests analogous to those in `test_qrdqn`. This commit adds a check whether the `logger` is initialized prior to attempting to use it. Interestingly, the analogous QRDQNTrainer class implementation does not use the `logger` property for logging, perhaps it's redundant? Reviewed By: czxttkl Differential Revision: D37529027 fbshipit-source-id: 5fe81cf715ee9f759b937290f1184d1c67e5325f
Summary: Pull Request resolved: facebookresearch#651 Adds the test_dqn.py with a set of unit tests for DQNTrainer class, mirroring those in test_qrdqn.py Reviewed By: czxttkl Differential Revision: D37536537 fbshipit-source-id: 60cef76adb62c54e66b3fda39596c1cf0ad20555
…#658) Summary: Pull Request resolved: facebookresearch#658 1. update machine images for some cpu-only tests 2. we have to switch back to use torchrec nightly (instead of stable) because the torchrec cpu stable version has caused some error in circle ci tests. See for example: https://app.circleci.com/pipelines/github/facebookresearch/ReAgent/2437/workflows/1c7658b7-1197-425d-9fe3-166876d492ac/jobs/23829 Reviewed By: speedystream Differential Revision: D38101590 fbshipit-source-id: d72f2cf0d204598ef648e0969522e9801029eca4
Summary: Pull Request resolved: facebookresearch#659 two things will help avoid openssl errors when installing python 3.8: upgrade pyenv add a retry logic for installing python 3.8. Reviewed By: speedystream Differential Revision: D38123668 fbshipit-source-id: 7e527a0caf2d302a81306b7f2005e92ce19a6f5e
Summary: To avoid test failures caused by importing torchrec, i finally decide the following import rules: For gpu machines, import torchrec (gpu, stable version) For cpu machines, import torchrec-nightly-cpu Reviewed By: speedystream Differential Revision: D38185498 fbshipit-source-id: 7988695f827cfd04d53f6d63630ac843eb6c23ee
Summary: Pull Request resolved: facebookresearch#657 1. add docstrings 2. test if models are torch.jit.trac-able Reviewed By: dkorenkevych Differential Revision: D38067071 fbshipit-source-id: 7863e0e1f3f618ee7fe46c6fa076fec1dd6fd48a
Summary: Pull Request resolved: facebookresearch#653 Add test_dqn_base.py file with unit tests for the methods in DQNTrainerBaseLightning class. Reviewed By: czxttkl Differential Revision: D37673366 fbshipit-source-id: 43482dde9be06a0df1e8dd3bb16e92d508bc8a13
…ebookresearch#654) Summary: Pull Request resolved: facebookresearch#654 Add docstrings to DQNTrainer and DQNTrainerBaseLightning classes and their methods. Reviewed By: czxttkl Differential Revision: D37875900 fbshipit-source-id: 52e9947f1c84f099bedb79a696de94a05c631f5c
Summary: Pull Request resolved: facebookresearch#663 This is the first part of the diff. I have moved the model in to reagent. Moreover, I have made some refactoring. Reviewed By: czxttkl Differential Revision: D38011818 fbshipit-source-id: cb76646ed0e3149887180cbe642b1035afaace9b
Differential Revision: D38447983 fbshipit-source-id: 03d4384d075a57bfbd9a76c23730307fc5255c90
Summary: Pull Request resolved: facebookresearch#665 we need to unfold embeddings from different sparse features Differential Revision: D38556778 fbshipit-source-id: 8eb646105991c0307d981bb3198c48e850cededa
Signed-off-by: Ryan Russell <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Ryan Russell [email protected]