Add multi-task models: SharedBottom, ESMM, MMOE, PLE

* add multitask mdoels

1. Add multi-task models: SharedBottom, ESMM, MMOE, PLE
2. Bugfix:
#240
#232

* support python 3.9/3.10 (#259)
* fix: variable name typo (#257)
Co-authored-by: Jason Zan <zanshuxun@aliyun.com>
Co-authored-by: Yi-Xuan Xu <xuyx@lamda.nju.edu.cn>

Author: 3 people

.github/ISSUE_TEMPLATE/bug_report.md

@@ -18,9 +18,9 @@ Steps to reproduce the behavior:
 4. See error
 
 **Operating environment(运行环境):**
- - python version [e.g. 3.5, 3.6]
- - torch version [e.g. 1.6.0, 1.7.0]
- - deepctr-torch version [e.g. 0.2.7,]
+ - python version [e.g. 3.6, 3.7]
+ - torch version [e.g. 1.9.0, 1.10.0]
+ - deepctr-torch version [e.g. 0.2.9,]
 
 **Additional context**
 Add any other context about the problem here.

.github/ISSUE_TEMPLATE/question.md

@@ -16,5 +16,5 @@ Add any other context about the problem here.
 
 **Operating environment(运行环境):**
  - python version [e.g. 3.6]
- - torch version [e.g. 1.7.0,]
- - deepctr-torch version [e.g. 0.2.7,]
+ - torch version [e.g. 1.10.0,]
+ - deepctr-torch version [e.g. 0.2.9,]

.github/workflows/ci.yml

@@ -17,19 +17,48 @@ jobs:
     timeout-minutes: 120
     strategy:
       matrix:
-        python-version: [3.6,3.7,3.8]
-        torch-version: [1.1.0,1.2.0,1.3.0,1.4.0,1.5.0,1.6.0,1.7.1,1.8.1,1.9.0,1.10.2,1.11.0]
+        python-version: [3.6,3.7,3.8,3.9,3.10.7]
+        torch-version: [1.2.0,1.3.0,1.4.0,1.5.0,1.6.0,1.7.1,1.8.1,1.9.0,1.10.2,1.11.0,1.12.1]
 
         exclude:
           - python-version: 3.6
             torch-version: 1.11.0
-          - python-version: 3.8
-            torch-version: 1.1.0
+          - python-version: 3.6
+            torch-version: 1.12.1
           - python-version: 3.8
             torch-version: 1.2.0
           - python-version: 3.8
             torch-version: 1.3.0
-
+          - python-version: 3.9
+            torch-version: 1.2.0
+          - python-version: 3.9
+            torch-version: 1.3.0
+          - python-version: 3.9
+            torch-version: 1.4.0
+          - python-version: 3.9
+            torch-version: 1.5.0
+          - python-version: 3.9
+            torch-version: 1.6.0
+          - python-version: 3.9
+            torch-version: 1.7.1
+          - python-version: 3.10.7
+            torch-version: 1.2.0
+          - python-version: 3.10.7
+            torch-version: 1.3.0
+          - python-version: 3.10.7
+            torch-version: 1.4.0
+          - python-version: 3.10.7
+            torch-version: 1.5.0
+          - python-version: 3.10.7
+            torch-version: 1.6.0
+          - python-version: 3.10.7
+            torch-version: 1.7.1
+          - python-version: 3.10.7
+            torch-version: 1.8.1
+          - python-version: 3.10.7
+            torch-version: 1.9.0
+          - python-version: 3.10.7
+            torch-version: 1.10.2
     steps:
 
     - uses: actions/checkout@v3

README.md

@@ -38,10 +38,14 @@ Let's [**Get Started!**](https://deepctr-torch.readthedocs.io/en/latest/Quick-St
 |                AutoInt                 | [CIKM 2019][AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://arxiv.org/abs/1810.11921)                              |
 |                  ONN                   | [arxiv 2019][Operation-aware Neural Networks for User Response Prediction](https://arxiv.org/pdf/1904.12579.pdf)                                                |
 |                FiBiNET                 | [RecSys 2019][FiBiNET: Combining Feature Importance and Bilinear feature Interaction for Click-Through Rate Prediction](https://arxiv.org/pdf/1905.09433.pdf)   |
-|                IFM                 | [IJCAI 2019][An Input-aware Factorization Machine for Sparse Prediction](https://www.ijcai.org/Proceedings/2019/0203.pdf)   |
-|                DCN V2                    | [arxiv 2020][DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://arxiv.org/abs/2008.13535)   |
-|                DIFM                 | [IJCAI 2020][A Dual Input-aware Factorization Machine for CTR Prediction](https://www.ijcai.org/Proceedings/2020/0434.pdf)   |
-|                AFN                 | [AAAI 2020][Adaptive Factorization Network: Learning Adaptive-Order Feature Interactions](https://arxiv.org/pdf/1909.03276)   |
+|                  IFM                   | [IJCAI 2019][An Input-aware Factorization Machine for Sparse Prediction](https://www.ijcai.org/Proceedings/2019/0203.pdf)   |
+|                  DCN V2                | [arxiv 2020][DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://arxiv.org/abs/2008.13535)   |
+|                  DIFM                  | [IJCAI 2020][A Dual Input-aware Factorization Machine for CTR Prediction](https://www.ijcai.org/Proceedings/2020/0434.pdf)   |
+|                  AFN                   | [AAAI 2020][Adaptive Factorization Network: Learning Adaptive-Order Feature Interactions](https://arxiv.org/pdf/1909.03276)   |
+|               SharedBottom             | [arxiv 2017][An Overview of Multi-Task Learning in Deep Neural Networks](https://arxiv.org/pdf/1706.05098.pdf)  |
+|                  ESMM                  | [SIGIR 2018][Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate](https://dl.acm.org/doi/10.1145/3209978.3210104)                       |
+|                  MMOE                  | [KDD 2018][Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts](https://dl.acm.org/doi/abs/10.1145/3219819.3220007)                   |
+|                  PLE                   | [RecSys 2020][Progressive Layered Extraction (PLE): A Novel Multi-Task Learning (MTL) Model for Personalized Recommendations](https://dl.acm.org/doi/10.1145/3383313.3412236)                   |
 
 
 

deepctr_torch/__init__.py

@@ -2,5 +2,5 @@
 from . import models
 from .utils import check_version
 
-__version__ = '0.2.8'
+__version__ = '0.2.9'
 check_version(__version__)

deepctr_torch/models/__init__.py

@@ -15,4 +15,5 @@
 from .ccpm import CCPM
 from .dien import DIEN
 from .din import DIN
-from .afn import AFN
+from .afn import AFN
+from .multitask import SharedBottom, ESMM, MMOE, PLE

deepctr_torch/models/basemodel.py

@@ -245,7 +245,13 @@ def fit(self, x=None, y=None, batch_size=None, epochs=1, verbose=1, initial_epoc
                         y_pred = model(x).squeeze()
 
                         optim.zero_grad()
-                        loss = loss_func(y_pred, y.squeeze(), reduction='sum')
+                        if isinstance(loss_func, list):
+                            assert len(loss_func) == self.num_tasks,\
+                                "the length of `loss_func` should be equal with `self.num_tasks`"
+                            loss = sum(
+                                [loss_func[i](y_pred[:, i], y[:, i], reduction='sum') for i in range(self.num_tasks)])
+                        else:
+                            loss = loss_func(y_pred, y.squeeze(), reduction='sum')
                         reg_loss = self.get_regularization_loss()
 
                         total_loss = loss + reg_loss + self.aux_loss
@@ -456,18 +462,24 @@ def _get_optim(self, optimizer):
 
     def _get_loss_func(self, loss):
         if isinstance(loss, str):
-            if loss == "binary_crossentropy":
-                loss_func = F.binary_cross_entropy
-            elif loss == "mse":
-                loss_func = F.mse_loss
-            elif loss == "mae":
-                loss_func = F.l1_loss
-            else:
-                raise NotImplementedError
+            loss_func = self._get_loss_func_single(loss)
+        elif isinstance(loss, list):
+            loss_func = [self._get_loss_func_single(loss_single) for loss_single in loss]
         else:
             loss_func = loss
         return loss_func
 
+    def _get_loss_func_single(self, loss):
+        if loss == "binary_crossentropy":
+            loss_func = F.binary_cross_entropy
+        elif loss == "mse":
+            loss_func = F.mse_loss
+        elif loss == "mae":
+            loss_func = F.l1_loss
+        else:
+            raise NotImplementedError
+        return loss_func
+
     def _log_loss(self, y_true, y_pred, eps=1e-7, normalize=True, sample_weight=None, labels=None):
         # change eps to improve calculation accuracy
         return log_loss(y_true,

deepctr_torch/models/dcnmix.py

@@ -71,9 +71,10 @@ def __init__(self, linear_feature_columns,
         self.add_regularization_weight(
             filter(lambda x: 'weight' in x[0] and 'bn' not in x[0], self.dnn.named_parameters()), l2=l2_reg_dnn)
         self.add_regularization_weight(self.dnn_linear.weight, l2=l2_reg_linear)
-        self.add_regularization_weight(self.crossnet.U_list, l2=l2_reg_cross)
-        self.add_regularization_weight(self.crossnet.V_list, l2=l2_reg_cross)
-        self.add_regularization_weight(self.crossnet.C_list, l2=l2_reg_cross)
+        regularization_modules = [self.crossnet.U_list, self.crossnet.V_list, self.crossnet.C_list]
+        for module in regularization_modules:
+            self.add_regularization_weight(module, l2=l2_reg_cross)
+
         self.to(device)
 
     def forward(self, X):

deepctr_torch/models/dien.py

@@ -217,7 +217,7 @@ def forward(self, keys, keys_length, neg_keys=None):
 
         masked_keys = torch.masked_select(keys, mask.view(-1, 1, 1)).view(-1, max_length, dim)
 
-        packed_keys = pack_padded_sequence(masked_keys, lengths=masked_keys_length, batch_first=True,
+        packed_keys = pack_padded_sequence(masked_keys, lengths=masked_keys_length.cpu(), batch_first=True,
                                            enforce_sorted=False)
         packed_interests, _ = self.gru(packed_keys)
         interests, _ = pad_packed_sequence(packed_interests, batch_first=True, padding_value=0.0,
@@ -353,7 +353,7 @@ def forward(self, query, keys, keys_length, mask=None):
         query = torch.masked_select(query, mask.view(-1, 1)).view(-1, dim).unsqueeze(1)
 
         if self.gru_type == 'GRU':
-            packed_keys = pack_padded_sequence(keys, lengths=keys_length, batch_first=True, enforce_sorted=False)
+            packed_keys = pack_padded_sequence(keys, lengths=keys_length.cpu(), batch_first=True, enforce_sorted=False)
             packed_interests, _ = self.interest_evolution(packed_keys)
             interests, _ = pad_packed_sequence(packed_interests, batch_first=True, padding_value=0.0,
                                                total_length=max_length)
@@ -362,15 +362,15 @@ def forward(self, query, keys, keys_length, mask=None):
         elif self.gru_type == 'AIGRU':
             att_scores = self.attention(query, keys, keys_length.unsqueeze(1))  # [b, 1, T]
             interests = keys * att_scores.transpose(1, 2)  # [b, T, H]
-            packed_interests = pack_padded_sequence(interests, lengths=keys_length, batch_first=True,
+            packed_interests = pack_padded_sequence(interests, lengths=keys_length.cpu(), batch_first=True,
                                                     enforce_sorted=False)
             _, outputs = self.interest_evolution(packed_interests)
             outputs = outputs.squeeze(0) # [b, H]
         elif self.gru_type == 'AGRU' or self.gru_type == 'AUGRU':
             att_scores = self.attention(query, keys, keys_length.unsqueeze(1)).squeeze(1)  # [b, T]
-            packed_interests = pack_padded_sequence(keys, lengths=keys_length, batch_first=True,
+            packed_interests = pack_padded_sequence(keys, lengths=keys_length.cpu(), batch_first=True,
                                                     enforce_sorted=False)
-            packed_scores = pack_padded_sequence(att_scores, lengths=keys_length, batch_first=True,
+            packed_scores = pack_padded_sequence(att_scores, lengths=keys_length.cpu(), batch_first=True,
                                                  enforce_sorted=False)
             outputs = self.interest_evolution(packed_interests, packed_scores)
             outputs, _ = pad_packed_sequence(outputs, batch_first=True, padding_value=0.0, total_length=max_length)

deepctr_torch/models/fibinet.py

@@ -45,9 +45,9 @@ def __init__(self, linear_feature_columns, dnn_feature_columns, bilinear_type='i
                                       device=device, gpus=gpus)
         self.linear_feature_columns = linear_feature_columns
         self.dnn_feature_columns = dnn_feature_columns
-        self.filed_size = len(self.embedding_dict)
-        self.SE = SENETLayer(self.filed_size, reduction_ratio, seed, device)
-        self.Bilinear = BilinearInteraction(self.filed_size, self.embedding_size, bilinear_type, seed, device)
+        self.field_size = len(self.embedding_dict)
+        self.SE = SENETLayer(self.field_size, reduction_ratio, seed, device)
+        self.Bilinear = BilinearInteraction(self.field_size, self.embedding_size, bilinear_type, seed, device)
         self.dnn = DNN(self.compute_input_dim(dnn_feature_columns), dnn_hidden_units,
                        activation=dnn_activation, l2_reg=l2_reg_dnn, dropout_rate=dnn_dropout, use_bn=False,
                        init_std=init_std, device=device)