Difference between v1 and v2 for xxlarge #266
Description
Hi,
I wanted to clarify a point from the paper and README that I am confused about.
In the paper, and the repo's README, it seems like the v1 model was trained only on wikipedia and the book corpus, to compare with BERT.
However, in the README, there's the following text:
On average, ALBERT-xxlarge is slightly worse than the v1, because of the following two reasons: 1) Training additional 1.5 M steps (the only difference between these two models is training for 1.5M steps and 3M steps) did not lead to significant performance improvement. 2) For v1, we did a little bit hyperparameter search among the parameters sets given by BERT, Roberta, and XLnet.
This implies that the xxlarge version of v1 was also trained on additional data.
The question is whether the v1 xxlarge model was solely trained on wiki+books, or was it trained on additional data?