add gpt-3 note

Author: healeycodes

notes/1781846462653.md

@@ -0,0 +1,31 @@
+Was my writing used to train GPT-3? Answer: probably.
+
+[Common Crawl](https://commoncrawl.org/) is one of the publicly known ingredients used to [train GPT-3](https://arxiv.org/pdf/2005.14165):
+
+> Table 2.2 shows the final mixture of datasets that we used in training. The CommonCrawl data was downloaded from
+41 shards of monthly CommonCrawl covering 2016 to 2019, constituting 45TB of compressed plaintext before filtering
+and 570GB after filtering, roughly equivalent to 400 billion byte-pair-encoded tokens.
+
+We can also check that some of my early blog posts were present in a Common Crawl snapshot by querying one of the monthly indexes:
+
+```text
+curl "https://index.commoncrawl.org/CC-MAIN-2019-51-index?url=healeycodes.com/*&output=json" |
+  jq -rs 'map(.urlkey) |
+    join("\n")'
+com,healeycodes)/
+com,healeycodes)/about
+com,healeycodes)/discuss/python/beginners/tutorial/2019/06/18/tricking-google-with-apostrophes.html
+com,healeycodes)/javascript/challenge/webdev/2019/11/17/hardest-javascript-puzzle.html
+com,healeycodes)/javascript/python/beginners/webdev/2019/04/11/talking-between-languages.html
+com,healeycodes)/javascript/webdev/node/tutorial/2019/11/11/lets-generate-captchas.html
+com,healeycodes)/robots.txt
+```
+
+However, that data was then filtered:
+
+>  [...] we took 3 steps to improve the average quality of our datasets:
+(1) we downloaded and filtered a version of CommonCrawl based on similarity to a range of high-quality reference
+corpora, (2) we performed fuzzy deduplication at the document level, within and across datasets, to prevent redundancy
+and preserve the integrity of our held-out validation set as an accurate measure of overfitting, [...]
+
+But I'd like to think my writing is high quality enough to survive such filtering!