ch-03 notebook

[IamAGP]

in section 3.4.1 , computing attention weights step by step , as shown in the screenshot here. attention score omega21 shouldn't it be 1.1 instead of 1.2 . just wanted to check once, my understanding of the calculation. Also a suggestion, throughout the notebook,can we have consistent naming like attention score and attention weight. attention score is unnnormalised and attention weight is normalised. do we need to define it as unnormalised attention score, coz they both are one and the same, right??

Completely unrelated, but a request, can u also dedicate one complete repo for DIFFUSION MODELS ? that would be amazing to cover two different paradigms (transformers and diffusion) and also different modality (one for images)..

[d-kleine]

Hi @IamAGP,

As a fellow reader, I double-checked the value for $\omega_{21}$, and it appears to be correct. For reference, the actual value for $\omega_{21}$ is 1.2705. The values shown in the figure are truncated after the first decimal place, therefore this value is truncated to 1.2:

LLMs-from-scratch/ch03/01_main-chapter-code/ch03.ipynb

Lines 210 to 211 in 35354fa

	"- (Please note that the numbers in this figure are truncated to one\n",
	"digit after the decimal point to reduce visual clutter; similarly, other figures may also contain truncated values)"

It looks like you used the truncated values from the figure for calculating the unscaled attention scores, which explains the slight difference.

[rasbt]

Thanks for answer. And that's correct, I truncated the numbers in the image. I remember that I used rounded values at first, but then it was more difficult to find the corresponding numbers in the array vs images. In that case, also the rounded numbers would have given slightly different results.

In hindsight, maybe one solution could be to force all numbers to be truncated or rounded in the notebook as well. (Maybe to 2 digits instead of 1 digit). But yeah, it's tricky.