When a single A100 80G ,memory is about 96G,Error loading 65B

[dpyneo]

First of all, thank you very much for this great project。When a single A100 80G loads the fifth weight file, it will prompt that the process is killed（signal）. Is this because of insufficient memory? My memory is about 96G。

[tloen]

That seems likely. Have you tried increasing the size of your swapfile?

[ChaoChungWu-Johnson]

I'm also having the troubles of insufficient cpu ram as well. Would you mind clarifying how much CPU ram is required when using in8 version for like 13B, 33B and 65B llama? I want to adjust hardware spec plan according to your advice. Thanks!

[dpyneo]

Sorry, the following is the time to load 13B (653s), 30B (1672s) and 65B (4242s). Because I use the weights placed on the mobile hard disk, it may take a little longer. And I found that after loading the two weight files of 13B, and then loading the four weight files of 30B, it will directly jump to the third weight file before starting. The first two weight files are directly loaded, and the memory required for loading 30B is about 96G, The video memory is about 37G. After increasing the swap to 120G, 30B can be loaded without being killed. When loading 65B, the memory needs about 96G+50G (swap), while the video memory needs about 70G