nanoGPT Reimplement

Reimplement nanoGPT, a local trainable and finetunable agent, into object-oriented style with typing checks.

Instead of using the example karpathy/chr_rnn/tinyshakespeare/input.txt, this repo support multiple .txt files from the folder input/ to be trained with.

Transformer - model architecture

Deviation: Unlike the Add & Norm after each layer of the original paper, this repo implements pre-norm formulation.

Getting Started

Create virtual environment and install pyhton libraries.

Hint: On the official webpage, scroll down to find Miniconda Installers section!

Conda for System Dependencies

% conda env create -f environment.yml

UV for no-sys-dependent packages

% uv venv
% uv add -r requirements.txt

[Optional] Compile as a module

% pip install -e .

Train and Validate

% pyhon src/main.py

After optional local compile

% nanogpt

Sample

Finetune

Testing

pytest .