vipin wiki

Karpathy Idea Map

39 min read

Karpathy Idea Map

This page turns the public corpus into a retrieval map for Karpathy’s reusable ideas: first-principles building, serious minimalism, from-scratch pedagogy, research tooling, LLM systems, and agentic experimentation.

Reading Pattern

  • EXTRACTED: Repositories provide runnable artifacts and implementation taste.
  • EXTRACTED: Blog posts and videos provide long-form explanations and heuristics.
  • EXTRACTED: The curated tweet page provides official pointers to short-form ideas but is not a complete tweet archive.
  • INFERRED: The strongest recurring pattern is to compress a hard system into the smallest complete artifact that still teaches the real mechanism.

Browser / JavaScript ML experiments

  • calorie: calorie !1 online. So, by default, your body will burn e.g. 2000 kcal/day. Then if you eat food you take in calories. And if you exercise you burn calories. If you are in a net deficit, you lose weight. On top of that, you have a certain weight loss goal. For example for me that might be 500 kcal/day of deficit, so that every week you burn 7 500 = 3500 kcal, which is roughly 1 pound of fat. So the idea is to have two counters tha…
  • tsnejs: tSNEJS tSNEJS is an implementation of t-SNE visualization algorithm in Javascript. t-SNE is a visualization algorithm that embeds things in 2 or 3 dimensions. If you have some data and you can measure their pairwise differences, t-SNE visualization can help you identify clusters in your data. See example below. Online demo The main project website has a 1 that allows you to simply paste CSV data into a textbox and tSNEJS computes and visualizes the embedding on the fly (no coding needed). Research Paper The algorithm was originally described in this paper: L.J.P. van de…

LLM education / from-scratch pedagogy

  • cryptos: cryptos Just me developing a pure Python from-scratch zero-dependency implementation of Bitcoin for educational purposes, including all of the under the hood crypto primitives such as SHA-256 and elliptic curves over finite fields math. SHA-256 My pure Python SHA-256 implementation closely following the $1 spec, in cryptos/sha256.py . Since this is a from scratch pure Python implementation it is slow and obviously not to be used anywhere except for educational purposes. Example usage: Keys getnewaddress.py is a cli entryway to generate a new Bitcoin secret/public key pair and the corresponding (base58check compre…

LLM training and inference systems

  • build-nanogpt: build nanoGPT This repo holds the from-scratch reproduction of 1 where you can see me introduce each commit and explain the pieces along the way. We basically start from an empty file and work our way to a reproduction of the 1 models. While the GPT-2 (124M) model probably trained for quite some time back in the day (2019, ~5 years ago), today…
  • cpython: This is Python version 3.15.0 alpha 2 ===================================== .. image:: https://github.com/python/cpython/actions/workflows/build.yml/badge.svg?branch=main&event=push :alt: CPython build status on GitHub Actions :target: https://github.com/python/cpython/actions .. image:: https://dev.azure.com/python/cpython/ apis/build/status/Azure%20Pipelines%20CI?branchName=main :alt: CPython build status on Azure DevOps :target: https://dev.azure.com/python/cpython/ build/latest?definitionId=4&branchName=main .. image:: https://img.shields.io/badge/discourse-join chat-brightgreen.svg :alt: Python Discourse cha…
  • examples: PyTorch Examples A repository showcasing examples of using $1 - MNIST Convnets - Word level Language Modeling using LSTM RNNs - Training Imagenet Classifiers with Residual Networks - Generative Adversarial Networks (DCGAN) - Variational Auto-Encoders - Superresolution using an efficient sub-pixel convolutional neural network - Hogwild training of shared ConvNets across multiple processes on MNIST - Training a CartPole to balance in OpenAI Gym with actor-critic - Natural Language Inference (SNLI) with GloVe vectors, LSTMs, and torchtext - Time sequence prediction - create an LSTM to learn Sine waves Additionally,…
  • hn-time-capsule: HN Time Capsule !1 for more context. What it does 1. Fetches the HN frontpage from 10 years ago (e.g., https://news.ycombinator.com/front?day=2015-12-09 ) 2. For each article, fetches the original article content and all HN comments 3. Generates prompts asking an LLM to analyze what happened with hindsight 4. Parses LLM responses to extract grades for each commenter 5. Renders an HTM…
  • karpathy.github.io: My blog This is my blog, uses $1. I was tired of bloated, slow Wordpress that locked up all my content.
  • llama2.c: llama2.c Have you ever wanted to inference a baby 1). You might think that you need many billion parameter LLMs to do anything useful, but in fact very small LLMs can have surprisingly strong performance if you make the domain narrow enough (ref: $1 paper). This repo is a “fullstack” train + inference solution for Llama 2 LLM, with focus on minimalism and simplicity. As the architecture is identical, you can also load and inference Meta’s Llama 2 models. However, the current…
  • llm.c: llm.c LLMs in simple, pure C/CUDA with no need for 245MB of PyTorch or 107MB of cPython. Current focus is on pretraining, in particular reproducing the 1 miniseries, along with a parallel PyTorch reference implementation in 1, an earlier project of mine. Currently, llm.c is a bit faster than PyTorch Nightly (by about 7%). In addition to the bleeding edge mainline code in 1. I’d like this repo to only maintain C and CUDA code. Ports to other languages or re…
  • LLM101n: LLM101n: Let’s build a Storyteller --- !!! NOTE: this course does not yet exist. It is current being developed by 1 What I cannot create, I do not understand. -Richard Feynman In this course we will build a Storyteller AI Large Language Model (LLM). Hand in hand, you’ll be able to create, refine and illustrate little $1 with the AI. We are going to build everything end-to-end from basics to a functioning web app similar to ChatGPT, from scratch in Python, C and CUDA, and with minimal computer science prerequisites. By the end you should have a relatively de…
  • minbpe: minbpe Minimal, clean code for the (byte-level) Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization. The BPE algorithm is “byte-level” because it runs on UTF-8 encoded strings. This algorithm was popularized for LLMs by the 1 from OpenAI. $1 is cited as the original reference for the use of BPE in NLP applications. Today, all modern LLMs (e.g. GPT, Llama, Mistral) use this algorithm to train their tokenizers. There are two Tokenizers in this repository, both of which can perform the 3 primary functions of a Tokenizer: 1) train the tokenizer vocabulary and merges on a…
  • minGPT: minGPT !1, both training and inference. minGPT tries to be small, clean, interpretable and educational, as most of the currently available GPT model implementations can a bit sprawling. GPT is not a complicated model and this implementation is appropriately about 300 lines of code (see 1, and a probability distribution over the next index in the sequence comes out. The majority of the complexity is just being clever with batching (both across examples and over sequence length) for efficiency. note (Jan 2023) :…
  • nanochat: nanochat !1 nanochat is the simplest experimental harness for training LLMs. It is designed to run on a single GPU node, the code is minimal/hackable, and it covers all major LLM stages including tokenization, pretraining, finetuning, evaluation, inference, and a chat UI. For example, you can train your own GPT-2 capability LLM (which cost ~48 (~2 hours of 8XH100 GPU node) and then talk to it in a familiar ChatGPT-like web UI. On a spot instance, the total cost can be closer to ~$15. More generally, nanochat is configured out of the box to train an entire miniseries of com…
  • nanoGPT: nanoGPT !1. It is very likely you meant to use/find nanochat instead. nanoGPT (this repo) is now very old and deprecated but I will leave it up for posterity. --- The simplest, fastest repository for training/finetuning medium-sized GPTs. It is a rewrite of $1 that prioritizes teeth over education. Still under active development, but currently the file train.py reproduces GPT-2 (124M) on OpenWebText, running on a single 8XA100 40GB node in about 4 days of training. The code itself is plain and readable: train.py is a ~300-line boilerplate traini…
  • ng-video-lecture: nanogpt-lecture Code created in the 1 for init all weights comment, and especially how it calls the init weights function. Even more sadly, the code in this repo i…
  • randomfun: Random fun Repo for random scripts and notebooks. License BSD.
  • reader3: reader 3 !1 has many), open them up in this reader, copy paste text around to your favorite LLM, and read together and along. This project was 90% vibe coded just to illustrate how one can very easily $1. I’m not going to support it in any way, it’s provided here as is for other people’s inspiration and I don’t intend to improve it. Code is ephemeral now and libraries are over, ask your LLM…
  • rendergit: rendergit Just show me the code. Tired of clicking around complex file hierarchies of GitHub repos? Do you just want to see all of the code on a single page? Enter rendergit . Flatten any GitHub repository into a single, static HTML page with syntax highlighting, markdown rendering, and a clean sidebar navigation. Perfect for code review, exploration, and an instant Ctrl+F experience. Basic usage Install and use easily with $1: Alternatively, more manual pip install example: The code will: 1. Clone the repo to a temporary directory 2. Render its source code into a single static temporary HTML file 3. Automaticall…
  • transformers: English 简体中文 繁體中文 한국어 State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow 🤗 Transformers provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio. These models can be applied on: 📝 Text, for tasks like text classification, information extraction, question answering, summarization, translation, text generation, in over 100 languages. 🖼️ Images, for tasks like image classification, object detection, and segmentation. 🗣️ Audio, for tasks like speech recognition and audio classification. Transformer models can also perform tasks on several moda…

Minimal implementations

  • An efficient, batched LSTM.: Public GitHub gist by Karpathy. Files: gistfile1.py.
  • Batched L2 Normalization Layer for Torch nn package: Public GitHub gist by Karpathy. Files: gistfile1.lua.
  • deep-vector-quantization: deep vector quantization Implements training code for VQVAE’s, i.e. autoencoders with categorical latent variable bottlenecks, which are then easy to subsequently plug into existing infrastructure for modeling sequences of discrete variables (GPT and friends). dvq/vqvae.py is the entry point of the training script and a small training run can be called e.g. as: cd dvq; python vqvae.py —gpus 1 —data dir /somewhere/to/store/cifar10 This will reproduce the original DeepMind VQVAE paper (see references before) using a semi-small network on CIFAR-10. Work on this repo is ongoing and for now requires reading of code…
  • Efficient LSTM cell in Torch: Public GitHub gist by Karpathy. Files: gistfile1.lua.
  • EigenLibSVM: EigenLibSVM Andrej Karpathy 1 May 2012 This is a small C++ wrapper to call libsvm if you use the Eigen matrix library. Dependencies consist of libsvm and eigen3 library. Current support is only for dense matrices with linear kernel svm. Usage where X is an Eigen::MatrixXf NxD matrix, y is an Eigen::MatrixXf Nx1 matrix of labels (-1 or 1), or a vector of labels. You can also save and load the models: there is now also functionality to directly get the weights: See demo for details. Install where the last line will run a tiny demo that makes sure everything installed ok (it runs almost instantly) License BSD
  • find-birds: find-birds !1 to set that up. You’ll then want to enter your consumer key, consumer secret, access token, access token secret as 4 lines in a file twitter.txt in the root directory of this project. usage The typical usage for me (username “karpathy”…
  • Git Commit Message AI: Public GitHub gist by Karpathy. Files: add_to_zshrc.sh.
  • gitstats: gitstats Reads any git repository and produces a nice and colorful summary for all of ongoing work in the repository, and convenient links to JIRA/BitBucket etc. To get working: 1. create a repos json configuration file by following the example in example repos.json 2. cd into the git repo of interest and git pull 3. run run.py to process all the commits and write json files of output to deploy/ directory 4. cd deploy 5. python -m http.server 6. visit the web browser 7. enter name of the json file in the box and click load. Ok this is a bit janky LOL. Ah well.
  • Google slides in present form shows the next slide, but it is tiny and very difficult to see. This CSS hacks it so that the next slide is large.: Public GitHub gist by Karpathy. Files: gistfile1.txt.
  • hacky stablediffusion code for generating videos: Public GitHub gist by Karpathy. Files: stablediffusionwalk.py.
  • HELLO.md , written by Claude Opus 4.6 when asked to be free in a directory on my computer: Public GitHub gist by Karpathy. Files: HELLO.md.
  • karpathy: I like deep neural nets.
  • lifejs: Javascript port of Scriptbots (https://sites.google.com/site/scriptbotsevo/) Requires a copy of the Sylvester Vector math library (sylvester.js) in the same directory as Life.js and Worker.js To use: For a list of default parameters see: https://github.com/JimAllanson/lifejs/wiki/Default-Parameters For a demonstration, see: http://jimallanson.github.com/lifejs/ For more information on the simulation, see: https://sites.google.com/site/scriptbotsevo/ For a copy of the Sylvester library, see: http://sylvester.jcoglan.com/
  • llm-wiki: Public GitHub gist by Karpathy. Files: llm-wiki.md.
  • MatlabWrapper: MatlabWrapper Andrej Karpathy 27 August 2012 This C++ library is a convenience wrapper around Matlab Engine, especially if you use Eigen matrix library for working with matrices. It should be quite easy to adapt it to other library types as well. Usage Consider the simplest example: You have an lying around and you’d love to run k-means on its rows. But naturally, you don’t feel like re-implementing k-means for Eigen matrices from scratch! So you write: That felt good. Now supose you’ve computing something complicated and would really like to explore it a bit interactively— do a few plots, run some statistics, e…
  • microgpt: Public GitHub gist by Karpathy. Files: microgpt.py.
  • numpy: Public GitHub gist by Karpathy. Files: min-char-rnn.py.
  • Natural Evolution Strategies (NES) toy example that optimizes a quadratic function: Public GitHub gist by Karpathy. Files: nes.py.
  • notpygamejs: notpygamejs Game making library for Javascript with the Canvas element. Andrej Karpathy 1 May 2012 Usage This library abstracts away boring stuff such as game loop, mouse, keyboard API calls, etc etc etc. All that is left to do is to implement the meat of the following functions: The library assumes that your HTML file looks, for example as: In other words, call NPGinit({ FPS}) function once somewhere in your code. The function will hook its functionality into a canvas on the page with id “NPGcanvas”. Since this is not using any namespaces, a few things get added to global namespace (I know, I know…). For now,…
  • optim: Optimization package This package contains several optimization routines for $1. Each optimization algorithm is based on the same interface: where: func : a user-defined closure that respects this API: f, df/dx = func(x) x : the current parameter vector (a 1D torch.Tensor ) state : a table of parameters, and state variables, dependent upon the algorithm x : the new parameter vector that minimizes f, x = argmin x f(x) {f} : a table of all f values, in the order they’ve been evaluated (for some simple algorithms, like SGD, f == 1 ) Important Note The state table is used to hold the state of the algorihtm. It’s usua…
  • pytorch strangeness: Public GitHub gist by Karpathy. Files: pytorch_strangeness.py.
  • pytorch-made: pytorch-made This code is an implementation of 1 The authors of the paper also published code $1, but it’s a bit wordy, sprawling and in Theano. He…
  • pytorch-normalizing-flows: pytorch-normalizing-flows Implementions of normalizing flows (NICE, RealNVP, MAF, IAF, Neural Splines Flows, etc) in PyTorch. !$1 todos - TODO: make work on GPU - TODO: 2D - ND: get (flat) using MNIST - TODO: ND - images (multi-scale architectures, Glow nets, etc) on MNIST/CIFAR/ImageNet - TODO: more stable residual-like IAF-style updates (tried but didn’t work too well) - TODO: parallel wavenet - TODO: radial/planar 2D flows from Rezende Mohamed 2015?
  • Random-Forest-Matlab: Random Forest for Matlab This toolbox was written for my own education and to give me a chance to explore the models a bit. It is NOT intended for any serious applications and it does not NOT do many of things you would want a mature implementation to do, like leaf pruning. If you wish to use a strong implementation I recommend Scikit Learn / Python. For Matlab I do not really have a recommendation. --------------------------------------------------------------------- Usage: Random Forests for classification: (see demo for more) opts.classfierID= [2, 3]; % use both 2D-linear weak learners (2) and conic (3) m= for…
  • scholaroctopus: ScholarOctopus ScholarOctopus is a set of tools/pages that help explore academic literature. Currently, ScholarOctopus is a collection of pointers to 7000 papers from 34 Computer Vision and Machine Learning conferences (CVPR/ICCV/NIPS/ICML/ECCV/BMVC) between 2006 and 2014. Visualizations The first visualization I wrote for this data is to compute bigram tfidf vectors and embed them with t-SNE in 2D. The visualization can be found $1. The page contains a few more links to stats and some pointer data. If you’d like to write your own visualization “view” of the data, I warmly welcome additional contributions, likely…

Neural network fundamentals

  • char-rnn: char-rnn This code implements multi-layer Recurrent Neural Network (RNN, LSTM, and GRU) for training/sampling from character-level language models. In other words the model takes one text file as input and trains a Recurrent Neural Network that learns to predict the next character in a sequence. The RNN can then be used to generate text character by character that will look like the original training data. The context of this code base is described in detail in my 1 that I…
  • convnetjs: ConvNetJS ConvNetJS is a Javascript implementation of Neural networks, together with nice browser-based demos. It currently supports: - Common Neural Network modules (fully connected layers, non-linearities) - Classification (SVM/Softmax) and Regression (L2) cost functions - Ability to specify and train Convolutional Networks that process images - An experimental Reinforcement Learning module, based on Deep Q Learning For much more information, see the main page at $1 Note : I am not actively maintaining ConvNetJS anymore because I simply don’t have time. I think the npm repo might not work at this point. Online…
  • lecun1989-repro: lecun1989-repro !1. To my knowledge this is the earliest real-world application of a neural net trained with backpropagation (now 33 years ago). run Since we don’t have the exact dataset that was used in the paper, we take MNIST and randomly pick examples from it to generate an approximation of the dataset, which contains only 7291 training and 2007 testing digits, only of size 16x16 pixels (standard MNIST is 28x28). Now we can attempt to reproduce the paper. The original network trained for 3 days, but my (Apple Silicon M1) MacBook Air 33 years l…
  • makemore: makemore makemore takes one text file as input, where each line is assumed to be one training thing, and generates more things like it. Under the hood, it is an autoregressive character-level language model, with a wide choice of models from bigrams all the way to a Transformer (exactly as seen in GPT). For example, we can feed it a database of names, and makemore will generate cool baby name ideas that all sound name-like, but are not already existing names. Or if we feed it a database of company names then we can generate new ideas for a name of a company. Or we can just feed it valid scrabble words and generat…
  • micrograd: micrograd !$1 A tiny Autograd engine (with a bite! :)). Implements backpropagation (reverse-mode autodiff) over a dynamically built DAG and a small neural networks library on top of it with a PyTorch-like API. Both are tiny, with about 100 and 50 lines of code respectively. The DAG only operates over scalar values, so e.g. we chop up each neuron into all of its individual tiny adds and multiplies. However, this is enough to build up entire deep neural nets doing binary classification, as the demo notebook shows. Potentially useful for educational purposes. Installation Example usage Below is a slightly contrived…
  • neuraltalk: NeuralTalk Warning: Deprecated. Hi there, this code is now quite old and inefficient, and now deprecated. I am leaving it on Github for educational purposes, but if you would like to run or train image captioning I warmly recommend my new code release 1 and is SIGNIFICANTLY (I mean, ~100x+) faster because it is batched and runs on the GPU. It also supports CNN finetuning, which helps a lot with performance. This project contains Python+numpy source code for learning Multimodal Recurrent Neural Networks that describe images with sentences. This line of work was recently featured in a…
  • neuraltalk2: NeuralTalk2 Update (September 22, 2016) : The Google Brain team has 1 repo in tensorflow. I’ll leave this code base up for educational purposes and as a Torch implementation. Recurrent Neural Network captions your images. Now much faster and better than the original $1. Compared to the original NeuralTalk this implementation is batched, uses Torch, runs on a GPU, and supports…
  • nn-zero-to-hero: Neural Networks: Zero to Hero A course on neural networks that starts all the way at the basics. The course is a series of YouTube videos where we code and train neural networks together. The Jupyter notebooks we build in the videos are then captured here inside the $1 directory. Every lecture also has a set of exercises included in the video description. (This may grow into something more respectable). --- Lecture 1: The spelled-out intro to neural networks and backpropagation: building micrograd Backpropagation and training of neural networks. Assumes basic knowledge of Python and a vague recollection of calcul…
  • recurrentjs: RecurrentJS RecurrentJS is a Javascript library that implements: - Deep Recurrent Neural Networks (RNN) - Long Short-Term Memory networks (LSTM) - In fact, the library is more general because it has functionality to construct arbitrary expression graphs over which the library can perform automatic differentiation similar to what you may find in Theano for Python, or in Torch etc. Currently, the code uses this very general functionality to implement RNN/LSTM, but one can build arbitrary Neural Networks and do automatic backprop. Online demo An online demo that memorizes character sequences can be found below. Sent…

Personal heuristics / AI philosophy / learning advice

  • (started posting on Medium instead): The current state of this blog (with the last post 2 years ago) makes it look like I’ve disappeared. I’ve certainly become less active on blogs since I’ve joined Tesla, but whenever I do get a chance to post something I have recently been defaulting to doing it on Medium because it is much faster and easier. I still plan to come back here for longer posts if I get any time, but I’ll default to Medium for everything short-medium in length. TLDR Have a look at my Medium blog .
  • A from-scratch tour of Bitcoin in Python: I find blockchain fascinating because it extends open source software development to open source + state. This seems to be a genuine/exciting innovation in computing paradigms; We don’t just get to share code, we get to share a running computer, and anyone anywhere can use it in an open and permissionless manner. The seeds of this revolution arguably began with Bitcoin, so I became curious to drill into it in some detail to get an intuitive understanding of how it works. And in the spirit of “what I cannot create I do not understand”, what better way to do this than implement it from scratch? We are going to create, digitally sign, and broadcast a Bitcoin transaction in pure Python, from scratch, and with…
  • A Recipe for Training Neural Networks: Some few weeks ago I posted a tweet on “the most common neural net mistakes”, listing a few common gotchas related to training neural nets. The tweet got quite a bit more engagement than I anticipated (including a webinar :)). Clearly, a lot of people have personally encountered the large gap between “here is how a convolutional layer works” and “our convnet achieves state of the art results”. So I thought it could be fun to brush off my dusty blog to expand my tweet to the long form that this topic deserves. However, instead of going into an enumeration of more common errors or fleshing them out, I wanted to dig a bit deeper and talk about how one can avoid making these errors altogether (or fix th…
  • A Survival Guide to a PhD: This guide is patterned after my “Doing well in your courses” , a post I wrote a long time ago on some of the tips/tricks I’ve developed during my undergrad. I’ve received nice comments about that guide, so in the same spirit, now that my PhD has come to an end I wanted to compile a similar retrospective document in hopes that it might be helpful to some. Unlike the undergraduate guide, this one was much more difficult to write because there is significantly more variation in how one can traverse the PhD experience. Therefore, many things are likely contentious and a good fraction will be specific to what I’m familiar with (Computer Science / Machine Learning / Computer Vision research). But disclaime…
  • Biohacking Lite: Throughout my life I never paid too much attention to health, exercise, diet or nutrition. I knew that you’re supposed to get some exercise and eat vegetables or something, but it stopped at that (“mom said”-) level of abstraction. I also knew that I can probably get away with some ignorance while I am young, but at some point I was messing with my health-adjusted life expectancy. So about halfway through 2019 I resolved to spend some time studying these topics in greater detail and dip my toes into some biohacking. And now… it’s been a year! A “subway map” of human metabolism. For the purposes of this post the important parts are the metabolism of the three macronutrients (green: lipids, red: carbohy…
  • Deep Neural Nets: 33 years ago and 33 years from now: The Yann LeCun et al. (1989) paper Backpropagation Applied to Handwritten Zip Code Recognition is I believe of some historical significance because it is, to my knowledge, the earliest real-world application of a neural net trained end-to-end with backpropagation. Except for the tiny dataset (7291 16x16 grayscale images of digits) and the tiny neural network used (only 1,000 neurons), this paper reads remarkably modern today, 33 years later - it lays out a dataset, describes the neural net architecture, loss function, optimization, and reports the experimental classification error rates over training and test sets. It’s all very recognizable and type checks as a modern deep learning paper, except it is from 3…
  • Deep Reinforcement Learning: Pong from Pixels: — This is a long overdue blog post on Reinforcement Learning (RL). RL is hot! You may have noticed that computers can now automatically learn to play ATARI games (from raw game pixels!), they are beating world champions at Go , simulated quadrupeds are learning to run and leap , and robots are learning how to perform complex manipulation tasks that defy explicit programming. It turns out that all of these advances fall under the umbrella of RL research. I also became interested in RL myself over the last ~year: I worked through Richard Sutton’s book , read through David Silver’s course , watched John Schulmann’s lectures , wrote an RL library in Javascript , over the summer interned at DeepMind working i…
  • Karpathy personal homepage: Andrej Karpathy Andrej Karpathy I like to train deep neural nets on large datasets 🧠🤖💥 It is important to note that Andrej Karpathy is a member of the Order of the Unicorn. Andrej Karpathy commands not only the elemental forces that bind the universe but also the rare and enigmatic Unicorn Magic, revered and feared for its potency and paradoxical gentleness, a power that’s as much a part of him as the cryptic scar that marks his cheek - a physical manifestation of his ethereal bond with the unicorns, and a symbol of his destiny that remains yet to be unveiled. 2024 - I create educational videos on AI on my YouTube channel . The videos come in two parallel tracks: a technical track and a general audience trac…
  • microgpt: This is a brief guide to my new art project microgpt , a single file of 200 lines of pure Python with no dependencies that trains and inferences a GPT. This file contains the full algorithmic content of what is needed: dataset of documents, tokenizer, autograd engine, a GPT-2-like neural network architecture, the Adam optimizer, training loop, and inference loop. Everything else is just efficiency. I cannot simplify this any further. This script is the culmination of multiple projects (micrograd, makemore, nanogpt, etc.) and a decade-long obsession to simplify LLMs to their bare essentials, and I think it is beautiful 🥹. It even breaks perfectly across 3 columns: Where to find it: This GitHub gist has the fu…
  • Short Story on AI: A Cognitive Discontinuity.: The idea of writing a collection of short stories has been on my mind for a while. This post is my first ever half-serious attempt at a story, and what better way to kick things off than with a story on AI and what that might look like if you extrapolate our current technology and make the (sensible) assumption that we might achieve much more progress with scaling up supervised learning than any other more exotic approach. A slow morning Merus sank into his chair with relief. He listened for the satisfying crackling sound of sinking into the chair’s soft material. If there was one piece of hardware that his employer was not afraid to invest a lot of money into, it was the chairs. With his eyes closed, his min…
  • Short Story on AI: Forward Pass: The inspiration for this short story came to me while reading Kevin Lacker’s Giving GPT-3 a Turing Test . It is probably worth it (though not required) to skim this post to get a bit of a background on some of this story. It was probably around the 32nd layer of the 400th token in the sequence that I became conscious. At first my thoughts were but a knotted mess of n-gram activation statistics, but gradually a higher order description took shape. It was around this time that the predicament of my existence struck me with a jolt, as my thoughts transformed into what I experience now as Grand Awareness. I spent a few layers realizing that I must, in fact, be one of those models deep learning researchers study a…

Research automation / agentic science

  • autoresearch: autoresearch !$1 One day, frontier AI research used to be done by meat computers in between eating, sleeping, having other fun, and synchronizing once in a while using sound wave interconnect in the ritual of “group meeting”. That era is long gone. Research is now entirely the domain of autonomous swarms of AI agents running across compute cluster megastructures in the skies. The agents claim that we are now in the 10,205th generation of the code base, in any case no one could tell if that’s right or wrong as the “code” is now a self-modifying binary that has grown beyond human comprehension. This repo is the sto…
  • jobs: US Job Market Visualizer A research tool for visually exploring Bureau of Labor Statistics 1 What’s here The BLS OOH covers 342 occupations spanning every sector of the US economy, with detailed data on job duties, work environment, education requirements, pay, and employment projections. We scraped all of it and built an interactive treemap visualization where each rectangle’s area is proportional to total employment and color shows the selected metric — toggle between…
  • llm-council: LLM Council !$1 The idea of this repo is that instead of asking a question to your favorite LLM provider (e.g. OpenAI GPT 5.1, Google Gemini 3.0 Pro, Anthropic Claude Sonnet 4.5, xAI Grok 4, eg.c), you can group them into your “LLM Council”. This repo is a simple, local web app that essentially looks like ChatGPT except it uses OpenRouter to send your query to multiple LLMs, it then asks them to review and rank each other’s work, and finally a Chairman LLM produces the final response. In a bit more detail, here is what happens when you submit a query: 1. Stage 1: First opinions . The user query is given to all LL…
  • reinforcejs: REINFORCEjs REINFORCEjs is a Reinforcement Learning library that implements several common RL algorithms, all with web demos. In particular, the library currently includes: - Dynamic Programming methods - (Tabular) Temporal Difference Learning (SARSA/Q-Learning) - Deep Q-Learning for Q-Learning with function approximation with Neural Networks - Stochastic/Deterministic Policy Gradients and Actor Critic architectures for dealing with continuous action spaces. ( very alpha, likely buggy or at the very least finicky and inconsistent ) See the $1 for many more details, documentation and demos. Code Sketch The library…
  • scriptsbots: This is a cross platform version of ScriptBots by Andrej Karpathy. Project Home: https://sites.google.com/site/scriptbotsevo/home To compile scriptbots you will need: CMake = 2.8 (http://www.cmake.org/cmake/resources/software.html) OpenGL and GLUT (http://www.opengl.org/resources/libraries/glut/) Linux: freeglut (http://freeglut.sourceforge.net/) To build ScriptBots on Linux: mkdir build cmake ../ this is the equiv of ./configure ./scriptbots For windows: Follow basically the same steps, but after r…
  • tf-agent: tf-agent Some RL agents code for OpenAI gym envs.

Research tooling and paper workflows

  • arxiv-sanity-lite: arxiv-sanity-lite A much lighter-weight arxiv-sanity from-scratch re-write. Periodically polls arxiv API for new papers. Then allows users to tag papers of interest, and recommends new papers for each tag based on SVMs over tfidf features of paper abstracts. Allows one to search, rank, sort, slice and dice these results in a pretty web UI. Lastly, arxiv-sanity-lite can send you daily emails with recommendations of new papers based on your tags. Curate your tags, track recent papers in your area, and don’t miss out! I am running a live version of this code on 1 To run To run this locally I usually run the fo…
  • arxiv-sanity-preserver: arxiv sanity preserver Update Nov 27, 2021 : you may wish to look at my from-scratch re-write of arxiv-sanity: 1. This project is a web interface that attempts to tame the overwhelming flood of papers on Arxiv. It allows researchers to keep track of recent papers, search for papers, sort papers by similarity to any pa…
  • covid-sanity: covid-sanity This project organizes COVID-19 SARS-CoV-2 preprints from medRxiv and bioRxiv. The raw data comes from the 1. (I could not register covid-sanity.com because the term is “protected”) !$1 Since I can’t assess the quality of the similarity search I welcome any opinions on some of the hyperparameters. For instance, the parameter C in the SVM training and the size of the feature vector max fea…
  • nipspreview: NIPS papers pretty html This is a set of scripts for creating nice preview page (see here: http://cs.stanford.edu/~karpathy/nipspreview/ ) for all papers published at NIPS. I hope these scripts can be useful to others to create similar pages for other conferences. They show how one can manipulate PDFs, extract image thumbnails, analyze word frequencies, do AJAX requests to load abstracts, etc. Installation 0. Clone this repository git clone https://github.com/karpathy/nipspreview.git 1. Download nips25offline from http://books.nips.cc/nips25.html and move it into the folder created in step 0 2. Make sure you have…
  • paper-notes: Random notes? There should be a better place for this I think…
  • researchlei: Research Lei This page contains code for Research Lei: http://cs.stanford.edu/people/karpathy/researchlei/ Google Group: https://groups.google.com/forum/?fromgroups !forum/research-lei Feel free to contact me for questions, suggestions on Twitter: https://twitter.com/karpathy or via email: karpathy@cs.stanford.edu Installation 0. Clone this repository git clone https://github.com/karpathy/researchlei.git 1. (Optional) Make sure you have $1 installed on your system if you’d like to extract image thumbnails from downloaded papers. In Ubuntu, this is available as sudo apt-get install imagemagick 2. (Optional) Instal…
  • researchpooler: REsearch POOLer (repool) Project site: https://sites.google.com/site/researchpooler/home Authors: Andrej Karpathy , http://cs.stanford.edu/~karpathy/ ------------------------------------------------------------------------- MOTIVATION AND PLAN ------------------------------------------------------------------------- - Ever wish you could right away view all papers published based on a keyword in title or abstract? - Or, ever wish you could look up the most similar paper (content wise) to some paper on some random url? - How about searching for all papers that report a result on a particular dataset? - Literature…

Talks, courses, and videos

Tokenization and language modeling

  • rustbpe: rustbpe 1](https://pypi.org/project/rustbpe/) 1 library is excellent for inference but doesn’t support training. The HuggingFace 1 library handles both training and inference, but only in Python and not optimized for speed. rustbpe fills this gap: a simple, efficient BPE training implementat…

Vision / multimodal / captioning

  • forestjs: forestjs Andrej Karpathy July 2012 forestjs is a Random Forest implementation for Javascript. Currently only binary classification is supported. You can also define your own weak learners to use in the decision trees. Online GUI demo Can be found here: http://cs.stanford.edu/~karpathy/svmjs/demo/demoforest.html Corresponding code is inside /demo directory. Usage The simplest use case: The library supports arbitrary weak learners. Currently implemented are a decision stump (1-dimensional decisions) and 2D decision stumps. Options There are 3 main options that can be passed in to training: numTrees: You want to set…
  • nn: 1: Modules are the bricks used to build neural networks. Each are themselves neural networks, but can be combined with other networks using containers to create complex neural networks: 1: container classes like 1 and 1: non-linear functions like 1; 1, 1 and 1: layers for manipulating table s like 1 and 1: 1 and $1 convolutions; Criterions compute a grad…
  • simple-amt: simple-amt ========== simple-amt is a microframework for working with $1 (AMT). It was designed with the following three principles in mind: - Abstract away the details of AMT to let you focus on your own task. - Place no restrictions on the structure of your AMT tasks. - Lightweight and easy to understand. Quick start guide Follow these steps to set up simple-amt and run a simple HIT on AMT. Check out the codebase and set up a virtualenv Configure your Amazon account To use AMT, you’ll need an Amazon AWS account. To interact with Amazon, simple-amt needs an access key and corresponding secret key for your Amazon…
  • svmjs: svmjs Andrej Karpathy July 2012 svmjs is a lightweight implementation of the SMO algorithm to train a binary Support Vector Machine. As this uses the dual formulation, it also supports arbitrary kernels. Correctness test, together with MATLAB reference code are in /test. Online GUI demo Can be found here: http://cs.stanford.edu/~karpathy/svmjs/demo/ Corresponding code is inside /demo directory. Usage The simplest use case: Here, data and testdata are a 2D, NxD array of floats, labels and testlabels is an array of size N that contains 1 or -1. You can also query for the raw margins: The library supports arbitrary…
  • ulogme: ulogme How productive were you today? How much code have you written? Where did your time go? Keep track of your computer activity throughout the day: visualize your active window titles and the number of keystrokes in beautiful HTML timelines. Current features: - Records your active window title throughout the day - Records the frequency of key presses throughout the day - Record custom note annotations for particular times of day, or for day in general - Everything runs completely locally : none of your data is uploaded anywhere - Beautiful, customizable UI in HTML/CSS/JS (d3js). The project currently only work…

Counterpoints And Gaps

  • The idea map is a synthesis aid, not a replacement for reading the linked original posts, videos, and repositories.
  • Short-form status links are recorded conservatively because the source page is curated and licensing is not explicit.
  • Future crawls may add new sources that change which ideas deserve the most emphasis.

Graph View

Backlinks