Skim

AI / ML Papers · 10

Contemporary · AI / ML Papers

Software 2.0

In traditional software ('Software 1.0'), a human engineer writes explicit step-by-step instructions telling the computer exactly what to do in every situation. Karpathy's argument: a large and growing share of valuable software is now better built a completely different way — you define a goal (a loss function measuring how wrong the model currently is), provide labeled examples, and let an optimization algorithm (like gradient descent) search through billions of possible internal settings until it finds ones that achieve the goal. The 'program' that results isn't human-readable code — it's a giant set of numbers (weights) that a human didn't write line by line.

Andrej Karpathy · 2017 swipe · next · J/K

Contemporary · AI / ML Papers

Efficient Estimation of Word Representations (word2vec)

Before word2vec, computers mostly treated words as arbitrary, unrelated symbols — 'cat' and 'dog' had no more mathematical relationship to each other than 'cat' and 'stapler.' Mikolov and colleagues at Google showed that by training a simple neural network to predict a word from its surrounding context (or vice versa) across huge amounts of text, you get a byproduct that turns out to be far more valuable than the prediction task itself: every word ends up represented as a point (a vector) in a high-dimensional space, positioned such that words used in similar contexts end up near each other, and consistent relationships (like 'male-to-female' or 'country-to-capital') show up as consistent directions you can add or subtract.

Mikolov et al. (Google) · 2013

Contemporary · AI / ML Papers

ImageNet Classification with Deep Convolutional Neural Networks (AlexNet)

Every year, the ImageNet competition tested how well computer programs could correctly label photos into categories. For years, progress came from hand-engineered techniques designed by computer vision experts. In 2012, a deep convolutional neural network trained by Krizhevsky, Sutskever, and Hinton entered and won by such a wide margin that it upended the field's consensus: a network with enough layers, trained on enough labeled images, using specific practical tricks to make training feasible, could learn better visual features on its own than experts could design by hand.

Krizhevsky, Sutskever, Hinton · 2012

Contemporary · AI / ML Papers

Deep Residual Learning for Image Recognition (ResNet)

Before ResNet, adding more layers to a deep neural network usually helped performance up to a point — and then made things worse, even on the training data itself, which ruled out overfitting as the explanation. The real problem was that training signals (gradients) had to travel backward through every layer during training, and in very deep networks that signal would shrink to almost nothing by the time it reached the earliest layers (the 'vanishing gradient' problem), making those layers nearly impossible to train properly. He, Zhang, Ren, and Sun's fix: add 'skip connections' that let the signal bypass a block of layers entirely if needed, so a layer only has to learn the difference (the 'residual') from what came before, rather than reconstructing the whole transformation from scratch.

He, Zhang, Ren, Sun (Microsoft) · 2015

Contemporary · AI / ML Papers

Attention Is All You Need

Before Transformers, the standard way to process a sequence of words (like a sentence) was recurrently — one word at a time, in order, feeding each step's output into the next. This was slow to train (you can't easily parallelize a process that depends on the previous step finishing first) and struggled to preserve context across long distances in a sentence. Vaswani and colleagues proposed processing the entire sequence at once, using a mechanism called 'self-attention' that directly calculates how relevant every word is to every other word in the sequence, regardless of how far apart they are.

Vaswani et al. (Google) · 2017

Contemporary · AI / ML Papers

Language Models are Few-Shot Learners (GPT-3)

GPT-3's core training task never changed from earlier language models — predict the next word in a sequence, over and over, on a vast amount of internet text. What changed was scale: 175 billion parameters, trained on an enormous dataset. At that scale, something unexpected showed up: the model could perform tasks it was never specifically trained to do — translation, arithmetic, simple coding — just by being shown a few examples of the task directly inside the prompt, with no retraining or fine-tuning required.

Brown et al. (OpenAI) · 2020

Contemporary · AI / ML Papers

Scaling Laws for Neural Language Models

Kaplan and colleagues at OpenAI ran systematic experiments training many language models of different sizes and found something remarkably clean: a model's performance (its error rate) improves in a predictable mathematical pattern (a power law) as you increase compute, dataset size, or parameter count — and crucially, the internal architecture details (like how deep versus how wide the network is) mattered far less than sheer scale. This gave labs a rare thing in machine learning research: a way to forecast how much better a bigger model would be, before actually building it.

Kaplan et al. (OpenAI) · 2020

Contemporary · AI / ML Papers

Training Compute-Optimal Large Language Models (Chinchilla)

DeepMind trained over 400 language models of varying sizes to empirically map out what actually minimizes error for a given amount of compute. Their finding directly corrected the earlier OpenAI scaling-laws guidance: rather than mostly growing parameter count, model size and training data should scale up together, roughly in equal proportion. Their 70-billion-parameter 'Chinchilla' model, trained on far more data (1.4 trillion tokens) than GPT-3's 175 billion parameters had been (roughly 300 billion tokens), ended up outperforming the much larger model.

Hoffmann et al. (DeepMind) · 2022

Contemporary · AI / ML Papers

DeepSeek-R1: Incentivizing Reasoning Capability via RL

DeepSeek, a Chinese AI lab, released an open-source reasoning model that matched top proprietary 'reasoning' models (like OpenAI's o1) on many benchmarks, while claiming a dramatically lower training and inference cost. The paper's core claim is that strong step-by-step reasoning ability can be taught to a language model largely through reinforcement learning — rewarding the model for correct final answers and consistent reasoning — rather than primarily through expensive, human-curated supervised fine-tuning data, which had been the more common assumption.

DeepSeek · 2025

Contemporary · AI / ML Papers

The Bitter Lesson

Sutton observed a repeating pattern across seven decades of AI research: researchers build systems that encode human expertise and intuition about a domain (like handcrafted chess strategies or linguistic grammar rules), these systems perform well initially, and then eventually get overtaken by much simpler, more general methods that just leverage more computation — search and learning — once enough compute becomes available. The 'bitter' part is that this keeps surprising and disappointing researchers who invested years building domain-specific expert knowledge into their systems, only to watch a more general, compute-hungry approach eventually surpass it.

Rich Sutton · 2019