Software 2.0
Andrej Karpathy · 2017
"Neural networks are a new software paradigm where code is learned from data, not written by hand."
In traditional software ('Software 1.0'), a human engineer writes explicit step-by-step instructions telling the computer exactly what to do in every situation. Karpathy's argument: a large and growing share of valuable software is now better built a completely different way — you define a goal (a loss function measuring how wrong the model currently is), provide labeled examples, and let an optimization algorithm (like gradient descent) search through billions of possible internal settings until it finds ones that achieve the goal. The 'program' that results isn't human-readable code — it's a giant set of numbers (weights) that a human didn't write line by line.
The mechanism is replacing hand-specification with search-plus-data. In Software 1.0, if you want a program to recognize cats in photos, an engineer would have to manually write rules about edges, shapes, and colors — a nearly impossible task to get right by hand. In Software 2.0, you instead show the system millions of labeled photos (cat / not cat) and let an optimizer adjust internal weights until the network's predictions match the labels as closely as possible. The 'source code' becomes two things: the architecture of the network, and the dataset it was trained on — meaning that improving the software increasingly means improving your data pipeline, not just your codebase.
According to Karpathy, what makes up the 'source code' of a Software 2.0 system?
Read more about the topic
The explanation above is written with AI assistance. These are the originals — go to them to check it.
- Software 2.0 (original essay)Andrej Karpathy on Medium
- The Rise of Software 2.0: You don't want to be left behindTowards Data Science
The Bitter Lesson
"General methods that leverage computation ultimately beat human-knowledge-engineered approaches in AI."