Why pyproject.toml Exists

2.2 Python's twist: the isolation step is NOT automatic

This is the part that trips up people coming from npm. In JavaScript, running npm install always creates a project-local node_modules/ folder - isolation is the default and you can't easily mess it up.

In Python, if you just run pip install requests, by default it installs globally on your whole machine, shared by every Python project you have. This is exactly the "dependency hell" problem from 2.1, self-inflicted. Python's fix is called a virtual environment (venv), and unlike npm, you must create it yourself, on purpose, every time.

   WITHOUT a venv (bad)                    WITH a venv (correct)

   Your whole computer                     Your computer
   ┌─────────────────────┐                 ┌─────────────────────┐
   │ python (global)      │                 │ python (global)      │
   │  requests 2.1        │                 │                       │
   │  ← used by ALL       │                 │  project_a/.venv/     │
   │    projects, and      │                │  └─ requests 2.1      │
   │    they all fight      │                │                       │
   │    over the version   │                │  project_b/.venv/     │
   └─────────────────────┘                 │  └─ requests 2.31     │
                                             │     (different! fine!)│
                                             └─────────────────────┘

Why is Python like this?

Python predates the modern "package manager" idea by decades - it shipped with pip as a bolt-on years after the language existed, and it inherited the assumption (common in older Unix tooling) that there's "one Python installation" on a machine that everything shares, similar to how system libraries work in C. npm, by contrast, was designed from day one (2010, well after people had learned this lesson the hard way) with per-project isolation as the default. Neither choice is "wrong" - but it means Python asks you to opt into the discipline that npm gives you for free.

Creating and using a venv

# inside your project folder
python -m venv .venv

# activate it (do this every time you open a new terminal for this project)
source .venv/bin/activate        # macOS/Linux
.venv\Scripts\activate           # Windows

# your terminal prompt now shows (.venv) - you are "inside" the isolated box
pip install requests             # installs ONLY into this project's .venv

To leave the box: deactivate.

What actually happens when you activate a venv?

Activation doesn't install anything new - it just edits your terminal session's PATH environment variable so that when you type python or pip, your shell finds the copies inside .venv/bin/ (or .venv\Scripts\ on Windows) before it finds the system-wide ones.

   BEFORE activation                       AFTER `source .venv/bin/activate`
   PATH = /usr/bin:/usr/local/bin          PATH = /my-agent/.venv/bin:/usr/bin:/usr/local/bin
   `python` → /usr/bin/python              `python` → /my-agent/.venv/bin/python
   (the global interpreter)                (the project's own interpreter, with its own
                                             separately installed packages)

That's the entire trick. A venv is just a folder containing a private copy of the Python interpreter (or a symlink to one) plus a private site-packages/ folder where pip install puts things, and "activating" is just a shell script that temporarily points your terminal at that folder instead of the system one.

Common mistake: installing a package before activating the venv. If your terminal prompt doesn't show (.venv), pip install is going to the global environment, silently recreating the exact mess venvs exist to prevent. Always check for (.venv) in your prompt before running pip install anything.

Common mistake: committing .venv/ to git. It can be hundreds of megabytes of files that are entirely reproducible from pyproject.toml, and it's platform-specific (a venv built on macOS won't work if a teammate clones it on Windows). Always .gitignore it - see 2.5.

Why this matters for the rest of the book: every project you build (Chapters 5–8) starts with python -m venv .venv and activating it. If you skip this, you'll install packages globally, and six months from now two of your projects will silently break each other.

2.3 pyproject.toml - Python's package.json / pom.xml

Modern Python projects declare their dependencies in a file called pyproject.toml (TOML = a simple, human-readable config format, a bit like a stricter INI file - it stands for "Tom's Obvious, Minimal Language," named after its creator).

# pyproject.toml
[project]
name = "my-agent"
version = "0.1.0"
description = "A local-first LLM agent"
requires-python = ">=3.11"
dependencies = [
    "openai>=1.40.0",
    "anthropic>=0.34.0",
    "python-dotenv>=1.0.0",
]

[project.optional-dependencies]
dev = ["pytest>=8.0.0"]

Side-by-side with what you already recognise:

package.json                       pom.xml                          pyproject.toml
──────────────────────────────    ──────────────────────────────   ──────────────────────────────
{                                 <project>                        [project]
  "name": "my-app",                 <groupId>com.me</groupId>        name = "my-agent"
  "version": "1.0.0",               <artifactId>my-app</artifactId>  version = "0.1.0"
  "dependencies": {                 <dependencies>                  dependencies = [
    "express": "^4.18.0"              <dependency>                    "openai>=1.40.0"
  }                                    <groupId>...</groupId>       ]
}                                    </dependency>
                                   </dependencies>
                                 </project>

Same purpose, different syntax, because each ecosystem evolved independently.

Reading version specifiers

Those >=1.40.0 strings look small but encode real rules. Here's the vocabulary:

Most real projects prefer the >=x,<y form or ~= because pinning with bare >= can silently pull in a future major version that changes its API and breaks your code. We use loose >= specifiers in this book's early chapters to keep examples simple, but in Chapter 8's real project you'll see the tighter range form.

Installing dependencies FROM the manifest

pip install -e .          # installs everything listed in pyproject.toml's dependencies

The -e flag means "editable" - it installs your own project itself as an importable package, linked back to your source folder instead of copied elsewhere, so edits to your code take effect immediately without reinstalling. This is the Python equivalent of npm install (reading package.json) or mvn install (reading pom.xml): read the manifest, fetch everything it lists, put it in the isolated box.

The older, simpler file you'll still see everywhere: requirements.txt

Before pyproject.toml became standard, and still very common in ML/research code you'll find on GitHub, projects just list packages in a plain text file:

# requirements.txt
openai>=1.40.0
anthropic>=0.34.0
python-dotenv>=1.0.0

pip install -r requirements.txt

There's no strict metadata (name, version) here - it's only the dependency list. Think of it as pyproject.toml's stripped-down ancestor. We'll use pyproject.toml for our real project in Chapter 8 because it's the modern standard, but don't be confused when you see requirements.txt in other people's repos - same job, older format.

A useful trick worth knowing: you can generate a requirements.txt from whatever is currently installed in your active venv with:

pip freeze > requirements.txt

This writes out the exact versions currently installed (not the loose >= ranges you might have originally asked for) - which brings us to lockfiles, the next section.

2.4 Lockfiles - pinning EXACT versions

dependencies = ["openai>=1.40.0"] says "at least 1.40.0" - but which exact version got installed on your machine last Tuesday might differ from mine today. A lockfile freezes the exact resolved versions of everything (including sub-dependencies you never even asked for directly), so installs are bit-for-bit reproducible.

   package-lock.json  (npm)     →  exact tree of every installed version
   poetry.lock / uv.lock (Py)   →  exact tree of every installed version

Concretely, imagine your manifest just says dependencies = ["openai>=1.40.0"]. openai itself depends on httpx, which depends on httpcore, which depends on h11. A lockfile writes down the exact version resolved for every single one of those, transitively:

   pyproject.toml (what YOU wrote)        uv.lock / poetry.lock (what got RESOLVED)
   openai>=1.40.0                    →    openai==1.42.0
                                           httpx==0.27.2
                                           httpcore==1.0.5
                                           h11==0.14.0
                                           ...

Without a lockfile, two machines running pip install on the same loose manifest, weeks apart, can silently end up with different sub-dependency versions - and then "works on my machine" bugs appear that are agonizing to track down, because the bug is in a package you never even directly imported.

Plain pip historically didn't generate one automatically (a real gap vs npm/Maven), which is why the community built extra tools on top:

For this book we keep it simple and use venv + pip + pyproject.toml because it needs no extra installs and teaches you the underlying mechanics. Once comfortable, look at uv (a fast modern all-in-one tool, pip install uv) - it plays the same unifying role that npm or mvn play in their ecosystems, doing manifest + lock + venv in one command.

Callout - do I need a lockfile for a personal learning project? Not strictly, no. If you're the only person ever running this code, on one machine, the risk a lockfile protects against barely applies to you. But the moment you deploy to a server, hand the project to a teammate, or come back to it in a year on a fresh machine, a lockfile is the difference between "it just works" and an afternoon of debugging version drift.

2.5 A real project skeleton (we'll use this shape from Chapter 5 onward)

my-agent/
├── .venv/                  ← isolated environment (never commit this to git)
├── pyproject.toml          ← the manifest: what this project needs
├── .env                    ← secrets (API keys) - never commit this either
├── .gitignore              ← tells git to ignore .venv/ and .env
├── src/
│   └── agent/
│       ├── __init__.py
│       ├── providers.py    ← Chapter 7
│       └── agent.py        ← Chapter 6
└── main.py                 ← entry point

.gitignore deserves a mention: it's the file that stops you from accidentally uploading your API keys or your entire installed-package folder to GitHub.

# .gitignore
.venv/
.env
__pycache__/
*.pyc

Two of these lines are about size (.venv/, __pycache__/, *.pyc - all regenerable, no reason to store them in version control), and one line is about security (.env - if that file ever reaches a public GitHub repo, assume the API keys inside it are compromised within minutes; automated bots scan public commits for exactly this pattern). We'll come back to .env and how your code actually reads secrets out of it in Chapter 5.

Common mistake: creating .gitignore after you've already committed .venv/ or .env once. Adding a file to .gitignore only stops git from tracking new changes to it - it does not remove something already committed from your project's history. If you ever commit a real secret by accident, treat the key as leaked: rotate/regenerate it at the provider, don't just delete the file.

2.6 Try it now

mkdir my-agent && cd my-agent
python -m venv .venv
source .venv/bin/activate      # or .venv\Scripts\activate on Windows
pip install requests
python -c "import requests; print(requests.__version__)"

If that prints a version number, your isolated environment works. This exact ritual - create venv, activate, install, run - is what you'll repeat for every project in this book, including the one we build starting in Chapter 5.

As a sanity check, try creating a second venv in a different folder and installing a different version of the same package, to see the isolation actually working with your own eyes:

mkdir other-project && cd other-project
python -m venv .venv
source .venv/bin/activate
pip install "requests==2.28.0"
python -c "import requests; print(requests.__version__)"   # 2.28.0, independent of my-agent's copy

2.7 Check yourself

1. In your own words, what are the three problems every dependency manager (npm, Maven, pip tooling) solves? Name the file or folder that solves each one, for both JavaScript and Python.
2. Why does pip install requests install globally by default, while npm install express never does? What line of reasoning explains the historical difference?
3. What does "activating" a venv actually change on your machine? (Hint: think about PATH, not about anything being "installed.")
4. What's the difference between openai>=1.40.0 in pyproject.toml and the line for openai that would appear inside a generated lockfile like uv.lock?
5. Explain why .gitignore should list .venv/ for a size reason and .env for a security reason - and why those are actually two different kinds of problems.
6. You clone a teammate's Python project onto a fresh laptop. List, in order, the exact terminal commands you'd run before you could execute python main.py successfully.