Often at the start of classes I teach, I get a question about virtual environments — why do we use them? What are they? How do they work? What is your opinion about uv (or the latest packaging tool)? And as I may have mentioned before on this blog, I enjoy a good question, so I thought I’d share a bit on why we do what we do with virtual environments at Diller Digital.
Virtual Environments
First of all, what is a virtual environment? For most applications on a computer or mobile device, you just install it once to get started and then download periodic updates to keep it current. Creative suites with lots of modules and extensions like PhotoShop have a single executable application with a single collection of libraries that extend their capabilities. The MathWorks’ MATLAB works this way, too, with a single runtime interpreter and a collection of libraries curated by The MathWorks. In this kind of monolithic setup, the publisher of the software implicitly guarantees the interoperability of the component parts by controlling the software releases and conditions of whatever extension marketplace they offer.
The world of Python is different: there is no single entity curating the extensions; different libraries are added for different use cases, resulting in different sets of dependencies for different applications; and there is no guarantee of backward compatibility. In fact, managing dependencies and preserving working environments is part of the price to pay for Free Open Source Software (FOSS). And that’s where virtual environments come into play. If the world of FOSS can feel like the Wild West, then think of virtual environments as a way to make a safe “corral” where you have a Python interpreter (runtime) and a set of libraries that work together and can control when and if they are updated so they don’t break code that is working.
As a practical example, let’s consider the TensorFlow library, which Diller Digital uses in one version of our Deep Learning for Scientists & Engineers course (we also provide a version that uses PyTorch). TensorFlow has components that are compiled in C/C++, and when Python 3.12 was released in October 2023, it included several internal changes that caused many C-extension builds to fail, including TensorFlow’s. At that time, the latest version of TensorFlow was 2.14, and it could only be run with Python versions between 3.8 and 3.11. It was early 2024 before TensorFlow 2.16 was released, which finally enabled use of Python 3.12. By that time, Python 3.13 was being released, and the cycle started again. Thus, users of TensorFlow have specific version requirements for their Python runtime, and it sometimes varies by hardware.
Consider, though, that a user of TensorFlow may also have another unrelated project that could really benefit from a recent release of another library, say Pandas, that is only available (or only provides wheels for) newer versions of Python. In this case, the TensorFlow dependencies conflict with the Pandas dependencies, and if Python were a monolithic application, you would have to choose between one or the other.
Virtual environments ease that dilemma by creating independent runtime environments, each with their own Python executable (potentially of different versions) and set of libraries. Recent versions of Python (if downloaded and installed from Python.org) can sit next to each other in an operating system and run independently. In the macOS, these are generally found in the Library/Frameworks/ directory and can be accessed with python3 (which is an alias to the latest one installed) or using the full version number.
❯ which python3
/Library/Frameworks/Python.framework/Versions/3.13/bin/python3
❯ which python3.12
/Library/Frameworks/Python.framework/Versions/3.12/bin/python3.12
❯ which python3.13
/Library/Frameworks/Python.framework/Versions/3.13/bin/python3.13
❯ which python3.14
/Library/Frameworks/Python.framework/Versions/3.14/bin/python3.14Note how each minor version (the 12 or 13 in 3.12 or 3.13) has its own runtime.
In Windows it’s similar, but the locations are different.
C:\Users\tim> where python
C:\Users\tim\AppData\Local\Programs\Python\Python314\python.exe
C:\Users\tim\AppData\Local\Programs\Python\Python313\python.exe
C:\Users\tim\AppData\Local\Programs\Python\Python312\python.exe
C:\Users\tim\AppData\Local\Microsoft\WindowsApps\python.exeThat last entry in the Windows example is sneaky — execute it and you’re taken to the Windows App store. I don’t recommend installing that way.
What we often do in Diller Digital classes is to create a virtual environment dedicated to the class. This makes sure we don’t break anything already set up and working, and if future changes to the libraries break the examples we used in class, there will at least be a working version in that environment. The native-Python way to do this (by which I mean “the way that uses only standard libraries that ship with Python”) is using the venv library. Many Python libraries have a “main” mode that you can invoke with the -m flag, so setting up a new Python environment looks like this:
python -m venv my_new_environmentWhich python you use to invoke venv determines the version of the runtime that gets installed into the virtual environment. In Windows, if I wanted something other than the version that appears first in the list, I’d have to specify the entire path. So, for example, to create a version that used Python 3.12, I’d type:
C:\Users\tim\AppData\Local\Programs\Python\Python312\python.exe -m venv my_new_environmentIn macOS as I showed above, I’d type
python3 -m venv my_new_environment and it would use version 3.13. The virtual environment amounts to a new directory in the place where I invoked the command with contents like this (for macOS environments):
my_new_environment├── bin
├── etc
├── include
├── lib
├── pyvenv.cfg
└── shareThere may be some minor variations with platform, but critically, there’s a lib/python3.13/site-packages directory where libraries get installed, and a bin directory with executables (it’s the Scripts directory on Windows). Note that the Python runtime executables are actually just links to the Python executable used to create the virtual environment. When you “activate” a virtual environment, two things happen:
1 – The prompt changes, so that my_new_environment now appears at the start of the command prompt, and
2 – Some path-related environment variables are modified such that my_new_environment/bin is added to system path, and Python’s import path resolves to my_new_environment/lib/python3.13/site-packages.
The environment is activated with a script located at my_new_environment/bin/activate (macOS) or my_new_environment\Scripts\activate (Windows).
❯ source my_new_environment/bin/activate
my_new_environment ❯ which python
/Users/timdiller/my_new_environment/bin/python
my_new_environment ❯ python
Python 3.13.9 (v3.13.9:8183fa5e3f7, Oct 14 2025, 10:27:13) [Clang 16.0.0 (clang-1600.0.26.6)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/Library/Frameworks/Python.framework/Versions/3.13/lib/python313.zip', '/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13', '/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/lib-dynload', '/Users/timdiller/my_new_environment/lib/python3.13/site-packages']That’s it. That’s the magic. The fun begins when you start installing packages…
PIP, the Package Installer for Python
In Python, the standard installation tool is called pip, the Package Installer for Python. For many use cases, pip works just fine, and if I were to only ever use a Mac with an SSD, I would probably just stick with pip. Although you can use pip to install packages from anywhere (including local code or from a code repository like GitHub) pip‘s default repository is PyPI, the Python Package Index, a central repository for community-contributed libraries. All of the major science and engineering packages including NumPy, SciPy, and Pandas are published through PyPI.
When someone uses pip install to install a package into their Python environment, broadly speaking, two things happen:
1 – pip resolves the dependencies for the package to be installed (properly listing dependencies is up to the author(s)) and computes a list of everything that needs to be installed or updated; this is a non-trivial task involving a mini-language for specifying acceptable versions and SAT Solving, and accomplishing it efficiently was an early distinctive advantage for early 3rd-party packaging tools like Enthought’s EDM and Continuum’s (now Anaconda’s) conda; since version 20.3, pip generally handles dependency resolution as well as any of its competitors (more on this below).
2 – pip places copies of the packages into the site-packages directory, downloading new versions as needed. pip keeps a cache of libraries so that it doesn’t have to download each one from PyPI every time. Unlike some of the newer tools, pip does not try to de-duplicate installed files across environments with hard links or shared package stores. Each environment is independent and isolated.
pip defines the canonical package management behavior for Python: safe, debug-friendly, and transparent, if somewhat inefficient in terms of file system utilization. In practice, I’ve found little to complain about using pip on the macOS, but in Windows, the file-copying step can be painfully slow, especially if there is any kind of malware scanning tool involved.
Enter the 3rd-party Package Managers
In the early days of scientific computing in Python, Enthought published the Enthought Python Distribution (EPD), a semantic-versioned Python distribution with a runtime and a reasonably complete set of libraries that were extensively tested and guaranteed to “play nicely” together. In 2011, Continuum brought a competing product to market, Anaconda, with a focus on broad access to open source packages, including the newest releases. Anaconda shipped with a command line tool conda for managing its own version of virtual environments. Meanwhile, Enthought focused on rigorously-tested, curated package sets with security-conscious institutional customers in mind and evolved EPD into the Enthought Deployment Manager (EDM), which also defined “applications” that could be deployed within an institution using the edm command-line tool. This involved managing virtual environments in a registry or database that could be synced with an institutional server. Both edm and conda implement managed environments and employ shared package stores and hard links where possible to reduce the storage footprint of virtual environments. But, notably, they also both follow a cultural norm of “playing nicely” (more or less — edm is substantially better about this than conda) with pip and following the same (more or less) command-line interface. conda has the notable disadvantage of a substantially more-invasive installation. In my experience, it is harder to debug conda environments (I do this a lot on the first days of class) and can take substantial effort to reverse an installation of Anaconda and remove all of the “magic” it contributes.
Meanwhile, improvements to pip over the last several years, especially since late 2020, effectively erased the dependency-solving advantages offered by edm and conda, and the main advantages offered by them are space efficiency and familiarity. Enthought’s packages continue to be rigorously tested and curated, and Anaconda’s conda-forge remains a trusted source for the latest versions of packages. But neither of them was particularly optimized for the fast, ephemeral environments needed for continuous integration workflows.
Astral – uv
That brings us to a newer player, uv, published by Astral, whose focus is on high-performance tooling for Python written in Rust. (Rust is a compiled language that prioritizes memory and thread safety and is often found in web services and system software.) uv addresses the needs presented by continuous integration environments, where source code repositories like GitHub enable automatic actions for testing and building code as part of the regular development process. Many providers of cloud services like GitHub charge for compute time, so that minimizing the time to set up environments for testing translates directly to cost savings. Depending on the situation, uv can operate from 10-100 times faster than pip.
That performance comes with tradeoffs, however. uv intentionally diverges from some long-standing pip conventions in order to optimize for speed, reproducibility, and CI-friendly workflows. Most notably, it decouples installs from shell activation, aggressively deduplicates packages across environments, and it treats dependency resolution as a first-class, cached artifact in a lock file.
For example, whereas venv is quite explicit about which runtime is used and what the virtual environment is named, uv will implicitly create an environment with a default name as needed, and activation is often unnecessary. And consider caching, uv‘s aggressive use of references to a global package store deliberately discards the independent isolation of venv environments; environments are no longer self-contained as they are under venv management.
These choices optimize for speed and reproducibility in CI pipelines, but come at the cost of some of the simplicity and transparency that make venv and pip effective teaching tools. Those design choices make a lot of sense for automated build systems that create and destroy environments repeatedly, but they are less compelling in interactive or instructional settings, where environment creation happens infrequently and clarity matters more than raw speed. They also add a lot of additional conceptual “surface area” to cover during class.
Conclusion
We acknowledge the curated safety and testing of edm environments, the ubiquity of conda environments and the conda-forge ecosystem, and the impressive engineering and benefit to modern CI pipelines of uv. However, we generally prefer to stick with the standard, simple, tried-and-true venv + pip tooling for teaching and day-to-day development. It’s the easiest foundation to build on if students want to explore the other options.
