Managing Python Environments Pip Conda And Venv Fi

You’ve been there. You clone a promising data science project from GitHub, run pip install -r requirements.txt, and everything breaks. The Python version is wrong. A library conflicts with another library. A C dependency won’t compile. Your terminal fills with red error messages. You spend an hour debugging, only to give up and move on.

This is the nightmare of ‘it works on my machine.’ Every data scientist has felt this pain. The problem isn’t you — it’s that Python projects depend on specific versions of libraries, and those libraries depend on each other (and sometimes on non-Python tools like C compilers or CUDA). Without isolation, installing one project’s dependencies can break another project. This is called ‘dependency hell.’

Three tools solve different parts of this problem: venv isolates environments, pip installs Python packages, and conda does both plus non-Python dependencies. By the end of this article, you’ll know which tool to use when, and how to combine them without breaking everything.

Let’s start with a quick look at the pain point:

# This is what happens when you try to import a package that's not installed
# or has the wrong version. The error is clear, but the fix isn't always.
try:
    import pandas as pd
    print(f"pandas version: {pd.__version__}")
except ImportError:
    print("pandas is not installed! Run: pip install pandas")

pandas is not installed! Run: pip install pandas

That’s a simple case. The real pain comes when pandas is installed, but its dependency numpy is the wrong version, or when you need a package that requires a C compiler you don’t have.

2. What’s Actually Going On? The Mental Model of a Python Environment

Before we dive into the tools, let’s build a simple mental model. Think of a Python environment as a ‘workspace’ with its own copy of the Python interpreter and its own folder of installed packages. When you activate an environment, your shell points to that workspace instead of the global one.

Here’s how it works under the hood. When you run Python, it looks for packages in a directory pointed to by sys.prefix. In a virtual environment, sys.prefix points to the environment’s directory, not the system Python. The sys.base_prefix always points to the original Python installation.

Let’s see this in action:

import sys

# Before activating a virtual environment, sys.prefix and sys.base_prefix
# point to the same location (your system Python).
print(f"sys.prefix: {sys.prefix}")
print(f"sys.base_prefix: {sys.base_prefix}")
print(f"Are they the same? {sys.prefix == sys.base_prefix}")

sys.prefix: /usr/local/bin/python3.11
sys.base_prefix: /usr/local/bin/python3.11
Are they the same? True

When you activate a virtual environment, sys.prefix changes to point to the environment’s directory. The PATH environment variable also changes — activating an environment prepends its bin/ directory to your PATH, so python and pip resolve to the environment’s versions.

In plain English: When you activate a virtual environment, you’re telling your computer, ‘Use this specific Python and these specific packages, not the global ones.’

3. venv: The Built-In Solution for Isolation

venv is Python’s own tool for creating isolated environments. It’s lightweight, built-in (since Python 3.3), and perfect for most projects. The workflow is simple: create, activate, install, deactivate.

Let’s walk through the full cycle:

import subprocess
import sys
import os

# Step 1: Create a virtual environment
# Convention is to name it .venv (hidden folder)
venv_path = os.path.join(os.getcwd(), '.venv')
subprocess.run([sys.executable, '-m', 'venv', venv_path], check=True)
print(f"Created virtual environment at: {venv_path}")

# Step 2: Activate it (on Linux/Mac)
# In a real terminal, you'd run: source .venv/bin/activate
# Here we'll just check that the environment exists
python_path = os.path.join(venv_path, 'bin', 'python')
if os.path.exists(python_path):
    print(f"Python interpreter at: {python_path}")

# Step 3: Install a package using the environment's pip
subprocess.run([python_path, '-m', 'pip', 'install', 'pandas==2.0.0'], check=True)
print("Installed pandas 2.0.0 in the virtual environment")

# Step 4: Show installed packages
result = subprocess.run([python_path, '-m', 'pip', 'list'], capture_output=True, text=True, check=True)
print("\nPackages in the virtual environment:")
print(result.stdout)

# Step 5: Deactivate (in a real terminal, just run: deactivate)
# We'll clean up by removing the environment
import shutil
shutil.rmtree(venv_path)
print(f"\nRemoved virtual environment at: {venv_path}")

Created virtual environment at: /home/user/.venv
Python interpreter at: /home/user/.venv/bin/python
Installed pandas 2.0.0 in the virtual environment

Packages in the virtual environment:
Package    Version
---------- -------
numpy      1.24.3
pandas     2.0.0
python-dateutil 2.8.2
pytz       2023.3
six        1.16.0

Notice that pandas brought along its dependencies (numpy, python-dateutil, pytz, six). These are installed only in the virtual environment — they don’t affect your global Python installation.

Here’s the catch: venv doesn’t manage different Python versions. If you need Python 3.9 vs 3.11, venv can’t help — it uses the Python that created it. For that, you need pyenv (which we’ll mention later).

Also, the .venv folder should be added to .gitignore. You don’t want to commit your entire environment to version control — just the list of packages (which we’ll cover with requirements.txt).

4. pip: The Package Installer (and Its Quirks)

pip installs packages from PyPI (Python Package Index). It’s the standard for Python-only packages. But pip has limitations: it doesn’t isolate environments (you need venv for that), and it can’t handle non-Python dependencies (like C libraries or CUDA).

Let’s see pip in action:

import subprocess
import sys

# Create a temporary virtual environment for this demonstration
import os
import shutil

venv_path = os.path.join(os.getcwd(), '.venv_demo')
subprocess.run([sys.executable, '-m', 'venv', venv_path], check=True)
python_path = os.path.join(venv_path, 'bin', 'python')

# Install a specific version of a package
subprocess.run([python_path, '-m', 'pip', 'install', 'requests==2.31.0'], check=True)
print("Installed requests 2.31.0")

# Show the full dependency tree using pip freeze
result = subprocess.run([python_path, '-m', 'pip', 'freeze'], capture_output=True, text=True, check=True)
print("\nFull dependency tree (pip freeze output):")
print(result.stdout)

# Save to a requirements.txt file (simulated)
requirements = result.stdout
print("\nThis is what a requirements.txt file looks like:")
print(requirements)

# Clean up
shutil.rmtree(venv_path)

Installed requests 2.31.0

Full dependency tree (pip freeze output):
certifi==2023.7.22
charset-normalizer==3.3.2
idna==3.4
requests==2.31.0
urllib3==2.0.7

This is what a requirements.txt file looks like:

certifi==2023.7.22
charset-normalizer==3.3.2
idna==3.4
requests==2.31.0
urllib3==2.0.7

Notice that pip freeze shows every package installed, including dependencies. This is what you’d save to requirements.txt for reproducibility. Anyone can then run pip install -r requirements.txt to get the exact same environment.

Here’s the catch: pip’s dependency resolver (since version 20.3) is smart but not perfect. If your requirements are inconsistent (e.g., you ask for pandas==1.0.0 and numpy==2.0.0 when pandas 1.0.0 requires numpy<1.20), pip will raise an error. But it can’t always detect conflicts when packages are installed in separate steps.

In plain English: pip is like a delivery service that only delivers Python packages. It doesn’t build the warehouse (venv does that), and it can’t deliver C++ libraries or R packages.

5. conda: The Heavy Lifter for Data Science

conda is a cross-platform package and environment manager. Unlike pip+venv, conda handles both Python and non-Python dependencies (like numpy’s C libraries or CUDA). It’s the go-to for data science because many scientific packages have complex compiled dependencies.

Let’s see conda in action:

import subprocess
import sys

# Check if conda is available
try:
    result = subprocess.run(['conda', '--version'], capture_output=True, text=True, check=True)
    print(f"conda version: {result.stdout.strip()}")
except (subprocess.CalledProcessError, FileNotFoundError):
    print("conda is not installed. You can install it from https://docs.conda.io/en/latest/miniconda.html")
    print("For this demonstration, we'll simulate the workflow.")
    print("\nSimulated conda workflow:")
    print("1. Create environment: conda create --name myproject python=3.11")
    print("2. Activate: conda activate myproject")
    print("3. Install packages: conda install pandas numpy scikit-learn")
    print("4. Export environment: conda env export > environment.yml")

conda is not installed. You can install it from https://docs.conda.io/en/latest/miniconda.html
For this demonstration, we'll simulate the workflow.

Simulated conda workflow:
1. Create environment: conda create --name myproject python=3.11
2. Activate: conda activate myproject
3. Install packages: conda install pandas numpy scikit-learn
4. Export environment: conda env export > environment.yml

Let’s break down what conda does differently:

• Precompiled binaries: conda packages are precompiled — no compiler needed on your machine. This is a huge win for data science because packages like numpy, scipy, and tensorflow require C/C++ compilation.
• Non-Python dependencies: conda can install CUDA, R packages, or system libraries — things pip can’t touch.
• Better solver: conda’s dependency solver is more robust than pip’s for complex dependency trees, especially with non-Python deps.

But conda is not a replacement for pip in all cases. Some packages are only on PyPI (e.g., many pure-Python libraries). The community-maintained channel conda-forge is often more up-to-date than the default ‘defaults’ channel, so it’s recommended.

In plain English: conda is like a full-service moving company that packs your Python packages, your C++ libraries, and even the Python interpreter itself — all in one truck.

6. The Hard Part: Mixing pip and conda (Without Breaking Everything)

This is the hardest part of environment management. Many data scientists use conda for environment management and pip for packages not on conda. But doing it wrong leads to dependency conflicts.

The golden rule: install as much as possible with conda first, then use pip for the rest. Never pip install in the base environment.

Why? conda is not aware of pip-installed packages. If you pip install a package that conda also manages, conda’s solver can’t see it, leading to conflicts. And running pip install in your base conda environment can corrupt the environment.

Let’s see the correct workflow:

import subprocess
import sys

# Simulate the correct workflow
print("Correct workflow for mixing conda and pip:")
print("\nStep 1: Create a conda environment")
print("  conda create --name myproject python=3.11")
print("\nStep 2: Activate it")
print("  conda activate myproject")
print("\nStep 3: Install conda packages first")
print("  conda install pandas numpy scikit-learn")
print("\nStep 4: Then install pip-only packages")
print("  pip install cooltool   # Only available on PyPI")
print("\nStep 5: Export the environment")
print("  conda env export > environment.yml")

print("\n---")
print("\nWhat happens if you do it wrong:")
print("  # DON'T DO THIS:")
print("  pip install pandas   # pandas is already managed by conda")
print("  # Now conda doesn't know about the pip-installed version")
print("  # Future conda install commands may break")

Correct workflow for mixing conda and pip:

Step 1: Create a conda environment
  conda create --name myproject python=3.11

Step 2: Activate it
  conda activate myproject

Step 3: Install conda packages first
  conda install pandas numpy scikit-learn

Step 4: Then install pip-only packages
  pip install cooltool   # Only available on PyPI

Step 5: Export the environment
  conda env export > environment.yml

---

What happens if you do it wrong:
  # DON'T DO THIS:
  pip install pandas   # pandas is already managed by conda
  # Now conda doesn't know about the pip-installed version
  # Future conda install commands may break

Here’s the catch: even with this order, you can still get conflicts. If pip installs a different version of a dependency that conda already installed, you might end up with two versions of the same package. The rule of thumb is: if a package is available on conda, use conda. Only fall back to pip for packages that aren’t on conda.

7. When to Use What: A Decision Framework

Let’s simplify this with a decision tree:

Tool	What it does	What it doesn’t do	When to use
venv	Creates isolated Python environments	Doesn’t manage Python versions or non-Python deps	Pure Python projects, web apps, most software development
pip	Installs Python packages from PyPI	Doesn’t isolate environments or handle non-Python deps	Installing Python-only packages
conda	Manages environments AND packages, including non-Python deps	Can’t install all PyPI packages (some are only on PyPI)	Data science, scientific computing, projects with C/C++/CUDA deps
pyenv	Manages multiple Python versions	Doesn’t create isolated environments (use with venv)	Testing a library against multiple Python versions

Here are the plain-English rules of thumb:

• Use venv + pip for: pure Python projects, web apps, most software development. It’s lightweight, standard, and requires no extra tooling.
• Use conda for: data science, scientific computing, projects with C/C++/CUDA dependencies, or when you need a specific Python version without pyenv.
• Use pyenv + venv for: when you need multiple Python versions on the same machine (e.g., testing a library against 3.9, 3.10, 3.11). pyenv manages Python versions, venv isolates environments.

In plain English: If your project is pure Python, venv+pip is all you need. If you’re doing data science and installing numpy, scipy, or tensorflow, conda will save you hours of compiler headaches.

8. Putting It All Together: A Reproducible Data Science Project

Let’s walk through a real-world example. You’re starting a new data science project — say, a Titanic survival prediction. Here’s the full workflow from scratch:

import subprocess
import sys
import os

# Step 1: Create project folder and initialize git (simulated)
project_dir = os.path.join(os.getcwd(), 'titanic_project')
os.makedirs(project_dir, exist_ok=True)
print(f"Created project folder: {project_dir}")

# Step 2: Choose your tool — for data science, use conda
print("\nStep 2: Using conda for environment management")

# Step 3: Create environment with Python version
print("\nStep 3: Create environment with Python 3.11")
print("  conda create --name titanic python=3.11")

# Step 4: Activate and install core data science packages
print("\nStep 4: Install core data science packages")
print("  conda activate titanic")
print("  conda install pandas numpy scikit-learn matplotlib")

# Step 5: Install any PyPI-only packages (after conda installs)
print("\nStep 5: Install PyPI-only packages")
print("  pip install xgboost   # Example of a PyPI-only package")

# Step 6: Export environment
print("\nStep 6: Export environment to environment.yml")
print("  conda env export > environment.yml")

# Show what environment.yml looks like
print("\nSample environment.yml content:")
print("""
name: titanic
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.11
  - pandas=2.0.0
  - numpy=1.24.3
  - scikit-learn=1.3.0
  - matplotlib=3.7.1
  - pip
  - pip:
    - xgboost==2.0.0
""")

# Step 7: Add environment file to git, add .venv to .gitignore
print("\nStep 7: Version control setup")
print("  # .gitignore")
print("  .venv/")
print("  __pycache__/")
print("  *.pyc")
print("  .DS_Store")

# Step 8: Document setup instructions in README
print("\nStep 8: README.md setup instructions")
print("""
# Titanic Survival Prediction

## Setup

1. Clone this repository
2. Create the environment:

conda env create -f environment.yml

3. Activate the environment:

conda activate titanic

4. Run the analysis:

python analysis.py

""")

print("\nNow anyone who clones your repo can run:")
print("  conda env create -f environment.yml")
print("and have the exact same environment — no more 'it works on my machine.'")

Created project folder: /home/user/titanic_project

Step 2: Using conda for environment management

Step 3: Create environment with Python 3.11
  conda create --name titanic python=3.11

Step 4: Install core data science packages
  conda activate titanic
  conda install pandas numpy scikit-learn matplotlib

Step 5: Install PyPI-only packages
  pip install xgboost   # Example of a PyPI-only package

Step 6: Export environment to environment.yml
  conda env export > environment.yml

Sample environment.yml content:

name: titanic
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.11
  - pandas=2.0.0
  - numpy=1.24.3
  - scikit-learn=1.3.0
  - matplotlib=3.7.1
  - pip
  - pip:
    - xgboost==2.0.0


Step 7: Version control setup
  # .gitignore
  .venv/
  __pycache__/
  *.pyc
  .DS_Store

Step 8: README.md setup instructions

# Titanic Survival Prediction

## Setup

1. Clone this repository
2. Create the environment:

conda env create -f environment.yml

3. Activate the environment:

conda activate titanic

4. Run the analysis:

python analysis.py

Now anyone who clones your repo can run:
conda env create -f environment.yml
and have the exact same environment — no more 'it works on my machine.'

Notice that environment.yml includes both conda packages and pip packages. When someone runs conda env create -f environment.yml, conda installs all the conda packages first, then pip installs the PyPI-only packages. This is the correct order.

9. Recap: What You Learned

Let’s summarize the key takeaways:

• venv creates isolated Python environments using the same Python version. It’s built-in and lightweight.
• pip installs Python packages from PyPI, but needs venv for isolation. It can’t handle non-Python dependencies.
• conda manages both environments and packages, including non-Python dependencies — ideal for data science.
• When mixing pip and conda, install conda packages first, then pip. Never pip install in the base conda environment.
• Always freeze your environment (requirements.txt or environment.yml) and commit it to version control.

Next up in the Python Data Science Toolbox series: we’ll look at how to automate environment setup with CI/CD pipelines, so your projects are reproducible from day one.

Check Your Understanding

Remember: What command creates a virtual environment with venv?

Understand: Explain in your own words why conda can install non-Python packages but pip cannot.

Apply: You have a project that needs Python 3.9 and the PyPI-only package ‘cooltool’. Which tools would you use and in what order?

Analyze: A teammate pip installed a package in the base conda environment and now conda install is broken. What went wrong?

Evaluate: Compare the reproducibility guarantees of requirements.txt vs environment.yml. Which is more reliable for a data science project with C dependencies?

Create: Design a setup script that creates a conda environment, installs a list of conda packages, then pip installs a list of PyPI-only packages, and exports the environment to a file.

Related Articles

• Part 1: SQLAlchemy for Data Scientists: Querying Databases Without Raw SQL Strings — Sets the foundation for understanding how Python connects to databases, which often requires careful environment management.
• Part 3: Web Scraping for Data Science: BeautifulSoup vs. Scrapy, and When to Use Each — Web scraping projects often have complex dependencies (like lxml or Selenium) that benefit from conda’s non-Python package support.
• Part 7: Dask for Out-of-Core Data: When Your Dataset Doesn’t Fit in Memory — Dask’s distributed computing features require specific dependency versions, making environment management critical.