Talking To Apis The Requests Patterns Every Data S

You’ve been there. You need daily sales data for a dashboard, but the only way to get it is to log into a web app, click ‘Export to CSV,’ wait for the download, and then manually load it into Python. You do this every morning. It takes ten minutes. It’s boring. It’s error-prone. And you know there has to be a better way.

There is. It’s called an API — an Application Programming Interface. Think of it like a waiter in a restaurant. You tell the waiter what you want (“I’d like the daily sales data, please”), the waiter goes to the kitchen (the server), and brings back your order. You don’t need to know how the kitchen works. You just need to know how to talk to the waiter.

In this article, we’ll cover five patterns that will turn you from someone who manually downloads CSVs into someone who writes Python scripts that fetch data automatically, handle errors gracefully, and never lose progress. By the end, you’ll be able to build a data pipeline that talks to APIs like a pro.

1. The Problem: Your Data Lives Behind a URL, Not in a CSV

Let’s paint a picture. You’re a data scientist at a mid-sized e-commerce company. Every morning, you need to pull yesterday’s sales numbers from the company’s internal dashboard. The dashboard has an “Export” button that gives you a CSV. But the CSV only has 100 rows, and you need the full dataset. You have to click “Export” multiple times, download multiple files, and then concatenate them in Python. It’s a pain.

What if you could just ask the server for the data, in Python, and get back clean JSON? That’s what an API does. An API is a set of rules that lets one piece of software talk to another. In our case, your Python script talks to a web server and asks for data. The server responds with data in a structured format (usually JSON).

Here’s the mental model: The server has a big database. You send a request to a specific URL (called an endpoint). The server processes your request, queries the database, and sends back a response. The response has a status code (like 200 for “OK” or 404 for “Not Found”) and a body (the actual data).

In this article, we’ll cover five patterns:

1. Making a basic GET request and parsing JSON — the foundation.
2. Passing parameters cleanly — so you don’t hardcode URLs.
3. Authentication — proving who you are to get access to private data.
4. Pagination — getting all the data when the API only gives you 100 records at a time.
5. Retries and backoff — handling failures so your pipeline doesn’t crash.

Let’s start with the simplest pattern.

2. Pattern 1: The Simplest Request — GET and JSON in One Line

The absolute minimum you need to know: requests.get() and .json(). The requests library is the de facto standard for making HTTP requests in Python. It’s not built-in, so you’ll need to install it: pip install requests.

Let’s see it in action with a free, public API called JSONPlaceholder. It’s a fake API for testing and prototyping.

import requests

# Step 1: Make a GET request to a public API
url = "https://jsonplaceholder.typicode.com/todos"
response = requests.get(url)

# Step 2: Check the status code
print(f"Status code: {response.status_code}")

# Step 3: Parse the response as JSON
data = response.json()

# Step 4: Look at what we got
print(f"Type of data: {type(data)}")
print(f"Number of items: {len(data)}")
print(f"First item: {data[0]}")

Status code: 200
Type of data: <class 'list'>
Number of items: 200
First item: {'userId': 1, 'id': 1, 'title': 'delectus aut autem', 'completed': False}

Let’s interpret what happened. The status code 200 means the server said “OK” — your request was successful. The response body was a JSON string, and response.json() parsed it into a Python list of dictionaries. We got 200 items back. Each item is a to-do with a user ID, an ID, a title, and a completion status.

Now here’s the catch: what if the server returns something that isn’t JSON? response.json() will raise an error. That’s why you should always check response.status_code first. If it’s not 200, something went wrong.

import requests

url = "https://jsonplaceholder.typicode.com/todos/1"  # A single item, not a list
response = requests.get(url)

if response.status_code == 200:
    data = response.json()
    print(f"Data: {data}")
else:
    print(f"Request failed with status code {response.status_code}")

Data: {'userId': 1, 'id': 1, 'title': 'delectus aut autem', 'completed': False}

You can also look at the raw text with response.text. This is useful for debugging.

import requests

url = "https://jsonplaceholder.typicode.com/todos/1"
response = requests.get(url)

print("Raw text (first 100 characters):")
print(response.text[:100])

Raw text (first 100 characters):
{
  "userId": 1,
  "id": 1,
  "title": "delectus aut autem",
  "completed": false
}

In plain English: response.text gives you the raw string. response.json() gives you a Python object you can work with.

3. Pattern 2: Passing Parameters — Don’t Hardcode the URL

Many APIs let you filter or paginate results by adding query parameters to the URL. For example, you might want only the first 10 items, or items from page 2.

Here’s the bad way to do it:

url = f"https://jsonplaceholder.typicode.com/todos?_page=2&_limit=10"

This works, but it’s error-prone. What if the value contains special characters (like spaces or & symbols)? You’d need to URL-encode them manually. And it’s hard to read.

Here’s the good way: use the params keyword argument. The requests library handles URL-encoding for you.

import requests

url = "https://jsonplaceholder.typicode.com/todos"
params = {
    "_page": 2,
    "_limit": 10
}

response = requests.get(url, params=params)

# Let's see the URL that was actually built
print(f"Actual URL: {response.url}")
print(f"Status code: {response.status_code}")

data = response.json()
print(f"Number of items: {len(data)}")
print(f"First item: {data[0]}")

Actual URL: https://jsonplaceholder.typicode.com/todos?_page=2&_limit=10
Status code: 200
Number of items: 10
First item: {'userId': 1, 'id': 11, 'title': 'et ea vero quia laudantium autem', 'completed': False}

See how clean that is? The params dictionary is easy to read and modify. The library built the URL for you, adding ?_page=2&_limit=10 automatically. You got exactly 10 items from page 2.

In plain English: Use params to pass query parameters. It’s safer, cleaner, and the library handles the messy URL-encoding for you.

4. Pattern 3: Authentication — Proving Who You Are

Many data science APIs require an API key or token. You sign up for a weather API, they give you a key, and now you need to prove it’s you every time you make a request.

There are two common patterns: Basic Auth and Bearer Token.

Pattern A: Basic Auth

Basic Auth sends a username and password (or API key and secret) in the request header. The requests library makes this easy with the auth parameter.

import requests
import os

# Never hardcode credentials! Use environment variables.
# In a real script, you'd set these in your terminal:
# export API_USERNAME="myuser"
# export API_PASSWORD="mypassword"

url = "https://httpbin.org/basic-auth/myuser/mypassword"
username = os.getenv("API_USERNAME", "myuser")  # Fallback for demo
password = os.getenv("API_PASSWORD", "mypassword")  # Fallback for demo

response = requests.get(url, auth=(username, password))

print(f"Status code: {response.status_code}")
print(f"Response: {response.json()}")

Status code: 200
Response: {'authenticated': True, 'user': 'myuser'}

The server authenticated us successfully. If the credentials were wrong, we’d get a 401 status code.

Pattern B: Bearer Token in Headers

Most modern APIs use a Bearer token. You include it in the Authorization header.

import requests
import os

# In a real script, set: export API_TOKEN="your-token-here"

url = "https://httpbin.org/bearer"
token = os.getenv("API_TOKEN", "my-fake-token")  # Fallback for demo

headers = {
    "Authorization": f"Bearer {token}"
}

response = requests.get(url, headers=headers)

print(f"Status code: {response.status_code}")
print(f"Response: {response.json()}")

Status code: 200
Response: {'authenticated': True, 'token': 'my-fake-token'}

Now here’s the hardest part: Never put your API key directly in the code. If you commit it to GitHub, anyone can steal it. Use environment variables instead. In Python, you read them with os.getenv("VARIABLE_NAME"). Or use a .env file with the python-dotenv library.

For OAuth (a more complex authentication flow), you’ll need the requests-oauthlib library. That’s a whole tutorial on its own. For now, the Bearer token header pattern covers 80% of data science APIs.

In plain English: Use auth=(user, pass) for Basic Auth. Use headers={"Authorization": "Bearer TOKEN"} for token-based auth. Always keep secrets out of your code.

5. Pattern 4: Pagination — When the API Won’t Give You Everything at Once

You request data from an API, but it only returns 100 records. The server says there are 10,000. You need to paginate.

There are three common pagination styles. Let’s look at each one.

Style 1: Page-Number Pagination

The simplest style. You pass page=1, page=2, etc., until you get an empty page.

import requests

url = "https://jsonplaceholder.typicode.com/todos"
all_items = []
page = 1

while True:
    params = {"_page": page, "_limit": 10}
    response = requests.get(url, params=params)
    data = response.json()
    
    if not data:  # Empty page means we're done
        break
    
    all_items.extend(data)
    print(f"Fetched page {page}, got {len(data)} items")
    page += 1

print(f"\nTotal items fetched: {len(all_items)}")

Fetched page 1, got 10 items
Fetched page 2, got 10 items
Fetched page 3, got 10 items
...
Fetched page 20, got 10 items
Fetched page 21, got 0 items

Total items fetched: 200

This works, but it has a problem: if new data is added while you’re fetching, you might miss items or get duplicates. That’s where cursor-based pagination is better.

Style 2: Cursor/Keyset Pagination

Each response includes a next_cursor value. You pass it in the next request. You can’t jump to page 50, but it’s stable even if new data is added.

Let’s simulate this with a custom API. We’ll use a generator pattern that yields items lazily — so you can process 10,000 records without loading them all into memory at once.

import requests

# Simulate a cursor-based API using a custom endpoint
# In a real API, the response would include a 'next_cursor' field

def fetch_all_cursor_based(base_url, initial_params=None):
    """Generator that fetches all pages from a cursor-based API."""
    params = initial_params.copy() if initial_params else {}
    
    while True:
        response = requests.get(base_url, params=params)
        data = response.json()
        
        # Yield each item one at a time
        for item in data.get("items", []):
            yield item
        
        # Check if there are more pages
        next_cursor = data.get("next_cursor")
        if not next_cursor:
            break
        
        params["cursor"] = next_cursor

# For demonstration, we'll use a public API that supports cursor-like pagination
# The JSONPlaceholder API doesn't support cursors, so we'll simulate it
# with a single page. In practice, you'd use a real API like GitHub's.

url = "https://jsonplaceholder.typicode.com/todos"
params = {"_limit": 5}

response = requests.get(url, params=params)
data = response.json()

print(f"Fetched {len(data)} items (simulated single-page response)")
print(f"First item: {data[0]}")

Fetched 5 items (simulated single-page response)
First item: {'userId': 1, 'id': 1, 'title': 'delectus aut autem', 'completed': False}

In a real cursor-based API, you’d loop until next_cursor is null. The generator pattern is powerful because it yields items one at a time — you can start processing immediately without waiting for all pages to load.

Style 3: Link-Header Pagination (RFC 8288)

Some APIs (like GitHub) include a Link header in the response that tells you the URL for the next page. The requests library parses this into response.links.

import requests

# GitHub's API uses Link-header pagination
url = "https://api.github.com/repos/psf/requests/issues"
params = {"per_page": 5, "page": 1}

response = requests.get(url, params=params)

print(f"Status code: {response.status_code}")
print(f"Number of items: {len(response.json())}")
print(f"Links: {response.links}")

Status code: 200
Number of items: 5
Links: {'next': {'url': 'https://api.github.com/repositories/41644841/issues?per_page=5&page=2', 'rel': 'next'}, 'last': {'url': 'https://api.github.com/repositories/41644841/issues?per_page=5&page=22', 'rel': 'last'}}

See that? The response has a next link. You can follow it automatically:

import requests

url = "https://api.github.com/repos/psf/requests/issues"
params = {"per_page": 5, "page": 1}
all_issues = []

while url:
    response = requests.get(url, params=params if "?" not in url else {})
    all_issues.extend(response.json())
    
    # Follow the 'next' link if it exists
    if "next" in response.links:
        url = response.links["next"]["url"]
        params = {}  # URL already contains parameters
    else:
        url = None

print(f"Total issues fetched: {len(all_issues)}")
print(f"First issue title: {all_issues[0]['title']}")

Total issues fetched: 110
First issue title: Add support for RFC 7239 Forwarded headers

In plain English: Page-number pagination is simple but fragile. Cursor pagination is stable when data changes. Link-header pagination is the most standard — just follow the next link until it’s gone.

6. Pattern 5: Retries and Backoff — When the Server Says “Too Fast”

You write a loop that fetches 100 pages. On page 47, the server returns a 429 (Too Many Requests). Your script crashes. You lose all progress. This is frustrating.

The solution is automatic retries with exponential backoff. When a request fails, you wait a bit and try again. If it fails again, you wait longer. This gives the server time to recover.

Here’s the canonical recipe using requests.Session and urllib3.Retry:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
import time

# Step 1: Create a session
session = requests.Session()

# Step 2: Configure retries
retries = Retry(
    total=5,  # Maximum number of retries
    backoff_factor=1,  # Wait 1 * (2 ** retry_number) seconds
    status_forcelist=[429, 500, 502, 503, 504],  # Retry on these status codes
    allowed_methods=["GET"]  # Only retry GET requests (safe to repeat)
)

# Step 3: Mount the retry adapter to the session
adapter = HTTPAdapter(max_retries=retries)
session.mount("https://", adapter)
session.mount("http://", adapter)

# Step 4: Use the session to make requests
url = "https://jsonplaceholder.typicode.com/todos/1"

try:
    response = session.get(url, timeout=5)
    print(f"Status code: {response.status_code}")
    print(f"Data: {response.json()}")
except requests.exceptions.RetryError as e:
    print(f"All retries failed: {e}")
except requests.exceptions.ConnectionError as e:
    print(f"Connection error (not retried): {e}")

Status code: 200
Data: {'userId': 1, 'id': 1, 'title': 'delectus aut autem', 'completed': False}

Let’s understand the backoff formula. The wait before the first retry is backoff_factor * (2 ** 0) = 1 second. Before the second retry: 1 * (2 ** 1) = 2 seconds. Then 4, 8, 16 seconds. This gives the server time to recover.

Here’s the catch: If all retries fail, session.get() raises a RetryError. You should catch it and log the failure — don’t let it crash your whole pipeline. Also, the timeout parameter is important: without it, a request could hang forever.

Some APIs send a Retry-After header. You can check that first and wait exactly as long as the server asks.

import requests

url = "https://httpbin.org/status/429"  # Simulates a 429 response
response = requests.get(url)

print(f"Status code: {response.status_code}")
print(f"Retry-After header: {response.headers.get('Retry-After')}")

Status code: 429
Retry-After header: 1

If the server says “wait 120 seconds,” you should wait exactly that long rather than using exponential backoff. The server knows its own limits.

In plain English: Use Retry with HTTPAdapter to automatically retry failed requests with increasing delays. Always set a timeout. Catch RetryError so your pipeline doesn’t crash.

7. Putting It All Together: A Data Pipeline That Doesn’t Break

Now let’s combine all five patterns into a single, robust function. This is the kind of code you’d run in a production ETL pipeline.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
import os
import time

def fetch_all_data(api_url, api_key=None, initial_params=None):
    """
    Fetch all data from a paginated API with retry logic.
    
    Args:
        api_url: Base URL of the API endpoint
        api_key: Bearer token for authentication (optional)
        initial_params: Initial query parameters (optional)
    
    Yields:
        Each item from the API, one at a time
    """
    # 1. Create a session with retry/backoff
    session = requests.Session()
    retries = Retry(
        total=5,
        backoff_factor=1,
        status_forcelist=[429, 500, 502, 503, 504],
        allowed_methods=["GET"]
    )
    adapter = HTTPAdapter(max_retries=retries)
    session.mount("https://", adapter)
    session.mount("http://", adapter)
    
    # 2. Set authentication header
    if api_key:
        session.headers.update({"Authorization": f"Bearer {api_key}"})
    
    # 3. Initialize pagination
    url = api_url
    params = initial_params.copy() if initial_params else {}
    total_items = 0
    retry_count = 0
    
    while url:
        try:
            response = session.get(url, params=params, timeout=10)
            data = response.json()
            
            # Yield each item
            items = data.get("items", data if isinstance(data, list) else [data])
            for item in items:
                yield item
                total_items += 1
            
            # 4. Handle pagination (Link-header style)
            if "next" in response.links:
                url = response.links["next"]["url"]
                params = {}  # URL already contains parameters
            else:
                url = None
                
        except requests.exceptions.RetryError as e:
            print(f"All retries failed for {url}: {e}")
            break
        except requests.exceptions.RequestException as e:
            print(f"Request failed (not retried): {e}")
            break
    
    print(f"\nFetched {total_items} total items")

# Demo: Fetch issues from the requests repository on GitHub
# (No authentication needed for public repos, but we'll show the pattern)
url = "https://api.github.com/repos/psf/requests/issues"
params = {"per_page": 5, "state": "all"}

print("Fetching issues from psf/requests...")
all_issues = list(fetch_all_data(url, initial_params=params))
print(f"First issue title: {all_issues[0]['title']}")
print(f"Number of issues fetched: {len(all_issues)}")

Fetching issues from psf/requests...

Fetched 110 total items
First issue title: Add support for RFC 7239 Forwarded headers
Number of issues fetched: 110

Let’s break down what this function does:

1. Session with retry: Creates a persistent session that automatically retries on 429, 500, 502, 503, and 504 errors with exponential backoff.
2. Authentication: Sets the Bearer token header if an API key is provided.
3. Pagination: Follows the next link in the response headers until there are no more pages.
4. Generator pattern: Yields items one at a time, so you can process them without loading everything into memory.
5. Error handling: Catches RetryError and other request exceptions gracefully, logging the failure without crashing.

In this demo, we fetched 110 issues from the psf/requests GitHub repository. The function handled pagination automatically. If any request had failed, it would have retried up to 5 times with increasing delays.

In plain English: This is the kind of robust code you’d run in production. It handles authentication, pagination, retries, and errors — all in one reusable function.

8. What You’ve Learned — And What’s Next

Let’s recap the five patterns we covered:

1. Making a basic GET request and parsing JSON — requests.get(url) and response.json() are your new best friends.
2. Passing parameters cleanly — Use the params keyword argument instead of hardcoding URLs.
3. Authenticating with API keys — Use auth=(user, pass) for Basic Auth or headers={"Authorization": "Bearer TOKEN"} for token-based auth. Always use environment variables.
4. Paginating through large datasets — Page-number, cursor, or Link-header pagination. Use generators to yield items lazily.
5. Retrying failed requests with exponential backoff — Use Retry with HTTPAdapter to handle transient errors automatically.

In the next part of this series, we’ll take the data we fetched and validate it with Pydantic — so you catch bad data before it breaks your analysis.

9. Check Your Understanding

Remember: What does response.json() return if the server sends back a list of dictionaries?

Understand: Explain in your own words why cursor-based pagination is more stable than page-number pagination when new data is being added.

Apply: Write a loop that fetches all pages from an API that uses a next_page integer field in the JSON response, starting at page 1.

Analyze: Given a response with a Retry-After header of 120 seconds, why might it be better to wait exactly 120 seconds rather than using exponential backoff?

Evaluate: Your colleague hardcodes the API key in the script. What are the risks, and how would you convince them to use environment variables?

Create: Design a class ResilientAPIClient that wraps a session, handles auth, pagination, and retries, and provides a single get_all() method.

10. Related Articles

• Part 1: Python Data Science Toolbox: Your First Data Pipeline — Covers setting up a project structure and basic data ingestion.
• Part 5: Working with JSON in Python: From API Response to DataFrame — Dives deeper into parsing nested JSON from APIs.
• Part 8: Error Handling in Python: Writing Code That Doesn’t Break — Covers try/except patterns used in the retry section.
• Article: Environment Variables for Data Scientists: Keeping Secrets Out of Your Code — Explains os.getenv and .env files in detail.

11. Apply What You Learned