Httpx

httpx Best Practices

httpx is the de-facto modern HTTP client for Python, offering both synchronous and asynchronous APIs, native HTTP/2 support, and advanced features essential for high-throughput services. Adhere to these guidelines for optimal performance, reliability, and code clarity.

Code Organization and Structure

Always use a Client or AsyncClient instance

Avoid top-level helper functions (httpx.get, httpx.post) for anything beyond simple, one-off scripts. Client instances enable connection pooling, HTTP/2, and persistent configuration.

❌ BAD:

import httpx

# In a loop or repeated calls, this creates new connections every time
for _ in range(5):
    response = httpx.get("https://api.example.com/data")
    # ... process response

✅ GOOD:

import httpx

# Client is instantiated once and reused
with httpx.Client() as client:
    for _ in range(5):
        response = client.get("https://api.example.com/data")
        # ... process response

Instantiate a single client per process or request scope

Reuse Client/AsyncClient instances. Creating new clients in a hot loop defeats connection pooling and can exhaust resources. Pass a scoped client or use a global instance (with caution in async apps).

❌ BAD:

# In an async web handler, this creates a new client per request
async def get_user_data(user_id: str):
    async with httpx.AsyncClient() as client: # Client created per request
        response = await client.get(f"https://api.example.com/users/{user_id}")
        return response.json()

✅ GOOD:

# Instantiate client once at application startup
# For web frameworks, use dependency injection or app-level state
import httpx

# Global client (careful with stateful clients in multi-tenant apps)
# For web frameworks like FastAPI, prefer dependency injection.
_async_client: httpx.AsyncClient | None = None

async def get_async_client() -> httpx.AsyncClient:
    global _async_client
    if _async_client is None:
        _async_client = httpx.AsyncClient(base_url="https://api.example.com")
    return _async_client

async def get_user_data(user_id: str):
    client = await get_async_client() # Reuse the same client
    response = await client.get(f"/users/{user_id}")
    return response.json()

Use context managers for client lifecycle

Always use with httpx.Client() as client: or async with httpx.AsyncClient() as client: to guarantee proper shutdown of background workers and sockets.

❌ BAD:

client = httpx.Client()
response = client.get("https://api.example.com/data")
# client.close() might be forgotten, leading to resource leaks

✅ GOOD:

with httpx.Client() as client:
    response = client.get("https://api.example.com/data")
    # Client is automatically closed on exit

Common Patterns and Anti-patterns

Prioritize AsyncClient in async contexts

In async web frameworks (FastAPI, Starlette, Quart) or any asyncio/trio application, use AsyncClient and await client.get(...) to keep the event loop free.

❌ BAD:

# In an async function, using a synchronous client blocks the event loop
import httpx
import asyncio

async def fetch_sync_in_async():
    client = httpx.Client() # This is a synchronous client
    response = client.get("https://slow.example.com") # BLOCKS
    client.close()
    return response.text

✅ GOOD:

import httpx
import asyncio

async def fetch_async():
    async with httpx.AsyncClient() as client:
        response = await client.get("https://fast.example.com") # Non-blocking
        return response.text

Stream large payloads to reduce memory pressure

For large files or responses, use client.stream(...) and iterate over response.aiter_bytes() (or response.iter_bytes() in sync mode) to process data as it arrives.

❌ BAD:

# Loads entire response into memory
response = client.get("https://large-file.example.com/data.zip")
file_content = response.content
# Process file_content

✅ GOOD:

async with client.stream("GET", "https://large-file.example.com/data.zip") as response:
    response.raise_for_status()
    async for chunk in response.aiter_bytes():
        # Process chunk as it arrives
        pass

Configure client-level defaults for DRY code

Pass auth, headers, params, and base_url to the client constructor to apply them to all requests made by that client instance.

❌ BAD:

# Repeating headers and auth for every request
client.get("https://api.example.com/users", headers={"Authorization": "Bearer token"})
client.post("https://api.example.com/items", headers={"Authorization": "Bearer token"}, json={...})

✅ GOOD:

client = httpx.Client(
    base_url="https://api.example.com",
    headers={"Authorization": "Bearer token"},
    auth=httpx.BasicAuth("user", "pass") # Can also set auth here
)
client.get("/users")
client.post("/items", json={...})

Performance Considerations

Enable HTTP/2 explicitly

For modern services, enable HTTP/2 support for multiplexing and improved performance. Install httpx[http2].

❌ BAD:

client = httpx.Client() # Defaults to HTTP/1.1

✅ GOOD:

client = httpx.Client(http2=True) # Enables HTTP/2

Set explicit, granular timeouts

Always configure timeouts. The default 5-second timeout is a safeguard, but customize it per-endpoint or client for specific needs (e.g., longer read timeouts for large responses).

❌ BAD:

response = client.get("https://api.example.com/slow-endpoint") # Uses default 5s timeout

✅ GOOD:

# Client-level timeout
client = httpx.Client(timeout=httpx.Timeout(10.0, connect=5.0, read=30.0))
response = client.get("https://api.example.com/slow-endpoint")

# Request-level override
response = client.get("https://api.example.com/another-endpoint", timeout=5.0)

Control redirects explicitly

httpx disables redirects by default. Only enable follow_redirects=True when necessary to avoid unexpected behavior or performance overhead.

❌ BAD:

# Assuming redirects are handled automatically (they are not by default)
response = client.get("https://example.com/old-path")
print(response.url) # Might still be the old URL if not redirected

✅ GOOD:

response = client.get("https://example.com/old-path", follow_redirects=True)
print(response.url) # Will be the new URL after redirect

Common Pitfalls and Gotchas

Handle HTTP errors gracefully

Always check for non-2xx responses. Use response.raise_for_status() to automatically raise httpx.HTTPStatusError for bad responses, or catch specific httpx.HTTPError subclasses.

❌ BAD:

response = client.get("https://api.example.com/non-existent")
# Code proceeds even if status_code is 404 or 500
if response.status_code == 200:
    data = response.json()
else:
    print("Request failed") # Generic error handling

✅ GOOD:

import httpx

try:
    response = client.get("https://api.example.com/non-existent")
    response.raise_for_status() # Raises HTTPStatusError for 4xx/5xx responses
    data = response.json()
except httpx.HTTPStatusError as e:
    print(f"HTTP error occurred: {e.response.status_code} - {e.response.text}")
except httpx.RequestError as e:
    print(f"An error occurred while requesting {e.request.url!r}: {e}")

Understand response.url is a URL object

response.url is an httpx.URL object, not a string. Access its components or convert to string explicitly.

❌ BAD:

response = client.get("https://example.com/path?query=1")
if response.url == "https://example.com/path?query=1": # String comparison fails
    pass

✅ GOOD:

from httpx import URL

response = client.get("https://example.com/path?query=1")
if response.url == URL("https://example.com/path?query=1"): # Compare with URL object
    pass
if str(response.url) == "https://example.com/path?query=1": # Explicit string conversion
    pass
print(response.url.host) # Access components

Type Hints

Annotate httpx types for clarity and static analysis

Use httpx.Client, httpx.AsyncClient, httpx.Response, and httpx.Request for type hinting. This improves IDE support and aligns with modern Python practices.

❌ BAD:

def fetch_data(client, url):
    response = client.get(url)
    return response.json()

✅ GOOD:

import httpx
from typing import Dict, Any

def fetch_data(client: httpx.Client, url: str) -> Dict[str, Any]:
    response: httpx.Response = client.get(url)
    response.raise_for_status()
    return response.json()

Virtual Environments, Packaging, and Testing

Always use virtual environments

This is standard Python practice. Ensure httpx and its optional dependencies (e.g., httpx[http2,brotli]) are installed in a dedicated virtual environment.

python -m venv .venv
source .venv/bin/activate
pip install "httpx[http2,brotli]"

Declare httpx and extras in pyproject.toml or requirements.txt

For reproducible builds, explicitly list httpx and any required extras in your project's dependency management file.

pyproject.toml:

[project]
dependencies = [
    "httpx[http2,brotli]",
]

requirements.txt:

httpx[http2,brotli]

Use pytest-httpx for robust testing

For unit and integration tests, pytest-httpx provides excellent fixtures for mocking httpx requests, allowing you to simulate API responses without making actual network calls. For testing internal ASGI applications, httpx.ASGITransport is the correct tool.

✅ GOOD (using pytest-httpx):

import httpx
import pytest

# Install: pip install pytest-httpx

def test_fetch_user_data(httpx_mock):
    httpx_mock.add_response(url="https://api.example.com/users/1", json={"id": 1, "name": "Test User"})

    with httpx.Client() as client:
        response = client.get("https://api.example.com/users/1")
        assert response.status_code == 200
        assert response.json() == {"id": 1, "name": "Test User"}

✅ GOOD (using ASGITransport for internal ASGI app):

import httpx
from httpx import ASGITransport
from my_app import app # Assuming 'app' is your ASGI application

def test_my_asgi_app():
    transport = ASGITransport(app=app)
    with httpx.Client(transport=transport, base_url="http://test") as client:
        response = client.get("/health")
        assert response.status_code == 200
        assert response.json() == {"status": "ok"}