httplib2 is a third-party Python HTTP client whose Http object sends requests with a single request() method and returns a (response, content) tuple, with built-in support for HTTP caching, redirects, compression and authentication. It is mature and still maintained, and Google’s API client library (google-api-python-client) depends on it.
import httplib2
http = httplib2.Http()
response, content = http.request("http://localhost:8000/")
status = response.status # 200
html = content.decode() # the page as textAfter installing the library and starting a small local test server, the examples send HEAD, GET, POST, PUT and DELETE requests with custom headers, credentials, caching and timeouts, using httplib2 0.32. A comparison with requests, httpx and urllib and a link-checker example come last.
1. Install httplib2 and Check the Version
We install the package from PyPI with pip.
pip install httplib2The module exposes its version as httplib2.__version__.
import httplib2
version = httplib2.__version__ # '0.32.0'2. A Local Test Server
A small server built on Python’s http.server module does the same job without any setup. It serves a home page, a greeting for GET and POST, an echo of the User-Agent header and a page protected with basic authentication.
import base64
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import urlparse, parse_qs
class Handler(BaseHTTPRequestHandler):
def send_text(self, body, status=200, extra=None):
data = body.encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "text/html; charset=utf-8")
self.send_header("Content-Length", str(len(data)))
self.send_header("Cache-Control", "max-age=60")
for k, v in (extra or {}).items():
self.send_header(k, v)
self.end_headers()
if self.command != "HEAD":
self.wfile.write(data)
def do_GET(self):
url = urlparse(self.path)
if url.path == "/":
self.send_text("<html><head><title>Something.</title></head><body>Something.</body></html>")
elif url.path == "/greet":
name = parse_qs(url.query).get("name", ["stranger"])[0]
self.send_text(f"Hello {name}")
elif url.path == "/agent":
self.send_text(self.headers.get("User-Agent", ""))
elif url.path == "/secure/":
expected = "Basic " + base64.b64encode(b"user7:7user").decode()
if self.headers.get("Authorization") == expected:
self.send_text("This is a secure page.")
else:
self.send_text("Unauthorized", 401, {"WWW-Authenticate": 'Basic realm="Restricted Area"'})
else:
self.send_text("Not Found", 404)
do_HEAD = do_GET
def do_POST(self):
length = int(self.headers.get("Content-Length", 0))
form = parse_qs(self.rfile.read(length).decode("utf-8"))
self.send_text("Hello " + form.get("name", ["stranger"])[0])
def log_message(self, *args):
pass
if __name__ == "__main__":
ThreadingHTTPServer(("localhost", 8000), Handler).serve_forever()We save it as server.py and start it in a separate terminal with python server.py. Every example in this tutorial sends its requests to http://localhost:8000.
3. Syntax and Return Value of request()
All requests go through one method of the Http object.
Http.request(uri, method="GET", body=None, headers=None, redirections=5) -> (Response, bytes)- uri is the full URL, including http:// or https://.
- method is the HTTP method as a string, such as “HEAD” or “POST”.
- body is the request body as a string or bytes, and headers is a dictionary.
- Return value is a tuple. The first item is a Response, a dictionary of lowercase header names with extra attributes such as status, reason and fromcache. The second item is the body as bytes.
4. Use httplib2 to Read a Web Page
A GET request is the default. The body arrives as bytes, so we decode it to text.
http = httplib2.Http()
response, content = http.request("http://localhost:8000/")
page = content.decode()
# '<html><head><title>Something.</title></head><body>Something.</body></html>'4.1. Stripping HTML Tags
A regular expression removes the tags from a small page. It only strips the markup and does not sanitize the content, so for real pages an HTML parser such as Beautiful Soup is the safer choice.
import re
stripped = re.sub(r"<[^<]+?>", "", content.decode()) # 'Something.Something.'5. Check Response Status
The status attribute of the response holds the HTTP status code. httplib2 does not raise an exception for 4xx and 5xx codes, so we check the status ourselves.
http = httplib2.Http()
ok = http.request("http://localhost:8000/")[0].status # 200
missing = http.request("http://localhost:8000/news/")[0].status # 4046. Send HTTP HEAD Request
A HEAD request returns only the headers of a resource, which is a cheap way to check its type and size before downloading it.
http = httplib2.Http()
resp = http.request("http://localhost:8000/", "HEAD")[0]
content_type = resp["content-type"] # 'text/html; charset=utf-8'
length = resp["content-length"] # '74'
server = resp["server"] # 'BaseHTTP/0.6 Python/3.14.6'7. Send HTTP GET Request with Parameters
Query parameters go into the URL. urllib.parse.urlencode() builds the query string and escapes special characters such as spaces.
from urllib.parse import urlencode
http = httplib2.Http()
query = urlencode({"name": "Peter"})
greeting = http.request(f"http://localhost:8000/greet?{query}")[1].decode() # 'Hello Peter'8. Send HTTP POST Request
A POST request sends data in the body instead of the URL. For a form submission, we encode the fields and set the matching Content-Type header.
from urllib.parse import urlencode
http = httplib2.Http()
body = urlencode({"name": "Peter"})
resp, content = http.request(
"http://localhost:8000/greet",
method="POST",
headers={"Content-Type": "application/x-www-form-urlencoded"},
body=body,
)
posted = content.decode() # 'Hello Peter'For a JSON API, the body is json.dumps(data) and the header is application/json.
8.1. PUT and DELETE Requests
Any HTTP method name works in the method argument. httplib2 also stores the ETag of a cached resource and sends it with a later PUT as If-Match, which protects against overwriting someone else’s change.
http = httplib2.Http()
put_status = http.request("http://localhost:8000/greet", "PUT", body="x")[0].status # 501
delete_status = http.request("http://localhost:8000/greet", "DELETE")[0].status # 501The test server does not implement these methods, so it answers 501 Not Implemented, which shows that httplib2 returns error codes instead of raising them. One Http object also keeps connections alive and reuses them for later requests to the same host, which is why we create it once and share it.
9. Send User Agent Information
By default, httplib2 identifies itself as Python-httplib2/0.32.0 (gzip). A custom User-Agent header replaces it, which some APIs require to identify the client.
http = httplib2.Http()
default_agent = http.request("http://localhost:8000/agent")[1].decode()
# 'Python-httplib2/0.32.0 (gzip)'
custom = http.request("http://localhost:8000/agent", headers={"User-Agent": "Python script"})[1].decode()
# 'Python script'10. Add Username and Password to Request
The add_credentials() method stores a user name and password. httplib2 sends them only after the server answers with a 401 and a WWW-Authenticate header, and it supports Basic and Digest authentication.
anon = httplib2.Http()
denied = anon.request("http://localhost:8000/secure/")[0].status # 401
auth = httplib2.Http()
auth.add_credentials("user7", "7user")
resp, content = auth.request("http://localhost:8000/secure/")
allowed = (resp.status, content.decode()) # (200, 'This is a secure page.')Real code reads credentials from environment variables or a secrets store instead of the source file, and sends them only over HTTPS.
11. Caching Responses
Caching is the feature that sets httplib2 apart. When the Http object gets a cache directory, it stores responses and follows the Cache-Control, ETag and Last-Modified headers. The test server sends max-age=60, so the second request is served from the cache without contacting the server.
cached = httplib2.Http(".cache")
first = cached.request("http://localhost:8000/")[0].fromcache # False
second = cached.request("http://localhost:8000/")[0].fromcache # TruePassing headers={“cache-control”: “no-cache”} forces a fresh request for one call.
12. Timeouts and Errors
Without a timeout, a request can wait a long time for a server that does not answer. The timeout argument sets a limit in seconds, and network failures raise exceptions such as ConnectionRefusedError, TimeoutError or httplib2.ServerNotFoundError.
safe = httplib2.Http(timeout=5)
try:
safe.request("http://localhost:9/")
except (ConnectionError, TimeoutError, httplib2.HttpLib2Error) as e:
error = type(e).__name__ # 'ConnectionRefusedError'13. httplib2 vs requests, httpx and urllib
httplib2 is a solid choice when HTTP caching matters or a library such as an older Google API client already depends on it. For new code, requests and httpx offer a friendlier API.
| Feature | httplib2 | requests | httpx | urllib.request |
|---|---|---|---|---|
| Install needed | Yes | Yes | Yes | No, standard library |
| Built-in HTTP cache | Yes | No (requests-cache add-on) | No (hishel add-on) | No |
| JSON helpers | No | .json() | .json() | No |
| Async support | No | No | Yes | No |
| HTTP/2 | No | No | Yes (optional) | No |
14. Real-World Example: Checking a List of URLs
A link checker sends a HEAD request to each URL and reports the status, falling back to an error name when the server cannot be reached. The timeout keeps one dead host from stalling the whole run.
def check_links(urls):
client = httplib2.Http(timeout=5)
report = {}
for url in urls:
try:
report[url] = client.request(url, "HEAD")[0].status
except (ConnectionError, TimeoutError, httplib2.HttpLib2Error) as e:
report[url] = type(e).__name__
return report
links = check_links(["http://localhost:8000/", "http://localhost:8000/old-page", "http://localhost:9/"])
# {'http://localhost:8000/': 200, 'http://localhost:8000/old-page': 404, 'http://localhost:9/': 'ConnectionRefusedError'}15. httplib2 FAQs
Most httplib2 questions are about HTTPS, JSON and whether to choose it at all.
15.1. Does httplib2 support HTTPS?
Yes. It verifies certificates with the bundled CA list by default. The ca_certs argument points to a custom bundle, and disable_ssl_certificate_validation=True exists for testing but must not be used in production.
15.2. How do we read a JSON response?
The body is bytes, so we pass it to json.loads(content), which accepts bytes as well as text. The Python JSON tutorials cover parsing in detail.
15.3. Is httplib2 still maintained?
Yes. It receives regular releases on PyPI, but it is in maintenance mode with few new features. requests and httpx are the more common choices for new projects.
16. Conclusion
httplib2 sends every request through Http.request() and returns the response headers and the body as bytes. It handles GET, HEAD and POST, custom headers, Basic and Digest authentication, timeouts and, unlike most clients, a standards-based HTTP cache. For new projects without caching needs, requests or httpx are easier to work with.
The first version of this tutorial was written by Jan Bodnar, who runs zetcode.com.
17. References
Happy Learning !!
