Python Tutorial (42) - requests module

Time: Column:Python views:206

The Python requests module is a commonly used HTTP library that simplifies sending HTTP requests to websites and retrieving response results.

Compared to the urllib module, requests is more concise and user-friendly.

To send HTTP requests using requests, you must first import the module:

import requests

After importing, you can send HTTP requests using methods provided by requests to a specified URL. For example:

Example:

# Import the requests package
import requests

# Send a request
x = requests.get('https://www.pmeve.com/')

# Return the web content
print(x.text)

After calling a request using requests, a response object is returned, which contains detailed response information such as status code, headers, and content:

print(response.status_code)  # Get the response status code
print(response.headers)  # Get the response headers
print(response.content)  # Get the response content

More response information is detailed below:

Attribute or MethodDescription
apparent_encodingThe encoding of the response
close()Closes the connection to the server
contentReturns the response content as bytes
cookiesReturns a CookieJar object containing cookies
elapsedReturns a timedelta object showing the time between sending the request and receiving the response
encodingThe encoding used to decode r.text
headersReturns the response headers in a dictionary format
historyReturns a list of response objects for each redirection (URL history)
is_permanent_redirectReturns True if the response is a permanent redirect, otherwise False
is_redirectReturns True if the response was redirected, otherwise False
iter_content()Iterates over the response content
iter_lines()Iterates over the lines in the response
json()Returns the JSON object from the response (only works if the response is in JSON format, otherwise raises an error)
linksReturns the parsed link headers
nextReturns the next request object in the redirection chain
okChecks the status_code value and returns True if less than 400, otherwise False
raise_for_status()Returns an HTTPError object if an error occurred
reasonDescription of the HTTP status (e.g., "Not Found" or "OK")
requestReturns the request object that generated this response
status_codeReturns the HTTP status code (e.g., 404 for "Not Found" or 200 for "OK")
textReturns the content of the response as a Unicode string
urlReturns the URL of the response

Example:

# Import the requests package
import requests

# Send a request
x = requests.get('https://www.pmeve.com/')

# Return the HTTP status code
print(x.status_code)

# Return the status description
print(x.reason)

# Return the encoding
print(x.apparent_encoding)

Output:

200
OK
utf-8

Requesting JSON Data

To request a JSON file and retrieve its content:

Example:

# Import the requests package
import requests

# Send a request
x = requests.get('https://www.pmeve.com/try/ajax/json_demo.json')

# Return the JSON data
print(x.json())

Output:

{
  "name": "Website",
  "num": 3,
  "sites": [
    {
      "name": "Google",
      "info": ["Android", "Google Search", "Google Translate"]
    },
    {
      "name": "PMeve",
      "info": ["PMeve Tutorials", "PMeve Tools", "PMeve WeChat"]
    },
    {
      "name": "Taobao",
      "info": ["Taobao", "Online Shopping"]
    }
  ]
}

Requests Methods

The requests methods are shown in the following table:

MethodDescription
delete(url, args)Sends a DELETE request to the specified URL
get(url, params, args)Sends a GET request to the specified URL
head(url, args)Sends a HEAD request to the specified URL
patch(url, data, args)Sends a PATCH request to the specified URL
post(url, data, json, args)Sends a POST request to the specified URL
put(url, data, args)Sends a PUT request to the specified URL
request(method, url, args)Sends a request using the specified method to the specified URL

Example Using requests.request():

# Import the requests package
import requests

# Send a request
x = requests.request('get', 'https://www.pmeve.com/')

# Return the HTTP status code
print(x.status_code)

Output:

200

Setting Request Headers

To set request headers:

Example:

# Import the requests package
import requests

# Define parameters
kw = {'s': 'python tutorial'}

# Set request headers
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"
}

# Use params to pass a dictionary or string query parameters
response = requests.get("https://www.pmeve.com/", params=kw, headers=headers)

# Check the response status code
print(response.status_code)

# Check the response encoding
print(response.encoding)

# Check the complete URL
print(response.url)

# Check the response content (Unicode format)
print(response.text)

Output:

200
UTF-8
https://www.pmeve.com/?s=python+%E6%95%99%E7%A8%8B
... other content...

POST Request Example

The post() method sends a POST request to a specified URL, typically formatted as follows:

requests.post(url, data={key: value}, json={key: value}, args)

The url is the target URL.

  • The data parameter is a dictionary, list of tuples, bytes, or file objects to send to the specified URL.

  • The json parameter is a JSON object to send to the specified URL.

  • The args include additional parameters such as cookies, headers, or verify options.

Example:

requests.post(url, data={key: value}, json={key: value}, args)

Output:

<p style='color:red;'>This content was requested using the POST method.</p><p style='color:red;'>Request time: 2024-05-26 17:30:47</p>

POST Request with Parameters

Example:

# Import the requests package
import requests

# Form parameters
myobj = {'fname': 'PMeve', 'lname': 'Boy'}

# Send a POST request
x = requests.post('https://www.pmeve.com/try/ajax/demo_post2.php', data=myobj)

# Return the web content
print(x.text)

Output:

<p style='color:red;'>Hello, PMeve Boy, how was your day?</p>

Additional Request Parameters

You can include additional parameters such as headers, query parameters, and request bodies in your requests:

headers = {'User-Agent': 'Mozilla/5.0'}  # Set request headers
params = {'key1': 'value1', 'key2': 'value2'}  # Set query parameters
data = {'username': 'example', 'password': '123456'}  # Set request body

response = requests.post('https://www.pmeve.com', headers=headers, params=params, data=data)

This code sends a POST request with custom headers, query parameters, and request body.

In addition to basic GET and POST requests, requests supports other HTTP methods such as PUT, DELETE, HEAD, and OPTIONS.