Kracekumar

Posted on Oct 11, 2021 • Originally published at kracekumar.com on Oct 10, 2021

Generate HTTP Request

HTTPie is a command-line utility for making HTTP requests with more straightforward syntax(controversial, I agree). The interesting feature is --offline flag which prints HTTP raw request text. The client sends the HTTP request to the server, and the server responds to the request. It's an alternate to curl.

HTTP Syntax

HTTP Flow and syntrax from Wikipedia.

A client sends request messages to the server, which consist of

a request line, consisting of the case-sensitive request method, a space, the request target, another space, the protocol version, a carriage return, and a line feed (e.g. GET /images/logo.png HTTP/1.1)

zero or more request header fields, each consisting of the case-insensitive field name, a colon, optional leading whitespace, the field value, and optional trailing whitespace (e.g. Accept-Language: en), and ending with a carriage return and a line feed.

an empty line, consisting of a carriage return and a line feed;

an optional message body.

In the HTTP/1.1 protocol, all header fields except Host are optional.

A request line containing only the path name is accepted by servers to maintain compatibility with HTTP clients before the HTTP/1.0 specification in RFC 1945.

Throughout the post, I'll use --offline feature to understand how the HTTP request structure looks for educational purposes.

🉑 Accept only JSON response data 🉑

HTTPie uses : to separate header key and values in the terminal. Accept:application/json.
Accept header tells the server, the client accepts only specific MIME types, here JSON.

$http --offline httpbin.org/get Accept:application/json
GET /get HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: httpbin.org
User-Agent: HTTPie/2.5.0
# Sample request sent to the server
$ http httpbin.org/get Accept:application/json
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 310
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:21:40 GMT
Server: gunicorn/19.9.0

{
    "args": {},
    "headers": {
        "Accept": "application/json",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",
        "User-Agent": "HTTPie/2.5.0",
        "X-Amzn-Trace-Id": "Root=1-61629484-3b25a3631e2a89bf60f2600e"
    },
    "origin": "xxx.xxx.xxx.xxx",
    "url": "http://httpbin.org/get"
}

You can pass more than one value in the Accept header seperated by comma.
The response from the server contains Content-Type as JSON. The server can choose to ignore the Accept header.

📚 Request the Tamil language version of duckduckgo 📚

Accept-Language header instructs the web server to deliver particluar language version of the page.
Here Accept-Language is set to ta. ta is ISO 639-1 code.

$http --offline  https://duckduckgo.com Accept-Language:ta
GET / HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: ta
Connection: keep-alive
Host: duckduckgo.com
User-Agent: HTTPie/2.5.0

$http  https://duckduckgo.com Accept-Language:ta
HTTP/1.1 200 OK
Cache-Control: no-cache
Connection: keep-alive
Content-Encoding: gzip
Content-Security-Policy: default-src 'none' ; connect-src  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ https://duck.co ; manifest-src  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; media-src  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; script-src blob:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ 'unsafe-inline' 'unsafe-eval' ; font-src data:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; img-src data:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; style-src  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ 'unsafe-inline' ; object-src 'none' ; worker-src blob: ; child-src blob:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; frame-src blob:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; form-action  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ https://duck.co ; frame-ancestors 'self' ; base-uri 'self' ; block-all-mixed-content ;
Content-Type: text/html; charset=UTF-8
Date: Sat, 09 Oct 2021 21:44:33 GMT
ETag: W/"6161a338-16b8"
Expect-CT: max-age=0
Expires: Sat, 09 Oct 2021 21:44:32 GMT
Permissions-Policy: interest-cohort=()
Referrer-Policy: origin
Server: nginx
Strict-Transport-Security: max-age=31536000
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1;mode=block

<!DOCTYPE html>
...
<body id="pg-index" class="page-index body--home">
    <script type="text/javascript" src="/tl5.js"></script>
<script type="text/javascript" src="/lib/l123.js"></script>
<script type="text/javascript" src="/locale/ta_IN/duckduckgo50.js"></script>
<script type="text/javascript" src="/util/u588.js"></script>
<script type="text/javascript" src="/d3012.js"></script>
...
<noscript>
    <div class="tag-home">
        <div class="tag-home__wrapper">
            <div class="tag-home__item">
                Privacy, simplified&period;
                <span class="hide--screen-xs"><a href="/about" class="tag-home__link"> மேலும் கற்க</a></span>
            </div>
        </div>
    </div>
</noscript>
...
</html>

The header can carry more than one value separated by a comma.
When more than one value is present, an extra parameter q=0.5 represents the weightage among the values. Example: Accept-Language: ta,fr;q=0.50.
One of the link description is in Tamil, மேலும் கற்க.

🔑 Post Authorization token as part of login 🔑

Authorization header can be used to provide credentials to authenticate the user.
Example: Authorization:"Bearer my-dear-darkness".
Because of space between Bearer and the value quotes is mandatory.

$http --offline POST httpbin.org/auth Authorization:"Bearer my-dear-darkness"
POST /auth HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Authorization: Bearer my-dear-darkness
Connection: keep-alive
Content-Length: 0
Host: httpbin.org
User-Agent: HTTPie/2.5.0

🍪 Send Cookie as part of a request 🍪

The server can send HTTP Cookie as part of the response, and the client can send the Cookies in subsequent requests.
Cookie header carries the cookies previously sent by the user.
Semi-colon separates multiple cookie pairs.
Example: "name=not-unique;value=23"

$http --offline http://pie.dev/cookies Cookie:"name=not-unique;value=23"
GET /cookies HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Cookie: name=not-unique;value=23
Host: dev.pie
User-Agent: HTTPie/2.5.0
$http http://pie.dev/cookies Cookie:"name=not-unique;value=23"
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 69be1b984c743c0c-BLR
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:24:13 GMT
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=2Zt9wQcKQYStY5dPukoEXFwn7K6pKNaVTiKljZ5h9oL3mcQZj0khYq8Kmzo8PmQinPncStZeHASetQqHzCODe0wbrljPEIJxCWGdRWMbry9rWOG%2FBheDvJs7"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked
access-control-allow-credentials: true
access-control-allow-origin: *
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

{
    "cookies": {
        "name": "not-unique",
        "value": "23"
    }
}

✉️ Send JSON Request ✉️

HTTPie supports sending form-encoded values or JSON values. = sign indiciates JSON key-value pair.
= is useful for primitive values like number, string, null, boolean.
Example: lang-py

$http --offline PUT http://pie.dev/put lang=py version=3.10
PUT /put HTTP/1.1
Accept: application/json, */*;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 33
Content-Type: application/json
Host: pie.dev
User-Agent: HTTPie/2.5.0

{
    "lang": "py",
    "version": "3.10"
}
$HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 69be1e017fb53c12-BLR
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:25:52 GMT
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=GMW5PQHMwjxpVUeDG7uWo6M%2FiKLMzmQLd1e5BIG3AXljmuQgwCP9nzrFPdaidR2wL14eisfiViDhumDHpepgNB6yIrsVKXRybHa5tRqmH7lUDoQdRqCK1Ijg"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked
access-control-allow-credentials: true
access-control-allow-origin: *
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

{
    "args": {},
    "data": "{\"lang\": \"py\", \"version\": \"3.10\"}",
    "files": {},
    "form": {},
    "headers": {
        "Accept": "application/json, */*;q=0.5",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "49.207.222.139",
        "Cf-Ipcountry": "IN",
        "Cf-Ray": "69be1e017fb53c12-FRA",
        "Cf-Visitor": "{\"scheme\":\"http\"}",
        "Connection": "Keep-Alive",
        "Content-Length": "33",
        "Content-Type": "application/json",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/2.5.0"
    },
    "json": {
        "lang": "py",
        "version": "3.10"
    },
    "origin": "xxx.xxx.xxx.xxxx",
    "url": "http://pie.dev/put"
}

✉️ Send non-primitive JSON values ✉️

:= means the JSON value is non-primitve values like array/list and dictionary.
Single quote carries the value.
Example: os:='["GNU/Linux", "Mac OSX"]'.

$http --offline  PUT http://pie.dev/put lang=py version=3.10 os:='["GNU/Linux", "Mac OSX"]'
PUT /put HTTP/1.1
Accept: application/json, */*;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 65
Content-Type: application/json
Host: pie.dev
User-Agent: HTTPie/2.5.0

{
    "lang": "py",
    "os": [
        "GNU/Linux",
        "Mac OSX"
    ],
    "version": "3.10"
}
$http  PUT http://pie.dev/put lang=py version=3.10 os:='["GNU/Linux", "Mac OSX"]'
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 69be224ce97c3c0c-BLR
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:28:48 GMT
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=dgY0R%2FqomBbumKs0IVB7GjPR2chhHJ9wVKAzW33IOnvZ%2Fs4l2LYnvKKeXo6Xhd162AYKGzyIrK4pNrYdH8SEs1NvGmYfJqEDIPmfUOELqpC6HK9iP1zIENa0"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked
access-control-allow-credentials: true
access-control-allow-origin: *
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

{
    "args": {},
    "data": "{\"lang\": \"py\", \"version\": \"3.10\", \"os\": [\"GNU/Linux\", \"Mac OSX\"]}",
    "files": {},
    "form": {},
    "headers": {
        "Accept": "application/json, */*;q=0.5",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "49.207.222.139",
        "Cf-Ipcountry": "IN",
        "Cf-Ray": "69be224ce97c3c0c-FRA",
        "Cf-Visitor": "{\"scheme\":\"http\"}",
        "Connection": "Keep-Alive",
        "Content-Length": "65",
        "Content-Type": "application/json",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/2.5.0"
    },
    "json": {
        "lang": "py",
        "os": [
            "GNU/Linux",
            "Mac OSX"
        ],
        "version": "3.10"
    },
    "origin": "xx.xxx.xxx.xxx",
    "url": "http://pie.dev/put"
}

📤 Upload files 📤

-f flag indicates the data is form-encoded values.
@ symbol indicates the value is a file.
Example: cv@hello-world.txt. cv form field name.

$cat hello-world.txt
hello-world
$http --offline -f POST pie.dev/post name='Krace' cv@hello-world.txt
POST /post HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 277
Content-Type: multipart/form-data; boundary=183f70d3da41432d95bcd839e2cc20e2
Host: pie.dev
User-Agent: HTTPie/2.5.0

--183f70d3da41432d95bcd839e2cc20e2
Content-Disposition: form-data; name="name"

Krace
--183f70d3da41432d95bcd839e2cc20e2
Content-Disposition: form-data; name="cv"; filename="hello-world.txt"
Content-Type: text/plain

hello-world

--183f70d3da41432d95bcd839e2cc20e2--
$ http -f POST pie.dev/post name='Krace' cv@hello-world.txt
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 69be2564a8fc3c0c-BLR
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:30:54 GMT
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=yNaTSZJ7ouYH%2FSJZw6LIR5vNl6KNTFeDKF1u8V60abb3ClKLzdOj0zkchcIAWqTvZDbxXm5MnffDkCLdqviMQuAo7DqFA2GH%2Bm%2FiZ6sH90oOr0HyFGAuS4Gp"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked
access-control-allow-credentials: true
access-control-allow-origin: *
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

{
    "args": {},
    "data": "",
    "files": {
        "cv": "hello-world\n"
    },
    "form": {
        "name": "Krace"
    },
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "49.207.222.139",
        "Cf-Ipcountry": "IN",
        "Cf-Ray": "69be2564a8fc3c0c-FRA",
        "Cf-Visitor": "{\"scheme\":\"http\"}",
        "Connection": "Keep-Alive",
        "Content-Length": "277",
        "Content-Type": "multipart/form-data; boundary=252c6fcd1dcc40e09de54958660d672d",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/2.5.0"
    },
    "json": null,
    "origin": "xxx.xxx.xxx.xxxx",
    "url": "http://pie.dev/post"
}

Explore more in HTTPie website.

🐈 How to verify the the generated request works? 🐈

Netcat is a utility for sending and receiving data in network connection using TCP or UDP. Netcat take hostname, port, and body as arguments and sends it to the server and displays the response.

$http --offline  PUT http://httpbin.org/put lang=py version=3.10 os:='["GNU/Linux", "Mac OSX"]' | nc httpbin.org 80
HTTP/1.1 200 OK
Date: Sun, 10 Oct 2021 08:27:58 GMT
Content-Type: application/json
Content-Length: 631
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

{
  "args": {},
  "data": "{\"lang\": \"py\", \"version\": \"3.10\", \"os\": [\"GNU/Linux\", \"Mac OSX\"]}",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "application/json, */*;q=0.5",
    "Accept-Encoding": "gzip, deflate",
    "Content-Length": "65",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "HTTPie/2.5.0",
    "X-Amzn-Trace-Id": "Root=1-6162a40e-34b9a83f40868b4a73e8fa09"
  },
  "json": {
    "lang": "py",
    "os": [
      "GNU/Linux",
      "Mac OSX"
    ],
    "version": "3.10"
  },
  "origin": "xxx.xxx.xxx.xxx",
  "url": "http://httpbin.org/put"
}

Netcat doesn't support encrypted network connections.

❓How to understand what HTTP request the client sends to the server?
💡Use HTTPie --offline feature to print the HTTP request.

Details on how to send various HTTPie options in the blog post - https://t.co/mIEG2FpBkP #TIL #Python #CLI pic.twitter.com/kBvfc9QQAt
— kracekumar || கிரேஸ்குமார் (@kracetheking) October 10, 2021

References

HTTPie - https://httpie.io/
HTTPBin - http://httpbin.org
NetCat - https://en.wikipedia.org/wiki/Netcat
HTTP MDN Docs: https://developer.mozilla.org/en-US/docs/Web/HTTP
HTTP Header Image: https://upload.wikimedia.org/wikipedia/commons/thumb/5/5b/HTTP_logo.svg/2880px-HTTP_logo.svg.png

DEV Community

Generate HTTP Request

HTTP Syntax

🉑 Accept only JSON response data 🉑

📚 Request the Tamil language version of duckduckgo 📚

🔑 Post Authorization token as part of login 🔑

🍪 Send Cookie as part of a request 🍪

✉️ Send JSON Request ✉️

✉️ Send non-primitive JSON values ✉️

📤 Upload files 📤

🐈 How to verify the the generated request works? 🐈

References

Top comments (0)

Read next

The Future of AI: Transformations and Possibilities

What are your goals for week 45 of 2024?

[From Whizlab] AWS Cheatsheet

Setting Up Tailwind in React Native SDK 51 with NativeWind