2026 Google Scraping API Guide: Tools, Tips & Best Practices
Complete guide to Google Scraping APIs: official options, managed SERP APIs, DIY scrapers, steps to scale, costs, and compliance.
Feb 9, 2026
Simple beginner guide to sending, viewing, removing, and debugging HTTP headers with curl, with examples, quoting tips, and troubleshooting.
When you work with HTTP APIs or debug web requests, headers are how clients and servers exchange metadata — content type, auth tokens, caching rules, language preferences, and more. This guide is friendly for beginners: starting from quick command runs, learn the small set of key flags, solving shell-quoting headaches, and debugging common header problems.

What you will learn:
Send single and multiple headers with curl.
View response headers and inspect what your client actually sent.
Remove a default header curl would normally send.
Quoting and shell tips for Bash, PowerShell and cmd.exe.
Quick fixes for common beginner problems on Linux/macOS and Windows.
Helpful tips: auth headers, cookies, proxies, redirects, and privacy.
Make sure curl is installed and check the version:
curl --version
You don’t need a very new curl for the basics in this guide. If a command fails, update curl or use the file-based workarounds shown below.
These use public test endpoints that echo what you send so you can immediately see results.
# 1) Send a custom header and see it echoed back
curl -H "X-MyHeader: 123" https://httpbin.org/headers
# 2) Show response headers + body
curl -i https://httpbin.org/get
# 3) Inspect exactly what curl sent (request lines start with >)
curl -v -H "X-MyHeader: 123" https://httpbin.org/get
Tips:
1. Use https://httpbin.org or https://postman-echo.com for safe testing — they echo back headers and bodies so you can confirm behavior without hitting your own servers.
2. Need to test from another IP or region? Try the same command through a proxy:
curl -x http://proxy.example.com:8000 -H "X-MyHeader: 123" https://httpbin.org/get
Proxies are useful for geo-testing or avoiding local rate limits — but only use trusted proxies.
Headers are Name: value pairs included in requests and responses that tell the other side how to treat the request/response. Common examples:
Headers are case-insensitive. Some headers can appear multiple times in a response (e.g., multiple Set-Cookie headers).
curl will always send a Host header. It typically sends a User-Agent (e.g., curl/8.x.x) and sometimes an Accept. Passing -H "Name: value" for the same name overrides the default. To remove a default header entirely, pass it with an empty value:
curl -H "User-Agent:" https://httpbin.org/headers
Use these as your daily toolbox.
curl -H "X-MyHeader: 123" https://httpbin.org/headers
curl -H "Accept: application/json" -H "X-Trace-ID: abc123" https://httpbin.org/headers
echo '{"city":"Paris"}' > payload.json
curl -X POST -H "Content-Type: application/json" -d @payload.json https://httpbin.org/post
curl -i https://httpbin.org/get
curl -v https://httpbin.org/get
In -v output: lines beginning with > are request lines (what curl sent); lines beginning with < are response lines (what server sent).
Command:
curl -i https://httpbin.org/get
Example output (trimmed & annotated):
HTTP/1.1 200 OK # <- response status line
Content-Type: application/json # <- response header
Date: Tue, 10 Feb 2026 12:00:00 GMT
{ "args": {}, "headers": { ... }, "url": "https://httpbin.org/get" } # <- response body
-i prints response headers first, then the body.
Command:
curl -v -H "X-MyHeader: 123" https://httpbin.org/get
Key lines you’ll see:
> GET /get HTTP/1.1 # <- request line
> Host: httpbin.org # <- request header
> X-MyHeader: 123 # <- your custom header
< HTTP/1.1 200 OK # <- response status
< Content-Type: application/json # <- response header
Look for > lines to confirm the header was sent.
echo '{"name":"Alice"}' > payload.json
curl -X POST -H "Content-Type: application/json" -d @payload.json https://httpbin.org/post
Using @payload.json avoids shell-quoting problems across different shells.
curl -D headers.txt https://httpbin.org/get
# then inspect
cat headers.txt
Many APIs use Authorization: Bearer <token> or Basic auth. Examples:
Bearer token
curl -H "Authorization: Bearer YOUR_TOKEN" https://httpbin.org/bearer
Basic auth (curl helper)
curl -u username:password https://httpbin.org/basic-auth/username/password
Important: Don’t paste real tokens into public shells or shared logs. Use environment variables and files when possible:
curl -H "Authorization: Bearer $MY_TOKEN" https://httpbin.org/bearer
Cookies are sent via Cookie request header and received via Set-Cookie response headers.
Save cookies to a file
curl -c cookies.txt https://httpbin.org/cookies/set?name=value
Send cookies from a file
curl -b cookies.txt https://httpbin.org/cookies
Cookies and headers often interact — saving cookies with -c and reusing them with -b is common in testing login flows.
Quoting and escaping differ by shell. When in doubt, use the file fallback.
Bash / macOS (recommended) — single quotes are easiest:
curl -H 'Content-Type: application/json' -d '{"name":"Alice"}' https://...
PowerShell — single quotes are literal; double quotes allow interpolation. Example:
curl -H 'Content-Type: application/json' -d '{ "name": "Alice" }' https://...
If it fails, save the body to a file and use -d @file.
Windows cmd.exe — single quotes don’t behave as in Unix shells; use double quotes and prefer a payload file:
echo {"name":"Alice"} > payload.json
curl -H "Content-Type: application/json" -d @payload.json https://...
Best fallback: put the body in a file and use -d @payload.json — this avoids shell-escaping entirely and is the most robust approach.
When using -L (follow redirects), curl may remove sensitive headers like Authorization when redirecting to a different host — this is intentional to avoid leaking credentials.
If you need the header at the final host, reapply the header at the final URL or use --location-trusted (cautious: this will resend credentials to redirected hosts). Prefer handling auth at the final endpoint in scripts.
curl -v: quick check of what was sent (>) and what was received (<).
curl -D file: captures response headers for later inspection.
--trace / --trace-ascii (advanced): prints raw requests and responses, helpful for protocol-level debugging but may contain sensitive data — don’t store traces publicly.
Proxies are useful to reproduce region-specific responses, test geo-blocking, or run many requests from different IPs.
HTTP proxy
curl -x http://user:[email protected]:8000 -H "X-MyHeader: 123" https://httpbin.org/get
SOCKS5 proxy
curl --socks5-hostname 127.0.0.1:1080 -H "X-MyHeader: 123" https://httpbin.org/get
Security notes:
Proxies can read (and log) headers and bodies — never send secrets through untrusted proxies.
Use HTTPS when sending tokens through proxies so the proxy sees only the destination (SNI) but not plaintext body (except for some metadata).
Review your proxy provider’s privacy/logging policy before sending anything sensitive.
Using proxies to bypass rate limits or access-restricted content may violate a site’s terms of service — respect ToS and laws.
1. Header not sent?
Run curl -v and look for your header in the > lines.
2. Server ignores the header?
Confirm header name and format (Name: value) and check the API docs; use curl -i to inspect response headers or error messages.
3. JSON breaks on Windows?
Use -d @payload.json or adjust quoting for PowerShell vs cmd.exe.
4. Header disappears after redirect?
Run curl -v -L ... and inspect the >/< lines; reapply auth at final URL if needed.
5. Sensitive data in logs?
Never save -v, --trace, or debug outputs that contain tokens. Use -D for non-sensitive header capture when possible.
-H "Name: value" — add/override header
-i — include response headers + body
-I — HEAD request (headers only)
-D file — dump response headers to file
-v — verbose (shows > request and < response)
-L — follow redirects
-d @file — request body from file
-x http://proxy:port — use HTTP proxy
--socks5-hostname host:port — use SOCKS5 proxy
-c cookies.txt — save cookies
-b cookies.txt — send cookies from file
-H "User-Agent:" — send no User-Agent
--trace / --trace-ascii — raw trace (advanced; sensitive)
Q: Will -H override curl’s default headers?
A: Yes — passing the same header name replaces the default value.
Q: How can I see what curl actually sent?
A: Use -v — request lines start with >.
Q: Can I send many headers?
A: Yes — repeat -H for each header. If bodies get messy, use -d @file.
Q: How do I remove a header curl normally adds?
A: Pass it with an empty value, e.g. -H "User-Agent:".
Start with -H, -i, and -v to build confidence. Avoid shell quoting problems by storing request bodies in files and using -d @file. Use -D to capture response headers in scripts. Treat tokens and debug traces as sensitive data — never publish them. Proxies are powerful testing tools but carry privacy and ToS risks — use trusted providers and HTTPS.
Next >
Cancel anytime
No credit card required