Mastering curl get: From Basics to Pro

cURL is versatile and powerful in web development, API testing, and data retrieval. It performs an HTTP GET by default — use it to fetch endpoints, save responses, add headers/auth, follow redirects, resume downloads, automate with retries/backoff, and diagnose network or TLS issues. This guide walks you from one-line commands to production-ready scripts with examples and troubleshooting tips.

Who this guide is for

• Absolute beginners who want to fetch or test endpoints from the terminal.
• Developers, QA engineers and SREs reproducing or debugging API calls.
• Engineers building light automation or safe download scripts.

If you only need one command, go to One-line Quick Start. If you plan to script or automate, follow Beginner → Intermediate → Advanced.

One-line Quick Start (Run These Now)

Simple GET (prints body):

curl https://api.example.com/endpoint

Expected: response body (HTML or JSON) printed to terminal.

Save to file:

curl -o response.json https://api.example.com/endpoint

Expected: file response.json created with response body.

Show headers + body:

curl -i https://api.example.com/endpoint

Expected: HTTP response headers appear, then the body.

What are curl & HTTP GET, and Why Use curl for GET?

What is curl?

curl is a compact, cross-platform command-line tool (and library) for transferring data with URLs. It supports many protocols (HTTP/HTTPS, FTP, etc.), advanced options (headers, cookies, TLS settings, HTTP/2/3), and is designed to run interactively or in scripts/CI.

What is an HTTP GET?

An HTTP GET requests a representation of a resource (web page, file, or JSON). Parameters are typically placed in the URL query string. GET requests are intended to retrieve data without changing server state.

Why use curl for GET requests?

Instant, reproducible testing of endpoints.

Scriptable for automation, monitoring, and CI tasks.

Low dependency and available in minimal environments.

Full debugging visibility: headers, TLS handshake, redirects (-v, -i, --trace).

Fine control: headers, cookies, partial download, retry/backoff, and bandwidth throttling.

When not to use curl:

Avoid using curl for large-scale scraping or high-concurrency crawling — prefer official bulk APIs or dedicated data platforms and always respect site policies.

Installing curl: Quick Setup & Verification

Before we jump into commands, ensure cURL is installed.

Quick check (run this first)

curl --version

This prints the curl version and compiled features (e.g., HTTP/2 or HTTP/3, SSL backend). If it prints a version, you’re ready.

Install / update (common commands)

Debian/Ubuntu:

sudo apt update && sudo apt install curl

Red Hat / Fedora:

sudo dnf install curl

Arch family:

sudo pacman -S curl

macOS: often preinstalled; verify with curl --version. Use your system’s package manager to update if needed.

Windows: recent system versions include a curl binary. In PowerShell, call curl.exe if behavior differs.

Quick troubleshooting

command not found: ensure installation completed and PATH includes the binary; reopen your terminal.

Older curl missing flags: update via your package manager.

Beginner: Must-know Commands

Default GET

curl https://example.com/path

Expected: raw response printed to terminal.

Save output

curl -o file.txt https://example.com/path

Expected: file.txt contains the response.

Use -O to save with the remote filename when appropriate.

Inspect headers

Headers + body:

curl -i https://example.com/path

Headers only (HEAD request):

curl -I https://example.com/path

Query parameters

Append manually:

curl 'https://api.example.com/search?q=hello%20world&limit=10'

Safer (automatic encoding) — preferred in scripts:

curl -G --data-urlencode "q=hello world" -d "limit=10" https://api.example.com/search

Important: without -G, -d will send a POST. With -G, -d becomes URL query parameters for GET.

Add custom headers

curl -H "Accept: application/json" -H "User-Agent: CLI-Client/1.0" https://api.example.com/endpoint

What to check: verify HTTP status and inspect response structure (JSON keys or HTML content).

Intermediate: Common Tasks

Authentication

Basic auth

curl -u 'username:password' https://api.example.com/secure

Bearer token

curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com/user

Tip: store tokens in environment variables; do not hardcode them.

Cookies and sessions

Save cookies after login(cookie jar):

curl -c cookies.txt -d "username=myuser&password=$PASS" https://site.example/login

Reuse saved cookies:

curl -b cookies.txt https://site.example/dashboard

Login + follow redirects while saving cookies:

curl -s -c cookies.txt -d "username=myuser&password=$PASS" https://site.example/login

curl -s -b cookies.txt -L https://site.example/dashboard

Follow redirects & check final URL

curl -L https://short.example/abc

curl -s -o /dev/null -w '%{url_effective}\n' -L https://short.example/abc

For details, please check our curl follow redirects Guide.

Resume & partial downloads

Resume a download:

curl -C - -o bigfile.zip https://cdn.example.com/bigfile.zip

Download a byte range:

curl -r 0-999 -o head.part https://cdn.example.com/video.mp4

Advanced: Automation, Reliability & Diagnostics

Verbose & tracing

Quick verbose:

curl -v https://api.example.com/

Full trace (ASCII) to file:

curl --trace-ascii trace.txt https://api.example.com/

Retries & delays

Built-in retry options:

curl --retry 5 --retry-delay 2 --retry-connrefused -O https://api.example.com/resource

Use scripted exponential backoff for complex needs (see Recipes).

Fail on HTTP errors (useful in scripts)

curl -f -s -S -o response.json https://api.example.com/data || echo "Request failed"

Notes: -f exits non-zero on HTTP 4xx/5xx; -s silences progress; -S shows errors when -s is used.

Capture HTTP status

status=$(curl -s -w '%{http_code}' -o response.json https://api.example.com/data)

echo "HTTP status: $status"

Save headers and body separately

curl -s -D headers.txt -o body.json https://api.example.com/data

Check: headers.txt contains response headers; body.json contains the payload.

Parallel fetching (use carefully)

cat urls.txt | xargs -n1 -P6 curl -O

Warning: respect rate limits and server load — prefer bulk APs when available.

Bandwidth throttling

curl --limit-rate 100K -o partial.json https://api.example.com/large

HTTP/2 & HTTP/3

HTTP/2:

curl -I --http2 https://api.example.com/

HTTP/3 (if supported):

curl --http3 https://api.example.com/

Note: not all builds support newer protocols; check curl --version.

Troubleshooting

Connection refused / timeout

curl -v --connect-timeout 10 https://api.example.com/

SSL certificate errors

For testing only (unsafe): -k disables verification.

Safer: provide CA bundle:

curl --cacert /path/to/ca.pem https://secure.example.com/

404 Not Found

Verify path & query string. Test with:

curl -I https://api.example.com/resource

403 Forbidden

Check credentials and required headers (Authorization, User-Agent).

429 Too Many Requests

Inspect Retry-After:

curl -sI https://api.example.com/endpoint | grep -i '^Retry-After:'

Add delays/backoff and reduce concurrency.

Redirect loops

Use -v to see the Location chain and identify the loop.

Practical Scripts & Recipes

Recipe 1. Fetch JSON and check status

status=$(curl -s -w '%{http_code}' -o response.json \

  -H "Accept: application/json" -H "Authorization: Bearer $API_TOKEN" \

  "https://api.example.com/user")

if [ "$status" -ne 200 ]; then

  echo "Request failed with status $status"

  exit 1

fi

# response.json contains the payload

Recipe 2. Paginated fetching (cursor style)

cursor=""

while :; do

  url="https://api.example.com/items?limit=100${cursor:+&cursor=$cursor}"

  result=$(curl -s -H "Authorization: Bearer $API_TOKEN" "$url")

  echo "$result" > "page_${cursor:-start}.json"

  next=$(echo "$result" | python -c "import sys,json; j=json.load(sys.stdin); print(j.get('next',''))")

  [ -z "$next" ] && break

  cursor=$next

  sleep 1  # polite delay

done

Recipe 3. Safe download with retries & resume

attempt=0

max=5

delay=1

url="https://cdn.example.com/dataset.zip"

outfile="dataset.zip"

until curl --retry 3 --retry-delay 5 -C - -o "$outfile" "$url"; do

  attempt=$((attempt+1))

  if [ $attempt -ge $max ]; then

    echo "Failed after $attempt attempts"

    exit 1

  fi

  echo "Retrying in $delay seconds..."

  sleep $delay

  delay=$((delay*2))

done

Security, Ethics & Best Practices

Do not hardcode secrets. Use environment variables or secret managers.

Respect site policies and robots.txt. For heavy data needs, request official API access.

Throttle and back off. Use polite delays, --limit-rate, and exponential backoff.

Verify TLS in production. Do not use -k for production; use --cacert if needed.

Sanitize logs. Avoid logging secrets or tokens.

Most Used Flags List

-i    include response headers

-I    headers only (HEAD)

-H    add header (repeatable)

-G    convert -d into query params (GET)

-d    data (with -G becomes query)

-L    follow redirects

-o    save to file (name)

-O    save using remote name

-u    basic auth user:pass

-b    cookie string or file

-c    cookie jar to save cookies

-v    verbose

--trace-ascii FILE    detailed trace

--limit-rate RATE     throttle

--retry N  --retry-delay S  --retry-connrefused

-r    byte range

-f    fail on HTTP error (4xx/5xx)

-w '%{http_code}'     write status code

--connect-timeout S   limit connect wait time (seconds)

--data-urlencode      url-encode data for GET query params

FAQs

Q: How do I check only the HTTP status code?

A: curl -s -o /dev/null -w '%{http_code}\n' https://api.example.com/

Q: How to resume a download?

A: curl -C - -o filename URL

Q: How to include a bearer token?

A: curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com/endpoint

Q: How to avoid exposing API keys?

A: Store them in environment variables and reference $API_TOKEN in scripts.

Final Checklist Before Running Scripts

URL correct and reachable

Secrets stored securely (env vars / secret manager)

Query parameters encoded or sent with -G/--data-urlencode

Required headers included(Accept/Authorization)

Retry/backoff and rate control present for loops 

Verbose logging enabled during development only

Final Thoughts

curl is a deceptively small tool that unlocks a huge range of practical workflows — from simple one-off checks to robust, production-ready fetch and retry scripts. Start with the Quick Start examples to build confidence, then layer in authentication, cookie handling, and retries as your use case demands.

Always prioritize safety: keep secrets out of code, respect server load and site policies, and test TLS and protocol compatibility on staging systems before deploying scripts in production.