Step-by-step guide to deploy SerpBear with GoProxy for accurate, scalable, and ban-resistant Google rank tracking.
Unreliable ranking data can hurt an SEO campaign. You can tweak content, build links, and optimize UX — but if your rank tracker is blocked, geo-wrong, or noisy, decisions become guesses. SerpBear gives you a self-hosted, unlimited keyword tracker — and when paired with a professional proxy pool like from GoProxy, it turns into a production-grade instrument for accurate, geo-correct SERP monitoring.
Deploy SerpBear with Docker, point its scraper at a GoProxy pool (API or local proxy list), follow conservative Starter defaults (concurrency, jitter, sticky sessions), validate with quick curl test, then scale safely using the staged rules below. This guide combines SerpBear’s official requirements with operational defaults, monitoring thresholds, and troubleshooting.
Who this guide is for
Beginners / Solo site owners: want a clear Starter path and a one-page checklist.
Intermediate SEOs / DevOps: want reliable defaults, tuning, and automation examples.
Agencies / Power users: want scale patterns, multi-tenant mapping, and proxy health practices.
If you’re new, follow the Starter section first. Intermediates and agencies should read the Operational and Scaling sections after that.
SerpBear determines ranking by fetching search engine result pages. Search engines detect high-volume automated traffic and will respond with CAPTCHAs, throttles, or IP blocks. Proxies solve the main problems:
Result: fewer failed fetches, more accurate local data, and scalable monitoring.
SerpBear requires a third-party scraper or proxy IPs to fetch Google results. Always follow local laws and search engines’ terms. Use polite scraping (rate limits, respectful scheduling). For commercial use, consider official APIs or legal review.
| Stage | Keywords | Pool size | Concurrency (per domain / global) | Jitter | Sticky session |
| Starter | ≤ 50 | 5–10 IPs | 1 / 1–2 | 5–12 s | 60 s |
| Operational | 50–500 | ~30% proxy-to-keyword or 1 proxy / 50–100 daily checks | 2 / 4–8 | 3–8 s | 60–120 s |
| Production | >500 | scale to daily checks (hundreds of IPs) | 2–4 / horizontally scaled | 2–6 s | 60–120 s |
Rule: start at Starter, validate metrics, then move to Operational → Production only when objective triggers are met.
Use these conservative defaults so your first runs don’t trigger blocks:
Sticky session (per IP): 60 s (Starter) → 60–120 s (Operational).
Concurrency per domain: 1 (Starter) → 2 (Operational).
Global concurrency: 1–2 (Starter) → 4–8 (Operational).
Randomized delay (jitter): 5–12 s (Starter) → 3–8 s (Operational).
Rotate IP after: 1–2 requests or 60 s (Starter); 1–3 requests or 60–120 s (Operational).
Proxy sizing: SerpBear docs baseline ≈ 30% proxy-to-keyword (e.g., 100 keywords → at least ~30 IPs). Operational rule ≈ 1 proxy / 50–100 daily checks.
Proxy type: datacenter proxies are acceptable (per docs); use residential selectively for sensitive/local checks.
These low-effort habits reduce detection risk:
Rotate a small set of User-Agents at session start (don’t change mid-session).
Send full headers: Accept, Accept-Language, Referer, Connection.
For mobile checks, use realistic mobile UA + viewport pairing.
Use randomized jitter; avoid regimented intervals.
Keep per-IP bursts small.
Log requests and proxy metadata for debugging
This is the friendly first-run path for beginners.
Folder
/SerpBear/
├─ docker-compose.yml
├─ .env
├─ proxies.txt # optional
└─ data/
docker-compose.yml (example)
version: '3.8'
services:
SerpBear:
image: towfiqi/SerpBear:latest
container_name: SerpBear
restart: unless-stopped
ports:
- "3000:3000"
volumes:
- ./data:/app/.SerpBear
- ./proxies.txt:/app/proxies.txt:ro # optional: mount if using local proxy list
env_file: .env
.env (edit)
NEXT_PUBLIC_APP_URL=http://localhost:3000
USER=admin
PASSWORD=ChangeMeStrong!
SECRET=REPLACE_WITH_SECURE_STRING
SCRAPER_PROVIDER=generic_scraper
GOPROXY_API_KEY=your_goproxy_api_key_here
GOPROXY_POOL=global
SMTP_HOST=
SMTP_PORT=
SMTP_USER=
SMTP_PASS=
Generate a secure SECRET
# OpenSSL
openssl rand -base64 48
# Or Python
python -c "import secrets;print(secrets.token_urlsafe(48))"
Note: Do not commit .env to source control. For production, use Docker secrets or a cloud provider secret manager.
Optional proxies.txt sample
http://user:[email protected]:8000
http://user:[email protected]:8000
Start
docker-compose up -d
Open http://localhost:3000, log in with USER/PASSWORD, add one domain and 5–10 keywords (mix countries/devices), set schedule to manual, and click Fetch Now to test.
Container path note: some images expect /app/proxies.txt or /app/.SerpBear/proxies.txt. If mounting fails, check:
docker exec -it SerpBear ls -la /app /app/.SerpBear
1. Proxy IP / geo
curl -x http://user:pass@ip:port https://ipinfo.io/json
Confirm the country matches your intended pool.
2. Single fetch
In SerpBear UI: Domains → Your domain → Keywords → select one → Fetch Now.
Check Activity/Logs for Proxy Success and Fetch OK.
3. Logs
docker logs -f SerpBear | grep -i proxy
docker logs --tail 200 SerpBear
Look for "Proxy Success: IP rotated to [IP]", "ERROR: Proxy auth failed", CAPTCHA, or HTTP 429.
Move Starter → Operational when:
Move Operational → Production when Operational metrics stable for 14–30 days and cost per successful fetch fits your ROI. If success drops ≥ 5% in 24h or CAPTCHA spikes, rollback to safer settings.
Keep proxies.txt optional — prefer provider API keys in SerpBear settings when available.
To scale throughput, run multiple scraper instances + a job queue distributing keywords.
Group keywords by country and priority — daily for high priority, weekly for long tail.
Scraper payload example (pseudo JSON) if scraper accepts proxy param:
{
"api_key":"YOUR_SCRAPER_KEY",
"url":"https://www.google.com/search?q=your_keyword",
"proxy":"http://user:[email protected]:8000"
}
Prefer GoProxy pool endpoints for auto-rotation and geo selection when available.
A/B test (7–14 days)
Two matched keyword sets (A = proxied, B = unproxied).
Track daily: fetch success rate, average position variance (mean(|pos_today − pos_baseline|)), #CAPTCHAs.
Compute:
Decision: scale proxies if proxied group shows materially higher success and lower variance justifying the cost.
Simple spreadsheet formulas
avg_variance = AVERAGE(ABS(pos_today - pos_baseline))
success_rate = successful_fetches / total_attempts
Webhook alerts: notify when position change > N.
n8n flow outline:
1. HTTP Request: GET SerpBear API /keywords?changed=true
2. Function/Filter: aggregate large changes
3. CSV: format report
4. Email/Slack: send to stakeholders
Archive: export daily CSVs to cloud storage for retention.
Key metrics
Fetch success rate target: >95%.
CAPTCHA rate: ≈0%.
Median fetch latency: <500 ms.
Alerts
Fetch success < 95% for 24h → alert.
CAPTCHA incidence > 1% in last 100 fetches → alert.
Median latency > 500 ms for a proxy → decommission/rotate it.
Cost tactics
Tier proxies: residential for top 10–20% keywords, datacenter for bulk/low priority.
Batch low-priority keywords across days.
Monitor provider meters daily.
Horizontal scraping: multiple scraper nodes, job queue, and shared proxy pool.
Tenant isolation: map clients to separate pools and tags; separate DB schemas or strict tagging.
Proxy health probes: synthetic checks for latency, success, CAPTCHAs; auto-retire bad IPs.
Failover: if client failure rate > 10% → switch pool and alert.
Security/compliance: encrypt logs, rotate credentials, minimize admin surface, use secrets manager.
Proxy auth / timeout
curl -x http://user:pass@ip:port https://httpbin.org/ip
Wrong geo
curl -x http://user:pass@ip:port https://ipinfo.io/json
Frequent CAPTCHAs: increase rotation, use residential, lower concurrency, increase jitter.
Encoding problems: ensure UTF-8; use encodeURIComponent for query encoding in the scraper.
Logs:
docker logs -f SerpBear | grep -i proxy
docker logs --tail 200 SerpBear
Pairing SerpBear with GoProxy gives you geo-accurate, resilient rank data without heavy SaaS fees — provided you adopt conservative concurrency, sticky sessions, and pragmatic monitoring. Sign up and get a trial today!
Next >
Cancel anytime No credit card required