Crawl any website
Start URLs, scoped link following, robots.txt and crawl-delay obeyed by default. Extracts metadata, headings, OpenGraph, JSON-LD, text and links, plus per-site CSS-selector fields and item lists.
Web data acquisition you can run, measure and trust.
Web Harvester is a self-hosted engine for collecting web data at scale: crawl any site, collect search results, rotate proxies from any provider, and detect blocks without bypassing them. Every claim links to its evidence.
$ git clone https://github.com/HenryMorganDibie/web-harvester
$ cd web-harvester && go test ./...
$ go run ./cmd/harvester -config configs/config.example.yaml -mode mock
[metrics] elapsed=2s success=5 failure=0 retried=0 dropped=0 completed=5 req/s=2.50
# full pipeline: queue, workers, proxy rotation, rate limiter, fetch, parse, sink
20.76M
requests in a 90-minute soak test
0
jobs dropped across that run
~3,800/s
sustained orchestration throughput
5 MB
final heap, no memory or goroutine leak
Orchestration-only figures: mock fetcher, simulated latency, zero network. They show the engine is not the bottleneck; they are not live-scraping throughput.
Capabilities
One acquisition stack, two front ends.
A general-purpose crawler and a search-results pipeline share the same workers, proxy pool, rate limiter, block detection and storage.
Search results pipeline
Structured organic results, featured snippets, "people also ask" and AI Overviews, direct or through a SERP API provider, with drift detection when page layouts stop matching.
Health-tracked proxy pool
Round-robin, random or success-weighted rotation, escalating cooldowns, live reload that keeps health history, and any HTTP(S) proxy provider.
Adaptive rate limiting
Per-proxy and per-host token buckets that slow down on pressure and honour Retry-After exactly.
Blocks detected, never bypassed
CAPTCHAs, challenges, consent walls and rate limits are classified and reported. The affected proxy is cooled down; access controls are not bypassed.
Plain HTTP or headless Chromium, with failover
Fetch with Go's HTTP client, render with Playwright, or run hybrid: start fast over HTTP and fail over to a real browser only where a page needs it.
Production plumbing
Redis Streams queue, PostgreSQL or JSON Lines, Prometheus and Grafana, Docker Compose and systemd.
Architecture
Every stage is an interface.
Swap the queue, fetcher or sink without touching the rest. The job API only ever produces work, so acquisition logic cannot leak into submission.
QueueIn-memory or Redis Streams with consumer groups
WorkersConcurrent pool, retries with backoff and jitter
Proxy poolHealth scores, cooldowns, three strategies
Rate limiterToken buckets per proxy and per host
FetcherHTTP, Chromium, hybrid or SERP provider
ParserStructured extraction with drift detection
SinkPostgreSQL or JSON Lines
Classifier on every fetch resultPrometheus metrics and Grafana dashboardJob API and cron scheduling as pure producers
Proxy provider measurement
Know how a proxy provider behaves before you depend on it.
The repository includes a small, auditable runner that measures a residential or mobile provider the way a scraper experiences it, and a protocol for turning that into a defensible report.
- Confidence intervals, not single numbers. Small samples are flagged rather than over-read.
- Usable results, not just HTTP 200. A challenge or consent page counts as a failure.
- Billed bytes reconciled against the provider's own usage figures.
- Sensitive data stays private. Exit IPs are hashed and credentials are redacted; only selected aggregate findings are published.
- Does rotating really rotate?
- A fresh tunnel per request, because reused tunnels keep their exit IP; reuse is recorded.
- Do sticky sessions hold?
- Exit IPs per session, and a cap on requests per session.
- Is state targeting accurate?
- Location read from the measured request itself, across all 50 states plus D.C.
- Why did requests fail?
- Provider error codes captured, separated from timeouts and target responses.
- Do retries recover?
- First-attempt against eventual success, recovery time, bytes spent on failures.
- What does it cost?
- Every byte exchanged, compared with what the provider bills.
US proxy benchmark results · 2026
Observed performance, with the sample sizes to back it up.
A two-phase evaluation of residential and mobile proxy pools using Web Harvester. Results are specific to these test conditions.
98.2%
Residential exits geolocated to the requested state
1,502 of 1,530 requests
600/600
Usable protected-page responses with sticky sessions renewed every 15 requests
97.9%
Usable responses with rotating sessions
1,175 of 1,200 requests
100–230 ms
Faster median latency for renewed sticky sessions
Across all six comparisons
What we learned: Renewed sticky sessions were faster in every comparison. A single long-lived sticky session performed worse on the tested protected page, with 16–49 usable responses per 100-request run before HTTP 4xx responses. Mobile state geolocation is less conclusive because carrier gateways can appear in a different state.
Evidence
Measured, not claimed.
Reproducible with one command, with results linked to their supporting evidence.
Search integration evaluation · 2026
50 queries. 49 returned mapped organic results.
A 50-query evaluation split evenly between easy and difficult searches, including a complex multi-leg flight query.
98%
Queries returned at least one mapped organic result
49 of 50 queries
1.39 s
Median end-to-end latency
4.37 s
95th-percentile latency
Scope: This evaluation measures result mapping and latency, not semantic relevance, answer correctness, direct Google scraping, or proxy performance. One difficult query returned no mapped organic results.
Third-party integration pilot · October 2026
External search connected to a live deal dataset.
A bounded integration run connected an external search provider, fetched source pages through the configured proxy, compared discovery results against Deal Intelligence's visible canonical dataset, and generated a review artifact. The workflow and its regression checks completed successfully.
8/8
External search queries succeeded
74
Unique result URLs discovered
65
URLs with recent publisher dates
22/30
Source pages fetched successfully
4,506
Existing deal records loaded for comparison
Integration verified: the third-party search connection, source-page acquisition path, live dataset comparison and review-artifact generation all completed successfully. The run was review-only and made no production database changes. This is a bounded integration validation. Comprehensive market coverage and analyst verification are separate evaluation steps.
View the successful workflow run → · Full integration results →
$ go run ./cmd/loadtest -duration 90m -sample-interval 2m -concurrency 200
actual elapsed: 1h30m0s (requested: 1h30m0s)
total success=20756754 failure=186 dropped=0 retried=186
latency: min=20ms p50=50ms p95=77ms p99=79ms max=200ms
average throughput: 3844 req/s
final memory: alloc=5MB sys=61MB goroutines=1
Work with me
Need data you can rely on?
I build, measure and harden web data pipelines. Engagements are scoped up front, with a fixed deliverable and a written report.
Proxy provider benchmarks
An independent, reproducible evaluation of a residential or mobile proxy network.
- Location accuracy by state or country
- Sticky against rotating sessions
- Failure, retry and cost analysis
Data pipelines and integrations
Custom crawlers and extraction on this engine, wired into your stack.
- Site-specific extraction rules
- Proxy and SERP provider integrations
- PostgreSQL, Redis and API delivery
Production hardening
Make an existing collection system reliable and observable under real load.
- Resilience, retries and rate limiting
- Metrics, dashboards and alerting
- Deployment and runbooks