The Problem with Polling Large Batch Crawls
Polling — repeatedly calling GET /api/v1/website/jobs/:jobId on a timer — seems like the obvious way to check when your job is done. For a 50-URL batch that takes 90 seconds, it's fine. For a 10,000-URL enrichment job that takes 20 minutes, it becomes a serious problem.
Wasted Compute
A 15-second polling loop over a 20-minute job makes ~80 HTTP requests that all return "status": "scraping". Every one of those requests burns bandwidth and execution time for zero value.
Added Latency
Your polling loop fires every N seconds. In the worst case you receive the "completed" signal N seconds after the job finishes. With webhooks, your endpoint receives the callback shortly after job completion with no polling overhead.
Fragile Architecture
Polling loops require a persistent process. If your server restarts or your Lambda times out during a 20-minute job, you lose the completion event entirely. Webhooks are fire-and-receive — no persistent connection needed.
Webhooks Are Even More Critical at Scale
The Website Scraper API supports up to 10,000 URLs per job. A large enrichment run at full capacity can take 15–30 minutes depending on domain response times. Webhooks are the only sane architecture for jobs at this scale.
Registering Your Webhook Endpoint
Before submitting any jobs, register up to 5 webhook endpoints in the Bolt Scraper dashboard or via the API. Each endpoint has a name, a URL, an optional shared secret for signature verification, and platform/event filters.
POST https://boltscraper.com/api/v1/webhooks Authorization: Bearer bs_<YOUR_API_KEY> Content-Type: application/json { "name": "My n8n Pipeline", "webhook_url": "https://n8n.mycompany.com/webhook/bolt-website", "secret": "my-shared-secret-token-32chars", "send_auto": true, "platform": "website", "events": "all" }
Once registered, every completed website scrape job automatically dispatches a POST request to your endpoint. No need to pass a webhook URL in each job submission — the registered endpoint is always on.
The website_job.completed Webhook Payload
When a job finishes, the webhook delivers a structured JSON payload to your endpoint. It contains job metrics, a signed CSV download URL, and the first 100 results inline — so your automation can act on the data immediately without an additional API call.
{
"event": "website_job.completed",
"platform": "website",
"job_id": "job_a3f8c2...",
"job_name": "Q4 Roofing Enrichment",
"status": "COMPLETED",
// Job metrics
"total_urls": 2847,
"processed_count": 2847,
"success_count": 2601,
"failed_count": 246,
"completed_at": "2026-09-19T12:04:33.000Z",
// Signed CSV download URL (valid for 24h)
"download_csv_url": "https://boltscraper.com/api/v1/website/jobs/job_a3f8/download?token=abc123...",
// First 100 result rows inline (is_truncated: true if more exist)
"results_in_payload": 100,
"is_truncated": true,
"results": [
{
"url": "https://example-co.com",
"email_1": "[email protected]",
"phone_1": "+15550100192",
"linkedin_url": "https://linkedin.com/company/example-co",
"instagram_url": "",
"status": "success"
}
// ... 99 more rows
]
} The download_csv_url is a pre-signed, token-authenticated URL that works directly from n8n's HTTP Request node or Make.com's HTTP module — no additional authorization header required.
Webhook Request Headers
Every webhook delivery includes several identifying headers that allow your endpoint to quickly classify the incoming request:
| Header | Example Value | Purpose |
|---|---|---|
X-Bolt-Event | website_job.completed | Event type — route to correct handler |
X-Bolt-Platform | website | Disambiguate from Maps scraper events |
X-Bolt-Webhook-Id | wh_a3f8c2... | Which registered endpoint triggered this delivery |
X-Bolt-Signature | sha256=a3f8c2... | HMAC-SHA256 signature for payload verification |
User-Agent | BoltScraper-Webhook-Delivery/1.0 | Identify Bolt Scraper in server access logs |
Verifying the Webhook Signature Securely
The X-Bolt-Signature header contains an HMAC-SHA256 signature of the raw request body, computed using your shared secret. Verifying this signature before processing the payload ensures only genuine Bolt Scraper deliveries are acted upon.
import hmac, hashlib, json from fastapi import FastAPI, Request, HTTPException app = FastAPI() WEBHOOK_SECRET = "my-shared-secret-token-32chars" def verify_signature(body: bytes, signature_header: str) -> bool: expected = hmac.new( WEBHOOK_SECRET.encode(), body, hashlib.sha256 ).hexdigest() return hmac.compare_digest( f"sha256={expected}", signature_header or "" ) @app.post("/webhook/bolt-website") async def handle_webhook(request: Request): body = await request.body() sig = request.headers.get("X-Bolt-Signature", "") if not verify_signature(body, sig): raise HTTPException(status_code=401, detail="Invalid signature") payload = json.loads(body) event = payload.get("event") if event == "website_job.completed": download_url = payload["download_csv_url"] job_name = payload["job_name"] success_ct = payload["success_count"] print(ff"✅ {job_name}: {success_ct} contacts ready → {download_url}") # Trigger downstream: import to CRM, notify Slack, etc. return {"received": True}
Always Use hmac.compare_digest
Never compare signatures with ==. Python's string comparison short-circuits on the first mismatched character, making it vulnerable to timing attacks. hmac.compare_digest takes constant time regardless of how many characters match.
Delivery Logs and Replay
The dashboard's Webhook section shows the last 20 delivery attempts per endpoint, including HTTP status, response time, and the raw request payload that was sent. If your endpoint was down when a job completed, use the Replay button in Dashboard → Integrations → Webhooks to re-dispatch the payload to your endpoint without re-running the scrape.
Part of the Website Scraper Playbook
Connect webhooks to the rest of your enrichment pipeline:
Stop Polling. Start Receiving.
Register a webhook, submit your URLs, and let Bolt Scraper ping your pipeline the instant results are ready — with a signed CSV download link in the payload.
Instant cloud setup • No infrastructure required • n8n & Make.com compatible
Frequently Asked Questions
GET /api/v1/website/jobs/:jobId/download endpoint with your API key to generate a fresh download link.website_job.stopped event instead of website_job.completed. The payload structure is identical — check the event field to route accordingly. The download_csv_url is still included and contains all results processed before the cancellation./replay/:jobId API endpoint.COMPLETED. End-to-end latency from job completion to your endpoint receiving the payload is typically under 2 seconds under normal conditions.