What You Can Track with Automated Crawls
Competitor intelligence isn't just for enterprise teams. Any business running paid ads, building SEO content, or setting prices benefits from knowing when a competitor makes a significant move. The Website Scraper API extracts several fields per URL that are particularly valuable for competitive monitoring:
| Extracted Field | Monitoring Use Case | Change Signal |
|---|---|---|
page_title | Detect brand/product renaming, A/B test messaging shifts | Title differs from last week's crawl |
meta_description | Track SEO repositioning, new value propositions | Description changed — likely a conversion test |
email_1 / phone_1 | Detect contact point changes — new office, new team, acquisition signals | Domain or contact changed entirely |
final_url | Domain redirects, acquisitions, site migrations | Final URL differs from submitted URL |
linkedin_url, facebook_url | New social channels launched, rebrand signals | New URL found that wasn't present before |
Setting Up Automated Weekly Crawls
The monitoring architecture is simple: a weekly cron job submits your competitor list to the API, the scraper runs in the cloud, and a webhook delivers results to your comparison logic. No headless browsers, no proxy rotation, no server maintenance.
import requests API_KEY = "bs_YOUR_API_KEY" BASE_URL = "https://boltscraper.com/api/v1/website" HEADERS = {"Authorization": f"Bearer {API_KEY}"} # Competitor domains to monitor weekly COMPETITORS = [ "https://competitor-a.com", "https://competitor-a.com/pricing", "https://competitor-a.com/features", "https://competitor-b.com", "https://competitor-b.com/pricing", # Include specific subpages you care most about ] r = requests.post(f"{BASE_URL}/jobs", headers=HEADERS, json={ "job_name": "Weekly Competitor Monitor", "urls": COMPETITORS }) data = r.json() print(f"✅ Monitoring job started: {data['job_id']}") # Configured webhook receives an automated callback on completion — no polling needed
Run this script on a weekly cron via GitHub Actions, a VPS cron job, or n8n's Schedule trigger. When the crawl finishes, your registered webhook endpoint receives an automated callback with results — no polling loop required.
Include Specific Subpages for Higher Signal Quality
Don't just crawl the root domain. Add /pricing, /features, /about, and specific product landing pages directly to your URL list. The scraper treats each as an independent URL, so you get granular, per-page comparison data rather than aggregated root-page signals.
Detecting Changes Between Weekly Crawls
The core of your monitoring system is a comparison function that runs in your webhook handler. It loads the previous week's results from a database or flat file, compares them against the new crawl, and generates a diff:
import json, requests from fastapi import FastAPI, Request app = FastAPI() DB = {} # Replace with Postgres, SQLite, or a JSON file SLACK = "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK" TRACKED_FIELDS = ["page_title", "meta_description", "final_url", "email_1"] @app.post("/webhook/bolt-website") async def handle_monitor(request: Request): payload = await request.json() if payload.get("event") != "website_job.completed": return {"skipped": True} changes = [] for result in payload.get("results", []): url = result["url"] prev = DB.get(url, {}) for field in TRACKED_FIELDS: old_val = prev.get(field, "") new_val = result.get(field, "") if old_val != new_val and new_val: changes.append({ "url": url, "field": field, "old": old_val or "(empty)", "new": new_val }) DB[url] = result # Update stored snapshot if changes: msg = f"🔔 *{len(changes)} competitor changes detected*\n\n" for c in changes[:10]: msg += f"• `{c['url']}`\n **{c['field']}**: `{c['old'][:60]}` → `{c['new'][:60]}`\n\n" requests.post(SLACK, json={"text": msg}) return {"changes": len(changes)}
Recommended Alert Rules by Business Type
Not every field change warrants waking someone up. Here's a practical guide to which changes matter most depending on your industry:
E-Commerce & SaaS
- Critical:
meta_descriptionon/pricingchanges — possible price drop or new tier - High:
page_titleon homepage changes — rebrand or A/B messaging shift - Medium: New
linkedin_urlfound — team expansion or relaunch signal
B2B Lead Generation
- Critical:
email_1changes — new ownership or key contact replacement - High:
final_urlredirects to a new domain — acquisition or migration - Medium:
phone_1changes — new office or contact centre
No Proxy Servers or Headless Browsers Required
Traditional competitor monitoring stacks require Puppeteer or Playwright clusters, residential proxy pools (expensive), and dedicated VPS infrastructure. The Bolt Scraper API handles all of this in the cloud. You write the diff logic and alerting — Bolt handles the crawling.
Part of the Website Scraper Playbook
Combine monitoring with the full enrichment pipeline:
Monitor Your Competitors Automatically
Set up weekly crawls over any competitor domain list. Get Slack or email alerts the moment a price page, title, or contact changes. No proxies, no infrastructure — just the API.
Instant cloud setup • CSV & JSON export • Up to 10,000 URLs per job
Frequently Asked Questions
https://competitor.com/pricing. The scraper fetches that specific page and returns its title, meta description, and any contact/social links found on it, giving you page-level granularity rather than just root domain data.data-cfemail) natively. For pages behind Cloudflare's full bot management (JS challenge / hCaptcha), the scraper will receive the challenge page rather than the target content — these pages typically have a very low yield. Most standard business websites, even those using Cloudflare CDN, do not enable challenge modes on their public marketing pages.