# Blog: Automated Competitor Website Monitoring with the Scraper API

**URL:** `/blogs/guides/competitor-website-monitoring-scraper-api/`  
**Series:** Website Scraper Playbook — Supporting Guide #4  
**Pillar:** [Extract Emails & Social Profiles at Scale](/blogs/guides/extract-emails-social-links-website-scraper-api/)

---

## Purpose

Shows product and growth teams how to build a recurring competitive intelligence system using scheduled crawls + webhook callbacks + diff logic. No Puppeteer/Playwright required.

## Content Structure

1. **Problem** — Manual competitor checks are inconsistent; teams miss pricing/messaging changes
2. **Education** — What signals to monitor: meta descriptions, page title, phone, social links, email changes
3. **Implementation** — Weekly Python cron script; GitHub Actions integration
4. **Bolt Scraper Example** — Python diff function comparing current vs. previous crawl results
5. **Change detection** — What fields to diff (title, meta_description, email_1, phone_1, social URLs)
6. **Webhook cross-link** — Automated callback delivers results to your comparison endpoint
7. **CTA** — Free trial

## Key Implementation Details

- The cron script calls `POST /jobs` with a list of competitor URLs (including subpages like `/pricing`, `/features`)
- The registered webhook receives an automated callback when the crawl completes
- Diff logic runs in the webhook handler: loads previous week's data, compares, alerts on changes
- Recommended fields to diff: `page_title`, `meta_description`, `email_1`, `phone_1`, social URLs

## Subpage Monitoring Strategy

Don't just monitor root domains. Add targeted subpages:
```
https://competitor-a.com
https://competitor-a.com/pricing
https://competitor-a.com/features
```
Each URL is treated independently, giving per-page granularity.

## Language Rules

- ✅ "receives an automated callback on completion" 
- ❌ "webhook fires the moment" / "instant" delivery
- ❌ Real competitor names in code (use `competitor-a.com`, `competitor-b.com`)
- `YOUR_API_KEY` is OK inside a Python string literal in code examples (e.g. `API_KEY = "YOUR_API_KEY"`)

## Cross-links Out

- [Pillar guide](/blogs/guides/extract-emails-social-links-website-scraper-api/)
- [Webhooks guide](/blogs/guides/website-scraper-webhooks-guide/)
- [Lead enrichment guide](/blogs/guides/enrich-lead-lists-website-scraper-api/)
- [Google Sheets integration](/blogs/guides/connect-website-scraper-api-google-sheets-airtable/)

## Last Audited

2026-09-19 — Inline `border-left` styles removed; webhook language corrected; "fires automatically on completion" → "receives an automated callback on completion".