owlabee

Web and domains

Sitemap Monitor: Track New, Removed and Changed Pages

A site's sitemaps list every page it wants indexed, with last-modified dates. This Actor reads them, keeps a snapshot per site, and on each run reports which pages appeared, disappeared, changed their lastmod date or changed their content.

Open the Actor on Apify Store

Website Change Monitor — Sitemap Monitor & Page Diff. Pay per result; see the Store page for current pricing.

The source and its limits

The Actor follows robots.txt Sitemap: lines, sitemap indexes, and plain, gzipped and text sitemaps, falling back to /sitemap.xml and /sitemap_index.xml. That is the sitemaps protocol sites already publish for search engines.

  • Discover mode lists every page URL with lastmod, changefreq and priority.
  • Monitor mode keeps a snapshot per site and emits one record per change: new-url, removed-url, lastmod-changed, content-changed and status-changed.

Content changes are detected without a browser. Each HTML page is scoped to a CSS selector (main, article, body by default), stripped of scripts, styles and whitespace noise, optionally stripped of numbers and dates, and hashed. A changed page gets a word-level preview of what was added and removed, and a severity of low, medium or high with a reason.

Limits worth knowing

  • Only pages listed in the sitemaps are tracked.
  • maxPages cuts the list in sitemap order. In monitor mode set it above the site's page count, or the tail of the sitemap will look removed.
  • Pages are fetched as plain HTML (up to 1 MB, 20-second timeout), at about two requests per second per host, and robots.txt Disallow rules are honored. JavaScript-rendered content is not executed.
  • The first monitor run is a baseline: every page becomes a baseline record and changes appear from the second run on.

Fields you get

FieldWhat it holds
changeTypediscovered, baseline, new-url, removed-url, lastmod-changed, content-changed, status-changed or error
url, site, sitemapUrlThe page, the input site and the sitemap it came from
lastmod, changefreq, priorityAs listed in the sitemap
status, title, contentHashHTTP status, page title and the content hash
changedFieldsEvery field that changed when several changed at once
changedPreviewAdded and removed word counts, previews and a similarity ratio
severity, severityReasonlow, medium or high, with a one-phrase reason
before, afterPrevious and current page snapshots

Example output

A content-changed record from a monitor run (shape from the Actor's README; values illustrative, snapshots omitted):

{
  "changeType": "content-changed",
  "site": "https://example.com",
  "url": "https://example.com/pricing",
  "changedFields": [
    "contentHash",
    "lastmod"
  ],
  "title": "Pricing — Example",
  "lastmod": "2026-10-16",
  "status": 200,
  "contentHash": "3f9a1c…",
  "changedPreview": {
    "addedWords": 3,
    "removedWords": 1,
    "addedPreview": "SSO now included",
    "removedPreview": "Add-on",
    "similarity": 0.94,
    "changedRatio": 0.06,
    "numbersChanged": false
  },
  "severity": "low",
  "severityReason": "6% of the text changed",
  "detectedAt": "2026-10-16T12:00:09.000Z"
}

Use cases

New pages on a competitor's site

Monitor with checkContent: false for a lighter, sitemap-only diff that reports new and removed URLs.

Documentation or policy page changes

Narrow with includePatterns and a contentSelector such as main, and get content-changed records with a diff preview.

Full URL inventory

Discover mode lists every URL in a site's sitemaps with its last-modified date, for SEO audits or crawl planning.

How to run it

You need an Apify account and its API token (Console → Settings → API & Integrations). Each run is billed per result to your account; set a maximum total charge for the run to cap spend.

With curl

The sync endpoint starts a run, waits up to 300 seconds and returns the dataset items. Sitemap-level monitoring (new and removed pages, lastmod changes) for one site:

export APIFY_TOKEN=<your Apify API token>

curl -sS -X POST \
  "https://api.apify.com/v2/acts/owlabee~website-change-monitor/run-sync-get-dataset-items" \
  -H "Authorization: Bearer ${APIFY_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
  "sites": [
    "https://www.sitemaps.org/sitemap.xml"
  ],
  "mode": "monitor",
  "checkContent": false,
  "maxPages": 200,
  "stateStoreName": "website-change-monitor-sitemaps-org",
  "resetBaseline": false,
  "notifyOnlyOnChanges": true
}'

From an AI agent (MCP)

Apify hosts an MCP server. This config exposes only this Actor to the agent:

{
  "mcpServers": {
    "owlabee-website-change-monitor": {
      "url": "https://mcp.apify.com/?tools=owlabee/website-change-monitor",
      "headers": {
        "Authorization": "Bearer ${APIFY_TOKEN}"
      }
    }
  }
}

Then ask, for example: Tell me which pages were added to or removed from sitemaps.org since the last check.

Python, Node.js and more examples

Runnable curl.sh, python.py, node.mjs and mcp.json files for this Actor are in examples/website-change-monitor on GitHub.

No code: Zapier, Make or n8n

Apify's official apps on these platforms can run this Actor and pass its results on. Step-by-step recipes: Website Change Alerts in Slack with n8n.

FAQ

Does it work with sitemap indexes and gzipped sitemaps?

Yes. Indexes are followed recursively, .xml.gz files are decompressed and plain-text sitemaps are accepted.

How do I avoid false positives from dates and counters?

Point contentSelector at the element with the real content and turn on ignoreNumbers.

How is severity decided?

A removed page or a page that starts failing is high, a new page is medium, a lastmod change with unchanged content is low, and a content change is scored by the share of words that changed.

Can I get alerts?

Set webhookUrl. By default it is called only when changes were found, with a run summary and up to 1,000 change records.

Does it render JavaScript?

No. Pages are fetched as plain HTML; use a selector that targets server-rendered markup.

Related guides