Indexing: sitemap watching and IndexNow
Sitemap checks, the IndexNow key, submissions and their responses, never-indexed pages and limits.
Indexing works on the website picked in the header switcher. Open Dashboard → Indexing. The page has three views: Pages (every page in your sitemaps, with status filters), History (every IndexNow batch and its response) and Settings (sitemaps, your key, sending and alerts). The switch at the top turns Indexing on or off for the website.
Sitemaps
- When you switch Indexing on with no sitemaps added, we look for them in your robots.txt, then at /sitemap.xml and the other usual locations.
- You can add sitemaps yourself in Settings. They must be on the website's own domain or one of its subdomains.
- Sitemap index files are followed, and gzipped sitemaps are read. Sitemaps are fetched with the SerpfullAuditor user agent, respecting robots.txt.
- Each check runs on a schedule set by your plan, from every 24 hours on Starter to every hour on Enterprise. Check sitemaps now runs one straight away, with a 10-minute pause between manual checks.
New, changed and removed pages
- Your opening check is the baseline: every page is recorded but none is sent.
- After that, a page that appears in a sitemap is new, and a page whose last-modified date changes is changed. Pages without a last-modified date get a fingerprint of their visible text, checked in turn within your plan's allowance per check, so edits still show.
- A page that leaves your sitemaps is marked removed once a check reads every sitemap successfully. A failed or cut-short check never removes pages.
Your IndexNow key
- Indexing creates a key for each website. Download the key file from Settings and upload it to the root of the site, so it opens at https://your-domain/your-key.txt and contains only the key.
- Press Verify key. We fetch the file from our servers and check it matches. Nothing is sent until the key is verified.
- If search engines later answer 403, the key is marked unverified, sending pauses and you get an alert. Put the file back and verify again.
Submissions
- With automatic sending on, new and changed pages are sent after each check, in batches of up to 10,000 pages, to the shared IndexNow endpoint. Only pages on the verified host are sent.
- Tick pages in the Pages view and press Send now to send them yourself, including baseline pages.
- Responses: 200 and 202 are accepted (202 means the key is still being checked on their side); 400 and 422 are rejected and the pages are marked failed; 403 means the key wasn't valid; 429 means too many requests, so the pages stay pending and are retried after an hour.
- A submission asks search engines to crawl pages sooner. Whether and when a page is indexed is up to each search engine.
Never-indexed pages
Connect Search Console (Integrations → Google Search Console) and link the website to its property. A page is flagged as never indexed when it has been in your sitemaps for a set number of days (14 by default, 3 to 90 in Settings), Search Console's URL inspection reports it as not indexed, and it had no impressions in the last 28 days. Without a linked property the check shows what it needs. We never query search results pages to guess.
Index checks
- Every 15 minutes Serpfull inspects a slice of your pages, spreading the day's budget across the day: Starter 50, Growth 150, Scale 400, Pro 1,000 and Enterprise 2,000 pages a day, and never more than Search Console's own limit of 2,000 a day and 600 a minute per property.
- Pages go first when they are past the never-indexed threshold and not known to be indexed, then pages sent through IndexNow since their last check (after 12 hours), then pages never checked, oldest first. Pages not indexed are checked again after 3 days and indexed pages after 30 days.
- The Index column in Pages shows Indexed, Not indexed or Not checked yet, with Search Console's coverage reason, the last crawl and impressions. Pages with impressions in the last 28 days count as indexed.
Limits
- Sitemaps watched: Starter 2, Growth 5, Scale 10, Pro 25, Enterprise 50.
- Pages watched: 1,000, 5,000, 25,000, 100,000 and 500,000.
- Pages sent per month: 1,000, 5,000, 25,000, 100,000 and 500,000. Accepted and rejected batches count; throttled and failed ones don't.
- Hours between sitemap checks: 24, 12, 6, 3 and 1.
- Submission history kept: 30, 90, 180, 365 and 730 days.
- Search Console index checks per day: 50, 150, 400, 1,000 and 2,000.
Alerts and webhooks
indexing.urls.changed (new, changed and removed pages after a check, with up to 50 of them) and indexing.submission.completed (every batch, with its status and response code). Use them as Workflow triggers or add them in Integrations → Webhooks. A rejected key also reaches the notification bell unless you switch alerts off.
Updated