The SerpfullAuditor crawler
How our page auditor identifies itself, what it requests and how to control it.
SEO audits are fetched from our servers with this user agent:
Mozilla/5.0 (compatible; SerpfullAuditor/1.0; +https://serpfull.com/documentation/seo-audit-crawler)What it requests
- The audited page, once per audit, with a 15-second timeout and a 3 MB limit. Redirects are followed only within the same website.
- robots.txt for each website it visits, cached for an hour.
- A sample of links and images on the page (up to 20 links and 8 images, with HEAD requests where supported) to find broken links and heavy images, at most one request per other website per audit.
- Only public addresses on ports 80 and 443.
Politeness
Audits of the same website, from all customers combined, start at least 10 seconds apart and no more than two run at once. Scheduled audits run weekly by default.
Blocking or allowing it
It follows robots.txt rules for the SerpfullAuditor token, falling back to the * group. A disallowed URL isn't fetched and the audit reports why. If robots.txt can't be reached (a server error or timeout), we treat the site as disallowed for that hour.
User-agent: SerpfullAuditor
Disallow: /private/Questions or concerns about requests from SerpfullAuditor? Contact support@serpfull.com with the URL and time.
Updated