# The SerpfullAuditor crawler

How our page auditor identifies itself, what it requests and how to control it.

SEO audits are fetched from our servers with this user agent:

```bash
Mozilla/5.0 (compatible; SerpfullAuditor/1.0; +https://serpfull.com/documentation/seo-audit-crawler)
```

## What it requests

- The audited page, once per audit, with a 15-second timeout and a 3 MB limit. Redirects are followed only within the same website.
- robots.txt for each website it visits, cached for an hour.
- A sample of links and images on the page (up to 20 links and 8 images, with HEAD requests where supported) to find broken links and heavy images, at most one request per other website per audit.
- Only public addresses on ports 80 and 443.

## Politeness

Audits of the same website, from all customers combined, start at least 10 seconds apart and no more than two run at once. Scheduled audits run weekly by default.

## Blocking or allowing it

It follows robots.txt rules for the SerpfullAuditor token, falling back to the * group. A disallowed URL isn't fetched and the audit reports why. If robots.txt can't be reached (a server error or timeout), we treat the site as disallowed for that hour.

```bash
User-agent: SerpfullAuditor
Disallow: /private/
```

Questions or concerns about requests from SerpfullAuditor? Contact support@serpfull.com with the URL and time.