--- title: "PostCliqBot: what our crawler does | PostCliq" description: "PostCliqBot is the read-only crawler behind PostCliq. What it fetches, how often, how to slow it down, how to block it, and who to contact." url: "https://postcliq.com/bot" --- # PostCliqBot If you found PostCliqBot in your server logs, this page tells you exactly what it is, what it did on your site, and how to stop it. No sign-up, no form. ## The short version PostCliqBot is the crawler behind PostCliq, a website audit tool made by WP Creative in Australia. It visits a site only when a PostCliq user has asked for that site to be audited. It is read-only. It requests pages with GET, reads the HTML, and leaves. It does not submit forms, does not log in, does not attempt to bypass a firewall or a CAPTCHA, does not probe for vulnerabilities, and does not change anything on your site. It honours `robots.txt` . If you tell it to go away, it goes away, and it stops at the `robots.txt` before it fetches anything else. ## How to identify it Every request carries this user agent string: `PostCliqBot/0.1 (+https://postcliq.com/bot)` Some pages are also rendered in a headless Chromium browser so we can see the page the way a visitor does. Those requests carry the browser's own device user agent with the same `PostCliqBot/0.1 (+https://postcliq.com/bot)` string appended, so the identification is present on every request either way. **A caution, stated plainly.** A user agent string is not proof of identity. Anyone can send that string. We do not yet publish a list of source addresses you could verify against, and we do not yet sign our requests. We intend to adopt Web Bot Auth, the HTTP Message Signatures profile built on RFC 9421 that lets a crawler prove its identity cryptographically, which was adopted as an IETF working group document on 1 September 2026 ( Cloudflare's documentation , checked 2026-09-20). Until we do, treat the string as a claim rather than as proof, and if traffic claiming to be us is behaving in a way this page does not describe, it is not us. Tell us and we will help you work out what it is. ## What it does, precisely - **Fetches** **`robots.txt`** **first** , parses it per RFC 9309, and applies the `PostCliqBot` group if one exists, otherwise the `*` group, using longest-match rules. - **Fetches your sitemaps** if `robots.txt` lists them, or at the common default locations. Sitemap traversal is bounded: three levels deep, fifty files, five thousand entries. - **Fetches HTML pages only.** Anything that is not HTML is recorded as such and not processed further. - **Renders a small sample of pages** in a headless browser, at a mobile viewport and a desktop viewport, to measure what a visitor actually sees. - **Stops.** A run ends and does not resume. ## What it never does - No form submissions. POST and PUT navigations are refused in code, not by policy alone. - No logins, no password attempts, no session reuse. - No firewall, CAPTCHA, bot-protection or paywall bypass. If we are blocked, the audit records that we were blocked and reports it to the user as missing coverage. - No file downloads, no WebSockets, no server-sent events. - No vulnerability scanning, port scanning or exploit attempts. - No writes of any kind. Nothing on your site changes because we visited. ## How often it visits, and how hard **Frequency.** A site is audited at most once per rolling seven days, and only while a PostCliq user has that site in their account. An audit costs us real money to run, so there is no incentive at our end to visit more often. **Rate.** Per origin, at most two requests per second, with at most two concurrent connections plus one browser session. **Size and time.** A single run is capped at fifty pages, twenty-five megabytes in total, two megabytes for any one HTML response, four hundred requests, and eight minutes of wall clock time. Whichever limit is reached first ends the run. **Backing off.** If you return a 429, we honour `Retry-After` in either the seconds or the HTTP date form, capped at sixty seconds, and back off exponentially if you send no `Retry-After` at all. For a typical small business site, a whole audit is a few dozen requests spread over about two minutes. ## How to slow it down Add a Crawl-delay to your robots.txt. We read it and apply it, up to a maximum of thirty seconds between requests. robots.txt ``` User-agent: PostCliqBot Crawl-delay: 10 ``` ## How to block it Add this to your robots.txt: robots.txt ``` User-agent: PostCliqBot Disallow: / ``` We fetch robots.txt before anything else, so this takes effect on the next run, with no approval step and no delay at our end. You can also block the user agent at your CDN or web server. That works too, and we would rather you did that than spend time wondering whether we will respect the file. We will, but you should not have to take our word for it. What blocking costs you. Nothing, unless somebody at your organisation is a PostCliq user. If they are, their audit of your site will return no findings and will say it could not crawl the site, rather than silently producing a thin report. If you are blocking us by accident, that is usually what alerts someone. Allowing a specific section only works as you would expect, because we apply longest-match rules with Allow winning ties. ## What we keep To produce a report we store evidence of what we fetched: response status codes and headers, the HTML we received, page screenshots, and timing measurements. That evidence exists so a user can inspect any finding rather than taking a score on trust, and so we can show our working if a finding is disputed. We do not collect personal information from your pages on purpose, and the audit has no route to anything behind a login. Our handling, retention and deletion terms are on the [terms and privacy page](https://postcliq.com/terms) . ## If something looks wrong If PostCliqBot is causing load problems, appearing more often than this page describes, requesting things it should not, or behaving in any way this page does not account for, tell us. Include the date and time with the timezone, the user agent string you saw, the source address, and a few log lines. We will look at it and reply. Email: `[[email protected]](https://postcliq.com/cdn-cgi/l/email-protection)` We would rather hear from you than be blocked quietly. A crawler that gets blocked everywhere is useless to us, so the incentives here point the same way yours do. ## Common questions. Is PostCliqBot an AI training crawler? No. It does not collect content for training models. It fetches pages to run technical and content checks for the person who asked for the audit, and the evidence it keeps exists so that person can inspect the findings. Will it try again if I block it? No. Each run reads robots.txt first and respects it. There is no retry queue, no alternative user agent, and no fallback address. Can I find out who asked for the audit? No, and we will not tell you. That is the requester's information, not ours to disclose. If you believe someone is auditing a site they have no relationship with, contact us and we will look into it. Do you obey noindex or nofollow? We read them and report them, because whether they are set correctly is one of the things an audit checks. They are not access controls, so they do not stop us fetching a page. robots.txt is the control that stops us. Why does the version say 0.1? Because it is honest. The crawler is new. This page will be updated as it changes.