How to approach the work
Create four groups: public search pages, public utility pages, duplicate URLs and private material. Keep important service pages crawlable. A public thank-you page may need noindex; confidential client files need access protection. A robots rule is not a privacy system.
What to check and change
Test rules against exact paths, including case and trailing slashes. Inspect both HTML robots tags and HTTP X-Robots-Tag headers. If a crawler cannot fetch a page because robots.txt blocks it, it cannot read a noindex instruction on that page. Remove contradictory instructions only after confirming the intended outcome.
A practical example
Example: a staging site goes live while a sitewide noindex remains in the template. Updating the sitemap alone will not remove that instruction. Fix the template, confirm the response on multiple page types, and then inspect the important URLs.
What to measure and ask your agency
Keep a dated change log and compare excluded URLs with the intended groups. Your SEO agency should supply the exact rule, affected paths, expected behaviour and rollback step before a sitewide change.
Continue reading
Technical reference: Google Search Central
Work with an SEO agency
Send your website, target market and the business outcome you want. We will discuss the technical and content priorities, who will implement the work and how the outcome will be measured.
Discuss your project by email