Example

robots.txt example for SaaS

A SaaS robots.txt should separate valuable public paths from low-value noise. This example focuses on sitemap discovery, crawl guidance, and avoiding accidental emphasis on app-only routes.

llms.txt vs robots.txt Agent readiness checklist for SaaS

Primary job

Crawl guidance

Keep public

Home, pricing, docs

Keep noisy

Preview and app-only routes

Example file

A practical SaaS robots.txt pattern

The goal is not to hide everything. The goal is to make the public surface easier to crawl without wasting attention on low-value paths.

SaaS robots.txt example

A simple pattern that keeps the public product surface crawlable while pointing to the sitemap.

User-agent: *
Allow: /
Disallow: /app/
Disallow: /api/
Disallow: /preview/
Disallow: /internal/

Sitemap: https://example.com/sitemap.xml

Disallowing a path does not make it canonical elsewhere. It only changes crawl behavior.

Use it well

What this example is trying to prevent

App-only or authenticated areas accidentally competing with public product pages.

Preview, internal, or temporary paths wasting crawl attention.

A public site with no obvious sitemap discovery hint.

Confusion between crawl control and canonical preference.

Boundaries

What to review before publishing

Public pages stay open

homepage

pricing

docs

trust pages

Noisy paths stay contained

/preview/

/internal/

/app/

/api/

Other layers still matter

canonical tags

sitemap.xml

llms.txt for preferred references

Continue through the cluster

llms.txt vs robots.txt

See where each file helps and where it does not.

llms.txt example for SaaS

See a practical SaaS-oriented llms.txt shape and what each block is doing.

Agent readiness checklist for SaaS

A practical checklist for pricing, docs, trust, and action paths.

Next step

Use robots for crawl boundaries, not for every architecture problem

If the public site is still confusing after this file looks clean, the next fix probably belongs in your page structure or canonical signals rather than more robots rules.

llms.txt vs robots.txt Run a fresh scan