Robots.txt Generator

What robots.txt can and cannot do

robots.txt is a voluntary crawl budget hint at the site root. Well-behaved bots read User-agent and Allow/Disallow rules. It is not authentication, not a paywall, and not a way to hide confidential files from humans who know the URL.

This page builds a simple file you can download and host at /robots.txt. For search appearance controls, use meta robots or HTTP headers on the pages themselves.

Worked example: allow all vs blocking AI crawlers

Allow-all for everyone:

User-agent: *
Allow: /

A stricter draft that blocks common AI training crawlers (names change over time) adds separate User-agent blocks with Disallow: /. Review the agent list before you publish; blocking the wrong bot can hurt search traffic.

Limits

Malicious scrapers ignore robots.txt. Sitemap lines are optional pointers, not indexing guarantees. Path rules are prefix-based, so check real URLs in the robots.txt tester before you deploy.

Sitemap line and crawl budget

Optional Sitemap: https://example.com/sitemap.xml lines help bots discover URLs but do not force indexing. Disallowing large filter facades can save crawl budget on big catalogs. After you publish, re-fetch robots.txt in Search Console and fix redirect chains to the root file.

Worked example: hiding an admin folder and listing the sitemap

User-agent: * then Disallow: /admin/ then Sitemap: https://example.com/sitemap.xml. This asks all crawlers to skip the admin folder and points them to the sitemap. The file must live at the root, https://example.com/robots.txt; a robots.txt in a subfolder is ignored.

Pitfalls

• Disallow is not noindex. A blocked URL can still be indexed from links, without content. To keep a page out of Google, allow crawling and use a noindex meta tag.
• Blocking CSS and JS. Google renders pages; blocking assets can hurt how it understands them.
• Not security. robots.txt is public and polite crawlers follow it voluntarily. Do not list secret paths.
• One slash too many. Disallow: / blocks the whole site.

Serve robots.txt from your app

If your host serves static files, put robots.txt in the site root and you are done. If your app handles every path itself, add a route like these so /robots.txt returns plain text with a 200 status.

JavaScript: Express

import express from "express"; // npm i express

const ROBOTS = `User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml
`;

const app = express();
app.get("/robots.txt", (req, res) => res.type("text/plain").send(ROBOTS));
app.listen(3000);

Python: Flask

from flask import Flask, Response  # pip install flask

ROBOTS = """User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml
"""

app = Flask(__name__)

@app.get("/robots.txt")
def robots_txt():
    return Response(ROBOTS, mimetype="text/plain")

Go: net/http (Go 1.22 or later)

package main

import (
	"net/http"
)

const robots = `User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml
`

func main() {
	http.HandleFunc("GET /robots.txt", func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "text/plain; charset=utf-8")
		w.Write([]byte(robots))
	})
	http.ListenAndServe(":8080", nil)
}

Each snippet was run on October 14, 2026 with Express 5.3.0, Flask 3.1.3 and Go 1.24.

Robots.txt standard and Google docs

• RFC 9309: Robots Exclusion Protocol: the IETF standard, published 2022
• Google: create and submit a robots.txt file
• Google: how Google interprets robots.txt: status codes, caching and the 500 KiB limit
• Sitemaps protocol: the Sitemap: line format
• Express 5 API
• Flask documentation
• Go docs: net/http

FAQ