Why sitemaps still matter
XML sitemaps help search engines discover URLs, especially on large sites, new launches, and pages with weak internal linking.
Robots.txt communicates crawl preferences and sitemap locations — foundational files every production site should ship deliberately.
This generator produces starter assets quickly for MVPs, staging reviews, and small business sites without enterprise crawl tooling.
How the URL scanner works
The scanner fetches your starting URL and follows same-origin internal links up to a 100-page cap.
HTML pages are collected with last-modified hints when available. Binary assets and external domains are excluded.
Rate limits keep the free tier responsive for all users — avoid scanning massive sites repeatedly in short windows.
What you receive after a scan
A sitemap.xml listing discovered URLs with optional lastmod values suitable for Search Console submission.
A robots.txt template referencing your sitemap location and sensible default disallow rules you can customize.
Copy buttons and download options for direct deployment or version control commits.
Deploying sitemap and robots files
Place sitemap.xml at your domain root or submit the path in Google Search Console.
Host robots.txt at https://example.com/robots.txt — crawlers expect it at the root only.
After deployment, request indexing for priority URLs and monitor Coverage reports for errors.
Limitations of lightweight crawlers
JavaScript-heavy SPAs may hide links from HTML-only discovery — validate critical routes manually.
No crawl replaces log analysis, canonical audits, or hreflang validation on international sites.
Enterprise catalogs should use dedicated crawl platforms with custom extraction and staging comparisons.
Combine with SEO audit workflows
Run the SEO audit tool on high-traffic URLs discovered in your sitemap export.
Pair with the favicon generator when launching new properties that need complete technical assets.
Read ABSEO insights on crawl budget and enterprise-scale technical SEO for deeper programs.
Frequently asked questions
How many pages can be scanned?
Up to 100 same-origin HTML pages per scan session.
Does this crawl JavaScript apps?
The scanner follows links in fetched HTML. Client-rendered routes may be missed without server-side links.
Can I scan password-protected sites?
No. Only public URLs without authentication are supported.
Where should I upload sitemap.xml?
Typically at your domain root, then submit the URL in Google Search Console.
Does robots.txt block indexing?
Disallow rules limit crawling, not necessarily indexing. Use noindex tags for de-indexing pages.
Are scans stored on your server?
Scan inputs are processed to generate files. Do not submit confidential staging URLs.
Can I edit the generated robots.txt?
Yes. Treat output as a starting template and customize rules for your environment.
How often should I regenerate sitemaps?
Regenerate after major site launches, migrations, or when adding large new URL batches.
How to generate sitemap and robots files
Scan a public site and export crawl assets.
Step 1
Enter site URL
Paste your public homepage or section root URL.
Step 2
Start scan
Run the crawler to discover internal links.
Step 3
Review URLs
Confirm discovered pages match expectations.
Step 4
Copy sitemap.xml
Download or copy the generated XML file.
Step 5
Copy robots.txt
Customize and deploy robots.txt at your domain root.
Step 6
Submit to Search Console
Add the sitemap URL and monitor coverage reports.