What Is a Robots.txt Generator?

A Robots.txt Generator is an online tool that helps website owners create a robots.txt file containing crawler directives. The file can provide instructions about which URL paths automated crawlers should or should not request.

Tool95’s Robots.txt Generator lets you create common directives without manually formatting the entire file. You can enter user-agent, allow, disallow and sitemap information and generate a ready-to-copy robots.txt file.

What Is a robots.txt File?

A robots.txt file is a plain-text file normally placed at the root of a website. It can contain rules intended for automated crawlers.

For example, a website at https://example.com/ would normally make the file available at:

The file is publicly accessible and should not be used as a method for protecting confidential information.

How to Create a robots.txt File

1. Choose the user-agent to which your rules should apply.
2. Enter paths that you want to disallow.
3. Add allow paths if they are needed.
4. Enter your sitemap URL if you want to include a Sitemap directive.
5. Optionally add a crawl-delay value where appropriate.
6. Click Generate Robots.txt.
7. Review the generated directives carefully.
8. Copy the code or download the robots.txt file.
9. Upload the file to the root directory of your website.

Basic robots.txt Syntax

A basic robots.txt file commonly uses directives such as User-agent, Disallow, Allow and Sitemap.

What Does User-agent Mean?

The User-agent directive identifies the crawler or group of crawlers that a set of rules applies to.

The asterisk character (*) is commonly used as a wildcard representing all crawlers.

What Does Disallow Mean?

Disallow specifies a URL path that a crawler is requested not to crawl for the applicable user-agent.

A Disallow rule is not a replacement for authentication or access control. If information is private, protect it at the server or application level.

What Does Allow Mean?

Allow can be used to specify paths that a crawler is permitted to access when applicable rules are also present.

Crawler implementations can differ, so test important robots.txt configurations carefully.

What Is a Sitemap Directive?

A Sitemap directive can tell crawlers where a website’s XML sitemap is located.

The sitemap URL should be a complete, accessible URL pointing to the correct XML sitemap.

Why Add a Sitemap to robots.txt?

Including a sitemap location in robots.txt can make the sitemap discoverable to crawlers that retrieve the robots.txt file.

A sitemap and robots.txt serve different purposes. The sitemap describes URLs that may be useful for discovery, while robots.txt can provide crawler access directives.

Googlebot robots.txt Rules

You can target Googlebot specifically by using its user-agent name. This can be useful when a rule is intended for Google’s web crawler rather than every crawler.

Bingbot robots.txt Rules

Bingbot is the crawler user-agent associated with Bing search. A robots.txt file can contain rules specifically addressed to Bingbot.

Googlebot-Image

Googlebot-Image can be specified when creating rules intended for Google’s image crawler.

What Is Crawl-delay?

Crawl-delay is a robots.txt directive used by some crawler implementations to request a delay between crawler requests. However, support and interpretation of Crawl-delay can vary between search engines and other crawlers.

Do not assume that every crawler will honor Crawl-delay in the same way.

Should You Block Your Admin Page?

Some website owners choose to include administrative paths in their robots.txt rules. However, robots.txt is publicly readable and should never be treated as a security mechanism.

Administrative areas should have appropriate authentication, authorization and server-side security regardless of robots.txt settings.

Robots.txt and SEO

Robots.txt can affect which URLs search-engine crawlers are able to request. Incorrect rules can therefore interfere with crawling and search visibility.

A robots.txt file should be created with care, especially on large websites where many directories and URL patterns are involved.

Robots.txt Does Not Remove a Page From the Internet

A Disallow rule does not delete a URL, remove its content or make private information secure. It only provides a crawler instruction about requesting the specified path.

If a resource must be private, use authentication or another proper access-control mechanism.

Robots.txt vs Meta Robots

Feature	robots.txt	Meta robots
Location	Usually at the website root	Inside an HTML page’s head section
Main purpose	Provides crawler access instructions	Provides indexing and link-following directives for a page
Scope	Can apply to URL paths	Applies to the individual page containing the metadata
Security mechanism	No	No

Common robots.txt Mistakes

• Blocking important website sections by mistake.
• Using robots.txt as a security mechanism.
• Entering an incorrect sitemap URL.
• Using unnecessary or overly broad Disallow rules.
• Forgetting that robots.txt is publicly accessible.
• Assuming every crawler interprets every directive identically.
• Blocking resources required to properly understand a webpage.

Robots.txt for WordPress Websites

WordPress websites can use robots.txt rules to provide crawler instructions. Before manually replacing or editing an existing configuration, check whether your hosting environment or SEO tools already generate or manage the file.

Always review the final robots.txt file after making changes.

Robots.txt for Ecommerce Websites

Large ecommerce websites can contain many URL variations created by filters, sorting parameters, internal search and other functions. Robots.txt rules may be used as part of a broader crawling strategy, but each rule should be evaluated carefully.

Blocking the wrong URL patterns can prevent crawlers from accessing important pages or resources.

Robots.txt for Large Websites

Large websites may require more carefully planned crawling rules because they can contain thousands or millions of URLs. A simple robots.txt file should not be expanded with arbitrary rules without understanding the site’s URL structure.

Robots.txt Best Practices

• Keep the file clear and easy to review.
• Use specific rules where appropriate.
• Check important pages before blocking directories.
• Include the correct sitemap URL when useful.
• Never rely on robots.txt for confidential information.
• Test changes before deploying them to production.
• Review the file after major website structure changes.
• Understand crawler-specific behavior when using specific user-agents.

How to Upload robots.txt

1. Generate and review the robots.txt file.
2. Save it with the exact filename robots.txt.
3. Upload it to the root directory of your website.
4. Confirm that the file is publicly reachable.
5. Review the final content after uploading.

Where Should robots.txt Be Located?

The standard location is the root of the host. For example:

A robots.txt file placed in an unrelated subdirectory does not serve as a replacement for the site’s root robots.txt file.

Can robots.txt Block All Crawlers?

A commonly used pattern is:

This requests that all crawlers covered by the wildcard not crawl anything under the host. Such a rule should only be used when that is actually the intended behavior because it can prevent search crawlers from accessing the site’s pages.

Can robots.txt Allow Everything?

A simple robots.txt file can specify:

This leaves the Disallow value empty. Website owners should still review any other crawler directives and site configuration that may affect crawling.

Benefits of Tool95’s Robots.txt Generator

• Free online robots.txt generation.
• Supports multiple common user-agents.
• Custom user-agent support.
• Allow and Disallow path controls.
• Sitemap URL support.
• Optional Crawl-delay field.
• Instant preview of generated code.
• One-click code copying.
• Download the robots.txt file directly.
• Works on desktop, tablet and mobile devices.
• No software installation required.

Robots.txt Testing Checklist

• Confirm the file is named robots.txt.
• Confirm it is located at the correct root URL.
• Check every Disallow path.
• Check that important pages are not accidentally blocked.
• Verify the sitemap URL.
• Review user-agent targeting.
• Test the final file after publishing.

Frequently Asked Questions