/
Flexible customization of robots.txt

Flexible customization of robots.txt

Released in V4.4 Do you need more control over how search engines crawl your site? Whether it is for specific SEO strategies, Google Shopping feeds, or hiding sensitive catalog sections, you can now fully customize your robots.txt file.

Solution

We have introduced a flexible mechanism to manage search engine indexing, moving away from the previously hard-coded file. This allows customers to define specific rules to allow or disallow bots, or to completely replace the default file with a custom one. This flexibility is essential for compliance with advertising strategies (e.g., allowing Googlebot-Image) and optimizing organic search traffic.

We now offer two methods to configure your robots.txt, depending on your needs:

1. Additional Directives (Recommended for safety) You can inject additional rules (User-agent, Allow, Disallow, Sitemap) on top of the system defaults.

  • Pros: The default security rules (blocking /checkout/, /account/) remain intact.

  • Behavior: Your new rules are validated and appended to the existing file.

2. Full File Override (Advanced) You can upload a completely custom robots.txt file via the theme configuration.

  • Pros: Maximum flexibility; you have complete control over every line of the file.

  • Behavior: If a valid file is found, it replaces the system defau

Getting started

To Append Rules:

  • Config Key: config.robots.additionalDirectives

  • Format: List of valid directives (User-agent, Allow, Disallow, Sitemap).

  • Validation: Invalid syntax will be skipped with a warning log.

To Full Override:

  • File Location: theme.robots.txt

  • Requirement: Must be a .txt file with UTF-8 encoding.

Known Limitations & Constraints

  • Override Risk: If you use the Full Override (theme.robots.txt), the system defaults are completely removed. You are responsible for manually adding protection for sensitive paths (e.g., /checkout/, /account/) to prevent them from being indexed.

  • Validation: In "Append" mode, if a directive contains invalid syntax, it is silently skipped (logged as a warning) rather than breaking the entire file.

  • Conflict Resolution: You cannot use "Append" mode to remove a default rule; you can only add to them. If you need to remove a default rule, you must use the "Full Override" method.