Check the existing Shopify robots.txt before proposing a change, identify the exact crawl problem and test the intended rule against representative URLs. Preserve the current template and a restoration route. Use the worksheet below to distinguish crawl access from indexing and to make a small, reviewable change instead of replacing the store’s rules wholesale.
What problem should a robots.txt change solve?
Write the unwanted request pattern or access problem as a specific question. Identify the crawler, the affected URL family and the business content that should remain discoverable. Gather the actual current file and example URLs before editing. A general desire for better SEO is too vague to define an acceptance check.
Google describes robots.txt as controlling which site URLs a crawler can access. Documentation. Shopify supplies a default robots.txt file for its stores. Documentation. Shopify says the default robots.txt works for most stores. Documentation. Begin the review from those defaults and ask what evidence justifies changing them. Keep unrelated catalogue, metadata and internal-navigation improvements in their own work items so the effect of the crawl change remains understandable.
Distinguish an unwanted crawl request from an unwanted search result. Google says a disallowed URL can still be indexed if other sites link to it. Documentation. Decide whether your objective concerns access, indexing or private information, then choose an appropriate control for that objective. A crawl rule should not be treated as the access policy for a customer account or confidential document.
How should the Shopify template be changed?
Shopify uses a robots.txt.liquid template for custom crawl rules. Documentation. Shopify recommends using Liquid customisations to preserve future automatic updates. Documentation. Ask the implementer to show the current custom template, its relationship to the generated file and the narrow proposed difference. Preserve a dated copy of the existing content and record the reason for each addition or removal.
Shopify discourages replacing the template with a static set of plain-text rules. Documentation. Prefer a review that can explain one intended change without carrying forward a copied block whose purpose nobody understands. If a previous customisation exists, investigate its origin and current need. Do not discard it merely because its author is unavailable or its comment looks old.
Treat an editing session as a controlled production change. Prepare the comparison and acceptance URLs first, arrange the appropriate operator access and record who can restore the previous state. The worksheet below describes that proposed review process; the actual store’s template and deployment arrangements still need inspection.
An original crawl-rule verification worksheet
Use this practical checklist for a single proposed change. Keep examples precise enough that someone else can repeat the comparison. The purpose is to make the scope and expected behaviour reviewable before the public file changes.
| Checkpoint | Evidence to prepare | Question to resolve |
|---|---|---|
| Current state | Existing public file and custom template reference | What rules are actually being served? |
| Intended crawler | Exact user-agent under review | Which group of rules is being evaluated? |
| Target family | URLs intended to change behaviour | What request pattern is unwanted? |
| Protected examples | Product, collection and useful content URLs | Should these remain available to the crawler? |
| Required resources | Scripts, styles or images used by the page | Can the crawler still understand the content? |
| Proposed difference | Small template edit with a reason | Can each change be explained? |
| Restoration | Previous content and responsible operator | How will an unintended block be reversed? |
Add both matching and non-matching examples to the working copy. Include a normal product, a collection and a URL resembling the intended target without actually belonging to it. Ask the reviewer to explain the expected decision for each example rather than relying only on the appearance of the rule.
Record the exact published file after the change, alongside the approved difference. If the generated output differs from the expected template result, investigate that before interpreting later crawl reports. Keep the implementation observation and the search-engine observation in separate parts of the record.
What can a verification exercise establish?
Google notes that crawler support for robots.txt rules is not universal. Documentation. Name the crawler whose behaviour you are assessing and use its relevant documentation or supported inspection route. A local rule comparison can show how a proposed rule is expected to match examples. It cannot establish that every crawler has already fetched or obeyed the updated public file.
Google advises against blocking resources needed to understand a page. Documentation. Include required resource URLs in the review where the change could affect them. Evaluate the actual page dependency rather than assuming that every script or image path can be blocked without consequence. Keep the page’s understandable content as the acceptance question.
After publishing an authorised edit, check that the public file contains the intended output. Then inspect the important page examples through the appropriate search-engine tooling as part of the site’s normal verification process. Preserve the observation date and the difference between current output and a report based on an earlier crawl.
Hypothetical example: an over-broad collection rule
Imagine a proposed rule is intended to reduce requests to a particular filtered collection pattern, but the working edit also matches an ordinary collection page. This hypothetical exercise gives the reviewer a useful reason to maintain non-matching examples.
Place the unwanted filtered URL and the ordinary collection URL on the worksheet. Write the expected treatment for each. Compare the generated rule output against both before approving the change. If the rule cannot distinguish the intended family, revise the proposal or reconsider whether a crawl change is the right intervention.
Keep the product and collection content review separate. If the ordinary collection is thin or confusing, improving its content and navigation may require its own editorial work. A robots.txt edit should have a clear technical purpose rather than becoming a shortcut for unresolved page-quality questions.
How should the previous state be restored?
Shopify says removing the custom robots.txt.liquid file restores the default file. Documentation. Review that documented route with the store’s authorised operator before the change. Restoring a platform default and restoring a previous custom template are different choices; record which state the recovery plan intends to return to and preserve the relevant evidence.
Keep a short change note with the reason, examples checked, observed output and owner. That record makes the next review easier when a catalogue or theme change prompts another crawl question.
For related discovery work, see the Bing SEO guide. Use the Shopify availability guide for broader store constraints, and SEO for the commercial brief. Get started with the actual file and the request pattern you want to examine.
By Spout Team. AI-assisted drafting by Codex using the linked primary documentation.