Page Not Crawlable Due to robots.txt in Merchant Center

What this issue means

Before showing a product, Google crawls its landing page to check price, availability and policy compliance. Robots.txt is a plain text file at yourdomain.com/robots.txt that tells crawlers which paths they may fetch. If a rule disallows the product URL, Google can't do its checks, and the product stays disapproved for Shopping ads and free listings until crawling is allowed.

Google also reports this issue when the landing page returns an HTTP error to the crawler, so check both.

Why it happens

• A blanket disallow left over from staging. User-agent: * with Disallow: / copied from a development site is the classic cause after launches and migrations.
• Parameter blocking. SEO advice to block query strings (Disallow: /*?) also blocks feed URLs like /products/shirt?variant=123 that Shopify and other platforms use for variants.
• Blocked collection or category paths. Some feeds link to products under a collection path (/collections/sale/products/...) that's disallowed.
• Rules for specific bots. A rule targeting Googlebot or Googlebot-Image that was meant to block something else.
• Mobile subdomain rules. A separate m. subdomain has its own robots.txt that blocks crawlers.
• Security plugins or CDNs rewriting robots.txt or serving a different file to bots.

How to fix it

1. Read your live robots.txt. Open https://yourdomain.com/robots.txt (and any subdomain used in your feed). Look for Disallow lines under User-agent: *, User-agent: Googlebot and User-agent: Googlebot-Image.
2. Find the rule that matches your feed URL. Take an affected product URL from Products > Needs attention and compare its path and parameters against each disallow rule. Remember * is a wildcard.
3. Allow Google to crawl products. Google's own guidance is to let both Googlebot and Googlebot-Image crawl your full site. The simplest version is two groups: User-agent: Googlebot followed by an empty Disallow: line, then User-agent: Googlebot-Image followed by an empty Disallow: line. An empty Disallow means nothing is blocked for that crawler. If you need to keep private areas (cart, account, checkout) blocked, keep those specific paths and remove rules that match product pages and their parameters. You can also add Allow rules for patterns like /products/.
4. Edit on your platform.
   • Shopify: Shopify's default robots.txt doesn't block product pages. If you customized it, edit the robots.txt.liquid template in your theme code. Look for added rules blocking ?variant= or collection paths. Deleting the custom template restores Shopify's default.
   • WooCommerce/WordPress: Check whether a physical robots.txt file exists in your site root (it overrides WordPress's virtual one). Then check SEO plugins (Yoast, Rank Math, All in One SEO), which have robots.txt editors. Also confirm Settings > Reading > Discourage search engines is unchecked.
   • Custom sites and CDNs: Make sure the CDN serves the same robots.txt to everyone and doesn't cache an old version.
5. Test. Use the URL Inspection Tool in Search Console on an affected URL (desktop and smartphone) and confirm crawling is allowed and the page returns a 200 status.
6. Request a website check in Merchant Center, or reupload your product data through a new fetch to prompt reprocessing. Google also suggests you can increase Google's crawl rate if needed.

How long until it's fixed

Google says changes can take 24 to 72 hours to appear on the Needs attention page, and errors should clear within a couple of days after products are reprocessed. Google caches robots.txt, so the new file may not be picked up instantly. If you requested a website check, allow up to 12 hours for that review.

How to prevent it

• Never deploy a staging robots.txt to production; add a check to your launch checklist.
• Avoid blanket Disallow: /*? rules on ecommerce sites that use parameters for variants.
• Review robots.txt after installing SEO plugins, changing themes or migrating platforms.
• Monitor Search Console for "Blocked by robots.txt" warnings on product URLs.
• Use our Merchant Center feed checker to catch URL problems early.

Robots.txt mistakes can wipe out a whole catalog overnight. Mercantads can audit your crawl setup and Merchant Center feed together and restore products quickly. Talk to our team or see our Merchant Center services.