fentybeauty.com

PerplexityBot could not read your product page. These are the agents that act for a live shopper, so this is the single most expensive block on the site.

• perplexitybot: blocked (503)

If the homepage check passed but this one did not, look for a rule scoped to product paths: price-scraping protection, rate limits on /products/*, or a "challenge on high-value pages" setting.

Blocks like this almost never come from robots.txt. They come from bot management at the edge (Cloudflare Bot Fight Mode or "Block AI bots", Akamai Bot Manager, DataDome, HUMAN/PerimeterX, Kasada, Imperva) or from a hosting firewall. The response headers usually name the layer that challenged the request; add an allow rule there for the shopping agents.

Cloudflare: in Security → Bots (or AI Crawl Control), set the AI bots control to allow the shopping agents, and make sure Super Bot Fight Mode is not challenging verified bots. For anything still caught, add a WAF custom rule with the action Skip (skipping bot management and managed challenges) for:

(http.user_agent contains "ChatGPT-User") or (http.user_agent contains "OAI-SearchBot") or (http.user_agent contains "Claude-User") or (http.user_agent contains "PerplexityBot") or (http.user_agent contains "meta-externalfetcher") or (http.user_agent contains "AURbot")

Other WAFs: look for a "known bots" or "verified bots" category and allow it; otherwise allow-list the user agents above. Rate-limit rules scoped to product or search paths are a frequent hidden cause.

If an outcome below reads js-shell, that agent was not blocked: it received an empty JavaScript app shell with no product content. Server-render the page so the content is in the HTML.

On Shopify, the storefront CDN is Shopify's and you cannot edit its WAF. Blocks of this kind usually come from a bot-blocking or anti-fraud app, a password-protected store, or a Cloudflare proxy placed in front of the domain (Shopify Plus). Remove the app, disable the password, or add the allow rule at that proxy.

Your robots.txt has no Content-Signal directive. Content Signals (contentsignals.org) is a robots.txt extension that states, separately, whether your content may be used for search, as real-time AI input (which is what shopping agents do), and for training. An explicit signal scores fully; silence scores half.

You can welcome shopping agents while still declining training. Add to the * group (or to each group you care about):

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Keep ai-input=yes; set ai-train however you like. If Cloudflare manages your robots.txt, the equivalent setting lives in AI Crawl Control → Content Signals.

On Shopify, robots.txt is generated from the theme. In Online Store → Themes → Edit code, add a robots.txt.liquid template (Add a new template → robots.txt) if you do not have one, keep the default groups, and append your rules after the loop. SEO apps that "manage" robots.txt for you are a common source of AI-bot blocks; check their settings too.

Nothing usable was found at /.well-known/http-message-signatures-directory (HTTP 404). Web Bot Auth is the IETF draft that lets agents sign requests with HTTP Message Signatures; parties that sign publish their Ed25519 public keys in this directory, and verifiers fetch it to check the Signature and Signature-Input headers. Visa's Trusted Agent Protocol, Mastercard's Agent Pay, and Cloudflare's signed-agent verification are all built on it.

Two steps matter for a storefront:

• Verify signatures at your edge so signed agents are let through bot management instead of challenged. On Cloudflare this is the Verified Bots / signed agents setting; other WAFs are adding the same.
• Publish your own directory for any requests you sign (your own agent, service-to-service calls, or a verifier that expects one to exist). Serve it with Content-Type: application/http-message-signatures-directory+json:

{
  "keys": [
    {
      "kty": "OKP",
      "crv": "Ed25519",
      "kid": "2026-01",
      "x": "<base64url-encoded public key>",
      "nbf": 1767225600,
      "exp": 1798761600
    }
  ]
}

On Shopify, themes cannot serve files under /.well-known/, so treat this as out of scope unless you run a proxy (Shopify Plus with Cloudflare) that can answer the path. It is worth 2 points.

Your Offer is missing shippingDetails and hasMerchantReturnPolicy. Agents rank options by delivery time, delivery cost, return risk, and how confident they are in the price; these three fields are where they look.

Add to the offers object:

"priceValidUntil": "2026-12-31",
"shippingDetails": {
  "@type": "OfferShippingDetails",
  "shippingRate": { "@type": "MonetaryAmount", "value": "0", "currency": "USD" },
  "shippingDestination": { "@type": "DefinedRegion", "addressCountry": "US" },
  "deliveryTime": {
    "@type": "ShippingDeliveryTime",
    "handlingTime": { "@type": "QuantitativeValue", "minValue": 0, "maxValue": 1, "unitCode": "DAY" },
    "transitTime": { "@type": "QuantitativeValue", "minValue": 2, "maxValue": 5, "unitCode": "DAY" }
  }
},
"hasMerchantReturnPolicy": {
  "@type": "MerchantReturnPolicy",
  "applicableCountry": "US",
  "returnPolicyCategory": "https://schema.org/MerchantReturnFiniteReturnWindow",
  "merchantReturnDays": 30,
  "returnMethod": "https://schema.org/ReturnByMail",
  "returnFees": "https://schema.org/FreeReturn"
}

Use your real values; one representative shipping rate and destination is enough. Set priceValidUntil to a date you can honour (the end of the current season or year, regenerated as it passes).

On Shopify, add these to the theme's JSON-LD block, driving the values from theme settings (return window, free-shipping threshold) so merchandisers can change them without touching Liquid. Shopify's own agentic surfaces read policies from Settings → Policies, so keep both in sync.

Requesting https://fentybeauty.com/products/oil-control-duo-invisimatte-powder-imposter-dry-shampoo with Accept: text/markdown returned HTML. Agents that support content negotiation get a page that is a tenth of the size and needs no parsing. This is worth one point and is a nice-to-have.

When the Accept header prefers text/markdown, respond with a Markdown body and Vary: Accept:

GET /products/trail-runner-2 HTTP/1.1
Accept: text/markdown

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept

# Trail Runner 2
**$129.00** · In stock · SKU TR2-BLK-10

Lightweight waterproof trail shoe. Sizes 6–13. Free shipping over $75, 30-day returns.

The simplest place to do this is edge middleware (a Cloudflare Worker, Vercel or Netlify middleware, or a Next.js route) that converts the rendered HTML or renders a dedicated template. If you are on Cloudflare, the "Markdown for Agents" option in AI Crawl Control does the conversion for you.

On Shopify, themes cannot vary the response on Accept, so this is only possible behind a proxy (Shopify Plus). Skip it otherwise.

Nothing was found at /.well-known/mcp/server-card.json. The server card is the MCP discovery document that lets an agent find your server, its transport endpoint, and how to authenticate, without reading documentation.

Publish a JSON document describing the server (field names follow the current MCP server-card draft; adjust to the version you target):

{
  "serverInfo": { "name": "example-storefront", "title": "Example Storefront", "version": "1.0.0" },
  "description": "Search the Example catalog, read policies, and manage a cart.",
  "transport": { "type": "streamable-http", "endpoint": "https://www.example.com/api/mcp" },
  "capabilities": { "tools": {} },
  "authentication": { "required": false }
}

Serve it as application/json with CORS enabled, and keep the endpoint in sync with the one your UCP profile names.

No API catalog was found. RFC 9727 defines /.well-known/api-catalog, a linkset that points agents at your APIs and their descriptions (OpenAPI, MCP, UCP) in one place.

Serve this as application/linkset+json:

{
  "linkset": [
    {
      "anchor": "https://www.example.com/",
      "service-desc": [
        { "href": "https://www.example.com/api/openapi.json", "type": "application/vnd.oai.openapi+json" }
      ],
      "service-doc": [
        { "href": "https://www.example.com/developers", "type": "text/html" }
      ],
      "service-meta": [
        { "href": "https://www.example.com/.well-known/ucp", "type": "application/json" },
        { "href": "https://www.example.com/.well-known/mcp/server-card.json", "type": "application/json" }
      ]
    }
  ]
}

Also advertise it from the homepage with a Link header, which we accept on its own:

Link: </.well-known/api-catalog>; rel="api-catalog"

None of /.well-known/agent-card.json, /.well-known/agent.json, and /.well-known/agent-skills/index.json was found. An A2A agent card tells other agents that your store has an agent of its own and what it can do; an Agent Skills index lists reusable skill files. Either satisfies this one-point check.

A minimal A2A agent card at /.well-known/agent-card.json:

{
  "name": "Example Shopping Assistant",
  "description": "Find, compare, and buy Example products.",
  "url": "https://www.example.com/a2a",
  "version": "1.0.0",
  "protocolVersion": "0.3.0",
  "capabilities": { "streaming": true },
  "defaultInputModes": ["text/plain"],
  "defaultOutputModes": ["text/plain", "application/json"],
  "skills": [
    {
      "id": "find-products",
      "name": "Find products",
      "description": "Search the catalog by query, size, colour, and price.",
      "tags": ["shopping", "search"]
    }
  ]
}

Or, if you publish skills rather than an agent, serve /.well-known/agent-skills/index.json listing each skill's name, description, and the URL of its SKILL.md. Only publish a card for an endpoint that actually answers.

No MerchantReturnPolicy structured data was found on the product offer, the homepage, or the organization. A written policy page is good; machine-readable terms let agents compare return windows and fees across stores without reading prose.

Attach it to the product Offer (preferred, since that is where agents are when deciding) or to the Organization/OnlineStore on the homepage:

"hasMerchantReturnPolicy": {
  "@type": "MerchantReturnPolicy",
  "applicableCountry": "US",
  "returnPolicyCategory": "https://schema.org/MerchantReturnFiniteReturnWindow",
  "merchantReturnDays": 30,
  "returnMethod": "https://schema.org/ReturnByMail",
  "returnFees": "https://schema.org/FreeReturn",
  "refundType": "https://schema.org/FullRefund",
  "merchantReturnLink": "https://www.example.com/policies/refund-policy"
}

Use MerchantReturnNotPermitted for final-sale items and MerchantReturnUnlimitedWindow if you accept returns indefinitely. Keep it consistent with the written policy page.

On Shopify, add the object to the theme's product JSON-LD block, with the window and fees as theme settings, and link merchantReturnLink to {{ shop.refund_policy.url }}.

Your product markup has no aggregateRating. Agents weigh review scores heavily when choosing between similar products, and they only trust ratings they can read from structured data.

Add to the Product node, with real numbers from reviews shown on the page:

"aggregateRating": {
  "@type": "AggregateRating",
  "ratingValue": "4.6",
  "reviewCount": "128",
  "bestRating": "5"
}

Only emit it for products that actually have reviews, and keep it server-rendered: markup injected by a review widget's JavaScript is invisible to agents.

On Shopify, review apps (Judge.me, Yotpo, Okendo, Loox, Stamped) each have a "rich snippets" or "SEO/JSON-LD" setting that renders the rating in Liquid rather than injecting it client-side; turn it on. Alternatively merge product.metafields.reviews.rating.value and product.metafields.reviews.rating_count into the theme's product JSON-LD.