jcpenney.com

When we requested your homepage as GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, Meta-ExternalAgent, Meta-ExternalFetcher, Amazonbot, Bytespider, and AURbot, the response was a block, a challenge page, or an empty shell instead of the page a browser gets.

• aurbot: error (0)
• gptbot: error (0)
• amazonbot: error (0)
• bytespider: error (0)
• chatgpt-user: error (0)
• oai-searchbot: error (0)
• perplexitybot: error (0)
• meta-externalagent: error (0)
• meta-externalfetcher: error (0)

Blocks like this almost never come from robots.txt. They come from bot management at the edge (Cloudflare Bot Fight Mode or "Block AI bots", Akamai Bot Manager, DataDome, HUMAN/PerimeterX, Kasada, Imperva) or from a hosting firewall. The response headers usually name the layer that challenged the request; add an allow rule there for the shopping agents.

Cloudflare: in Security → Bots (or AI Crawl Control), set the AI bots control to allow the shopping agents, and make sure Super Bot Fight Mode is not challenging verified bots. For anything still caught, add a WAF custom rule with the action Skip (skipping bot management and managed challenges) for:

(http.user_agent contains "ChatGPT-User") or (http.user_agent contains "OAI-SearchBot") or (http.user_agent contains "Claude-User") or (http.user_agent contains "PerplexityBot") or (http.user_agent contains "meta-externalfetcher") or (http.user_agent contains "AURbot")

Other WAFs: look for a "known bots" or "verified bots" category and allow it; otherwise allow-list the user agents above. Rate-limit rules scoped to product or search paths are a frequent hidden cause.

If an outcome below reads js-shell, that agent was not blocked: it received an empty JavaScript app shell with no product content. Server-render the page so the content is in the HTML.

ChatGPT-User, OAI-SearchBot, PerplexityBot, Meta-ExternalFetcher, and AURbot could not read your product page. These are the agents that act for a live shopper, so this is the single most expensive block on the site: with four or more blocked, your AUR is capped at 29 regardless of everything else.

• aurbot: error (0)
• chatgpt-user: error (0)
• oai-searchbot: error (0)
• perplexitybot: error (0)
• meta-externalfetcher: error (0)

If the homepage check passed but this one did not, look for a rule scoped to product paths: price-scraping protection, rate limits on /products/*, or a "challenge on high-value pages" setting.

Blocks like this almost never come from robots.txt. They come from bot management at the edge (Cloudflare Bot Fight Mode or "Block AI bots", Akamai Bot Manager, DataDome, HUMAN/PerimeterX, Kasada, Imperva) or from a hosting firewall. The response headers usually name the layer that challenged the request; add an allow rule there for the shopping agents.

Cloudflare: in Security → Bots (or AI Crawl Control), set the AI bots control to allow the shopping agents, and make sure Super Bot Fight Mode is not challenging verified bots. For anything still caught, add a WAF custom rule with the action Skip (skipping bot management and managed challenges) for:

(http.user_agent contains "ChatGPT-User") or (http.user_agent contains "OAI-SearchBot") or (http.user_agent contains "Claude-User") or (http.user_agent contains "PerplexityBot") or (http.user_agent contains "meta-externalfetcher") or (http.user_agent contains "AURbot")

Other WAFs: look for a "known bots" or "verified bots" category and allow it; otherwise allow-list the user agents above. Rate-limit rules scoped to product or search paths are a frequent hidden cause.

If an outcome below reads js-shell, that agent was not blocked: it received an empty JavaScript app shell with no product content. Server-render the page so the content is in the HTML.

Your search results page (https://www.jcpenney.com/s/search?Ntt=gift) did not return usable content for every probe. Agents use site search to find a product when the user names it; a search page that is blocked or empty forces them to guess URLs or give up.

• browser: error (400)
• claude-user: error (400)
• chatgpt-user: error (0)

Typical causes and fixes:

• Results are rendered client-side. Third-party search (Algolia, Klevu, Searchspring, Nosto) often replaces the results grid with JavaScript. Keep the platform's server-rendered results as the HTML fallback, or render results on the server before enhancing them.
• A WAF rule treats query strings as scraping. Allow the shopping agents (see the homepage and product-page fixes) on the search path, and raise any rate limit on ?q=.
• The search URL is hard to find. Expose a plain <form action="/search"> with a q input in the header, and declare it in WebSite JSON-LD so agents can jump straight to results:

{
  "@context": "https://schema.org",
  "@type": "WebSite",
  "url": "https://www.example.com/",
  "potentialAction": {
    "@type": "SearchAction",
    "target": { "@type": "EntryPoint", "urlTemplate": "https://www.example.com/search?q={search_term_string}" },
    "query-input": "required name=search_term_string"
  }
}

Your robots.txt has no Content-Signal directive. Content Signals (contentsignals.org) is a robots.txt extension that states, separately, whether your content may be used for search, as real-time AI input (which is what shopping agents do), and for training. An explicit signal scores fully; silence scores half.

You can welcome shopping agents while still declining training. Add to the * group (or to each group you care about):

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Keep ai-input=yes; set ai-train however you like. If Cloudflare manages your robots.txt, the equivalent setting lives in AI Crawl Control → Content Signals.

Edit the robots.txt served from your web root (or the CDN/edge rule that generates it) and redeploy. Crawlers cache robots.txt for up to 24 hours, so re-scan after a day.

Nothing usable was found at /.well-known/http-message-signatures-directory (HTTP 404). Web Bot Auth is the IETF draft that lets agents sign requests with HTTP Message Signatures; parties that sign publish their Ed25519 public keys in this directory, and verifiers fetch it to check the Signature and Signature-Input headers. Visa's Trusted Agent Protocol, Mastercard's Agent Pay, and Cloudflare's signed-agent verification are all built on it.

Two steps matter for a storefront:

• Verify signatures at your edge so signed agents are let through bot management instead of challenged. On Cloudflare this is the Verified Bots / signed agents setting; other WAFs are adding the same.
• Publish your own directory for any requests you sign (your own agent, service-to-service calls, or a verifier that expects one to exist). Serve it with Content-Type: application/http-message-signatures-directory+json:

{
  "keys": [
    {
      "kty": "OKP",
      "crv": "Ed25519",
      "kid": "2026-01",
      "x": "<base64url-encoded public key>",
      "nbf": 1767225600,
      "exp": 1798761600
    }
  ]
}

Serve the file from the same origin as the storefront, with a long cache lifetime, and rotate keys by adding the new one before removing the old.

Your product markup has sku but no brand. Agents use a GTIN, MPN, or SKU to recognise the same item across retailers (for price comparison and reviews) and the brand to answer "is this the real thing".

Add to the Product node:

"sku": "TR2-BLK-10",
"gtin13": "0012345678905",
"mpn": "TR2-BLK",
"brand": { "@type": "Brand", "name": "Example" }

Use gtin13/gtin12/gtin14 for barcodes (EAN/UPC), mpn for the manufacturer part number, and sku for your own code. One identifier is enough for the check; a GTIN is the most useful.

Your Offer is missing shippingDetails, hasMerchantReturnPolicy, and priceValidUntil. Agents rank options by delivery time, delivery cost, return risk, and how confident they are in the price; these three fields are where they look.

Add to the offers object:

"priceValidUntil": "2026-12-31",
"shippingDetails": {
  "@type": "OfferShippingDetails",
  "shippingRate": { "@type": "MonetaryAmount", "value": "0", "currency": "USD" },
  "shippingDestination": { "@type": "DefinedRegion", "addressCountry": "US" },
  "deliveryTime": {
    "@type": "ShippingDeliveryTime",
    "handlingTime": { "@type": "QuantitativeValue", "minValue": 0, "maxValue": 1, "unitCode": "DAY" },
    "transitTime": { "@type": "QuantitativeValue", "minValue": 2, "maxValue": 5, "unitCode": "DAY" }
  }
},
"hasMerchantReturnPolicy": {
  "@type": "MerchantReturnPolicy",
  "applicableCountry": "US",
  "returnPolicyCategory": "https://schema.org/MerchantReturnFiniteReturnWindow",
  "merchantReturnDays": 30,
  "returnMethod": "https://schema.org/ReturnByMail",
  "returnFees": "https://schema.org/FreeReturn"
}

Use your real values; one representative shipping rate and destination is enough. Set priceValidUntil to a date you can honour (the end of the current season or year, regenerated as it passes).

Many stores define the return policy once at the Organization level and reference it from each Offer; either is fine as long as the Offer on the product page carries or links to it.

Agents read the raw HTML. If the product name and price only appear after client-side JavaScript runs, most agents see a blank page or a price-less one.

• Browser request: OK
• ChatGPT-User request: error (0)

The page is fine for a browser but not for the agent, so you are serving a different response by user agent: a lighter template, a challenge, or a cached "bot" variant. Serve agents the same server-rendered HTML.

Frameworks: Next.js (server components / getServerSideProps), Nuxt (ssr: true), Remix, SvelteKit, and Astro all render on the server by default; make sure the product route has not opted out.

You have BreadcrumbList on the product page but no ItemList on a category or search page. Breadcrumbs tell an agent where a product sits in the catalog; an ItemList lets it read a whole category or result set in one pass.

On category and search result pages:

{
  "@context": "https://schema.org",
  "@type": "ItemList",
  "itemListElement": [
    { "@type": "ListItem", "position": 1, "url": "https://www.example.com/products/trail-runner-2" },
    { "@type": "ListItem", "position": 2, "url": "https://www.example.com/products/ridge-hiker" }
  ]
}

Requesting https://www.jcpenney.com/p/worthington-womens-modern-perfect-trouser/ppr5008151662?pTmplType=regular&cm_re=ZD-_-HOME-_-HOTDEALS-_-GET-READY-WITH-US-DEAL-ZONE-_-SHOP-ALL_1& with Accept: text/markdown returned HTML. Agents that support content negotiation get a page that is a tenth of the size and needs no parsing. This is worth one point and is a nice-to-have.

When the Accept header prefers text/markdown, respond with a Markdown body and Vary: Accept:

GET /products/trail-runner-2 HTTP/1.1
Accept: text/markdown

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept

# Trail Runner 2
**$129.00** · In stock · SKU TR2-BLK-10

Lightweight waterproof trail shoe. Sizes 6–13. Free shipping over $75, 30-day returns.

The simplest place to do this is edge middleware (a Cloudflare Worker, Vercel or Netlify middleware, or a Next.js route) that converts the rendered HTML or renders a dedicated template. If you are on Cloudflare, the "Markdown for Agents" option in AI Crawl Control does the conversion for you.

No /llms.txt was found (HTTP 404).

llms.txt (llmstxt.org) is a short Markdown file at the site root that tells language models what the site is and where the important pages are. For a store, point at the catalog, the policies, and any agent endpoints. Serve it as text/plain or text/markdown with a 200 status:

# Example Outdoor

> Example sells trail-running shoes and apparel, shipping to the US and Canada. Free 30-day returns.

## Shop
- [All products](https://www.example.com/collections/all): the full catalog
- [Product sitemap](https://www.example.com/sitemap_products_1.xml)

## Policies
- [Shipping](https://www.example.com/policies/shipping-policy)
- [Returns](https://www.example.com/policies/refund-policy)

## For agents
- [UCP profile](https://www.example.com/.well-known/ucp)
- [Storefront MCP](https://www.example.com/api/mcp)

No profile was found at /.well-known/ucp. The Universal Commerce Protocol (ucp.dev) is how an agent discovers, in one request, that your store can be shopped programmatically: which services you expose, over which transports (REST and MCP), and whether checkout is supported. It is the largest single item in the transactability pillar, and without it (or an attested ACP integration) your AUR is capped at 79.

A complete profile declares a shopping service with an endpoint and a checkout capability:

{
  "ucp": {
    "version": "2026-01-11",
    "services": {
      "dev.ucp.shopping": {
        "version": "2026-01-11",
        "spec": "https://ucp.dev/specification/overview",
        "rest": {
          "schema": "https://ucp.dev/services/shopping/openapi.json",
          "endpoint": "https://www.example.com/api/ucp"
        },
        "mcp": {
          "schema": "https://ucp.dev/services/shopping/mcp.json",
          "endpoint": "https://www.example.com/api/ucp/mcp"
        }
      }
    },
    "capabilities": [
      {
        "name": "dev.ucp.shopping.checkout",
        "version": "2026-01-11",
        "spec": "https://ucp.dev/specification/checkout",
        "schema": "https://ucp.dev/schemas/shopping/checkout.json"
      },
      {
        "name": "dev.ucp.shopping.order",
        "version": "2026-01-11",
        "spec": "https://ucp.dev/specification/order",
        "schema": "https://ucp.dev/schemas/shopping/order.json"
      }
    ],
    "payment_handlers": []
  }
}

Serve it as application/json from the storefront origin. The endpoints must exist and speak the protocol; the MCP one is tested by the next check. Check ucp.dev for the current version string and schema URLs.

Serve the profile from the storefront origin and implement the shopping service behind it; most of the work is the checkout capability, which maps onto your existing cart and checkout APIs.

https://www.jcpenney.com/api/ucp/mcp did not return a tool list. The endpoint must speak MCP over Streamable HTTP: accept a JSON-RPC POST and answer with JSON or an SSE stream.

Test it the way we do:

curl -sS -X POST https://www.jcpenney.com/api/ucp/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'

• tools/list must work without authentication; require auth on the tool calls that need it, and advertise it through OAuth discovery (see that fix).
• Return 200 with Content-Type: application/json (or text/event-stream), never a 405 for POST or an HTML page.
• Make sure the URL in the UCP profile (services → dev.ucp.shopping → mcp.endpoint) points at this endpoint exactly.

Any MCP server SDK (TypeScript, Python, Go) with the Streamable HTTP transport will do; mount it on the storefront origin so the profile, the endpoint, and the cookies share a domain.

No MCP server answered tools/list at /api/mcp. This is the conventional storefront endpoint (Shopify's Storefront MCP lives here) where an agent can search the catalog, read policies, and manage a cart without scraping.

Expose an MCP server over Streamable HTTP at that path with tools such as search_shop_catalog, get_product_details, search_shop_policies_and_faqs, get_cart, and update_cart. tools/list must be callable without authentication.

If you already run the UCP MCP endpoint, mounting the same server at /api/mcp is enough.

Nothing was found at /.well-known/mcp/server-card.json. The server card is the MCP discovery document that lets an agent find your server, its transport endpoint, and how to authenticate, without reading documentation.

Publish a JSON document describing the server (field names follow the current MCP server-card draft; adjust to the version you target):

{
  "serverInfo": { "name": "example-storefront", "title": "Example Storefront", "version": "1.0.0" },
  "description": "Search the Example catalog, read policies, and manage a cart.",
  "transport": { "type": "streamable-http", "endpoint": "https://www.example.com/api/mcp" },
  "capabilities": { "tools": {} },
  "authentication": { "required": false }
}

Serve it as application/json with CORS enabled, and keep the endpoint in sync with the one your UCP profile names.

No API catalog was found. RFC 9727 defines /.well-known/api-catalog, a linkset that points agents at your APIs and their descriptions (OpenAPI, MCP, UCP) in one place.

Serve this as application/linkset+json:

{
  "linkset": [
    {
      "anchor": "https://www.example.com/",
      "service-desc": [
        { "href": "https://www.example.com/api/openapi.json", "type": "application/vnd.oai.openapi+json" }
      ],
      "service-doc": [
        { "href": "https://www.example.com/developers", "type": "text/html" }
      ],
      "service-meta": [
        { "href": "https://www.example.com/.well-known/ucp", "type": "application/json" },
        { "href": "https://www.example.com/.well-known/mcp/server-card.json", "type": "application/json" }
      ]
    }
  ]
}

Also advertise it from the homepage with a Link header, which we accept on its own:

Link: </.well-known/api-catalog>; rel="api-catalog"

None of /.well-known/oauth-authorization-server, /.well-known/openid-configuration, and /.well-known/oauth-protected-resource returned a JSON document. Agents that need to act on a customer's account (order status, saved addresses, loyalty) look for OAuth 2.0 discovery before they will attempt authenticated calls; it is also what MCP clients use to authorise against your MCP server.

If you run an authorization server on this origin, publish RFC 8414 metadata at /.well-known/oauth-authorization-server:

{
  "issuer": "https://www.example.com",
  "authorization_endpoint": "https://www.example.com/oauth/authorize",
  "token_endpoint": "https://www.example.com/oauth/token",
  "registration_endpoint": "https://www.example.com/oauth/register",
  "scopes_supported": ["cart:write", "orders:read"],
  "response_types_supported": ["code"],
  "grant_types_supported": ["authorization_code", "refresh_token"],
  "code_challenge_methods_supported": ["S256"]
}

If authentication is handled by a separate identity provider, publish RFC 9728 protected-resource metadata at /.well-known/oauth-protected-resource pointing at it:

{
  "resource": "https://www.example.com/api/mcp",
  "authorization_servers": ["https://auth.example.com"],
  "scopes_supported": ["cart:write", "orders:read"],
  "bearer_methods_supported": ["header"]
}

Support PKCE and dynamic client registration (RFC 7591) so agents can onboard without a manual app review.

None of /.well-known/agent-card.json, /.well-known/agent.json, and /.well-known/agent-skills/index.json was found. An A2A agent card tells other agents that your store has an agent of its own and what it can do; an Agent Skills index lists reusable skill files. Either satisfies this one-point check.

A minimal A2A agent card at /.well-known/agent-card.json:

{
  "name": "Example Shopping Assistant",
  "description": "Find, compare, and buy Example products.",
  "url": "https://www.example.com/a2a",
  "version": "1.0.0",
  "protocolVersion": "0.3.0",
  "capabilities": { "streaming": true },
  "defaultInputModes": ["text/plain"],
  "defaultOutputModes": ["text/plain", "application/json"],
  "skills": [
    {
      "id": "find-products",
      "name": "Find products",
      "description": "Search the catalog by query, size, colour, and price.",
      "tags": ["shopping", "search"]
    }
  ]
}

Or, if you publish skills rather than an agent, serve /.well-known/agent-skills/index.json listing each skill's name, description, and the URL of its SKILL.md. Only publish a card for an endpoint that actually answers.

Your product markup has no aggregateRating. Agents weigh review scores heavily when choosing between similar products, and they only trust ratings they can read from structured data.

Add to the Product node, with real numbers from reviews shown on the page:

"aggregateRating": {
  "@type": "AggregateRating",
  "ratingValue": "4.6",
  "reviewCount": "128",
  "bestRating": "5"
}

Only emit it for products that actually have reviews, and keep it server-rendered: markup injected by a review widget's JavaScript is invisible to agents.