AI crawlers allowed in robots.txt
app/robots.ts (or public/robots.txt on the pages router). If neither exists, whatever is answering /robots.txt is your host or your CDN.
Site platforms
On Next.js every one of these checks is a file you control, which makes it the easiest platform to fix and the easiest to get quietly wrong. The framework hands you robots, sitemap and metadata APIs, and none of them cover JSON-LD — so the schema checks are the ones most Next.js sites fail while having excellent everything else.
The failure worth watching for is the rendering one. A homepage that is a client component, or whose copy arrives from a client-side fetch, ships a shell to anything that does not run JavaScript. It looks perfect in a browser and reads as an empty page to the crawler that decides whether you get quoted.
How we recognise it: an X-Powered-By: Next.js header, __NEXT_DATA__ in the page, or /_next/static/ assets. Run the free readiness check and the fixes it gives you are the Next.js ones below.
Before changing anything. If this number is in the hundreds rather than the thousands, your copy is arriving after hydration and nothing else on this page matters until that is fixed.
In a terminal
curl -s https://yourdomain.com | wc -cThe Metadata API covers titles, descriptions and canonicals, and stops short of JSON-LD, so this one is a script tag written by hand. The root layout puts it on every page, which is where an Organization statement belongs.
app/layout.tsx
const org = {
'@context': 'https://schema.org',
'@type': 'Organization',
name: 'Your company',
url: 'https://yourdomain.com',
sameAs: ['https://www.linkedin.com/company/yourcompany'],
};
// inside the layout's returned JSX
<script
type="application/ld+json"
dangerouslySetInnerHTML={{ __html: JSON.stringify(org) }}
/>public/llms.txt is fine and it is a file somebody has to remember to update. A route handler builds it from the same modules your pages render from, so a new page appears in it without anybody editing anything.
app/llms.txt/route.ts
export const dynamic = 'force-static';
export function GET() {
return new Response(buildLlmsTxt(), {
headers: { 'content-type': 'text/plain; charset=utf-8' },
});
}CueScout runs eight checks on whether an answer engine can read a site at all. These are the ones whose answer is different on Next.js — the rest are the same everywhere.
app/robots.ts (or public/robots.txt on the pages router). If neither exists, whatever is answering /robots.txt is your host or your CDN.
public/llms.txt is served at the root as-is. For one generated from your content, a route handler at app/llms.txt/route.ts returning text/plain does it without a build step.
A <script type="application/ld+json"> in your root layout, with the JSON stringified into dangerouslySetInnerHTML. Next's Metadata API does not cover JSON-LD.
The `metadata` export in app/layout.tsx: title, description, and alternates.canonical.
A fail here means the homepage is a client component, or its copy arrives from a client-side fetch. Move the text into a server component — this is the single check where the framework is the whole story.
These describe Next.js’s own settings, which move without warning us. Last checked against the product on 2026-08-14.
Mostly. Server components render the copy into the HTML, and the metadata export handles the head. JSON-LD is the gap, and it is the same gap on every Next.js site.
Next.js server-renders what it can. A page marked "use client" whose text arrives from a useEffect fetch renders a shell on the server, and the shell is what the crawler gets.
A route if the content should track the site — the file only stays accurate as long as somebody remembers it exists, and the failure mode is quiet. A static file is fine for a site that rarely changes.
Docusaurus
static/ handles the files, headTags handles the schema, and pre-rendering handles the rest.
Site platformsFramer
The robots.txt setting, custom head code on paid sites, and why /llms.txt needs a proxy.
Site platformsWebflow
The robots.txt setting that only works on a custom domain, where JSON-LD goes, and why /llms.txt needs a proxy.
The readiness check is free, needs no signup, and takes about half a minute. It works out what your site is built on and gives you the fixes written for it.