Category: TOOLS

  • Why Isn’t My Page Indexed? A Diagnostic Guide (and a Free Bulk Checker)

    Why Isn’t My Page Indexed? A Diagnostic Guide (and a Free Bulk Checker)

    Everything in SEO assumes something basic that almost nobody checks: whether Google has actually indexed the page. If it hasn’t, your title tags, schema, backlinks, and content quality don’t matter. The page cannot rank because, as far as Google is concerned, it simply doesn’t exist.

    Indexation problems are quiet. There’s no error message and no notification. A page sits in your sitemap, looks perfectly normal in a browser, and simply never appears in search. On larger sites it’s routine to find hundreds of pages in this state, and most site owners have never looked.

    Our free Bulk Index Checker lets you paste up to 100 URLs and see which are indexed in one pass. This post covers how to read the result and work through the causes in order.

    TL;DR

    A page can’t rank if it isn’t indexed, and indexation failures are usually caused by one of six things: blocked crawling, a noindex directive, a canonical pointing elsewhere, thin or duplicate content, poor internal linking, or simply not having been crawled yet. Check status in bulk with the free Bulk Index Checker or page-by-page in Google Search Console, then address the causes in order, starting with the easiest and most reversible ones first.

    → Check up to 100 URLs at once

    What’s the difference between crawling and indexing?

    Crawling is Google fetching your page; indexing is Google deciding to store it and make it eligible to rank. A page can be crawled and still not indexed, which is the most common and most frequently misread situation. The distinction matters because the fixes are completely different.

    • Not crawled — Google hasn’t fetched the page yet. This is usually an access, discovery, or crawl-budget problem.
    • Crawled, not indexed — Google fetched the page but chose not to store it. This is typically a judgment about quality, duplication, or overall value.
    • Indexed — The page is stored and eligible to rank.

    That middle state is the one that confuses people. “Crawled – currently not indexed” in Search Console is not a bug and not a technical error. It’s Google saying the page didn’t seem worth keeping. No amount of resubmitting fixes it; the page itself has to become worth indexing.

    How do I check whether a page is indexed?

    Three ways: run the URLs through a bulk index checker, use the URL Inspection tool in Google Search Console for a single page, or search site:yourdomain.com/page-url for a rough signal. Search Console is authoritative; bulk tools are faster when you’re checking many pages.

    The practical workflow:

    1. Bulk-check the set — Paste up to 100 URLs into the Bulk Index Checker to find which ones are missing.
    2. Inspect the failures individually in Search Console’s URL Inspection tool to see Google’s specific reason.
    3. Group the failures by reason and fix the underlying patterns rather than working one page at a time.

    A note on the site: operator: it’s a quick sanity check, not a reliable measurement. Results are approximate and the operator behaves inconsistently. Treat an absence as a prompt to check properly, not as proof.

    Cause 1: Google can’t crawl the page

    Check robots.txt and server responses first: if crawling is blocked, nothing downstream matters. This is the fastest thing to rule out and the most problematic when it happens to be the cause.

    What to look for:

    • robots.txt disallow rules blocking the path, often a leftover from staging
    • Non-200 status codes — A page returning 404, 410, or 500 won’t be indexed. Check status codes in bulk with our Redirect & HTTP Status Checker.
    • Login or paywall gating the content
    • Server errors or timeouts during crawl windows (review Search Console’s Crawl Stats report)
    • Content requiring JavaScript to render. Google can render JS, but it is slower and less reliable than plain HTML. AI crawlers also largely skip JS execution, as discussed in our AI crawler post.

    Cause 2: A noindex directive

    Check for a noindex meta tag or X-Robots-Tag header. An explicit noindex is the most common reason a crawlable page stays out of the index. It is also easy to miss because the page looks completely fine in a browser.

    Where noindex hides:

    • The WordPress “Discourage search engines” setting (Settings → Reading), routinely left on after launch
    • SEO plugin defaults applied to a post type, category or tag archive
    • Staging configuration shipped to production
    • X-Robots-Tag HTTP headers set at the server or CDN, invisible in page source

    That last one catches people out. If the HTML has no noindex but the page still won’t index, check the response headers.

    Cause 3: A canonical pointing somewhere else

    If a page canonicals to a different URL, Google treats that other URL as the authoritative version and omits this page from the index. This is working as intended, so the real problem is usually an incorrect canonical tag.

    Common cases: a template bug canonicalising every page to the homepage, pagination canonicalising to page one, parameter URLs canonicalising to a version that no longer exists, or a canonical left pointing at a staging domain after migration.

    Search Console reports this as “Duplicate, Google chose different canonical than user” or “Alternate page with proper canonical tag.” The second is usually fine. The first means Google disagreed with you, which is worth investigating.

    Cause 4: Thin, duplicate or low-value content

    The “Crawled – currently not indexed” status usually means Google decided the page wasn’t worth storing. This often happens because content is thin, near-duplicate, or doesn’t add value beyond existing indexed pages. In short, it is a content issue formatted as a technical status.

    The pattern we see most often in audits:

    • Templated pages at scale — Location or service pages that only swap out a city or service name. These are viewed as doorway pages, and search algorithms enforce quality standards strictly on them.
    • Thin tag and category archives with no unique content
    • Near-duplicate product variants
    • Auto-generated or lightly edited bulk content (covered further in Should You Use AI to Write Your Content?)

    The fix isn’t resubmitting the page. It requires consolidation, clear differentiation, or removal, following the framework outlined in Content Refresh & Pruning.

    Cause 5: The page is orphaned or buried

    Pages without internal links pointing to them are hard to discover and receive no internal authority. This makes Google slower to find them and less inclined to index them. Sitemaps help, but sitemap inclusion alone is not a strong indexing signal.

    Fix by linking to the page from relevant existing content, keeping it within about three clicks of the homepage, and making sure it sits inside a logical section rather than floating. The full approach is in Internal Linking & Site Architecture.

    Cause 6: It just hasn’t been crawled yet

    New pages on sites with lower domain authority can take days or weeks to get indexed, which is standard behavior rather than a technical failure. Before troubleshooting, check when the page was published.

    To speed things up, submit the URL via Search Console’s URL Inspection tool, confirm it appears in your XML sitemap, link to it from frequently crawled pages, and share it on active channels to drive initial traffic. Note that these steps improve discovery, but they do not guarantee indexing if Google decides the content lacks value.

    To speed it up: submit the URL in Search Console’s URL Inspection, make sure it’s in your XML sitemap, link to it from pages that get crawled often, and share it somewhere that generates genuine traffic. None of these guarantee indexing — they improve discovery, not the quality judgement.

    A diagnostic table

    Search Console statusWhat it meansFirst action
    Discovered – currently not indexedKnown but not crawledCheck crawl budget, internal links, server speed
    Crawled – currently not indexedCrawled, judged not worth indexingImprove or consolidate the content
    Excluded by noindex tagExplicit directiveRemove the noindex
    Blocked by robots.txtCrawling blockedFix robots.txt
    Duplicate, Google chose different canonicalGoogle disagreed with your canonicalCheck for near-duplicates
    Page with redirectURL redirects elsewhereExpected — index the destination
    Soft 404Looks empty or error-likeAdd real content or return a proper 404

    What not to do

    • Don’t mass-resubmit URLs. Resubmission doesn’t change a quality judgement, and there are daily limits.
    • Don’t buy indexing services. They mostly submit URLs you can submit yourself, and they don’t fix why Google skipped the page in the first place.
    • Don’t assume every page should be indexed. Thin archives, internal search results and parameter URLs are often better left out. A smaller, higher-quality index is usually better.
    • Don’t panic about a few unindexed pages. Some non-indexation is normal and healthy.

    Frequently Asked Questions

    Why is my page crawled but not indexed? 

    Google fetched the page and decided it was not worth storing, usually because it is thin, near-duplicate, or adds little value beyond what is already indexed. Since this is a quality assessment rather than a technical bug, simply resubmitting the URL will not help. The content needs to be expanded with unique insights or consolidated into another page.

    How long does Google take to index a new page? 

    Anywhere from hours to several weeks, depending on your site’s authority, crawl frequency and internal linking. Established sites with frequent publishing often see indexing within a day; new or low-authority sites routinely wait weeks. Submitting in Search Console and linking internally both help discovery.

    How do I check if my pages are indexed in bulk? 

    Paste up to 100 URLs into a bulk index checker like OptiSEOn’s free tool to see which are missing, then inspect the failures individually in Google Search Console for Google’s own stated reason.

    Does submitting a URL in Search Console guarantee indexing? 

    No. Submitting a URL requests a crawl, but it does not force Google to index it. If Google crawls the page and finds low value, it will still leave it unindexed. Requesting indexing aids discovery, not quality evaluation.

    Should all of my pages be indexed? 

    No. Thin tag archives, internal search results, parameter URLs and paginated series often shouldn’t be. A focused index of genuinely valuable pages tends to perform better than a bloated one, so deliberate exclusions are a sign of good hygiene rather than a problem.

    Can a page be indexed and still get no traffic? 

    Yes, and it’s common. Indexing makes a page eligible to rank; it doesn’t mean it ranks well or that anyone searches for what it covers. If indexed pages get no impressions, the issue is keyword targeting or competitiveness rather than indexation.


    Find out which of your pages are actually indexed. Check up to 100 URLs for free. If a large portion is missing, there is usually a systematic issue behind it. Book a free audit and we will help you fix it.

  • Are AI Bots Actually Crawling Your Site? Your Server Logs Know

    Are AI Bots Actually Crawling Your Site? Your Server Logs Know

    There’s a gap in how almost everyone approaches AI search visibility, and it’s a strange one: we spend enormous effort trying to get cited by ChatGPT, Perplexity and Gemini without ever checking whether their crawlers have visited the site.

    robots.txt tells you what you asked bots to do, but it says nothing about what they actually did. A site can allow every AI crawler and receive almost no AI crawling. A site can block GPTBot and find Bytespider crawling straight through anyway. The only way to know is to check your server logs. Because logs are awkward to work with, hardly anyone checks them.

    Our free AI Crawler Log Analyzer exists for that: upload a server log and see GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest, with hits over time. As far as we can tell nobody else offers this free, which is odd, because it’s the only way to replace guesswork with evidence.

    TL;DR

    Server logs are the only reliable record of which AI crawlers actually visited your site, how often, and which pages they fetched. robots.txt only records your instructions, and analytics doesn’t capture bots at all. Upload a log to the free AI Crawler Log Analyzer to see real activity by user agent over time. Two findings consistently matter: AI crawlers fetch JavaScript but generally don’t execute it, and a meaningful share of requests claiming to be AI bots are spoofed.

    → Upload a log and see your AI crawler activity

    Why can’t I just check robots.txt or analytics?

    Because robots.txt is a request, not a record, and analytics tools deliberately filter out bots — neither one can tell you whether GPTBot visited your site yesterday. Logs are the only source that captures every request, including the ones no JavaScript tag ever fires on.

    Here is what each of the three sources can actually tell you:

    SourceTells youCan’t tell you
    robots.txtWhat you asked bots to doWhether they did it
    Analytics (GA4)Human sessions, some AI referralsBot crawling — bots don’t run JS tags
    Server / CDN logsEvery request, by user agent, with timestampsWhether a crawl led to a citation

    Our free robots.txt AI Crawler Checker handles the first column. It parses your rules the way engines actually resolve them, including the named-group precedence most checkers get wrong. The log analyzer handles the third column. They work as a natural pair: one shows intent, while the other shows reality.

    Which AI crawlers should I look for?

    The crawlers that matter in 2026 split into training crawlers, search indexers and real-time fetchers. The distinction determines whether a visit can ever send you traffic. Blocking the wrong category costs you citations, while allowing the wrong one feeds training models with no return.

    User agentOperatorPurpose
    GPTBotOpenAITraining
    OAI-SearchBotOpenAISearch indexing
    ChatGPT-UserOpenAIReal-time fetch for a user
    ClaudeBotAnthropicTraining
    Claude-SearchBotAnthropicSearch indexing (newer)
    Claude-UserAnthropicReal-time fetch
    PerplexityBotPerplexitySearch indexing
    Perplexity-UserPerplexityReal-time fetch
    Google-ExtendedGoogleGemini training/grounding
    meta-externalagentMetaTraining
    BytespiderByteDanceTraining
    CCBotCommon CrawlOpen training dataset

    A real-time fetch from ChatGPT-User or Perplexity-User is the most interesting line in your log: it means a human asked a question and the assistant went to your page to answer it. That’s a citation in progress.

    What is the crawl-to-refer ratio, and should I care?

    The crawl-to-refer ratio compares how many pages an AI operator crawls against how many visits it sends back. Across 2026, published figures have been extraordinarily volatile, which is precisely why your own data matters more than any industry average.

    Reported ratios for Anthropic’s crawlers across 2026 have ranged from roughly 988:1 to well over 30,000:1 depending on the source, the month and the measurement window. OpenAI’s have similarly swung between the low hundreds and well over a thousand. Google’s traditional search sits around 5:1 by comparison.

    (Those ranges come from Cloudflare Radar-derived reporting and several independent log studies published through 2026. It is worth re-checking these against current sources before quoting specific figures because they move fast.)

    We’re not going to lead with a dramatic single number, because any one of them would be out of date within weeks and would misrepresent how much the picture varies. The genuinely useful conclusions are structural and stable:

    • Training crawlers return almost nothing. They exist to harvest, not to refer.
    • Search and user-action bots do return traffic. They’re a different proposition entirely.
    • The spread between operators is enormous, so a blanket “allow AI bots” or “block AI bots” policy is a blunt instrument.
    • Your own ratio is calculable and is the only number that describes your site.

    To compute yours: count AI bot requests by user agent in your logs over 30 days, then divide by the referral sessions your analytics attributes to each matching platform over the same window. The methodology for the referral side is in how to measure AI search traffic.

    What should I actually look for in the log?

    Look for five main metrics: which AI bots visit, how often they visit, which pages they fetch, which pages they ignore, and whether the pattern is changing over time. The gaps in activity are usually more informative than the hits.

    • Coverage. Are your most important commercial pages being fetched, or only your blog? If AI crawlers never reach your service pages, they can’t cite them.
    • Frequency. Independent log studies in 2026 reported GPTBot revisiting high-traffic pages every couple of days, with ClaudeBot and Google-Extended considerably slower. Your own cadence tells you how quickly content changes get picked up.
    • Status codes served to bots. A crawler receiving 404s or 500s is being told your content doesn’t exist.
    • Absence. No AI crawler hits at all usually means a robots.txt block, a CDN bot rule, or a site too new to be discovered.
    • Trend. One month is a snapshot. The direction of travel is the signal.

    Do AI crawlers execute JavaScript?

    Generally no. Multiple independent analyses in 2026 found that GPTBot, ClaudeBot and PerplexityBot fetch JavaScript files as text but do not execute them, so content that only appears after JS runs is effectively invisible to them. This is one of the most actionable findings available.

    The practical consequence: if your pricing, your service descriptions or your key content render client-side, Googlebot may eventually see it, but AI crawlers largely won’t. A React or Vue site without server-side rendering can be fully indexed by Google and almost entirely opaque to ChatGPT.

    How to check: fetch your own page with JavaScript disabled, or view the raw HTML source rather than the rendered DOM. If the content you want cited isn’t in the raw HTML, AI crawlers aren’t seeing it. The fix is server-side rendering, static generation, or at minimum making critical content available in the initial HTML response.

    How do I know the bot was real?

    Verify by IP, not by user-agent string. Anyone can send a request claiming to be GPTBot, and studies in 2026 estimated that roughly 5–8% of requests claiming to be known AI crawlers are spoofed. User agents are self-declared and trivially faked.

    Verification approaches: the major operators publish IP ranges for their crawlers, and reverse DNS lookup on the requesting IP should resolve to the operator’s domain and forward-resolve back to the same IP. For most businesses, this matters less for analysis than for blocking decisions, but if you’re about to make a policy change based on heavy crawling from one operator, confirm it’s really them.

    What do I do with what I find?

    Three common outcomes and their fixes:

    No AI crawler activity at all. Check robots.txt with our free checker, check for CDN or WAF bot rules, and confirm the site is discoverable. Being uncrawlable makes everything else in AI visibility moot.

    Crawling but no citations. The crawlers can reach you but aren’t finding your content worth quoting. That points to a content structure and authority issue. See how to get cited by ChatGPT, Perplexity & Gemini and check whether AI systems can even identify your brand with the Entity Signal Checker.

    Heavy training crawls, no referrals. This is the value-exchange question. You may decide to block training crawlers while allowing search and user-action bots, which keeps you citable without feeding models for free. Our llms.txt honest guide covers the control surfaces and which of them actually work.

    Where this fits in OptiSEOn’s GEO work

    Log-based crawl verification is part of how we approach LLM and AI search optimization. It’s the first diagnostic we run, because every other AI visibility tactic assumes crawl access that often isn’t there. Proving it with real data rather than estimates is the difference between a strategy and a hope.

    Frequently Asked Questions

    How do I know if GPTBot is crawling my site? 

    Check your server or CDN logs for requests with the GPTBot user agent. This is the only reliable method, since analytics tools filter out bots and robots.txt only records your instructions. Upload a log to the free AI Crawler Log Analyzer to see activity by bot over time.

    Why doesn’t Google Analytics show AI crawler traffic? 

    Because analytics tools rely on JavaScript tags that fire in a browser, and crawlers generally don’t execute JavaScript. GA4 is designed to measure humans and actively filters known bots. Server logs capture every request regardless.

    What is a good crawl-to-refer ratio? 

    There’s no universal benchmark because it depends entirely on which operators crawl you and whether they run consumer-facing products that can send referrals. Published 2026 figures varied enormously by source and month. Calculate your own ratio per operator and track the direction over time rather than comparing to an industry average.

    Do AI crawlers render JavaScript? 

    Generally no. Independent analyses through 2026 found GPTBot, ClaudeBot and PerplexityBot fetch JavaScript files as text without executing them. Content that requires JavaScript to appear is effectively invisible to them, which makes server-side rendering important for AI visibility.

    Should I block AI crawlers? 

    It depends on the category. Training crawlers like GPTBot and ClaudeBot return little or no referral traffic; search and user-action bots like OAI-SearchBot, PerplexityBot and ChatGPT-User can send real visits. Many businesses allow the latter and make a deliberate choice about the former. Check your current rules with our robots.txt checker.

    Can AI bot traffic be faked? 

    Yes. User-agent strings are self-declared and trivially spoofed; 2026 studies estimated roughly 5–8% of requests claiming to be known AI crawlers were not genuine. Verify by IP range or reverse DNS before making policy decisions based on apparent crawl volume.


    Stop guessing whether AI systems can see your site. Upload a server log and find out for free with no signup required. If the answer is “they’re not crawling you,” that’s the first thing to fix, and we can help.

  • Free On-Page SEO Checker: Audit Any Page in Seconds

    Free On-Page SEO Checker: Audit Any Page in Seconds

    Most on-page SEO problems are boring. A missing meta description. Two H1s where there should be one. A canonical pointing at the wrong URL. Images with no alt text. None of it is interesting, all of it is fixable in minutes, and collectively it’s the difference between a page that competes and one that quietly underperforms.

    The trouble is that checking it by hand means opening dev tools, reading source, and knowing what you’re looking for. Most people don’t, so it never gets checked.

    Our free On-Page SEO Checker does that pass for you: enter a URL and it returns title, meta description, heading structure, canonical, HTTP status, image alt text, word count and schema presence. This post explains what it checks, what each result means, and which problems are actually worth your time.

    TL;DR

    An on-page SEO checker inspects a single page’s fundamentals (like title tag, meta description, headings, canonical, status code, alt text, word count, and structured data) and flags what’s missing or misconfigured. The free OptiSEOn On-Page SEO Checker runs all of those on any URL with no signup. On-page issues rarely tank a site on their own, but they’re the easiest wins available and compound across an entire site.

    → Run the free On-Page SEO Checker

    What is on-page SEO?

    On-page SEO covers everything on the page itself that influences how search engines understand and rank it, including the title, headings, content, internal links, images, structured data, and HTML technical signals. It’s distinct from off-page SEO (links and authority from elsewhere) and site-wide technical SEO.

    The useful framing: off-page SEO is other people’s opinion of you, technical SEO is whether machines can access you, and on-page SEO is whether a given page clearly says what it’s about. All three matter, but on-page is the only one you can fix entirely by yourself this afternoon.

    What does the checker actually look at?

    It fetches the page and reports eight key areas: title tag, meta description, heading structure, canonical tag, HTTP status code, image alt text, word count, and whether schema markup is present. Each result is either a clear status or a metric you can act on directly.

    CheckWhat it tells you
    Title tagPresent, length, and whether it reads like a real title
    Meta descriptionPresent and within a sensible length
    HeadingsH1 count and the H2/H3 hierarchy underneath
    CanonicalWhether one exists and where it points
    HTTP statusWhether the URL returns 200 or something else
    Alt textHow many images have it and how many don’t
    Word countRough content depth signal
    SchemaWhether structured data is present on the page

    Why do titles and meta descriptions matter most?

    Because they’re the main elements a searcher sees before deciding to click; a page can rank well and still lose traffic to a more compelling result below it. Titles also directly influence rankings. Meta descriptions don’t affect rankings directly, but they heavily impact your click-through rate.

    Common failures the checker catches:

    • No title at all, or a CMS default like “Home” or “Untitled”
    • Duplicate titles across pages, which muddies which page should rank for what
    • Missing meta descriptions, leaving Google to pull an arbitrary snippet
    • Titles too long and truncating mid-phrase

    On length: Google truncates titles by pixel width, not character count, so relying solely on a 60-character limit can leave text cut off if you use wider letters. Our Title & Meta Description Generator measures the actual rendered width and shows a live SERP preview so you can refine your snippet accurately.

    Why does heading structure matter?

    Headings lay out a page’s structure for both readers and search engines. A clear hierarchy with one primary H1 and descriptive H2s makes your content easy to parse. This is especially important today, as AI search systems rely on clear headings to extract direct answers.

    What the checker flags:

    • No H1, or several H1s competing
    • Skipped levels, such as jumping straight from an H1 to an H3
    • Headings used for styling rather than structure, which is common in page builders
    • Vague headings that describe nothing (“Our Approach”)

    The practical upgrade: phrase H2s as the questions people actually ask, then answer each in the first 40–60 words beneath it. That’s the structure that wins featured snippets and AI citations, and it’s covered properly in our featured snippets and AI Overviews guide.

    What is a canonical tag and why does it matter?

    A canonical tag tells search engines which URL represents the main version of a page, preventing duplicate or similar URLs from competing with each other. An incorrect canonical tag is one of the few on-page errors that can cause a page to be deindexed.

    The failure modes worth knowing:

    • Canonical pointing to the wrong page, often caused by a template bug where every page points to the homepage and effectively hides the rest of the site from search
    • Canonical pointing at a staging domain after a migration
    • Missing canonical on sites that generate parameter URLs
    • Canonical conflicting with a redirect or a noindex

    If the checker shows a canonical pointing somewhere unexpected, treat it as urgent rather than routine. It’s in the same class of problem as the ones in Why Did My Organic Traffic Drop?.

    Does word count affect rankings?

    No. Word count isn’t a direct ranking factor, and there is no target number to hit. It simply indicates whether a page covers its topic thoroughly compared to competing pages. A concise 400-word page that answers a query directly can easily outperform an artificially padded 2,500-word article.

    Use the number comparatively rather than absolutely. If every page-one result for your target query runs to 2,000 words of genuine depth and yours is 300, the gap is probably real. If you’re at 300 words because the question needs 300 words, leave it alone.

    Does schema markup show up in this check?

    The checker indicates whether structured data is present on the page, offering a quick way to spot pages that lack it entirely. Generating and validating the markup code itself can then be handled separately.

    Schema is the bridge between traditional SEO and AI visibility: it tells both Google and AI systems what a page is in machine-readable terms. If the checker reports none, our free Schema Markup Generator builds valid JSON-LD from the page’s real content, and 10 Schema Markup Types Every Business Needs covers which types belong where.

    What this checker doesn’t do

    Worth being straight about the boundaries, because on-page tools are routinely oversold:

    • It checks one page at a time, not your whole site. Site-wide crawling is a different job.
    • It doesn’t measure performance. Core Web Vitals need PageSpeed Insights and field data — see our Core Web Vitals guide.
    • It doesn’t assess content quality. No tool can tell you whether your page is actually useful.
    • It doesn’t check links or authority. That’s the Backlink Profile Checker.
    • A clean report doesn’t mean you’ll rank. On-page fundamentals are table stakes, not an advantage.

    Ultimately, resolving on-page issues removes technical barriers; it doesn’t automatically guarantee top rankings. Sustainable search visibility depends on delivering helpful content, demonstrated expertise, and strong site authority over time.

    Where to start if everything’s broken

    Priority order when a check comes back with a long list:

    1. Status code and canonical problems (these can prevent pages from appearing in search results entirely)
    2. Missing or duplicate titles (these directly impact rankings and click-through rates)
    3. Missing H1 or broken hierarchy (these hurt readability and content extraction)
    4. Missing meta descriptions on pages already receiving search impressions (a fast win for click-through rate)
    5. Missing schema on key templates
    6. Alt text (important for accessibility first, and image search second)
    7. Thin content (requires the most effort to expand and refine)

    Run the checker on your five highest-value commercial pages first. Those are where fixes pay back immediately.

    Frequently Asked Questions

    How do I check the SEO of a web page? 

    Run the URL through an on-page SEO checker, which inspects the title, meta description, headings, canonical tag, HTTP status, image alt text, word count and schema. OptiSEOn’s free checker covers all of these with no signup, and Google Search Console shows how the page actually performs in search.

    What should an on-page SEO audit include? 

    At minimum: title and meta description, heading hierarchy, canonical tag, HTTP status, internal links, image alt text, content depth, and structured data. A fuller audit adds page speed, mobile rendering, and whether the page matches the search intent it targets.

    How long should a title tag be? 

    Google truncates titles by pixel width rather than character count (roughly 580 pixels, or about 50 to 60 standard characters). Measuring rendered width provides a clearer preview; our meta generator calculates this automatically.

    Does every page need schema markup? 

    Most pages benefit. At minimum, use Organization schema site-wide, Article on content pages, LocalBusiness on contact pages, and Product on product pages. Schema doesn’t rank pages directly but it helps search engines and AI systems identify what a page is and makes rich results possible.

    How many H1 tags should a page have? 

    One. The H1 states what the page is about; multiple H1s dilute that signal and usually indicate headings being used for styling rather than structure. Use H2 and H3 for the hierarchy beneath it.

    Will fixing on-page SEO improve my rankings? 

    It eliminates optimization barriers rather than guaranteeing higher positions. Correcting a broken canonical or adding a title tag can yield quick improvements, whereas adding alt text to an established page may have a subtle impact. On-page optimization creates a solid foundation, while overall content quality and authority drive ranking growth.


    Check any page in seconds. Run the free On-Page SEO Checker, starting with your highest-value commercial pages. If you have a large list of issues and want expert help resolving them, OptiSEOn is a Dallas-based SEO, AEO, GEO, and LLM optimization agency: book a free audit.

  • Free Title & Meta Description Generator: Built From Your Page, Measured in Pixels

    Free Title & Meta Description Generator: Built From Your Page, Measured in Pixels

    Title tags and meta descriptions are the highest-leverage fifty words on any web page. They’re what someone actually reads in the search results before deciding whether to click your link or the result below it, and improving them can lift click-through rate on pages whose rankings never move at all.

    They’re also the element most commonly left to whatever a CMS auto-generates, or written to a character count that doesn’t reflect how Google actually truncates.

    Our free Title & Meta Description Generator addresses both problems: it builds candidates from your page’s real content and measures them in pixels rather than characters, because pixel width is what determines truncation.

    TL;DR

    Title tags and meta descriptions determine whether people click your result, and Google truncates them by pixel width rather than character count, which is why character-based guidance is unreliable. The free OptiSEOn generator reads your page’s H1, headings, schema, location, and opening paragraph to assemble candidates, measures against a 580px title and 920px description budget using Arial advance-width metrics, and includes a live SERP preview with an editor that re-measures as you type. Nothing is invented.

    → Generate titles and meta descriptions

    Why does pixel width matter more than character count?

    Google truncates titles and descriptions based on how wide they render, not how many characters they contain, and characters have wildly different widths. A title of sixty W’s is vastly wider than sixty i’s, so any “keep it under 60 characters” rule is an approximation that fails on real titles.

    This is why two titles of identical character length can behave differently in the SERP: one displays in full, the other gets cut mid-word with an ellipsis. Capital letters, wide characters like W and M, and narrow ones like i, l, and t all shift the real width substantially.

    The generator measures against a 580px title budget and 920px description budget using Arial advance-width metrics, matching the way Google actually renders and truncates. You see whether your title fits, not whether it satisfies a rule of thumb.

    What does the generator do?

    It reads the page at the URL you enter and assembles title and description candidates from that page’s own H1, headings, schema, location data, and opening paragraph, then measures each in pixels and shows them in a live SERP preview. Nothing is invented, and you can edit in place with the measurement updating as you type.

    What that means practically:

    • Built from your real content. Candidates come from what’s genuinely on the page, not from a model guessing what your business does.
    • Pixel-measured. Both title and description measured against Google’s rendering budget.
    • Live SERP preview. See how the result will actually look.
    • Editable with live re-measurement. Adjust wording and watch the pixel width update.
    • Free, no signup.

    The “nothing invented” constraint is deliberate, and it’s the same principle behind our schema markup generator and entity signal checker. A tool that fabricates a plausible-sounding description for a page it didn’t really read produces meta descriptions that misrepresent the page — which raises click-through briefly and bounce rate permanently.

    What makes a good title tag?

    A good title tag includes the primary keyword near the front, describes the page accurately, fits within the pixel budget, and gives someone a reason to click rather than just naming the topic. It’s a ranking element and an advertisement simultaneously.

    The rules that hold up:

    • Primary keyword early. Front-loading matters for both relevance and scannability.
    • Fit the pixel budget so it doesn’t truncate mid-phrase.
    • Be specific. “Local SEO Guide” loses to “Local SEO for Dallas Businesses: 2026 Map Pack Playbook.”
    • Unique per page. Duplicate titles across pages are a cannibalization signal (see our internal linking guide).
    • Brand at the end, if at all, on content pages. Front-loading your brand wastes the most valuable pixels unless the brand is the draw.
    • Match search intent. If the SERP is full of guides, “Guide” belongs in your title.

    What makes a good meta description?

    A good meta description summarises what the page delivers, includes the keyword naturally, fits the pixel budget, and gives a concrete reason to click. While it doesn’t directly affect rankings, it strongly affects click-through rate. Treat it as ad copy that happens to sit in search results.

    Worth knowing: Google rewrites meta descriptions frequently (often more than half the time) when it judges that a different snippet better matches the query. That’s not a reason to skip writing them. A well-written description is used more often than a poor one, and when Google does rewrite, it pulls from your page content, so well-structured content improves the substitute too.

    What works:

    • Deliver on the title’s promise with specifics
    • Include the keyword naturally (it gets bolded when it matches the query)
    • Give a concrete reason to click by highlighting what readers will learn or be able to do.
    • Active voice, no filler. “Learn about our services” wastes the whole budget.
    • Unique per page

    Why do my meta descriptions matter if Google rewrites them?

    Click-through rate is a real competitive lever, and a description Google keeps is better than one it replaces. Plus, the pages Google rewrites most are usually the ones whose descriptions were weakest to begin with. Writing them well raises the floor either way.

    There’s also a compounding angle. Improving CTR on pages that already rank is one of the fastest wins in SEO, because you’re not waiting for ranking movement at all. A page sitting at position 4 with a weak description is leaving clicks on the table today, and the fix takes ten minutes.

    To find those pages: in Search Console, sort by impressions and look for pages with high impressions and low CTR relative to their average position. Those are your rewrite candidates.

    Does this affect AI search too?

    Indirectly. AI systems read your page content rather than your meta description, but clear titles and well-structured pages help them identify what a page is about, and the same discipline that produces a good title produces a clear H1 and heading structure. The description itself isn’t the lever; the clarity behind it is.

    The thing that genuinely drives AI citation is the structure covered in how to get cited by ChatGPT, Perplexity & Gemini: question-format headings with direct answers underneath. Titles and descriptions are for humans in the SERP. Both matter, for different reasons.

    Frequently Asked Questions

    How long should a title tag be? 

    Google truncates by pixel width rather than character count, with a title budget of roughly 580px. That works out to somewhere around 50–60 characters for typical text, but wide characters consume the budget faster. Measuring in pixels, as our generator does, is more reliable than counting characters.

    How long should a meta description be? 

    Around 920px of rendered width, which is roughly 150–160 characters of typical text but varies with the actual characters used. Measure rather than count, and front-load the important information so that truncation doesn’t remove your reason to click.

    Do meta descriptions affect rankings?

    Not directly, as Google has stated meta descriptions aren’t a ranking factor. They affect click-through rate, which affects how much traffic a given ranking produces. Improving descriptions on pages that already rank well is one of the fastest available traffic wins.

    Why does Google rewrite my meta description? 

    Google substitutes its own snippet when it judges that text from your page better matches the specific query. This happens frequently. Writing good descriptions still helps, because strong ones are kept more often, and Google’s substitutes are drawn from your page content, so clear content improves both outcomes.

    Should every page have a unique title and meta description? 

    Yes. Duplicate titles and descriptions across pages make it harder for search engines to distinguish them and can contribute to cannibalization. Every indexable page should have its own, written for that page’s specific intent.

    Is the OptiSEOn title and meta generator free? 

    Yes, with no sign-up. Enter a URL, and it assembles candidates from that page’s own H1, headings, schema, location, and opening paragraph, measures them in pixels against Google’s truncation budgets, and shows a live SERP preview you can edit. It’s one of eleven free tools we build.


    Fix the fifty words that decide whether people click. Use the free Title & Meta Description Generator, starting with your highest-impression, lowest-CTR pages. Want the full list of which pages those are? Book a free audit.

  • Free Backlink Profile Checker: See Your Links, and Whether They’re Improving

    Free Backlink Profile Checker: See Your Links, and Whether They’re Improving

    Most backlink tools answer one question: how many links do you have right now? That’s a snapshot, and a snapshot is a surprisingly weak thing to make decisions from.

    The question that actually matters is whether your profile is getting stronger. Are new referring domains arriving? Is quality improving or is the growth all low-value noise? Is too much of your authority concentrated in a handful of domains that could disappear? A single number can’t tell you any of that.

    So our free Backlink Profile Checker is built around progress rather than snapshots. It scores your profile out of 100, breaks down the quality mix and concentration risk, and shows you exactly what changed between checks.

    TL;DR

    A backlink profile checker analyses the links pointing to your site: how many referring domains you have, their quality mix, your spam score, and how concentrated your authority is. The free OptiSEOn Backlink Profile Checker scores all of this out of 100 and tracks progress between checks, so you can see whether your link building is actually working. No signup.

    → Run the free Backlink Profile Checker

    What is a backlink profile?

    Your backlink profile is the complete picture of links pointing to your site. This includes how many distinct domains link to you, their authority, topical relevance, and how links are distributed across your pages. Quality and distribution matter far more than raw link counts.

    The components that matter:

    • Referring domains: The number of distinct sites linking to you. This is far more meaningful than total link count, as 50 links from 50 sites build a stronger profile than 500 links from just 5 sites.
    • Quality mix: The spread of authority across your referring domains.
    • Spam score: The proportion of links coming from low-quality or spammy sources.
    • Concentration risk: How much of your overall authority relies on a small cluster of domains.
    • Growth trend: Whether your link profile is growing, holding steady, or losing ground.

    What does the OptiSEOn Backlink Profile Checker show?

    It returns your referring domains, link quality mix, spam score and concentration risk, scored out of 100, plus progress tracking that shows exactly what changed since your last check. Enter a domain and it runs; no signup required.

    Progress tracking is central to how this tool works. Effective link building takes time, as outlined in our link building guide. A single snapshot can’t show if a campaign is making steady progress or stalling out, but tracking changes over time clarifies your actual trajectory.

    Note on the data: like most free checkers, this tool uses third-party index data via an API rather than crawling the entire web directly. Because no backlink index is exhaustive and providers measure differently, treat absolute figures as estimates. The real value lies in tracking relative changes for the same domain over time using consistent data.

    Why do referring domains matter more than total backlinks?

    Search engines place much higher weight on unique websites linking to you than on total individual links. A hundred links from one site count as a single endorsement. Broad source diversity signals genuine authority.

    This is also why “we’ll build you 500 backlinks a month” offers are meaningless at best. Volume from a narrow set of low-quality sources doesn’t move the metric that matters, and in 2026 it carries real penalty risk. Google’s SpamBrain evaluates link patterns across your profile rather than links in isolation.

    What is concentration risk?

    Concentration risk measures how dependent your site is on a handful of referring domains. If a major share of authority comes from just three or four sites, losing any of them directly harms your profile. It evaluates profile vulnerability rather than raw quality.

    High concentration typically comes from a single viral piece picked up by one publisher network, a long-standing directory or partner relationship carrying most of your links, or an over-reliance on one guest-posting outlet. None of those are bad in themselves; depending on them is.

    The fix is straightforward and slow: diversify sources. Original research, digital PR, expert commentary, podcast appearances, and genuine community presence spread your profile across more domains.

    What is a good backlink profile score?

    There is no universal score to aim for. Look at your profile relative to competing sites ranking for your target terms, and focus on whether your score trends upward over time. Benchmarks vary widely by industry.

    Run the checker on your top three competitors alongside your own domain. That comparison tells you more than any score in isolation: if they have four times your referring domains, you know the gap you’re closing. If you’re comparable and still outranked, links aren’t your problem and you should look at content, technical health, or entity signals instead.

    Should I disavow links the checker flags as spammy?

    In most cases, no. Google advises against disavowing links unless you face a manual penalty or clear manipulation. Search engines generally ignore low-quality links automatically. Disavowing valid links risks removing genuine authority.

    Disavow when: you have a manual action for unnatural links, you previously bought links or used a PBN, or you’re facing a clear negative SEO attack. Otherwise, leave it alone and spend the effort earning better links instead.

    How does this fit with the rest of your SEO?

    Links are one input among several. A strong profile won’t rescue thin content, and a weak one won’t sink genuinely excellent content in a low-competition niche. Check your backlink profile alongside:

    • Domain Rating: A relative authority metric rather than an official ranking factor.
    • Entity signals: How clearly search engines and AI models identify your brand.
    • Core Web Vitals: Page performance and technical site health.
    • Content quality and structure: The foundational content that backlinks serve to amplify.

    For businesses that need authority built rather than measured, our citation and authority building service handles consistent listings and entity signals across 260+ platforms.

    Frequently Asked Questions

    How do I check my backlinks for free? 

    You can use a free backlink checker like OptiSEOn’s. Enter your domain to review referring domains, quality distribution, spam score, and concentration risk without creating an account. Google Search Console also provides a sample of indexed backlinks in its Links report.

    What’s the difference between backlinks and referring domains? 

    Backlinks are individual links, while referring domains are the distinct websites providing those links. Referring domains carry much more weight because multiple links from a single site act as one overall endorsement.

    How many backlinks do I need to rank? 

    There is no single target number. Compare your referring domain count against competitors ranking for your target terms to set a practical baseline.

    Why do different backlink tools show different numbers? 

    Each tool relies on its own web crawler and index. Crawl frequencies and filtering rules vary by provider, so treat raw counts as estimates and focus on performance trends over time.

    Should I remove or disavow bad backlinks? 

    Usually no. Google recommends disavowing links only if you have a manual penalty or clear evidence of unnatural link building. Most low-quality links are ignored automatically.

    How often should I check my backlink profile? 

    Checking monthly is sufficient for most sites since link profiles change gradually. Monitor more frequently during active outreach or PR campaigns when new link acquisition is expected.


    See where your backlink profile stands and track its progress over time. Run the free Backlink Profile Checker, then compare your domain against top competitors. If you need help closing the gap, book a free audit with our team.

  • Free AI Citation Tester: Does ChatGPT Actually Recommend Your Brand?

    Free AI Citation Tester: Does ChatGPT Actually Recommend Your Brand?

    Here is a question many business owners haven’t considered yet: if a potential customer asks ChatGPT to recommend a company in your industry, will your business appear in the response?

    Beyond your Google search rankings or content quality, consider whether an AI assistant would actively suggest your brand when answering a buyer’s question.

    Many businesses don’t know the answer, which creates a significant blind spot. Potential buyers regularly consult AI platforms like ChatGPT or Perplexity for recommendations before visiting traditional search engine results. If your company isn’t included in those responses, you miss out on potential leads without seeing any lost traffic in your analytics.

    The free OptiSEOn AI Citation Tester helps solve this. It creates prompt sets based on actual buyer queries, lets you run them in ChatGPT, Perplexity, Claude, and Google AI with one click, and tracks your citation rate over time. Below, we explain how it operates, the design choices behind it, and how to use the findings.

    TL;DR

    An AI citation test checks if AI tools mention your business when answering common questions from potential customers. The free OptiSEOn AI Citation Tester generates a custom prompt set, connects you to ChatGPT, Perplexity, Claude, and Google AI, and helps track your citation rate using live responses you review directly without relying on simulated data. Results stay stored in your browser. If competitors are featured instead of your brand, strengthening entity signals, citable content, and external authority can help close the gap.

    → Build your free prompt set

    What is an AI citation test?

    An AI citation test evaluates whether AI assistants reference or cite your brand in response to customer questions, helping you track your visibility rate over time. It functions much like tracking keyword rankings in traditional search, but focuses on measuring brand inclusion within generated answers.

    Two outcomes worth distinguishing:

    • Mention when an AI includes your brand name in its output (for example, listing your company alongside competitors)
    • Citation when an AI links directly to your website as a source for its answer

    Both outcomes are valuable. A brand mention ensures you are considered during a buyer’s evaluation, while a direct citation can drive referral traffic and demonstrate that the AI views your site as an authoritative reference. We cover these distinctions further in our guide on getting cited by ChatGPT, Perplexity, and Gemini.

    What does the OptiSEOn AI Citation Tester do?

    The tool constructs realistic buyer prompts, lets you launch them into ChatGPT, Perplexity, Claude, and Google AI with one click, and calculates your brand mention rate based on live responses. Prompts are not submitted automatically by the site; you run each query and record the output you observe.

    The workflow:

    1. Build your prompt set. The tool generates the questions your buyers would realistically ask an AI assistant — the conversational, full-sentence kind, not keywords.
    2. Launch each prompt into ChatGPT, Perplexity, Claude, and Google AI with one click per platform.
    3. Log your results to track whether your brand was mentioned or cited, and note which competitors appeared.
    4. Track your citation rate over time as you do the work to improve it.

    Your results are stored in your browser only. No signup, nothing sent anywhere.

    Why do you make me run the prompts myself?

    Running automated queries across AI platforms incurs API costs, and platform terms restrict re-distributing answers as an automated ranking product. As a result, tools offering instant results often generate simulated responses rather than checking live AI models. Relying on simulated outputs can lead to misleading conclusions when planning strategy.

    This approach is central to how the tool was designed. Some AI monitoring software pays for direct API access, while other tools produce estimated outputs that model what an AI might say rather than pulling a live answer.

    Baseing strategic decisions on estimated data risks optimizing for simulated metrics while real-world visibility stays unchanged.

    We designed the tool so you can check live answers directly on each platform within minutes, giving you accurate feedback on what prospective customers see.

    This matches the approach of our Entity Signal Checker, which only scores signals verified against live sources.

    What prompts should I test?

    Focus on full-sentence queries that buyers naturally ask, including category questions, problem descriptions, product comparisons, and branded searches. Natural language prompts often differ significantly from traditional search keywords.

    The difference in practice:

    Google queryActual AI prompt
    dallas seo agencyWho’s a good SEO agency in Dallas for a small B2B company?
    best crmWhat CRM should a 12-person sales team already using HubSpot switch to?
    plumber near meMy water heater is leaking in North Dallas. Who should I call?

    The four prompt types worth covering:

    • Category prompts such as “Best [category] in [location]” or “Best [category] for [audience]” measure your market share of voice.
    • Problem prompts focus on how customers describe an issue before identifying specific service providers.
    • Comparison prompts evaluate queries like “Option A vs Option B” or “alternatives to [competitor].”
    • Branded prompts such as “Is [your brand] reliable?” measure how AI models describe your business specifically, covering both visibility and reputation.

    Branded searches often highlight unexpected issues. If an AI system misrepresents your services, location, or company details, that inaccuracy can impact potential sales. Incomplete or inconsistent entity signals are frequently the root cause.

    For additional strategies on building prompt banks alongside keyword research, see our 2026 keyword research guide.

    How often should I test?

    We recommend running tests weekly while actively optimizing for AI search, or monthly for ongoing monitoring. AI responses evolve as models update and competitor content changes, so tracking ongoing trends provides the clearest picture. Focus on overall performance over time rather than single test runs.

    Regular testing is especially helpful for two reasons:

    • Model updates alter outputs. An AI platform that mentions your business today might adjust its answers after an update, even if your site content stays the same.
    • Generative answers vary. The same query can yield slightly different responses across separate runs. A single test represents a point in time rather than a permanent result.

    Establish a baseline over a few weeks to track long-term progress. For details on setting up GA4 tracking for AI referral traffic, review our guide on measuring AI search traffic from ChatGPT, Perplexity, and Gemini.

    What do I do if competitors are named and I’m not?

    When AI platforms feature competitors over your brand, it usually stems from three areas: weak entity signals, unstructured content, or limited third-party mentions. Addressing these elements in order helps improve overall brand visibility.

    Diagnosing which one is your problem:

    • Weak entity signals make it harder for AI models to verify your business details. Run a test with our Entity Signal Checker to review Organization schema, sameAs links, NAP consistency, and Wikidata profiles.
    • Unstructured content can prevent AI models from extracting direct answers. Formatting content with clear question-based headings helps; see our guide on getting cited by ChatGPT, Perplexity, and Gemini along with our featured snippets guide.
    • Limited third-party mentions can restrict how often AI models reference your business. If AI responses in your industry pull data from review platforms, directory sites, or forum discussions, building an active presence on those channels can help. Read our guide on using Reddit for AI citations and explore our citation building services.

    First, verify that AI web crawlers can access your site. If your robots.txt file blocks bots like GPTBot or PerplexityBot, AI engines will not index your content. Test your file using our free robots.txt AI Crawler Checker.

    It is also helpful to review which external sources AI platforms cite when recommending competitors. This highlights key directories and publications in your industry.

    Understanding what an AI citation test reveals

    A citation test shows how your brand performs for specific prompts at a given moment, rather than providing a single fixed “AI rank.” AI answers are generated dynamically for each prompt, vary over time, and differ across platforms.

    The test offers useful insights by establishing a practical baseline, tracking visibility trends over time, highlighting featured competitors, and revealing how AI platforms describe your brand. Tracking these directional trends helps guide strategy effectively.

    Frequently Asked Questions

    How do I check if ChatGPT mentions my business?

     Query ChatGPT using natural prompts that your buyers might use, including category searches like “best [category] in [city]” and branded questions like “is [brand] reliable?” to see if your company appears. The free OptiSEOn AI Citation Tester generates relevant prompt sets and connects you directly to ChatGPT, Perplexity, Claude, and Google AI.

    What is the difference between an AI mention and an AI citation?

     An AI mention occurs when a platform includes your business name in a response. An AI citation happens when the platform links directly to your website as a source. Citations can drive referral traffic and demonstrate domain authority, while brand mentions help keep your business on a buyer’s shortlist.

    Why doesn’t the tool run AI queries automatically?

     Running automated API queries across AI platforms adds service costs, and platform terms restrict redistributing answers in automated rank-tracking products. Tools promising automated instant data often rely on simulated answers rather than live checks. Reviewing real prompts directly provides reliable data you can trust.

    How often should I test my AI citation rate?

     We suggest testing weekly if you are actively working on AI optimization, or monthly for ongoing monitoring. Because generative outputs can shift as platform models update, establishing a baseline over several weeks offers a clearer picture than relying on a single test run.

    Why do competitors show up in ChatGPT while my brand doesn’t?

     This typically relates to three factors: weak entity signals that make your brand harder to verify, content that is not formatted for direct extraction, or limited presence on third-party platforms referenced by AI models. Reviewing entity signals, content structure, external mentions, and your robots.txt access can help address these gaps.

    Can I improve my brand’s AI citation rate?

     Yes. Improving AI visibility involves strengthening entity signals (such as schema and directory profiles), formatting content for easy extraction, and building mentions across external sources trusted in your industry. Initial improvements often appear within 60 to 90 days, with broader gains taking 6 to 12 months.

    Is the OptiSEOn AI Citation Tester free?

     Yes, it is free with no registration required. The tool builds prompt sets, connects you to ChatGPT, Perplexity, Claude, and Google AI, and saves your findings locally in your browser. It is part of our collection of free SEO tools.


    See if AI platforms recommend your business using live responses. Build your free prompt set. If competitors are featured over your brand, OptiSEOn can help close the visibility gap. We are a Dallas-based agency offering SEO, AEO, GEO, and LLM optimization services. Schedule a free GEO strategy session.

  • Free Entity Signal Checker: Can AI Systems Actually Identify Your Brand?

    Free Entity Signal Checker: Can AI Systems Actually Identify Your Brand?

    There is a common issue in AI search that frequently gets misdiagnosed. A business might do everything right, including creating strong content, setting up clean technical SEO, and achieving high search rankings, yet still get passed over when someone asks ChatGPT or Perplexity for a recommendation in their industry. The default reaction is to assume the content needs work, but that is rarely the underlying issue.

    The real issue is that the AI cannot confidently figure out what your brand is. Beyond what a specific webpage states, AI models need to understand your company as a distinct, verifiable organization. If an AI system is unsure whether your business is legitimate or fits the requested category, it simply will not recommend you. It has no reason to risk giving an inaccurate answer based on an unverified brand.

    That confidence comes from entity signals: your Organization schema, your linked profiles, your Wikipedia and Wikidata presence, your consistent name-address-phone data. Most businesses have never audited them, because there was never an obvious reason to.

    To address this, we built a free Entity Signal Checker that checks these signals against real data sources and highlights which gaps to address first. Below, we break down what entity signals are, why AI platforms rely on them so heavily, and how to evaluate your results.

    TL;DR

    Entity signals are verifiable data points, such as Organization schema, sameAs links, Wikipedia and Wikidata entries, and consistent NAP details, that allow search engines and AI models to identify your brand as a real, distinct entity. The free OptiSEOn Entity Signal Checker verifies each signal against live sources (including site markup and the Wikipedia and Wikidata APIs), prioritizes fixes, and avoids scoring signals that cannot be verified. It requires no signup and runs live lookups.

    → Run the free Entity Signal Checker · Try stripe.com, notion.so, or your own domain.

    What is entity SEO?

    Entity SEO focuses on establishing your brand as a distinct, verifiable entity for search engines and AI systems, rather than treating your website as merely a collection of keyword-rich pages. It marks the difference between a search engine recognizing that your pages mention “Dallas SEO agency” and recognizing that your company actually is one.

    Search engines have operated on this principle since Google launched the Knowledge Graph in 2012, organizing the web around connected real-world entities like people, places, organizations, and concepts instead of standalone documents. Modern AI systems have expanded on this approach, since delivering direct recommendations requires far greater certainty than presenting a standard list of links.

    What are entity signals?

    Entity signals are specific, machine-verifiable data points that confirm your brand’s identity online. These include structured data defining your business, authoritative external references, linked profiles, and consistent NAP details across directory listings. While single signals are small on their own, combined they form a fully verified digital identity.

    The signals that matter most, and what each one does:

    SignalWhat it establishes
    Organization schemaDeclares in machine-readable terms who you are, what you do, and how to identify you
    sameAs linksConnects your website to your LinkedIn, X, Crunchbase, and other profiles as one entity
    Wikipedia presenceThe single strongest third-party confirmation available (if you meet notability rules)
    Wikidata entryStructured, machine-readable entity data — heavily used by AI systems
    Consistent NAPName, address, phone matching everywhere confirms you’re a real, locatable business
    llms.txtAn emerging format that outlines your site structure for AI crawlers (currently carries low impact, as noted below)

    Why do AI systems care about entity signals?

    Because AI systems provide direct recommendations rather than simple link lists, recommending a business requires confidence that the company exists, is legitimate, and fits the user’s intent. Entity signals build that confidence by enabling independent verification across multiple sources.

    For instance, when a user asks for “a good SEO agency in Dallas,” an AI model must verify which organizations are real businesses, located in Dallas, offering SEO services, and backed by sufficient evidence to justify a recommendation. A brand with complete Organization schema, a Wikidata page, consistent NAP listings, and linked profiles is straightforward to verify. Conversely, a brand lacking those signals looks like unverified text on a page, making it hard to distinguish from a personal blog or an inactive business.

    This is also why ambiguity is so costly. If your brand name is shared with other companies, or you use different name variations across platforms, AI systems can’t confidently attribute anything to you. Entity clarity isn’t a nice-to-have; it’s the precondition for being recommended at all.

    How does the OptiSEOn Entity Signal Checker work?

    The tool analyzes key identity signals, including Organization schema, sameAs links, Wikipedia, Wikidata, NAP details, and llms.txt, validating each against live sources and ranking issues by priority. Data is pulled directly from your site’s markup or via official Wikipedia and Wikidata APIs, offering real-time checks with no account required.

    What it does specifically:

    • Reads your site’s schema markup to confirm Organization schema is present and properly formed
    • Checks your sameAs links to ensure your website properly links to your official profiles
    • Queries the Wikipedia and Wikidata APIs to verify real presence rather than guessing
    • Checks NAP details for the consistency AI systems rely on
    • Checks for llms.txt as a supplementary signal
    • Prioritizes missing elements so you can focus on the most impactful fixes first

    Why doesn’t the tool score everything?

    Certain signals, such as Google Knowledge Panels or Crunchbase listings, cannot be verified without automated scraping. The tool highlights these for manual review instead of generating arbitrary scores. Presenting estimated data as verified fact can lead to misinformed decisions.

    We chose this approach intentionally. Many tools estimate or approximate unverified signals and bundle them into an overall score without clarifying which data points are actually confirmed. That can give a false impression of accuracy based on partial guesswork.

    We prefer to focus on verified data. Signals that can be confirmed directly are included in the score, while unverified items are flagged for manual review to keep the analysis clear and reliable.

    This same focus on accuracy applies to our AI Citation Tester, which lets you run live prompts directly instead of relying on simulated responses, as well as our robots.txt AI Crawler Checker, which evaluates crawler precedence rules accurately.

    How do I improve my entity signals?

    Start by implementing Organization schema and sameAs links on your website, establish consistent NAP details across web directories, create a Wikidata entry, and evaluate Wikipedia eligibility if applicable. Following this order lets you complete quick, controllable setup tasks before moving on to longer-term authority building.

    The practical sequence:

    1. Add complete Organization schema across your entire site, including your brand name, URL, logo, description, contact details, and address. You can generate JSON-LD markup with our free schema generator and review our schema guide for best practices.
    2. Include sameAs links in your Organization schema pointing to your official profiles on platforms like LinkedIn, X, Facebook, Instagram, Crunchbase, G2, and YouTube. This simple step helps connect your web presences into a single recognized entity.
    3. Standardize NAP details across all platforms. Use a single format for your business name, address, and phone number across your website, Google Business Profile, and directories. Minor variations, such as switching between “Suite 100” and “Ste 100,” can create confusion. Check our GBP guide for local optimization tips.
    4. Use one canonical brand name. Alternating between your full legal name, an abbreviation, and a stylized variant actively confuses entity recognition. Pick one and use it consistently everywhere.
    5. Create a Wikidata entry if your business does not have one yet. Wikidata has accessible entry guidelines, allows community contributions, and is regularly referenced by AI engines, making it an efficient way to strengthen your entity signals.
    6. Build out supporting profiles on sites like Crunchbase, G2, Capterra, industry directories, and review platforms. Having multiple independent sources confirm your details reinforces your brand identity. Learn more about how our citation and authority building services manage this process.
    7. Explore Wikipedia creation only if your business meets notability guidelines. While a Wikipedia page provides strong validation, strict editorial standards mean many smaller businesses will not qualify initially.

    How long does entity building take?

    Setting up schema and sameAs links can be done in a single day, while updating directory listings across the web takes several weeks. Establishing long-term authority so AI tools consistently recognize your brand develops over several months. Entity trust relies on multiple independent sources confirming your details over time.

    Here is a typical timeline: Technical updates like schema, sameAs links, and standardizing brand names can be completed right away. Cleaning up directory listings usually takes 30 to 90 days, depending on how many existing profiles need updates. Setting up a Wikidata entry can be completed quickly once prepared. Over 6 to 12 months, accumulating consistent references builds solid authority that AI engines recognize.

    While this process takes time, it also creates a strong competitive advantage. Established entity authority is difficult for competitors to replicate quickly.

    Entity signals are the foundation, not the whole building

    Clear entity signals make your brand eligible for recommendations, but getting recommended also requires structured content that addresses user queries along with solid overall domain authority. Proper entity setup verifies identity, but compelling content completes the picture.

    The rest of the system:

    Entity optimization is an ongoing effort that serves as a core foundation for OptiSEOn’s LLM optimization services.

    Frequently Asked Questions

    What is an entity in SEO? 

    An entity is a distinct, verifiable item, such as a person, place, organization, or concept, that search engines and AI systems recognize independently. Entity SEO focuses on helping search platforms identify your brand as a real-world organization rather than just a collection of keyword-focused pages.

    How do I check if AI can recognize my brand? 

    You can check your domain using OptiSEOn’s free Entity Signal Checker, which evaluates Organization schema, sameAs links, Wikipedia, Wikidata, and NAP data against live sources to prioritize improvements. You can also evaluate live brand recommendations with our AI Citation Tester.

    Do I need a Wikipedia page to be recognized by AI? 

    No. While Wikipedia is a strong signal, its strict notability requirements mean many businesses will not qualify. Wikidata offers a practical alternative with accessible entry guidelines and wide use across AI platforms. Implementing Organization schema, linking sameAs profiles, and keeping NAP details consistent are generally higher priorities for most companies.

    What are sameAs links and why do they matter? 

    sameAs links are schema properties that connect your website to your official profiles on platforms like LinkedIn, X, and Crunchbase. They confirm to AI platforms and search engines that these separate profiles belong to one organization, uniting your brand signals into a single recognized entity.

    Why does NAP consistency affect AI visibility? 

    Consistent Name, Address, and Phone data helps systems verify that your business is real and active. Inconsistent information across web directories can confuse algorithms, lowering confidence in your brand identity and impacting both local search rankings and AI recommendations.

    How long does entity building take to work? 

    On-page updates like schema and sameAs links can be completed quickly. Standardizing directory listings usually takes 30 to 90 days, while building sustainable entity authority for consistent AI recommendations develops over 6 to 12 months.

    Is the OptiSEOn Entity Signal Checker really free? 

    Yes, it is completely free with no registration required. Every check runs against live data sources like site markup or official APIs, and unverified signals are noted separately for manual review. It is part of our suite of free SEO tools.


    Check if AI systems can identify your brand. Run the free Entity Signal Checker to review your signals in seconds and highlight priority fixes. If you want expert support building your entity presence, OptiSEOn is a Dallas-based agency specializing in SEO, AEO, GEO, and LLM optimization. Schedule a free entity audit.

  • Free Schema Markup Generator: Build Valid JSON-LD From Any Page

    Free Schema Markup Generator: Build Valid JSON-LD From Any Page

    Schema markup is one of those rare SEO levers that’s both high-value and genuinely underused. Done right, it makes your pages eligible for rich results in Google, and it tells AI engines like ChatGPT, Perplexity, and Gemini exactly what your content is and who you are. Done wrong — or not at all — you’re leaving one of the clearest machine-readable trust signals on the table.

    So why doesn’t every site have it? Because the traditional way of adding schema is tedious and easy to get wrong. You pick a type from a dropdown, hand-type every field into a form, hope you got the syntax right, paste it in, and cross your fingers. Most small teams try it once, find it fiddly, and move on.

    We built a free tool to remove that friction. The OptiSEOn Schema Markup Generator takes a different approach: you paste a page URL, it figures out what kind of page it is, and it generates valid JSON-LD built from what’s actually on the page — no dropdowns, no manual data entry, no invented values. This post explains what it does, how to add the output correctly, and the one step you should never skip before shipping.

    TL;DR

    Schema markup is structured data (in JSON-LD format) that tells Google and AI engines what a page means. The free OptiSEOn Schema Markup Generator fetches any page URL, auto-detects its type (homepage, blog post, product, FAQ, contact page, and more), and generates valid, copy-paste JSON-LD built only from data found on the page — it never fabricates authors, dates, prices, or ratings. No signup, nothing stored. Always validate the output with Google’s Rich Results Test before you ship it.

    → Generate schema for any page · No signup · Nothing stored.

    What is a schema markup generator?

    A schema markup generator is a tool that produces structured data code — usually JSON-LD — that you add to a web page so search engines and AI systems can understand its content in a machine-readable way. Instead of writing the code by hand, you use the generator to produce valid markup for your page type, whether that’s an article, a product, an FAQ, or a local business.

    The OptiSEOn generator differs from most in one important way: rather than making you choose a type and fill in a blank form, it reads your actual page, detects what kind of page it is, and populates the markup from real content.

    What does the OptiSEOn schema generator do?

    It fetches the page URL you paste, identifies the page type, and generates valid JSON-LD structured data built from what’s genuinely on the page — ready to paste into your <head>. It works for homepages, blog posts, products, FAQs, and contact pages, and produces types including Organization, WebPage, Article, Product, FAQPage, and LocalBusiness, among others.

    What that means in practice:

    • Auto-detection. You don’t pick a type from a dropdown. The tool works out whether it’s looking at an article, a product page, an FAQ, and so on, and generates the matching schema.
    • Built from your real page. Titles, descriptions, and other values come from the page’s actual content — not a blank form you fill in and not made-up data.
    • Valid, copy-paste JSON-LD. The output is formatted and ready to drop into your page’s <head>.
    • Private and free. No signup, and nothing you enter is stored.

    The result is a strong, accurate first draft in seconds. The two things you still own — reviewing it and validating it — are covered below, because skipping them is where schema goes wrong.

    What is schema markup and JSON-LD, briefly?

    Schema markup is a shared vocabulary from schema.org that labels the parts of your page — this is the author, this is the price, this is a review — and JSON-LD is the format Google recommends for adding it, a small script placed in your page’s <head>. Together they turn ambiguous text into explicit, machine-readable facts.

    Without markup, a search engine has to guess whether “$199” is a price, a discount, or just a number in your copy. With Product schema, it knows. That clarity is what makes pages eligible for rich results (star ratings, FAQ accordions, breadcrumbs) and what helps AI engines identify and cite your content accurately.

    A simple Article JSON-LD block looks like this:

    json

    {

      “@context”: “https://schema.org”,

      “@type”: “Article”,

      “headline”: “Your Post Title”,

      “author”: { “@type”: “Person”, “name”: “Author Name” },

      “datePublished”: “2026-08-01”,

      “publisher”: {

        “@type”: “Organization”,

        “name”: “Your Company”

      }

    }

    For a full breakdown of which types matter and where to use each one, see our guide to the 10 schema markup types every business needs in 2026. This post is about generating them quickly; that one is about choosing them wisely.

    How do I generate and add schema markup?

    Paste your page URL into the generator, let it detect the page type and build the JSON-LD, copy the output, paste it into your page’s <head>, then validate it with Google’s Rich Results Test before publishing. The generation takes seconds; the validation is the step that protects you from shipping something broken.

    Step by step:

    1. Go to the Schema Markup Generator and paste the URL of the page you want to mark up. No signup.
    2. Let it analyze. The tool fetches the page, detects its type, and generates matching JSON-LD from the page’s real content.
    3. Review the output. Confirm the detected type is right and the values match what’s visibly on the page.
    4. Copy the JSON-LD and paste it inside the <head> of that page (or via your CMS’s structured-data / custom-code field). In WordPress, most SEO plugins have a slot for custom JSON-LD; in Webflow or Wix, use a head-code embed.
    5. Validate before shipping. Run the page or the code through Google’s Rich Results Test and the Schema.org validator. Fix any errors or warnings.
    6. Repeat per page type. Generate one for each distinct template — homepage, article, product, contact — since each needs different schema.

    Which schema types does the generator produce?

    It generates the schema types that match common page templates — Organization and WebPage for homepages, Article for blog posts, Product for product pages, FAQPage for FAQs, and LocalBusiness for contact and location pages, among others. Because it detects the page type first, you get the right schema for each page instead of forcing one type everywhere.

    Here’s how the common types map to pages:

    Page typeSchema generatedWhat it does
    HomepageOrganization, WebPageIdentifies your business as an entity
    Blog post / articleArticleAuthor, date, headline — feeds rich results and E-E-A-T
    Product pageProductName, description, price, availability
    FAQ sectionFAQPageMakes Q&A machine-readable for parsing and AI extraction
    Contact / locationLocalBusinessAddress, hours, geo — critical for local SEO

    Organization schema is worth adding site-wide as your foundational entity signal, and Article plus Person schema on content pages feed the author-authority signals that both Google and AI engines weigh — which we cover in our E-E-A-T guide.

    Why does “nothing invented” matter?

    The generator only uses data found on your page — it never invents authors, dates, prices, or ratings, because fabricated values are exactly what gets schema flagged and penalized by Google. Structured data that doesn’t match your visible page content, or that fakes reviews and ratings, is a policy violation that can trigger manual actions — not a shortcut to rich results.

    This is a guardrail, not a limitation, and it’s the same warning we gave in our schema markup guide: Review and AggregateRating schema must reflect real, displayed reviews; inventing them is one of the fastest ways to earn a penalty. A lot of schema tools and plugins will happily let you type in a 5-star rating for a page with no reviews. That “help” is a liability.

    So if the generator leaves a field empty, that’s usually correct — it means the data isn’t genuinely on your page, and adding it manually would mean fabricating it. The fix is to put the real information on the page first, then regenerate.

    Do I still need to validate the output?

    Yes — always run generated schema through Google’s Rich Results Test before publishing, because broken structured data can do more harm than none at all. Google ignores invalid markup, and markup that conflicts with your visible content can hurt trust signals. Validation takes thirty seconds and catches the errors that would otherwise ship silently.

    A quick validation checklist:

    • Google Rich Results Test — confirms eligibility for specific rich result types and flags errors
    • Schema.org validator — a stricter, vocabulary-level check
    • Eyeball the values — make sure every field matches what a visitor actually sees on the page
    • Check for duplicates — if your CMS or theme already outputs some schema, avoid conflicting or doubled markup

    The generator gives you valid JSON-LD to start from, but your specific CMS, theme, or existing plugins can introduce conflicts only a validation pass will catch. Never skip it.

    Schema is one signal. We optimize all of them.

    Structured data tells Google and AI engines what your pages mean — but it works best alongside content, authority, and technical SEO that all agree with it. Schema on its own doesn’t rank a page or earn a citation; it amplifies the signals already there. Mark up a thin page and you’ve accurately described a thin page.

    Schema sits inside a larger system, and the rest of that system is what turns “understood” into “ranked” and “cited”:

    That end-to-end work — structured data, content, authority, and technical SEO tuned together for both search engines and AI answers — is what we do at OptiSEOn. The generator gets your markup right; the service makes it count.

    Frequently Asked Questions

    What is a schema markup generator used for? A schema markup generator produces structured data code (usually JSON-LD) that you add to a web page so search engines and AI systems can understand its content. It saves you from hand-writing the code and reduces syntax errors. The OptiSEOn generator goes further by auto-detecting your page type and building the markup from the page’s real content.

    How do I add schema markup to my website? Generate the JSON-LD for your page, then paste it inside the page’s <head> — directly in the HTML, or through your CMS’s custom-code or structured-data field (most WordPress SEO plugins have one; Webflow and Wix use head-code embeds). Then validate it with Google’s Rich Results Test before publishing.

    What format should schema markup be in? JSON-LD is the format Google recommends. It’s a self-contained script placed in the page’s <head>, separate from your visible HTML, which makes it easier to add and maintain than older formats like microdata or RDFa. The OptiSEOn generator outputs JSON-LD.

    Does schema markup improve my Google rankings? Not directly. Schema doesn’t boost rankings on its own, but it makes pages eligible for rich results (star ratings, FAQ accordions, breadcrumbs) that improve click-through rate, and it helps AI engines understand and cite your content. The benefit is real but indirect — schema amplifies your other signals rather than replacing them.

    Is it safe to use auto-generated schema? Yes, provided the values are accurate and you validate before shipping. The risk with schema isn’t automation — it’s fabricated data. A generator that pulls only from your real page content (and leaves fields empty rather than inventing values) is safe; one that lets you type in fake reviews or ratings is a liability. Always run the output through Google’s Rich Results Test.

    Why is some schema data missing from my generated file? Usually because that data isn’t genuinely on your page. The OptiSEOn generator won’t invent authors, dates, prices, or ratings, since fabricated values are what get schema penalized. If a field is empty, add the real information to your page first, then regenerate — don’t fill it in with made-up data.

    Is the OptiSEOn schema generator really free? Yes — no signup, and nothing you enter is stored. Paste a URL, get valid JSON-LD auto-built from your page. It’s one of several free tools we build, alongside our robots.txt AI Checker, llms.txt Generator, Entity Signal Checker, and AI Citation Tester.


    Generate valid schema for any page in seconds — then validate and ship it with confidence. Use the free Schema Markup Generator. And if you want your structured data working in concert with content, authority, and technical SEO — for Google rankings and AI citations alike — OptiSEOn is a Dallas-based SEO, AEO, GEO, and LLM optimization agency. Book a free schema & SEO review.

  • The Free llms.txt Generator (and the Honest Take on Whether You Need One)

    The Free llms.txt Generator (and the Honest Take on Whether You Need One)

    A few months ago we published a post with a deliberately unglamorous conclusion: llms.txt is overhyped. The data we pulled showed it appearing in well under 1% of AI-cited URLs, and no major AI vendor had officially committed to honoring the spec. Our recommendation was blunt — don’t treat llms.txt as a primary AI-visibility lever, because it isn’t one.

    So it would be a little strange to now hand you a free llms.txt generator without addressing the obvious question: if you told me not to bother, why build the tool?

    Here’s the honest answer, and it’s the same one that runs through everything we publish. llms.txt is low effort and near-zero downside. It won’t hurt you, it might help later if the major AI platforms formally adopt the standard, and it’s a genuinely useful forcing function for auditing which of your pages actually matter. What made it not worth prioritizing was the manual effort relative to the payoff. Take the manual effort down to twenty seconds, and the math changes — a zero-downside, maybe-helpful task you can finish before your coffee is worth doing.

    That’s what the OptiSEOn llms.txt generator is for. This post explains what it does, how to use it well, and — because we’d rather keep your trust than oversell you a file — where it fits and where it doesn’t.

    TL;DR

    llms.txt is an emerging, community-proposed standard: a Markdown file at your domain root that tells AI systems what your site is and which pages matter. The free OptiSEOn llms.txt generator crawls your real pages, groups them into sections, and produces a valid, downloadable llms.txt in about 20 seconds — with every title and description pulled from your own pages, nothing invented. It’s low-effort and low-risk, but it is not a shortcut to getting cited by AI. Being readable is step one; being citation-worthy is the actual job.

    → Generate your free llms.txt · No signup · ~20 seconds.

    What is an llms.txt file?

    An llms.txt file is a Markdown file placed at your domain root (yoursite.com/llms.txt) that gives AI systems a curated, plain-language map of your site — what it is and which pages matter most. It was proposed in September 2024 as a community standard, and the idea is to hand large language models a clean index instead of making them guess from your raw HTML.

    A basic llms.txt looks like this:

    # Your Company

    > One-line summary of what your company does.

    ## Services

    – [SEO Services](https://yoursite.com/services): What you offer and for whom.

    – [Pricing](https://yoursite.com/pricing): Plans and what’s included.

    ## Guides

    – [How AI Search Works](https://yoursite.com/blog/ai-search): A plain-English overview.

    Clean, structured, human-readable, and machine-readable. The concept is sound. Whether the major AI platforms actually use it is the part worth being honest about — more on that below.

    What does the OptiSEOn llms.txt generator do?

    The OptiSEOn llms.txt generator reads your site’s sitemap and pages, groups them into logical sections, and builds a comprehensive, valid llms.txt file you can download and upload to your site root. Every title and description in the file comes directly from your own pages — the tool doesn’t invent copy or hallucinate descriptions.

    What that means in practice:

    • It uses your real content. Titles and descriptions are pulled from your actual pages, so the file reflects what’s genuinely on your site.
    • It structures automatically. Pages are grouped into sections (services, guides, products, and so on) rather than dumped into one flat list.
    • It outputs a valid file. The result follows the llms.txt format, ready to paste at yourdomain.com/llms.txt.
    • It’s free and needs no signup. Enter a domain, wait about twenty seconds, download.

    The tool gives you a strong first draft built from reality. What it deliberately does not do is pretend the file is finished the moment it’s generated — which brings us to how to use it well.

    How do I create an llms.txt file?

    Enter your domain into the generator, let it crawl your pages (~20 seconds), review and trim the generated file, then upload it to your site root at /llms.txt. The generation is automatic; the curation is the part that makes it actually good.

    Step by step:

    1. Go to the llms.txt generator and enter your domain. No signup.
    2. Let it crawl. It reads your sitemap and pages and groups them into sections.
    3. Review the draft. This is the important step most people skip. The tool gives you everything; you decide what belongs.
    4. Trim what doesn’t matter. Cut thin pages, tag/category archives, utility pages, and anything that doesn’t help an AI understand your site. A focused file of 15–30 meaningful pages beats a sprawling dump of 200.
    5. Upload to your root. Place the final file at yourdomain.com/llms.txt (UTF-8, named exactly llms.txt).
    6. Keep it updated. Revisit it when your site changes materially — new services, new cornerstone content, restructured navigation.

    What makes a good llms.txt file?

    A good llms.txt is curated, not exhaustive: it points AI systems at your genuinely important pages with accurate descriptions, and leaves everything else out. The value is in the editorial judgment — telling an AI which 20 pages define your site is more useful than listing all 200.

    The principles we’d apply:

    • Curate ruthlessly. Include cornerstone content, key service and product pages, and your best explanatory guides. Exclude thin pages, duplicates, archives, and utility URLs.
    • Keep descriptions accurate. Since the generator pulls from your real pages, the honesty is built in — but if you edit, keep descriptions truthful. Misleading descriptions help no one and erode trust signals.
    • Don’t link to gated or noindexed pages. If you don’t want it in Google, you don’t want it in llms.txt.
    • Update it as you go. A file that reflects your site as it was a year ago is worse than a slightly shorter one that’s current. Freshness matters across AI systems generally.

    The generator handles the tedious first pass. The curation is where a human — you — adds the value a crawler can’t.

    Does llms.txt actually work? The honest answer.

    As of 2026, no major AI vendor — OpenAI, Google, Anthropic, or Perplexity — has publicly committed to honoring llms.txt, and citation studies show it appearing in well under 1% of AI-cited URLs. So the honest answer is: implementing llms.txt is unlikely to measurably change your AI citations today. We laid out the full data in our honest guide to llms.txt, and nothing since has changed that conclusion.

    We’re telling you this on the same page where we’re offering you the tool, on purpose. Here’s the case for generating one anyway:

    • Near-zero downside. A well-formed llms.txt won’t hurt your SEO or your AI visibility.
    • Possible future upside. It’s a proposed standard with real momentum in developer circles (Anthropic, Stripe, Cursor, and Cloudflare publish them). If adoption formalizes, you’re already set up.
    • It’s now a 20-second task. The original objection was effort-versus-payoff. The tool removes the effort side of that equation almost entirely.
    • It’s a useful audit. Deciding which pages belong in your llms.txt forces you to identify your genuinely important pages — which is clarifying regardless of whether an AI ever reads the file.

    What we won’t do is tell you it’s a growth lever. It isn’t. It’s a cheap, sensible hygiene task with option value. Treat it as that and you’ll have the right expectations.

    Who should generate an llms.txt?

    Most businesses can generate one because the cost is trivial, but developer-focused and documentation-heavy sites get the clearest current benefit — their users frequently ask AI coding assistants questions that a curated doc index helps answer. For everyone else, it’s a reasonable low-priority hygiene task, not something to agonize over.

    A quick way to think about priority:

    If you are…llms.txt priority
    A developer tool or docs-heavy siteHigher — real current use case with AI coding assistants
    A SaaS or B2B site wanting AI visibilityLow-priority hygiene — generate it, then move on to what matters
    A local or small businessOptional — fine to do, not worth stress
    Short on time entirelySkip it and prioritize content, schema, and crawler access first

    Wherever you land, the sequencing is the same: do the things that actually move AI visibility first, and treat llms.txt as the easy box you tick once, not the project you build a quarter around.

    llms.txt is step one. Getting cited is the job.

    A good llms.txt helps AI systems read your site — but showing up in AI answers takes citation-worthy content, entity-consistent schema, and authority signals that agree with each other. Readability is necessary; it is not sufficient. An AI can parse your site perfectly and still never cite you, because parsing isn’t the same as being worth quoting.

    The work that actually earns citations is the work we’ve mapped across this blog:

    That end-to-end work is OptiSEOn’s LLM optimization service. The llms.txt file is the doormat; the service is the house.

    Frequently Asked Questions

    What is an llms.txt file used for? An llms.txt file gives AI systems a curated, plain-language map of your website — what it is and which pages matter most — as a Markdown file at your domain root. The intent is to help models understand your site without guessing from raw HTML. It’s an emerging community standard proposed in 2024, not an official requirement from any AI vendor.

    How do I create an llms.txt file for free? Use a generator like OptiSEOn’s free tool: enter your domain, let it crawl your pages and group them into sections (~20 seconds), review and trim the draft, then upload the file to your site root at /llms.txt. No signup is required, and every description is pulled from your real pages.

    Where do I put the llms.txt file? At the root of your domain, exactly at yourdomain.com/llms.txt, saved as UTF-8 and named precisely llms.txt. This mirrors how robots.txt lives at the root. Don’t nest it in a subfolder — AI systems that look for it expect it at the root path.

    Does llms.txt help with SEO or AI rankings? Not directly. No major AI vendor has committed to using llms.txt, and it appears in well under 1% of AI-cited URLs as of 2026. It won’t hurt you and may help if adoption grows, but it’s a low-downside hygiene task, not a ranking or citation lever. The things that actually earn citations are content structure, schema, and authority signals.

    What’s the difference between llms.txt and robots.txt? robots.txt controls access — which crawlers may read which pages. llms.txt provides curation — a hand-picked index of your important pages with descriptions. robots.txt is universally respected by crawlers; llms.txt is an emerging standard that isn’t yet formally honored by the major AI platforms. They solve different problems, and you can use both.

    How often should I update my llms.txt? Whenever your site changes materially — new services, new cornerstone content, or restructured navigation. There’s no fixed schedule, but a file that reflects a year-old version of your site is less useful than a current one. Regenerating and re-curating quarterly is a reasonable cadence for most sites.

    Is the OptiSEOn llms.txt generator really free? Yes — no signup, no email. It crawls your real pages, groups them into sections, and produces a valid, downloadable llms.txt in about 20 seconds. It’s one of several free tools we build, alongside our robots.txt AI Checker, Schema Generator, Entity Signal Checker, and AI Citation Tester.


    Generate your llms.txt in twenty seconds — then go do the work that actually matters. Use the free llms.txt generator, tick the box, and move on. If you want help with the part that genuinely moves AI visibility — content, schema, and authority — OptiSEOn is a Dallas-based SEO, AEO, GEO, and LLM optimization agency. Book a free AI visibility review and we’ll show you where the real gaps are.

  • Are You Accidentally Blocking ChatGPT? Check Your robots.txt in 30 Seconds

    Are You Accidentally Blocking ChatGPT? Check Your robots.txt in 30 Seconds

    Here’s a failure mode we run into constantly when we audit businesses at OptiSEOn: a company has spent months on content, schema, and AI visibility work — and a single line in a file most people never open is quietly hiding the entire site from ChatGPT, Perplexity, and Gemini.

    That file is robots.txt. It’s the first thing a crawler reads when it arrives at your domain, and it decides — before any of your content matters — whether that crawler is even allowed to look. If your robots.txt blocks AI crawlers, none of the downstream work can save you. You can’t be cited by an engine that was never allowed through the door.

    The frustrating part is that this usually happens by accident. Nobody decides “let’s block ChatGPT.” A developer copies a robots.txt template, a security plugin adds a blanket rule, a staging config ships to production — and the block is invisible until someone thinks to check.

    So we built a free tool that checks it for you. The OptiSEOn robots.txt AI Crawler Checker reads your live file, works out rule by rule which AI crawlers can actually reach your pages, and hands you a corrected file you can paste straight in. No signup. This post explains what it checks, why the checking is trickier than it looks, and what to do with the result.

    TL;DR

    Your robots.txt file tells crawlers which parts of your site they may access — and a misconfigured one can block AI crawlers like GPTBot, ClaudeBot, and PerplexityBot without you knowing. The free OptiSEOn robots.txt AI Crawler Checker reads your live file, evaluates each major AI crawler the way search engines actually do (named bot groups override the wildcard group; the longest matching path rule wins), and generates a corrected file. Being crawlable is necessary but not sufficient — it makes you eligible to be cited, not guaranteed to be.

    → Run the free robots.txt AI Crawler Checker · Try it on stripe.com, github.com, or your own domain.

    What is a robots.txt AI crawler checker?

    A robots.txt AI crawler checker is a tool that reads your site’s robots.txt file and reports which AI crawlers — the bots behind ChatGPT, Claude, Perplexity, Gemini, and others — are allowed or blocked from accessing your pages. It exists because AI crawlers use different user-agent names than Googlebot, so a file that’s fine for traditional SEO can still block every AI engine.

    The OptiSEOn checker goes one step further than a pass/fail readout: when it finds a crawler you’re blocking (or one you probably want to allow), it generates a corrected robots.txt you can download and deploy.

    Why would my site block AI crawlers by accident?

    Most accidental AI-crawler blocks come from generic robots.txt templates, security or privacy plugins that add blanket disallow rules, or a blanket User-agent: * / Disallow: / left over from a staging environment. Nobody sets out to block ChatGPT — the rule arrives through a default, a plugin, or a copied config, and stays invisible until something checks for it.

    The common culprits we see in audits:

    • A staging Disallow: / that shipped to production and was never removed
    • A security plugin that added AI-bot blocks as a “feature,” often framed as protecting against scraping
    • A CDN or host default that blocks unknown user agents
    • A copied template from a tutorial written before AI crawlers existed, so it neither allows nor accounts for them
    • An intentional block from 2023 — when many publishers blocked GPTBot during the training-data backlash — that nobody has revisited even though the business now wants AI visibility

    That last one matters. A decision that made sense two years ago may be actively working against you now, and it’s sitting in a file you haven’t opened since.

    Which AI crawlers should you check for?

    The AI crawlers that matter most in 2026 are OpenAI’s GPTBot, ChatGPT-User, and OAI-SearchBot; Anthropic’s ClaudeBot and Claude-User; PerplexityBot and Perplexity-User; Google-Extended; plus CCBot (Common Crawl) and Bingbot, which feed multiple AI systems downstream. Each has a distinct job, and blocking the wrong one can cost you citations while blocking the right one protects your content from training use.

    Here’s how the major ones break down:

    Crawler (user agent)Who runs itWhat it does
    GPTBotOpenAICrawls content for model training
    ChatGPT-UserOpenAILive retrieval when ChatGPT fetches a page to answer a user right now
    OAI-SearchBotOpenAIIndexes pages for ChatGPT’s search feature
    ClaudeBotAnthropicCrawls content for model training
    Claude-UserAnthropicLive retrieval when Claude fetches a page for a user
    PerplexityBotPerplexityIndexes pages so they can be cited in answers
    Perplexity-UserPerplexityLive retrieval triggered by a user’s question
    Google-ExtendedGoogleControls whether your content trains/grounds Gemini
    CCBotCommon CrawlOpen dataset that feeds many AI training pipelines
    BingbotMicrosoftPowers Bing and, downstream, Copilot

    The key distinction — and one most site owners miss — is training crawlers versus live-retrieval agents. GPTBot and ClaudeBot gather content to train future models. ChatGPT-User, Claude-User, and Perplexity-User fetch your page in real time because a user asked a question your page might answer. Many businesses want to block training while allowing live retrieval — so they don’t feed the models for free, but they can still get cited (and get referral traffic) when a user’s question surfaces their page. We walked through that trade-off in detail in our honest guide to llms.txt and AI crawler control.

    Why do most robots.txt checkers get this wrong?

    Most simple checkers only read the User-agent: * wildcard group, so when a site names a specific bot to allow it — the single most common pattern — they wrongly report that site as blocking everything. Real crawlers don’t work that way: a bot obeys the most specific group that names it and ignores the wildcard entirely, and within that group the longest matching path rule wins.

    This is the nuance the OptiSEOn checker was built around, and it’s worth understanding because it changes the answer completely. Consider this file:

    User-agent: *

    Disallow: /

    User-agent: GPTBot

    Allow: /

    A naive checker reads the first group, sees Disallow: /, and reports “this site blocks all crawlers.” Wrong. GPTBot has its own named group, so it ignores the wildcard group entirely and follows its own rule — Allow: /. This site allows GPTBot and blocks everything else. Getting that backwards could send you “fixing” a file that was already correct, or reassure you about a file that’s actually blocking you.

    The two rules that decide the real outcome:

    1. Most-specific-group wins. A crawler follows the group that names it. Only if no group names it does it fall back to User-agent: *. So naming a bot to allow it is a real, common, correct pattern — and it defeats any checker that only reads the wildcard.
    2. Longest matching path wins. Within the applicable group, the most specific (longest) path rule takes precedence, and on a tie the least restrictive rule (Allow) wins — which is how Google’s own parser resolves conflicts.

    Our checker evaluates every crawler through both rules, the way the engines themselves do. That’s the difference between a readout you can trust and one that looks authoritative but is quietly wrong.

    How to use the OptiSEOn robots.txt AI Crawler Checker

    Enter your domain, and the tool fetches your live robots.txt, evaluates each major AI crawler against your real rules, and shows you a clear allowed/blocked verdict per bot — then generates a corrected file to download. It reads your actual live file, not a cached copy, so the result reflects what crawlers see right now.

    The flow, start to finish:

    1. Go to the robots.txt AI Crawler Checker.
    2. Enter your domain (or try one of the built-in examples — stripe.com, github.com, nytimes.com — to see how well-known sites handle it). No signup required.
    3. Read the per-crawler verdict. The tool shows, bot by bot, which AI crawlers can reach your pages and which are blocked, applying the precedence rules above so allow-listed bots are reported correctly.
    4. Review the corrected file. If the tool finds crawlers you’re blocking that you likely want to allow, it generates a corrected robots.txt.
    5. Download and deploy. Paste the corrected file at yourdomain.com/robots.txt, replacing the old one. Re-run the checker to confirm.

    The whole thing takes under a minute, and it’s the fastest way we know to rule out the single most catastrophic (and most overlooked) AI-visibility problem.

    An honest caveat: Google-Extended doesn’t control AI Overviews

    One thing we won’t oversell, because getting it wrong is a common mistake: Google-Extended controls whether your content is used to train and ground Gemini — it does not control whether you appear in Google’s AI Overviews. AI Overviews are served through the regular Googlebot crawl, the same one that powers Search. You cannot block AI Overviews via robots.txt without also blocking yourself from Google Search entirely, which almost no one wants.

    So if your goal is “show up in Google’s AI answers,” the lever isn’t Google-Extended — it’s ranking well in ordinary Google Search, plus the content structure that makes your page extractable. We covered that extraction structure in our guide to winning featured snippets and AI Overviews. The checker tells you the truth about Google-Extended; it won’t pretend that toggling it changes your AI Overview presence, because it doesn’t.

    Getting crawled is step one. Getting cited is the goal.

    Allowing AI crawlers only makes you eligible to be cited — it doesn’t make it happen. Actually getting quoted in an AI answer takes content structured to answer real questions, schema that clearly states who you are, and authority signals across the web that corroborate it. The robots.txt check clears the doorway; it doesn’t walk you through it.

    Think of it as a gate, not a growth lever. An open gate is necessary — a blocked crawler can never cite you — but on its own it just puts you in the running. The work that turns eligibility into citations is the same work we’ve written about across this blog:

    The strategic frame for all of it is in how LLMs are replacing traditional search. The robots.txt check is just the first box you have to tick before any of that can pay off.

    Should you allow or block AI crawlers?

    Most businesses that want AI visibility should allow live-retrieval agents (ChatGPT-User, Claude-User, Perplexity-User) and search indexers (OAI-SearchBot, PerplexityBot), and can choose separately whether to allow training crawlers (GPTBot, ClaudeBot, CCBot) based on how they feel about their content training future models. There’s no universally correct answer — it depends on your goals.

    A reasonable default for a business that wants to be found in AI answers:

    • Allow live-retrieval and search-indexing bots — this is how you earn citations and referral traffic
    • Decide deliberately on training crawlers — blocking them protects your content from training use but has no measurable effect on whether you get cited today; allowing them may help long-term model familiarity
    • Never block Googlebot or Bingbot unless you intend to leave Search entirely

    Publishers with premium or licensable content often block training crawlers while allowing retrieval. Most SMBs and B2B companies competing for AI visibility allow both. The checker generates a corrected file for whichever posture you choose — it doesn’t force a policy on you.

    Frequently Asked Questions

    How do I know if my robots.txt is blocking ChatGPT? Run your domain through a robots.txt AI crawler checker like OptiSEOn’s free tool. It reads your live file and reports, bot by bot, whether OpenAI’s crawlers (GPTBot, ChatGPT-User, OAI-SearchBot) and other AI crawlers can reach your pages. Reading the file manually works too, but it’s easy to misjudge how precedence rules resolve.

    What’s the difference between GPTBot and ChatGPT-User? GPTBot is OpenAI’s crawler for gathering content to train future models. ChatGPT-User is OpenAI’s live-retrieval agent that fetches a specific page in real time because a user asked a question it might answer. Many businesses block GPTBot (to avoid training use) while allowing ChatGPT-User (to stay citable). They’re separate user agents with separate rules.

    Will blocking GPTBot stop me from appearing in ChatGPT? Not necessarily. GPTBot is the training crawler; ChatGPT’s live answers to user questions are fetched by ChatGPT-User, and its search feature uses OAI-SearchBot. If you block GPTBot but allow those two, you can still be retrieved and cited in real-time answers. Blocking all three, however, does remove you from ChatGPT’s reach.

    Can I block AI Overviews with robots.txt? No. Google AI Overviews are served through the standard Googlebot crawl, the same one that powers Google Search. The Google-Extended token controls Gemini training and grounding, not AI Overviews. You can’t opt out of AI Overviews via robots.txt without blocking Googlebot and disappearing from Search entirely.

    Is it safe to let AI crawlers access my whole site? For most businesses seeking AI visibility, yes — allowing AI crawlers is what makes you eligible to be cited and to earn referral traffic. The main reason to restrict them is if you have premium or licensable content you don’t want used for model training, in which case you can block training crawlers while still allowing live-retrieval agents.

    Does allowing AI crawlers guarantee I’ll get cited? No. Allowing crawlers only makes you eligible. Actual citations require content structured to answer questions, schema that identifies who you are, and authority signals that corroborate it. The robots.txt check removes a blocker; it doesn’t create visibility on its own.

    Is the OptiSEOn robots.txt checker really free? Yes — no signup, no email required. It reads your live robots.txt, evaluates each major AI crawler against your real rules, and generates a corrected file you can download. It’s one of several free tools we build for the SEO and AI-visibility community, alongside our Schema Generator, llms.txt Generator, Entity Signal Checker, and AI Citation Tester.


    Check your site now — it takes 30 seconds. Run the free robots.txt AI Crawler Checker and see exactly which AI crawlers can reach your pages. If it turns up a block (or you want help turning eligibility into actual citations), OptiSEOn is a Dallas-based SEO, AEO, GEO, and LLM optimization agency — book a free AI visibility review and we’ll walk through your results together.