Canonical URL & Robots

ByTheWeb GEO provides per-content Canonical URL and Indexing & Discovery controls when ByTheWeb GEO is managing your site’s SEO layer.

These controls let you:

• Set a custom Canonical URL for an individual Post, Page, or supported Taxonomy term.
• Keep the normal URL as the Canonical when no custom value is needed.
• Set an individual item to Index or Noindex.
• Let an individual item follow the default setting of its Post Type or Taxonomy.

For a product overview, see:

Canonical URL & Robots Feature Overview

These controls are local ByTheWeb GEO functionality and do not use AI credits.

When Are These Controls Available?

ByTheWeb GEO manages Canonical and Robots settings only when it is responsible for the site’s overlapping SEO functionality.

If Yoast SEO or Rank Math SEO is the active supported SEO provider, ByTheWeb GEO leaves these settings to that plugin.

In that situation, use the active SEO plugin to manage:

• Canonical URL
• Index/Noindex
• SEO Title
• Meta Description
• Social SEO fields

ByTheWeb GEO continues its non-overlapping GEO functionality.

For more information, see:

Requirements & Compatibility

Where Do I Find the Canonical and Robots Settings?

Open a supported Post, Page, or enabled Custom Post Type.

In the ByTheWeb GEO panel, open:

Advanced SEO, Indexing & Social

There you will find:

Canonical URL

and:

Indexing & Discovery

Supported Categories, Tags, and other enabled Taxonomy terms also provide these controls on the Term editing screen.

What Is the Canonical URL Field?

The Canonical URL field lets you specify the preferred URL for the current content item.

If you enter a URL, ByTheWeb GEO uses that URL as the Canonical for the item.

For example:

When ByTheWeb GEO is managing SEO, the applicable frontend output uses that value for the Canonical URL.

What Happens If I Leave Canonical URL Empty?

You normally do not need to manually enter the current page URL.

If the Canonical URL field is empty, ByTheWeb GEO uses:

• The normal WordPress permalink for a Post, Page, or supported Custom Post Type.
• The normal WordPress Term URL for a supported Category, Tag, or other Taxonomy term.

Enter a custom Canonical only when the preferred URL should be different from the current item’s normal URL.

Can I Set a Canonical to Another URL?

Yes.

The Canonical field accepts a URL different from the current page URL.

This can be used when another URL should be treated as the preferred version of the content.

Changing the Canonical does not redirect visitors to that URL.

It changes the Canonical URL output.

What Is Indexing & Discovery?

The Indexing & Discovery control determines the individual ByTheWeb GEO indexing setting for the current item.

For Posts and Pages, the available options are:

• Default (Follow Content Type Setting)
• Index
• Noindex

For Taxonomy terms, the equivalent Default option follows the Taxonomy setting.

What Does Default Mean?

Default means the individual item does not create its own Index or Noindex override.

For a Post, Page, or Custom Post Type item, it follows the applicable content-type setting.

For a Category, Tag, or other supported Taxonomy term, it follows the applicable Taxonomy setting.

The interface also shows the current inherited value as either:

Index

or:

Noindex

Where Do the Default Indexing Settings Come From?

The content-type defaults are configured in:

ByTheWeb > SEO Settings

Post Types and Taxonomies can have their own default Index or Noindex setting.

The individual Indexing & Discovery field can then override that default when necessary.

For example:

• Post Type default = Index
• Individual Post = Noindex

The individual Post is Noindex.

Or:

• Post Type default = Noindex
• Individual Post = Index

The individual Post is Index.

How Does Noindex Affect the Frontend Output?

When an item is effectively set to Noindex, ByTheWeb GEO outputs the applicable robots directive as:

noindex, nofollow

when ByTheWeb GEO is managing SEO.

When the item is effectively Index, ByTheWeb GEO outputs its normal Index and Follow directives together with its supported preview directives.

Does Noindex Make Content Private?

No.

Noindex is an indexing directive.

It does not:

• Password-protect a page
• Restrict access to logged-in users
• Prevent someone from opening the URL
• Act as a firewall
• Secure confidential information

Do not use Noindex as a privacy or access-control mechanism.

Does Noindex Affect the ByTheWeb XML Sitemap?

Yes.

When ByTheWeb GEO manages SEO and the Sitemap, content that is effectively Noindex is not eligible for the ByTheWeb XML Sitemap.

An individual Index or Noindex override can therefore change the item’s Sitemap eligibility.

For more information, see:

XML Sitemap

Does Noindex Affect llms.txt and llms-full.txt?

Yes.

ByTheWeb GEO’s LLM files are built from eligible public and indexable content.

An item that is effectively Noindex is not eligible for those files.

However, ByTheWeb GEO also provides a separate:

Exclude from LLM files

setting.

That setting lets you remove an individual item from llms.txt and llms-full.txt without changing its normal Index/Noindex setting or removing it from the XML Sitemap.

For the complete behavior, see:

llms.txt & llms-full.txt

Index/Noindex vs. Exclude from LLM Files

These controls are different.

Noindex

Affects the content’s ByTheWeb indexing status and therefore also affects eligibility for outputs that require indexable content, including the ByTheWeb XML Sitemap and LLM files.

Exclude from LLM files

Removes the item only from:

• llms.txt
• llms-full.txt

The item can remain Index and can remain eligible for the XML Sitemap.

Use the setting that matches what you actually want to control.

Are Categories and Tags Supported?

Yes.

When the relevant Taxonomy is enabled for ByTheWeb SEO, its Term editing screen can include:

• Canonical URL
• Indexing & Discovery

The Term can follow the Taxonomy default or use an individual Index or Noindex override.

If no custom Canonical is entered, ByTheWeb GEO uses the normal Term URL.

What Happens If Yoast SEO Is Active?

When Yoast SEO is the active supported SEO provider, ByTheWeb GEO does not create competing Canonical or Robots controls for the same SEO layer.

The ByTheWeb GEO editing interface displays a notice explaining that Yoast is managing those SEO fields.

Use Yoast to edit the Canonical and Robots settings.

ByTheWeb GEO can continue managing its GEO fields such as:

• AI Summary
• Short Answer
• FAQ Schema

What Happens If Rank Math SEO Is Active?

The same behavior applies to Rank Math SEO.

ByTheWeb GEO leaves the overlapping Canonical and Robots functionality to Rank Math rather than generating competing SEO output.

Use Rank Math for those fields while continuing to use ByTheWeb GEO for its non-overlapping GEO functionality.

Does WordPress Search Engine Visibility Override These Settings?

WordPress has a global setting under:

Settings > Reading > Search Engine Visibility

If the website is configured to discourage search engines from indexing the site, ByTheWeb GEO respects that global site-level setting in its indexing logic.

An individual page set to Index should not be treated as a way to override the site’s global WordPress privacy setting.

Do Canonical and Robots Controls Use AI Credits?

No.

Entering a Canonical URL or changing Indexing & Discovery does not use AI credits.

These are local SEO settings provided by ByTheWeb GEO.

Do I Need ByTheWeb AI?

No.

ByTheWeb AI is not required to use Canonical URL or Indexing & Discovery controls.

These fields are part of ByTheWeb GEO.

Does Changing the Canonical Change the Page URL?

No.

The Canonical field does not change:

• The WordPress permalink
• The Post slug
• The Taxonomy slug
• The URL visitors use to open the current page

It controls the Canonical URL output only.

Does Setting Noindex Remove the Page?

No.

The WordPress content remains in your website.

The setting changes the applicable indexing directive.

It does not delete, unpublish, or password-protect the Post or Term.

Why Can I Not See the Canonical or Robots Fields?

First check whether Yoast SEO or Rank Math SEO is active.

When a supported external SEO provider is managing the site’s SEO layer, ByTheWeb GEO intentionally leaves the overlapping SEO fields to that provider.

Also confirm that the current Post Type or Taxonomy is enabled for the applicable ByTheWeb SEO functionality.

For additional help, see:

Troubleshooting

Why Is My Page Missing from the Sitemap or LLM Files?

Check its effective Indexing & Discovery setting.

If the item is Noindex, it is not eligible for the applicable ByTheWeb Sitemap or LLM files.

Also check:

• The Post Type or Taxonomy settings.
• WordPress Search Engine Visibility.
• The dedicated Exclude from LLM files option if the problem affects only the LLM files.

Recommended Setup

1. Configure your normal Post Type and Taxonomy Index/Noindex defaults under ByTheWeb > SEO Settings.
2. Let individual content use Default when it should follow that normal behavior.
3. Use Index or Noindex only when the individual item needs an explicit override.
4. Leave Canonical URL empty when the item’s normal WordPress URL should be canonical.
5. Enter a custom Canonical only when another URL should be preferred.
6. Save or update the content.
7. Review your Sitemap or LLM files when changing Index/Noindex settings that affect their eligibility.

Summary

ByTheWeb GEO Canonical URL & Robots controls:

• Work when ByTheWeb GEO manages the site’s SEO layer.
• Are available for supported Posts, Pages, Custom Post Types, and Taxonomy terms.
• Use the normal permalink or Term URL when the Canonical field is empty.
• Allow a custom Canonical URL when needed.
• Provide Default, Index, and Noindex controls.
• Let Default inherit the relevant Post Type or Taxonomy setting.
• Allow an individual item to override its content-type default.
• Output noindex, nofollow for effectively Noindex content.
• Affect ByTheWeb Sitemap and LLM eligibility.
• Are separate from the dedicated Exclude from LLM files control.
• Do not use AI credits.
• Are left to Yoast SEO or Rank Math SEO when either is the active supported SEO provider.

Where Should I Go Next?

For the product overview:

Canonical URL & Robots Feature Overview

For site-wide indexing defaults and metadata templates:

Title & Description Templates

For Sitemap eligibility:

XML Sitemap

For LLM eligibility and individual LLM exclusion:

llms.txt & llms-full.txt

For crawler-specific robots.txt policies:

Robots.txt AI Policy Manager

For SEO provider compatibility:

Requirements & Compatibility

For indexing problems:

Troubleshooting