Google’s Search Console offers tools to identify crawling and indexing problems.
One of the most helpful but often overlooked is the “URL inspection” tool. To use, paste your URL into the “Inspect any URL” field on the top of every Search Console page. You can also access the “URL inspection” tab in the left desktop menu or from the “Pages” section by clicking any URL in the report.
URL Inspection Tool
Identify reasons for not indexing
The URL inspection tool shows obvious non-indexation issues (such as robots.txt blocking) and not-so-obvious.
One of the not-so-obvious in the Pages report is called “Duplicate without user-selected canonical,” wherein Google found two pages with the same content and chose to index one of them. The URL Inspection tool identifies the duplicate pages. Click one of the URLs in the report and then “Inspect URL.”
You may need to fix via 301 redirects or new canonical tags, although it’s unnecessary if Google indexes the right page.
Another indexation glitch to investigate is “Duplicate, Google chose different canonical than user,” where Google indexed a page despite a conflicting canonical tag. This often occurs when the non-http page has the proper canonical tag, but Google indexed the https version. The fix could require a developer to point all URLs to https.
Learn how Google discovered a page
Google discovers web pages via internal or external links and sitemaps. The URL inspection tool shows all links pointing to each indexed page. If a link points to a private page — one you didn’t intend to be crawled or indexed — identify how Google found it. Then correct (internal links) or redirect (external).
Confirm Google can render all page elements
Search Console can confirm whether Google can see, access, and crawl all elements on a web page.
Paste a URL into the inspection tool and click “View crawled text” to access the HTML rendered by Google.
Then copy that code and paste into any text or HTML editor. Use Ctrl+F (CMD+F) to search for the element you want Google to render. For example, you can verify that Google JavaScript links and renders the anchor texts as intended.
Submit your URL for indexation
After updating a page, ask Google to re-index it by clicking “Request indexation.” That will prioritize indexing the updated page over others on your site. Avoid using this option too often or for too many pages. A properly structured site is easily crawled by Google without having to request it.