SEO Course
What Is Technical SEO? Crawling, Indexing, Redirects (2026)
By Forsa | SEO Audit and Technical SEO Specialist
Technical SEO explained: how crawling, rendering and indexing work, robots.txt vs noindex, canonical tags, 301 vs 302 redirects and a prioritized checklist.
Updated 31 October 2026 · Part of the SEO guide series
Short answer: Technical SEO is the work that makes sure search engines can find, crawl, render, understand and index your pages, and show the right version of each one. It covers things like robots.txt, sitemaps, site structure, status codes, redirects, canonical tags, JavaScript rendering, page speed, mobile and HTTPS. It does not make weak content rank, but technical problems can stop strong content from ranking at all.
Technical vs on-page vs off-page SEO
| Technical SEO | On-page SEO | Off-page SEO | |
|---|---|---|---|
| Question | Can search engines access and index the site correctly? | Is each page useful and clear? | Do others vouch for the site? |
| Examples | Crawling, indexing, redirects, canonicals, speed | Content, titles, headings, internal links | Links, mentions, reviews |
See our on-page SEO guide for the page-level side.
How Google processes a page
- Discovery: Google finds URLs through links, sitemaps and redirects.
- Crawling: Googlebot requests the URL, if robots.txt allows it.
- Rendering: For pages that rely on JavaScript, Google renders the page to see the final content. Google describes three phases for JavaScript apps: crawling, rendering and indexing, with pages queued for each.
- Indexing: Google analyzes the content, chooses a canonical URL among duplicates and decides whether to store it.
- Serving: Indexed pages can appear in results when relevant.
A problem at any step stops the page before it can rank.
Crawling: robots.txt and what it can't do
robots.txt tells crawlers which URLs they may request. Google is clear that it is mainly for managing crawler traffic and is not a way to keep a page out of Google. A blocked page can still be indexed (without its content) if other pages link to it.
| You want to | Use |
|---|---|
| Stop crawling of low-value URLs (filters, internal search) | robots.txt disallow |
| Keep a page out of search results | noindex (and allow crawling so Google can see it) or password protection |
| Remove a page permanently | Return 404 or 410 |
Example: Reading a robots.txt File
User-agent: *
Disallow: /cart/
Disallow: /search?
User-agent: GPTBot
Disallow: /
Sitemap: https://www.example.com/sitemap.xml
This file lets all crawlers visit the site except for the cart and internal search URLs, blocks one AI crawler (GPTBot) from the whole site and points to the sitemap. A single misplaced line such as Disallow: / under User-agent: * would block every crawler from the whole site, which is a common launch mistake. If you want AI assistants to be able to read your content, check that none of their crawlers are blocked by accident with an AI crawler access check. Decide deliberately, because blocking a crawler also removes you from what that assistant can cite.
Crawl budget: who needs to worry
Google's crawl budget guidance is mainly for very large sites (around 1 million+ unique pages that change weekly) or sites with 10,000+ pages that change daily, plus sites with many URLs Google discovered but hasn't crawled. Most small and medium sites do not need to manage crawl budget; they need clean structure, internal links and no wasted URLs (endless parameters, soft 404s).
Rendering and JavaScript
If your main content only appears after JavaScript runs, Google has to render it first. To reduce risk:
- Serve important content and links in the initial HTML (server-side rendering, static generation or prerendering) where possible.
- Use real
<a href>links for navigation. - Do not hide primary content behind clicks or scrolling events Google won't trigger.
- Check pages with URL Inspection's "View crawled page" to see what Google rendered.
Indexing and canonical tags
When several URLs show the same or very similar content (for example with tracking parameters, HTTP/HTTPS or trailing slashes), Google picks one as canonical and usually shows only that one.
You can influence the choice. Google ranks the signals:
| Signal | Strength |
|---|---|
| Redirects to the preferred URL | Strong |
rel="canonical" link annotation |
Strong |
| Including the URL in your sitemap | Weak |
A canonical tag is a strong hint, not a command; Google can choose a different URL if other signals disagree. Keep signals consistent: canonical tag, internal links, sitemap and redirects should all point to the same URL.
301 vs 302 redirects
| Redirect | Type | Google shows | Use when |
|---|---|---|---|
| 301, 308 | Permanent | The new (target) URL | A page has moved for good; you won't revert |
| 302, 303, 307 | Temporary | The original (source) URL | Short-term moves, such as maintenance or a temporary promotion |
Google says server-side redirects are the most reliable. Avoid long redirect chains (A → B → C); point old URLs straight to the final destination, and update internal links to the final URL.
Status codes that matter
- 200: OK, the page can be indexed.
- 301/308, 302/307: redirects (see above).
- 404/410: page not found or gone; a strong signal not to crawl again.
- Soft 404: a "not found" page that returns 200. Google treats it as an error and it wastes crawling; return a real 404.
- 5xx: server errors; frequent 5xx can slow Google's crawling.
Page experience in brief
Core Web Vitals (LCP, INP, CLS), mobile-friendliness and HTTPS are part of page experience, which Google's ranking systems consider. Our Core Web Vitals guide covers thresholds and fixes; use PageSpeed Insights to test a URL.
Example: Fixing a Canonical Conflict
A store has 3 URLs for the same product: /shoes/runner, /shoes/runner/ and /shoes/runner?utm_source=news. The page's canonical tag points to /shoes/runner/, but the sitemap lists /shoes/runner and internal links use the version with parameters. Google sees mixed signals and may pick a version you did not choose. The fix is to redirect the other forms to /shoes/runner/, keep the canonical tag as it is, list only that URL in the sitemap and update internal links to match.
How to find technical problems
- A quick page check: a free SEO audit reports status, title, headings, meta tags, load time and mobile basics for a single page in seconds.
- Search Console: Page indexing report (why pages aren't indexed), URL Inspection, Core Web Vitals, HTTPS, Sitemaps.
- A crawler (desktop or cloud): broken links, redirect chains, duplicate titles, canonical conflicts, noindex pages.
- Server logs (larger sites): which URLs Googlebot actually requests.
Technical SEO checklist (in priority order)
- Important pages return 200, are not blocked by robots.txt and have no accidental
noindex. - One canonical version of each page; HTTP → HTTPS and www/non-www redirect to it.
- XML sitemap lists canonical, indexable URLs only and is submitted in Search Console. See what is a sitemap.
- Every important page is linked internally with crawlable links.
- Main content is in the HTML or reliably rendered.
- Redirects are permanent where the move is permanent, without chains.
- Real 404s for missing pages; no soft 404s.
- Core Web Vitals pass on key templates; mobile content matches desktop.
- Structured data is valid and matches visible content.
FAQs
What is technical SEO?
The part of SEO that makes sure search engines can crawl, render, index and correctly understand your site: robots rules, sitemaps, status codes, redirects, canonicals, JavaScript, speed, mobile and HTTPS.
What is the difference between a 301 and a 302 redirect?
A 301 is permanent: Google shows the new URL and treats it as canonical. A 302 is temporary: Google keeps showing the original URL. Use 301 when a move is permanent.
What is a canonical tag?
A rel="canonical" link that tells search engines which URL is the preferred version of duplicate or similar pages. Google treats it as a strong signal, though it may choose another URL if other signals conflict.
Does robots.txt stop a page from being indexed?
No. Google says robots.txt is not a mechanism to keep a page out of Google. Use noindex or password protection instead.
How Often Should I Run a Technical SEO Check?
Check Search Console weekly for new errors, run a crawl once a month or after major site changes, and review the full checklist every quarter. Always check after a redesign, migration or CMS update, because those are the moments when noindex tags, redirects and canonicals most often break.
Is Technical SEO Only for Developers?
No. Many checks, such as titles, canonicals, sitemaps and Search Console reports, can be done without code. Developers are needed for server changes, rendering and performance work, so write clear tickets that explain the problem and how to test the fix.
Do small sites need to worry about crawl budget?
Usually not. Google's crawl budget guidance is aimed at very large or very frequently updated sites. Small sites should focus on clean structure, internal links and avoiding wasted URLs.