Lecture 2: How Search Engines Work

Short answer: A search engine discovers pages through links and sitemaps, crawls them by downloading their code, processes them through rendering to see the final content, indexes them in a massive database, and then uses hundreds of ranking signals to decide which pages appear first for any given search. Understanding this full process is the foundation for every SEO decision you will make in this course.

What You'll Learn in This Lecture

• What Are Search Engines and What Do They Actually Do?
• What Do Search Engines Try to Achieve?
• How Do Search Engines Discover URLs?
• How Does Crawling Work in Detail?
• How Does Rendering Work and Why Does It Matter?
• How Does Indexing Work?
• How Does Ranking Work?
• How Do Search Algorithms Work?
• How Does Google Personalize Search Results?
• How Does Location Affect Search Results?
• How Does Language Affect Search Results?
• What Is Crawl Budget and When Does It Matter?
• What Is Index Bloat and How Does It Hurt Rankings?
• Why Do Some Pages Never Get Indexed?
• Why Do Some Pages Never Rank Even After Being Indexed?
• How Do Search Engines Understand Links?
• How Do Search Engines Understand Content?
• How Do Search Engines Understand Entities?
• How Do AI Search Engines Use Web Content?
• How to Apply All of This on a Real Website

What Are Search Engines and What Do They Actually Do?

A search engine is a system that continuously scans the web, stores information about billions of pages, and then matches specific pages to specific queries in a fraction of a second. The 3 major search engines in 2026 are Google (the dominant platform, handling over 90% of global searches), Bing (the second largest, powering Microsoft's AI-driven Copilot answers), and Yandex (the primary engine in Russia). For most SEO work, Google is the primary focus, but understanding how search engines work in general applies to all of them.

A search engine does not browse the web when you type a query. It has already done that work continuously, in advance, and stored everything in an enormous database called an index. When you search, the engine looks up your query against this pre-built index and ranks the best matches in under a second. This means the work that determines your search ranking happens long before anyone searches for your page, during the crawl, render, and index stages.

The 3 essential jobs every search engine performs are: crawling (finding and downloading pages), indexing (storing and organizing them), and ranking (ordering them for specific queries). These 3 stages are sequential and each one is a prerequisite for the next. A page that is not crawled cannot be indexed. A page that is not indexed cannot rank. Understanding where a page is failing in this sequence is the first step to fixing any SEO problem.

Example: A bakery in Denver, Colorado publishes a new page about "custom wedding cakes Denver." Without being crawled, indexed, and ranked, this page is essentially invisible. The owner might think their SEO is working because the page is live on their website, but if Googlebot has never visited it, it does not exist in any search engine's database. Running a Google search for "site:bakerydomainname.com wedding cakes" is the fastest way to check whether the page has been discovered and indexed yet.

What Do Search Engines Try to Achieve?

The goal of every major search engine can be stated in one sentence: give the right searcher the most useful answer for their specific query, as quickly as possible. Every ranking signal, every algorithm update, and every quality guideline that exists is ultimately an attempt to serve this one goal. When Google says it wants "helpful, people-first content," it is not setting an arbitrary standard. It is describing the content that best serves this fundamental mission.

This goal has 3 components that affect SEO strategy directly. First, "right searcher" means search results are personalized. The same query produces slightly different results for different people based on location, device, language, and sometimes prior search behavior. Second, "most useful answer" means relevance and quality are what get rewarded, not technical tricks or keyword manipulation. Third, "as quickly as possible" means page speed and technical performance are not separate from SEO; they are core parts of what makes an answer useful.

The practical consequence for SEO is that tactics designed to fool the algorithm, rather than to genuinely serve the searcher, have a poor long-term track record. Google's algorithm updates over the past 15 years (Panda, Penguin, Hummingbird, RankBrain, BERT, Helpful Content, and the SGE/AI Overview transition) have all, in different ways, been attempts to get rankings to more accurately reflect genuine content quality and user satisfaction. Working with that direction rather than against it is the most reliable long-term SEO strategy.

Example: In 2012, it was possible to rank well for "best running shoes" by repeating the phrase hundreds of times throughout a thin, unhelpful page, a practice called keyword stuffing. Google's 2012 Panda update targeted exactly this kind of low-quality content. By 2015, keyword stuffing was not just ineffective; it actively caused ranking drops. By 2026, a page that genuinely compares 15 specific running shoe models with detailed performance notes, expert author credentials, and real buyer reviews consistently outranks a keyword-stuffed page that contains only generic advice. The algorithm moved closer to what actually serves the searcher.

How Do Search Engines Discover URLs?

A search engine cannot rank a page it does not know exists. URL discovery happens through 3 main channels, and each one has implications for how quickly a new or updated page gets found.

The first and most reliable channel is following links. When a search engine's crawler is on any page it already knows about, it follows every hyperlink on that page to discover new pages. This is why internal linking matters so much: a new page with no internal links from the rest of your site may never be discovered, or may take weeks longer to be found than a page that has 10 links pointing to it from existing, already-indexed pages.

The second channel is XML sitemaps. A sitemap is a file that explicitly lists all the URLs on your website that you want search engines to know about. Submitting your sitemap through Google Search Console is the most direct way to tell Googlebot "these pages exist." This is especially important for large websites where deep pages might not have strong internal linking, and for new websites where the crawler has no existing pages to start from.

The third channel is direct submission. Google Search Console has a "URL inspection" tool that lets you paste any URL and request immediate indexing. This does not guarantee instant indexing, but it puts the URL in the priority queue for the crawler's next pass. This is most useful for time-sensitive pages like a news article or a product launch page where waiting for natural discovery is too slow.

Example: A software company in Austin, Texas publishes a new comparison page: "HubSpot vs Salesforce for Small Business." The page is live but linked from nowhere. Googlebot visits the homepage every few days but has no path to the new comparison page. After 3 weeks, the page is still not indexed. The SEO manager adds a link to the new page from the homepage's navigation, submits the URL through Search Console, and adds it to the sitemap. Within 48 hours, the page is discovered, crawled, and appears in Search Console's index. The content had been ready for 3 weeks; only the discovery pathway was missing.

How Does Crawling Work in Detail?

Crawling is the process of a search engine's automated program (called Googlebot, Bingbot, or simply a spider) visiting a web page, downloading its HTML code, and extracting information from it. The crawler visits thousands of pages per second at a global scale and makes systematic decisions about which pages to crawl, when to re-crawl them, and how deep to go within any given site.

The crawler's primary reading material is the raw HTML of a page: the title tags, meta descriptions, heading hierarchy, paragraph text, anchor text of links, image alt attributes, and structured data. It reads the page exactly as a browser would before JavaScript runs, which is why JavaScript-rendered content presents a special challenge covered in the next section.

Crawling frequency varies by page. Google crawls important, frequently-updated pages (like major news sites and large ecommerce homepages) sometimes several times per day. Small or infrequently-updated pages on low-authority sites may be crawled only once every few weeks or months. The more frequently a page is linked to, the more recent its content, and the higher the overall authority of the site, the more often the crawler will return.

Several things can block or slow crawling even on pages that should be accessible. A disallow rule in robots.txt blocks the crawler from even visiting certain URL patterns. A slow server response means the crawler gives up before the page fully loads. Too many redirect chains (A redirects to B, which redirects to C, which redirects to D) waste crawl resources. And a site that requires login to access content means the crawler can never read the actual page.

Example: A real estate platform in Miami, Florida has 80,000 property listing pages. Each listing page has a URL like /listings/12345. The site also has search filter URLs like /listings?beds=3&baths=2&sort=newest, of which there are potentially millions of combinations. By default, Googlebot would crawl both the real listing pages and the filter pages, wasting the majority of its crawl time on the same properties viewed through different filters rather than on the actual unique listing content. Adding a robots.txt disallow rule for /listings? pattern, combined with noindex tags on filter pages, refocuses all crawl attention on the 80,000 actual listing pages that should rank in search.

How Does Rendering Work and Why Does It Matter?

Rendering is the step where a search engine executes a page's JavaScript code to see the final, fully-assembled version of the page, exactly as a human visitor would see it in a web browser. This step matters because a huge portion of modern websites use JavaScript frameworks (React, Vue, Angular, Next.js) to build their content dynamically, meaning the raw HTML the crawler downloads initially contains very little actual content.

When a crawler first downloads a JavaScript-heavy page, it may see only a small HTML shell: a container element and several script tags. The actual article text, product names, prices, and navigation links only appear after the JavaScript runs. This is client-side rendering (CSR). If the crawler does not also render the page, it will index only the empty shell, not the actual content.

Google does render JavaScript, using a version of the Chrome browser to execute JavaScript before indexing. However, rendering is resource-intensive and happens on a delay. Google often crawls a page first (seeing the empty HTML shell), adds it to a rendering queue, renders it later (sometimes hours or days later), and only then indexes the complete content. During this delay, the page may not rank correctly for its actual content.

Server-side rendering (SSR) solves this problem by generating the complete HTML on the server before the browser or crawler receives it. When a crawler visits an SSR page, the very first response contains all the content. There is no rendering delay. This is why SSR and static site generation (SSG) are preferred for SEO-critical pages in 2026, and why switching from client-side to server-side rendering is one of the most impactful technical SEO changes a JavaScript-heavy site can make.

Example: A fintech startup in San Francisco, California builds their entire website as a React single-page application. Their "How It Works" and "Pricing" pages both use React to load content dynamically. A Googlebot visit to /pricing returns the raw HTML: <div id="root"></div>. Google's crawler queues the page for rendering. 3 days later, the rendering happens and the full pricing content is indexed. But in the 3 days between crawl and render, if Google needs to re-index the page after an update, the same delay repeats. After migrating to Next.js with server-side rendering, the full pricing content is present in the very first HTTP response. Indexing becomes immediate and reliable, and the pages begin ranking for pricing-related queries within 2 weeks.

How Does Indexing Work?

Indexing is the step where a search engine decides whether to permanently store a page in its searchable database and how to categorize it. Not every crawled page gets indexed. The search engine makes a quality and relevance determination for each page after crawling (and rendering, if needed), and some pages are excluded from the index entirely.

A page gets excluded from the index if it: carries a noindex meta tag (an explicit instruction not to index it), is blocked by canonical tags pointing elsewhere (telling Google the "real" version of this content is at a different URL), is considered a near-duplicate of another indexed page (which can lead to one being selected as the canonical and the other ignored), has too little content to be genuinely useful (thin content), or returns an error status code like 404 (not found) or 500 (server error).

When a page is indexed, Google stores specific attributes: the page title, meta description, page content, structured data, and the relationships between the page and others in the site. This stored information is what gets retrieved and compared against queries during the ranking phase. The accuracy and completeness of what is stored during indexing directly affects how well the page ranks.

You can check indexing status in 2 ways. First, search site:yourdomain.com in Google. Pages that appear in these results are indexed. Pages that do not appear are not. Second, use Google Search Console's URL inspection tool to see the exact indexing status of any specific URL, including when it was last crawled, what Googlebot saw, and whether any issues were detected.

Example: A healthcare information site in Boston, Massachusetts publishes 500 articles about medical conditions. 200 of these articles are very short (under 300 words) and essentially duplicate information already covered more thoroughly in the other 300 articles. Google indexes the 300 good articles but treats many of the 200 thin articles as low-quality and either does not index them or deindexes them after early testing. After a content audit, the team merges 150 of the thin articles into related, stronger articles and deletes the remaining 50. The site's overall indexing health improves, and several of the merged articles begin ranking significantly better because the consolidated versions have more depth than any individual thin article did.

How Does Ranking Work?

Ranking is the final stage: the search engine's algorithm evaluates all the indexed pages relevant to a specific query and decides their order. This happens every time someone searches, in milliseconds, across billions of query-page combinations. The algorithm uses hundreds of signals simultaneously to make this determination.

The most influential ranking signals in 2026 fall into 5 broad categories. First, relevance: does the page content match what the searcher is actually looking for? This is evaluated through semantic analysis, not just keyword matching. Second, authority: do other trusted websites link to this page, and is the site it comes from generally trusted? Third, experience: does the page load quickly, work on mobile, and provide a non-frustrating user experience? Fourth, expertise and trust: is the page created by someone with genuine knowledge of the topic, and does the site's overall quality signal trust? Fifth, freshness: for time-sensitive topics, is the content recent and updated?

No single signal determines ranking. A page with perfect content but no backlinks will usually lose to a page with slightly less perfect content but strong external links. A page with perfect content and strong links but a 10-second load time will lose to one that loads in 1.5 seconds. Rankings are the result of a holistic evaluation, not a single factor. This is why "magic bullet" SEO promises are always misleading.

Example: Two competing pages both target the keyword "best HVAC companies in Chicago." Page A has excellent content with specific company reviews, pricing ranges, and a customer satisfaction comparison. Page B has weaker content but 40 links from local news sites and contractor directories. Page A is on a site with a 4-second load time. Page B loads in 1.3 seconds. In the ranking algorithm's evaluation, Page B's combination of authority signals and technical performance outweighs Page A's content advantage. The lesson for the Page A owner is not to weaken their content but to improve page speed and earn local citation links, creating a more complete ranking signal profile.

How Do Search Algorithms Work?

A search algorithm is not a single formula. It is a layered system of hundreds of individual signals and sub-algorithms that each evaluate a specific aspect of a page's quality or relevance, and then combine their scores to produce a final ranking. Google's algorithm has never been publicly disclosed in full, but through years of research, documented updates, and Google's own statements, its major components are well understood.

The key algorithmic components in Google's 2026 ranking system include: PageRank (the original link-authority scoring system, still active in an evolved form), RankBrain (a machine learning system that interprets unusual or ambiguous queries and maps them to the most relevant results), BERT and MUM (natural language processing models that understand the meaning of queries and content at a nuanced level, not just at a keyword level), and the Helpful Content System (a site-wide signal that rewards sites producing primarily original, genuinely useful content and penalizes sites that primarily produce content for search engine manipulation).

Algorithm updates happen continuously. Small "core" updates happen daily. Broad core updates, which recalibrate how multiple signals are weighted, happen several times per year and often produce noticeable ranking shifts. Understanding that algorithms update constantly is essential because SEO is not a one-time task. A page that ranks well today can rank differently next month as the algorithm evolves, which is why ongoing monitoring and content maintenance are part of every professional SEO strategy.

Example: A personal finance blog in New York, New York ranked in the top 3 for "how to invest $10,000" for 2 years. In August 2023, Google launched a broad core update that reweighed E-E-A-T signals (Experience, Expertise, Authoritativeness, Trustworthiness) more heavily. The blog's articles had no named authors, no credentials, and no cited sources. They dropped from position 2 to position 18. Competing articles from financial advisors with named, credentialed authors rose to fill the top positions. The algorithm update did not change what good content looks like. It changed how much weight the algorithm placed on being able to verify the content's expertise. The blog recovered 6 months later after adding named author pages with credentials and adding source citations to every major claim.

How Does Google Personalize Search Results?

Google does not show the same results to every person for the same query. Personalization is built into Google's ranking system, and it adjusts results based on several factors specific to each individual searcher. This is why 2 people in different situations may see completely different top results for identical search terms.

The primary personalization signals are: device type (mobile vs. desktop results sometimes differ significantly), location (more on this in the next section), language preference (set in browser settings or by the language of the typed query), search history (for logged-in Google account users, recent searches may influence what appears), and personalized SafeSearch and content preferences. For most informational and commercial queries, personalization effects are moderate. For local, news, and entertainment queries, personalization can dramatically change what appears at the top.

The practical implication for SEO tracking is that checking your own rankings from your own device is unreliable. Your location, search history, and account settings all influence what you see. Professional rank tracking tools solve this by simulating searches from neutral, logged-out browser sessions in specific geographic locations, producing results that reflect what a typical searcher would see rather than what your personal profile shows you.

Example: An SEO manager at a software company in Seattle, Washington checks their rankings for "project management software" from their office browser and sees their product at position 4. A potential customer in Phoenix, Arizona searches the same phrase and sees a different result set influenced by Phoenix-area business directory signals. A colleague in London sees entirely different results because UK Google (.co.uk) serves different pages. When the SEO manager switches to an incognito window with a VPN set to a specific U.S. location, they see the same results their target audience sees, which is position 7, not position 4. The difference represents the personalization gap between their account's history and a neutral searcher's view.

How Does Location Affect Search Results?

Location is one of the most powerful personalization signals in Google's algorithm. For any query that implies a local intent (whether explicitly stated like "plumber near me" or implicitly local like "best dentist" or "pizza delivery"), Google's algorithm heavily weights proximity between the searcher and the business or content. This is called local search, and it is governed by a separate part of Google's algorithm from organic search.

Google uses multiple signals to determine a searcher's location: their device's GPS coordinates (most precise, used for mobile searches with location sharing enabled), their IP address (less precise, used when GPS is not available), their home location set in their Google account, and explicit location modifiers in the query ("restaurants in Chicago"). For businesses serving a specific geographic area, location signals are the primary determinant of whether they appear in local search results at all.

For non-local queries, location still matters but more subtly. Content published on a U.S.-hosted site using U.S. English tends to rank better in U.S. Google searches than content from a site hosted in the UK, even when the language is identical. This is why international websites typically use country-specific domains or subdirectories (/us/, /uk/) rather than a single global URL for all markets.

Example: A roofing company in Philadelphia, Pennsylvania has offices in both Philadelphia and nearby Cherry Hill, New Jersey. They use one website with one phone number and one address listed, the Philadelphia headquarters. When residents of Cherry Hill search "roofing company near me," the company rarely appears because their listed location is 8 miles away and competitors with Cherry Hill addresses appear first. After creating a separate Google Business Profile for their Cherry Hill service area (with the Cherry Hill address, a local phone number, and Cherry Hill-specific reviews), they begin appearing in the local results for nearby New Jersey searchers. Same company, 8-mile difference, dramatically different local ranking results.

How Does Language Affect Search Results?

Search engines are optimized to match content to searchers in their preferred language. The language of a query is the strongest signal: a search in Spanish returns Spanish-language results. Browser language settings and Google account language preferences provide additional signals. For multilingual websites or businesses serving multiple language markets, getting language signals right is a critical SEO factor.

Google uses the hreflang attribute on web pages to understand which language and country a specific version of a page is intended for. Without hreflang, Google may show the wrong language version of a page to searchers, or may consolidate all versions as duplicates and index only one. Proper hreflang implementation tells Google: "The /es/ version of this page is for Spanish speakers in Mexico, the /en-us/ version is for English speakers in the United States, and the /en-gb/ version is for English speakers in the United Kingdom."

Language-matching also affects the quality of ranking in complex ways. A page written in formal academic English may rank differently from a page covering the same topic in everyday conversational English, because Google's BERT model understands language at a nuanced level. The right "register" of language, formal or informal, technical or plain, depends on matching what the target searcher is likely to be looking for.

Example: A travel agency in Chicago, Illinois serves both English-speaking and Spanish-speaking customers. Their website has one English version of every page. When Spanish-speaking users search "mejores agencias de viajes en Chicago," the agency never appears because they have no Spanish-language content. After creating a /es/ subdirectory with properly translated Spanish versions of their 10 most important pages, with hreflang tags connecting each English page to its Spanish equivalent, they begin appearing in Spanish-language Google searches within 6 weeks and generate 28 new Spanish-speaking inquiries in the first 2 months, a market segment they had been completely invisible to before.

What Is Crawl Budget and When Does It Matter?

Crawl budget is the amount of time and server resources Googlebot allocates to crawling a specific website per day. For small websites with a few hundred pages, crawl budget is rarely a concern: Googlebot will crawl the entire site quickly and spend its remaining time re-crawling updated pages. For large websites with thousands or millions of pages, crawl budget becomes a strategic concern: if Googlebot wastes its limited allocation on low-value pages, important pages may be crawled infrequently or missed entirely.

Crawl budget is influenced by 2 factors in opposite directions. Crawl demand (how much Google wants to crawl a site) goes up when a site has high authority, frequent updates, and many external links. Crawl capacity limit (how much Googlebot can crawl without overloading the server) goes up when the server responds quickly and reliably. A slow server that frequently times out or returns errors will have its crawl rate limited by Google to avoid causing downtime, which can severely restrict how much of the site gets crawled.

The most common crawl budget problems are: thin or duplicate pages that consume crawl allocation without contributing ranking value (filtered search pages, parameterized URLs, paginated archive pages with no unique content), broken pages returning 404 or 500 errors that waste crawl attempts, and redirect chains that use multiple hops between crawl attempts. Fixing these issues frees the crawl budget to focus on the pages that actually matter for rankings.

Example: An online clothing retailer in Los Angeles, California has 25,000 real product pages but also 800,000 filtered search result pages (combinations like /dresses?color=red&size=S&sort=bestsellers&material=cotton). Googlebot allocates 50,000 crawl slots per day to the site. Without intervention, the crawler spends the majority of those slots on the 800,000 filter page combinations, leaving many of the 25,000 actual product pages crawled only once every few weeks. After adding disallow rules for filter URL patterns in robots.txt and noindex tags on the remaining filter pages, Googlebot's entire allocation focuses on the real product pages. New products are indexed within 24 hours instead of weeks, and the site's organic traffic increases 22% over the following 3 months as previously ignored product pages finally appear in search results.

What Is Index Bloat and How Does It Hurt Rankings?

Index bloat occurs when a website has a large number of low-quality, thin, or near-duplicate pages included in a search engine's index. This matters for rankings because search engines evaluate site quality in part at the site level, not just the page level. A site with thousands of empty tag pages, session-ID URL variations, or nearly identical location pages can be evaluated as a lower-quality site overall, dragging down the rankings of even the genuinely excellent pages on the same domain.

Common causes of index bloat include: WordPress tag and category archive pages with only 1 or 2 posts each (each creating a separate indexed URL with nearly identical content), parameterized URLs that create hundreds of versions of the same page (session IDs, tracking parameters, sort order variations), printer-friendly page versions, paginated archive pages showing the same posts with only a different range, and thin boilerplate location pages ("our plumbing services in [every city name]" with identical content across hundreds of pages).

The fix depends on the page type. Low-value archive pages can be noindexed (they still exist and serve user navigation but stay out of the index). Parameterized URL variations can be consolidated with canonical tags pointing to the non-parameterized version. Identical or near-identical location pages can be either merged into a single page or rewritten with genuinely unique local content if local ranking value is important. Running a crawl with a tool like Screaming Frog and counting the total indexed pages versus total pages with unique, substantial content is the starting point for diagnosing index bloat.

Example: A law firm in Dallas, Texas has a website with 50 real service pages and 2,000 indexed pages in Google Search Console. The 2,000 pages include: 50 real pages, 500 empty tag archive pages, 300 session ID variations of the homepage, and 1,150 "service in [city name]" pages with near-identical boilerplate content. Google treats the site as having 2,000 pages, most of which are thin. After cleaning the index (noindexing all archive pages, adding canonical tags to session ID URLs, and rewriting 20 of the location pages with genuine local content while noindexing the rest), the site's effective index drops to 120 high-quality pages. Within 4 months, several of their key service pages rise from position 8-12 to position 2-5 in organic results.

Why Do Some Pages Never Get Indexed?

A page that exists and loads correctly in a browser can still fail to appear in any search engine's index for several distinct reasons, and diagnosing which reason applies is the first step toward a fix.

The most common indexing blockers are: a noindex meta tag in the page's HTML head (which explicitly instructs crawlers not to index the page), a Disallow rule in the robots.txt file that blocks the crawler from even reaching the page, a canonical tag pointing to a different URL (telling the search engine to treat another page as the "official" version and skip this one), the page only being reachable through JavaScript (which requires rendering, which may not happen promptly for low-priority pages), the page returning a non-200 status code (like 404 Not Found or 301 Redirect), and a complete absence of internal links making the page undiscoverable by following links through the site.

Checking indexing status in Google Search Console is the definitive diagnosis tool. The URL inspection tool shows: whether the page has been indexed, when it was last crawled, what Googlebot saw when it crawled it (including the rendered HTML), and any specific reason Google found to exclude it from the index. This inspection output eliminates guesswork and points directly to the technical issue causing the indexing failure.

Example: A marketing agency in Atlanta, Georgia builds a new case study page that is crucial for sales. After 3 weeks, the page does not appear in any Google search. The SEO specialist uses the URL inspection tool in Search Console and finds: "Page is blocked by robots.txt." Investigation reveals that the developer who built the site added a blanket Disallow: /case-studies/ rule to robots.txt during development to prevent the staging site from ranking, and never removed the rule when the site went live. Removing the single robots.txt line allows Googlebot to crawl the page. Within 4 days, it is indexed. Within 3 weeks, it ranks on page 1 for the targeted case study keywords the agency had been trying to reach for months.

Why Do Some Pages Never Rank Even After Being Indexed?

Being indexed is necessary for ranking, but it does not guarantee ranking. A page that is fully indexed can still fail to appear in the first 10 pages of results, effectively making it invisible to searchers. There are 4 primary reasons this happens.

First, poor relevance match: the page's content does not adequately address the search intent behind the target query. A page titled "best laptops" that focuses primarily on gaming laptops will not rank well for "best laptops for graphic designers" because it does not answer that specific need. Second, insufficient authority: the page has no backlinks from other credible sites, and the domain it sits on has low overall authority. Search engines use external signals to establish trust, and content with no trust signals rarely competes against content with strong link profiles. Third, poor content depth: the page provides a surface-level overview of a topic while competing pages provide comprehensive, specific, expert-level coverage. For most competitive topics, shallow content simply does not rank well regardless of how well it is technically optimized. Fourth, user experience problems: slow loading, intrusive popups, confusing navigation, or a mobile layout that breaks significantly reduce the ranking potential of otherwise good content.

Example: A nutrition coaching company in Chicago, Illinois publishes a 400-word article targeting "meal prep for weight loss." They check their rankings monthly for 6 months and the page never appears in the top 50. An audit reveals 3 problems: the 3 competing pages in positions 1-3 all have 2,000 to 3,500 words with specific meal plans, calorie counts, and shopping lists (depth gap), the nutrition company's domain has only 12 backlinks from low-authority sites (authority gap), and the page loads in 7.8 seconds on mobile (speed gap). After expanding the article to 2,200 words with a real 7-day meal plan and calorie breakdown, improving page speed to 2.1 seconds, and earning 4 links from fitness blogs through a guest post campaign, the page rises from position 58 to position 11 within 8 weeks. Indexing was never the problem. Competitiveness was.

How Do Search Engines Understand Links?

Links are one of the most important ranking signals in search engine algorithms, and they work through 2 distinct mechanisms: link equity (the authority one page passes to another through a link) and anchor text (the words used in the clickable link, which give context about what the destination page covers).

Link equity flows from pages with high authority to pages they link to. A page linked by the New York Times, NASA, and Harvard University is treated as more credible by search engines than a page linked only by 3 small personal blogs. This is the original insight behind PageRank, the algorithm Google's founders Larry Page and Sergey Brin developed in 1996: the web itself is a voting system, and links are votes. Not all votes are equal; votes from already-trusted sources carry more weight.

Anchor text tells the search engine what the linked page is about from the perspective of the linking page. A link with anchor text "best running shoes for flat feet" is a strong relevance signal for the destination page's topic. A link with generic anchor text "click here" provides no topical context. This is why high-quality links with relevant, natural anchor text (not stuffed with keywords) are more valuable than generic links, and why manipulative anchor text schemes, like buying 100 links all using the exact target keyword, are specifically targeted by Google's Penguin algorithm.

Example: A home improvement website in Phoenix, Arizona earns 3 links. Link 1: from a national home improvement magazine with anchor text "best paint colors for small bedrooms." Link 2: from a 5-year-old, regularly updated interior design blog with anchor text "bedroom color ideas." Link 3: from a newly created, content-thin website with anchor text "best paint colors for small bedrooms" as part of a paid link scheme. Google's algorithm heavily weights Link 1 (high authority, relevant anchor text), gives moderate credit to Link 2 (medium authority, relevant but generic anchor), and either ignores or penalizes Link 3 (low authority, suspicious pattern). The site's ranking improvement comes entirely from the first 2 natural links, not from the paid scheme.

How Do Search Engines Understand Content?

Modern search engines understand content at a semantic level, not just at the level of matching keywords. Using natural language processing (NLP) models like Google's BERT and MUM, search engines can understand the meaning behind queries and content, including context, intent, and the relationship between concepts, regardless of whether the exact same words are used.

This means keyword optimization in 2026 is not about repeating exact phrases a set number of times. It is about writing content that thoroughly covers a topic, uses the natural vocabulary of that topic (including related concepts, synonyms, and real-world context), and addresses the actual questions searchers are asking. A page about "home HVAC maintenance" that never uses the words "heating" or "air conditioning" directly will likely rank for those terms if the content clearly covers the same topic, because semantic understanding connects the vocabulary.

Search engines also look for expertise signals within content. Specific, accurate technical detail, real data with named sources, direct explanations of mechanisms rather than vague generalities, and acknowledgment of nuance and edge cases all contribute to a content quality signal that tells the algorithm this page was written by someone who genuinely understands the subject. Generic AI-generated summaries that cover surface topics without specificity increasingly rank below content with genuine depth and original insight.

Example: A financial planning firm in Seattle, Washington publishes 2 articles about retirement savings. Article A: "Start saving early, diversify your portfolio, and maximize your 401(k) contributions." Article B: "For a 35-year-old earning $85,000 annually, contributing the 2024 IRS maximum of $23,000 per year to a 401(k) at a 7% average annual return produces approximately $1.3 million by age 65, enough to sustain roughly $52,000 per year in withdrawals for 25 years using a 4% drawdown rate." Google's semantic understanding recognizes Article B as more specific, more useful, and more expert. Despite the same topic and similar keyword usage, Article B outranks Article A for retirement planning queries because its content depth sends stronger expertise signals.

How Do Search Engines Understand Entities?

An entity is any distinct real-world thing: a person, a place, a business, a product, a concept, or an organization. Search engines build a knowledge graph of entities and the relationships between them. Understanding entities allows search engines to interpret queries precisely, even when the same word could refer to multiple different things.

The Google Knowledge Graph, first introduced in 2012, stores billions of entities and their properties. When you search "Apple CEO," Google knows "Apple" here means the technology company, not the fruit, because "CEO" is a property associated with organizations, not produce. When you search for "Mercury," Google uses the rest of the query to determine whether you mean the planet, the chemical element, the Roman god, or the car brand. Entities allow search engines to disambiguate and provide precise answers without requiring the searcher to be specific.

For SEO, entity optimization means making sure your website clearly establishes the entities it represents: your business name, your location, your category, your founders, your products or services, and how all of these connect. Schema markup (covered in detail in Lecture 14) is the structured data language that explicitly defines entities and their properties for search engines. A business with clear entity signals, a Google Business Profile, linked social profiles, named authors with credentials, and schema markup is far more likely to be understood correctly and displayed accurately than a business with weak entity signals.

Example: A technology consulting firm in Boston, Massachusetts is named "Apex Solutions." There is also a gym chain and a staffing firm named "Apex Solutions" in other states. Without clear entity signals, Google might confuse which "Apex Solutions" a searcher is looking for. After the consulting firm adds Organization schema with their specific category (IT consulting), city (Boston), founding year, and links to their LinkedIn Company Page and Crunchbase profile (the sameAs property in schema), Google's knowledge graph creates a distinct entity for this specific Apex Solutions. Their business information begins appearing correctly in the Knowledge Panel on the right side of relevant search results, and they stop being confused with the other businesses of the same name.

How Do AI Search Engines Use Web Content?

AI-powered search features, including Google AI Overviews, Microsoft Copilot, Perplexity, and ChatGPT web search, all use the same fundamental web infrastructure described in this lecture. They send crawlers to collect web content, they index and process that content, and they use retrieval systems to pull relevant content when answering queries. What is different is what happens after retrieval: instead of returning a ranked list of links, they generate a synthesized, prose answer that combines information from multiple sources.

The implication for SEO is that the same content quality signals that help pages rank in traditional search, directness, specificity, expertise, structured formatting, and factual accuracy, also make pages more likely to be cited as sources in AI-generated answers. But there are additional signals specific to AI retrieval: content that is organized into self-contained, clearly-headed sections is easier to chunk and retrieve. Content with explicit author credentials is more trusted as a citation source. And content with FAQPage schema, which formats questions and answers in machine-readable code, is directly usable by AI answer generation systems.

For a website that wants visibility in both traditional search and AI-generated answers in 2026, the strategy is to build content quality that serves both simultaneously. The full GEO (Generative Engine Optimization) course on this site covers the AI-specific layer in complete detail, but the foundation starts with the traditional SEO fundamentals in this course.

Example: A personal injury law firm in Los Angeles, California publishes a detailed FAQ page: "What to Do After a Car Accident in California." Each FAQ answer is 150 to 200 words, directly answers the question without preamble, and includes the specific California law or procedure where applicable. The page includes FAQPage schema. In traditional Google search, the page earns a featured snippet for several questions. In Google AI Overviews, the page is cited as a source when users ask about California car accident procedures. In Perplexity, it is cited when users ask "what should I do if I was hit by an uninsured driver in California?" The same content quality investments produce visibility across 3 different search formats simultaneously.

Common Mistakes to Avoid

• Assuming a published page is automatically indexed the same day or even the same week.
• Leaving noindex tags on important pages after migrating from a staging to production environment.
• Allowing parameterized or filter URLs to be crawled and indexed, wasting crawl budget on duplicate content.
• Checking personal rankings from a logged-in browser and treating them as the true search results your audience sees.
• Publishing thin content and expecting rankings to improve over time without addressing the depth gap versus competitors.
• Ignoring mobile rendering and speed issues, which affect rankings even for pages that look fine on desktop.
• Building backlinks with overly exact-match anchor text, which triggers spam filters in Google's link quality algorithm.

Action Checklist

• Open Google Search Console and check the Pages report for indexing issues: "Discovered, not indexed," "Crawled, not indexed," and "Excluded" pages.
• Use the URL Inspection tool on your top 5 most important pages to confirm they are indexed and see what Googlebot actually read.
• Search robots.txt on your domain and verify no important page types are accidentally blocked.
• Run a site: search in Google to see the approximate number of indexed pages and cross-check it against your total page count.
• Check your page load time on mobile using Google's PageSpeed Insights. Flag any pages loading in over 3 seconds for improvement.
• Identify whether your site uses client-side JavaScript rendering for key content. Investigate server-side rendering if AI citation visibility is a priority.
• Verify that your top pages have at least some internal links pointing to them from other pages on the site.

Practice Task

Pick one page on your site that is not ranking as well as you want. Run through the full diagnostic flow: indexing status, crawl access, rendering, relevance match, authority, and content depth. Use the table below to document what you find and what you will do about it.

Diagnostic Check	What to Look For	What You Found	Action
Indexing status	Is the page in Google's index?	Indexed / Not indexed / Excluded	Fix block / Request indexing / Investigate
Crawl access	Is robots.txt or noindex blocking access?	Open / Blocked by rule / Noindex tag found	Remove blocker
Rendering	Does URL inspection show the full page content?	Full content / Partial / Empty	Switch to SSR if empty
Relevance match	Does content fully address the search intent?	Full match / Partial / Wrong intent	Rewrite or restructure
Authority	How many backlinks does the page have?	Strong / Weak / None	Link building campaign
Content depth	Does it match competitor depth for the same query?	Deeper / Equal / Shallower	Expand content

Related Lessons Across SEO, AEO, GEO, SEM, and PPC

Use these connected lessons to move through organic search, answer engines, generative AI visibility, paid search, and PPC without losing the bigger strategy.

• Lecture - 7: How AI Chatbots (ChatGPT, Gemini, Perplexity) Answer Questions (AEO) - understand how answer systems choose sources.
• Lecture - 4: How to Write Content That AI Systems Can Retrieve, Summarize, and Trust (GEO) - make content easier for AI systems to retrieve.
• Lecture 25: AI and Automation in SEM: Smart Bidding, AI Overviews, and the Future (SEM) - see how automation changes paid search.
• Lecture - 1: What Is AEO? How Answer Engines Are Different From Search Engines (AEO) - see how answer engines build on SEO foundations.
• Lecture - 1: What Is GEO? How Generative AI Search Works in 2026 (GEO) - connect classic SEO with generative engine visibility.

Course Links

Each lecture builds on what came before. Understanding how search engines work (this lecture) is what makes everything in Lecture 3 and beyond make sense.

• Back to Lecture 1: SEO Fundamentals
• Continue to Lecture 3: SEO Setup and Website Foundation
• Run a Free SEO Audit on Your Site

Trusted References

For Google's own documentation on how search works, see the How Search Works overview and the Crawling and Indexing documentation. For crawl budget management guidance, see Google's crawl budget guidance for large sites.

FAQs

What Is the Difference Between Crawling and Indexing?

Crawling means a search engine's bot visits the page and downloads its code. Indexing means the search engine decides to store that page in its searchable database so it can appear in results. A page can be crawled but still not indexed if it is too thin, a near-duplicate of another page, carries a noindex tag, or fails other quality thresholds.

How Long Does It Take for a New Page to Get Indexed?

It depends on the site's authority and crawl history. New pages on high-authority, frequently-crawled domains can be indexed within hours of being linked to. New pages on low-authority, infrequently-crawled sites can take 2 to 4 weeks. Submitting the URL through Google Search Console significantly speeds up the process for any site.

Why Is a Page Indexed but Not Ranking?

Indexing only means the page is stored in Google's database. Ranking depends on how the page compares to all other indexed pages for the same query. A page can be indexed and still rank on page 10 or beyond if it lacks sufficient authority, content depth, or relevance match compared to stronger competitors.

Does Site Speed Affect Search Rankings?

Yes. Site speed is a confirmed Google ranking signal, measured through Core Web Vitals (Largest Contentful Paint, Interaction to Next Paint, and Cumulative Layout Shift). Slow pages rank below faster pages when content quality is otherwise equivalent. On mobile, speed differences of even a few seconds can produce measurable ranking differences for competitive queries.

Do AI Search Tools Crawl the Web the Same Way Google Does?

AI search tools like Perplexity, ChatGPT web search, and Google AI Overviews all depend on web crawling in some form. GPTBot (OpenAI's crawler), ClaudeBot (Anthropic's crawler), and PerplexityBot each crawl websites using their own user-agents. Keeping your robots.txt open to these crawlers and ensuring your content is server-rendered (not JavaScript-only) is the foundation for being included in AI-generated answers.