SEO Basics for Developers

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

SEO

Search engine optimisation: making pages easy for search engines to find, understand and show.

Crawling

A search engine's program visiting pages by following links and downloading them.

Indexing

Storing a page in the search engine's database so it can appear in results.

robots.txt

A file at the site root that tells crawlers which paths they may visit.

noindex

A robots instruction meaning "don't show this page in search results".

Canonical URL

The one main address for a page that exists at several addresses.

Sitemap

An XML file listing the pages you want search engines to find.

301 redirect

A server reply saying a page has moved permanently to a new address.

hreflang

An attribute that tells search engines which language or region a page version is for.

Search Console

Google's free tool that shows how your site is crawled, indexed and found.

1. 1

   Noindex the page

   In the first example, change the robots content to noindex, follow. Predict the first two console lines, then run. Then delete the robots meta entirely: what does a page with no robots tag mean?

2. 2

   Break hreflang

   Delete the hreflang="en" line. Predict the last console line. Why would Google distrust this hreflang set?

3. 3

   Fix the links

   In the second example, turn the upma <span> into a real <a href>, give "Coming soon" a real target or make it plain text, and rewrite "Click here" as "Poha with peanuts". Run until every line says crawlable with no vague text.

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

1. 1

   SEO (search engine optimisation) is making pages easy to find, understand and choose in search results. Content quality matters most, but developers own the technical part: if a page can't be crawled, rendered or indexed, nothing else counts. Google's crawler is called Googlebot; it discovers pages through links and sitemaps, downloads them, renders them (it runs JavaScript, but later and with limits) and stores what it understands in the index.

2. 2

   robots.txt is a text file at the site root (https://example.com/robots.txt) that tells crawlers which paths they may crawl. It controls crawling, not indexing: a blocked URL can still appear in results (with no description) if other sites link to it. It is public and voluntary, so it is not a security tool; never use it to hide private pages.

3. 3

   To keep a page out of the index, use <meta name="robots" content="noindex"> (or the X-Robots-Tag: noindex HTTP header for PDFs and other files). The crawler must be allowed to fetch the page to see this tag, so don't block a noindexed page in robots.txt. Other values: nofollow (don't follow links on this page), noarchive, nosnippet, max-snippet:[n], max-image-preview:large. On individual links, rel="nofollow", rel="ugc" (user-generated content) and rel="sponsored" (paid links) qualify the link.

4. 4

   The same content often lives at several URLs: with and without www, with tracking parameters (?utm_source=x), with a trailing slash, sorted differently. <link rel="canonical" href="https://example.com/shoes"> names the canonical URL, the main version that should be indexed and credited. Use an absolute URL, put one on every page (pointing to itself if it's the original), and make it agree with your sitemap, internal links and redirects. It's a strong hint, not an order. A sitemap is an XML file listing the URLs you want indexed, with <lastmod> dates; one file holds up to 50,000 URLs or 50 MB. Link it from robots.txt and submit it in Google Search Console.

5. 5

   URLs should be readable, lowercase and stable: /recipes/egg-free-pancakes, not /index.php?id=4821&cat=7. Use hyphens between words (Google treats them as word separators; underscores join words). When a URL must change, send a 301 redirect (permanent) from the old one, so links and ranking move with it. Semantic HTML is SEO too: one clear <h1>, logical headings, real <a href> links (crawlers don't click buttons or run onclick), descriptive link text instead of "click here", alt text on images, <main>, <nav> and <article>. Speed and stability (Google's Core Web Vitals: LCP, INP, CLS) are part of page experience.

6. 6

   If you have the same page in several languages or regions, hreflang links tell search engines which version to show to whom: <link rel="alternate" hreflang="hi" href="https://example.com/hi/">. Values are a language code (en, hi, fr), optionally plus a region (en-IN, en-GB; there is no en-UK). Every version must list all versions including itself, the links must be reciprocal (each page links back), and hreflang="x-default" names the fallback page for everyone else.

• ↔

  Client-side rendering is quick to build but relies on crawlers running JavaScript; server rendering or static generation (what Hectal uses) costs build or server work but gives every crawler complete HTML.

• ↔

  Many filter and sort URLs help users but create near-duplicates; canonicalise them or keep them out of links and sitemaps, at the cost of those variants never ranking on their own.

• I can explain crawling, rendering, indexing and ranking.

• I can write robots.txt and know it doesn't stop indexing.

• I can use noindex correctly, with crawling allowed.

• I put an absolute, self-referencing canonical on every page.

• I can write a sitemap and correct reciprocal hreflang links.

• I build crawlable links and a clear heading structure.