SEO Technical

Technical SEO Implementation (Next.js 2025)

Skill Files

This skill includes multiple reference files:

• SKILL.md (this file): Core technical SEO implementation guide
• nextjs-implementation.md: Next.js-specific code templates and patterns
• checklist.md: Pre-launch technical SEO checklist
• structured-data.md: JSON-LD schema markup templates

What This Skill Covers

1. Sitemaps → app/sitemap.ts for dynamic sitemap generation
2. Robots.txt → app/robots.ts for crawler directives
3. Meta Tags → OpenGraph, Twitter Cards, keywords, descriptions
4. Structured Data → JSON-LD for rich snippets
5. Canonical URLs → Prevent duplicate content issues
6. Performance SEO → Core Web Vitals considerations

Part 1: Sitemap Implementation

Next.js App Router Sitemap (app/sitemap.ts)

Next.js automatically serves /sitemap.xml when you create app/sitemap.ts:

import type { MetadataRoute } from "next";

const BASE_URL = process.env.NEXT_PUBLIC_SITE_URL || "https://example.com";

export default function sitemap(): MetadataRoute.Sitemap {
  const currentDate = new Date().toISOString();

  // Static pages
  const staticPages: MetadataRoute.Sitemap = [
    {
      url: BASE_URL,
      lastModified: currentDate,
      changeFrequency: "weekly",
      priority: 1.0,
    },
    {
      url: `${BASE_URL}/pricing`,
      lastModified: currentDate,
      changeFrequency: "monthly",
      priority: 0.8,
    },
    {
      url: `${BASE_URL}/about`,
      lastModified: currentDate,
      changeFrequency: "monthly",
      priority: 0.7,
    },
    {
      url: `${BASE_URL}/privacy`,
      lastModified: currentDate,
      changeFrequency: "yearly",
      priority: 0.3,
    },
    {
      url: `${BASE_URL}/terms`,
      lastModified: currentDate,
      changeFrequency: "yearly",
      priority: 0.3,
    },
  ];

  return staticPages;
}

Dynamic Sitemap with Database Content

import type { MetadataRoute } from "next";
import { db } from "@/lib/db";
import { blogPosts, products } from "@/lib/db/schema";

const BASE_URL = process.env.NEXT_PUBLIC_SITE_URL || "https://example.com";

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  // Fetch dynamic content
  const posts = await db.select().from(blogPosts).where(eq(blogPosts.published, true));
  const allProducts = await db.select().from(products);

  const staticPages: MetadataRoute.Sitemap = [
    { url: BASE_URL, lastModified: new Date(), changeFrequency: "weekly", priority: 1.0 },
  ];

  const blogPages: MetadataRoute.Sitemap = posts.map((post) => ({
    url: `${BASE_URL}/blog/${post.slug}`,
    lastModified: post.updatedAt || post.createdAt,
    changeFrequency: "weekly" as const,
    priority: 0.7,
  }));

  const productPages: MetadataRoute.Sitemap = allProducts.map((product) => ({
    url: `${BASE_URL}/products/${product.slug}`,
    lastModified: product.updatedAt,
    changeFrequency: "daily" as const,
    priority: 0.8,
  }));

  return [...staticPages, ...blogPages, ...productPages];
}

Large Sitemaps (50,000+ URLs)

Use generateSitemaps() for sitemap index:

import type { MetadataRoute } from "next";

const URLS_PER_SITEMAP = 50000;

export async function generateSitemaps() {
  const totalProducts = await getProductCount();
  const sitemapCount = Math.ceil(totalProducts / URLS_PER_SITEMAP);

  return Array.from({ length: sitemapCount }, (_, i) => ({ id: i }));
}

export default async function sitemap({ id }: { id: number }): Promise<MetadataRoute.Sitemap> {
  const start = id * URLS_PER_SITEMAP;
  const products = await getProducts({ start, limit: URLS_PER_SITEMAP });

  return products.map((product) => ({
    url: `${BASE_URL}/products/${product.slug}`,
    lastModified: product.updatedAt,
  }));
}

Sitemap Best Practices

Part 2: Robots.txt Implementation

Next.js App Router Robots (app/robots.ts)

import type { MetadataRoute } from "next";

const BASE_URL = process.env.NEXT_PUBLIC_SITE_URL || "https://example.com";

export default function robots(): MetadataRoute.Robots {
  const isProduction = process.env.NODE_ENV === "production";

  // Block everything in non-production
  if (!isProduction) {
    return {
      rules: { userAgent: "*", disallow: "/" },
    };
  }

  return {
    rules: [
      {
        userAgent: "*",
        allow: "/",
        disallow: [
          "/api/",
          "/dashboard/",
          "/admin/",
          "/private/",
          "/_next/",
          "/sign-in/",
          "/sign-up/",
        ],
      },
      // Block AI training bots (optional)
      { userAgent: "GPTBot", disallow: "/" },
      { userAgent: "ChatGPT-User", disallow: "/" },
      { userAgent: "CCBot", disallow: "/" },
      { userAgent: "anthropic-ai", disallow: "/" },
      { userAgent: "Google-Extended", disallow: "/" },
    ],
    sitemap: `${BASE_URL}/sitemap.xml`,
  };
}

Robots.txt Rules

Common AI Bots to Block/Allow

// Block AI training (keeps content out of training data)
{ userAgent: "GPTBot", disallow: "/" },           // OpenAI
{ userAgent: "ChatGPT-User", disallow: "/" },     // ChatGPT browsing
{ userAgent: "CCBot", disallow: "/" },            // Common Crawl
{ userAgent: "anthropic-ai", disallow: "/" },     // Anthropic
{ userAgent: "Google-Extended", disallow: "/" },  // Google AI training
{ userAgent: "Bytespider", disallow: "/" },       // ByteDance

// Allow AI search (keeps content in AI search results)
// Comment out the above to allow AI indexing

What NOT to Block

• Don't block /sitemap.xml
• Don't block CSS/JS files (/_next/static/)
• Don't block images you want indexed
• Don't block your homepage

Part 3: Metadata Implementation

Root Layout Metadata (app/layout.tsx)

import type { Metadata, Viewport } from "next";

const BASE_URL = process.env.NEXT_PUBLIC_SITE_URL || "https://example.com";

export const viewport: Viewport = {
  width: "device-width",
  initialScale: 1,
  themeColor: "#6366f1",
};

export const metadata: Metadata = {
  metadataBase: new URL(BASE_URL),

  // Title template for child pages
  title: {
    default: "Brand Name — Tagline",
    template: "%s | Brand Name",
  },

  // Description (150-160 chars ideal)
  description: "Your compelling meta description that includes primary keywords and encourages clicks.",

  // Keywords (less important now, but include)
  keywords: ["primary keyword", "secondary keyword", "brand name"],

  // Author info
  authors: [{ name: "Brand Name" }],
  creator: "Brand Name",
  publisher: "Brand Name",

  // Robots directives
  robots: {
    index: true,
    follow: true,
    googleBot: {
      index: true,
      follow: true,
      "max-video-preview": -1,
      "max-image-preview": "large",
      "max-snippet": -1,
    },
  },

  // OpenGraph (Facebook, LinkedIn, etc.)
  openGraph: {
    type: "website",
    locale: "en_US",
    url: BASE_URL,
    siteName: "Brand Name",
    title: "Brand Name — Tagline",
    description: "Your compelling description for social sharing.",
    images: [
      {
        url: "/og-image.png",
        width: 1200,
        height: 630,
        alt: "Brand Name - Description",
      },
    ],
  },

  // Twitter Card
  twitter: {
    card: "summary_large_image",
    title: "Brand Name — Tagline",
    description: "Your compelling description for Twitter.",
    images: ["/og-image.png"],
    creator: "@twitterhandle",
    site: "@twitterhandle",
  },

  // Canonical URL
  alternates: {
    canonical: BASE_URL,
  },

  // App categorization
  category: "Technology",

  // Verification codes
  verification: {
    google: "google-site-verification-code",
    // yandex: "yandex-verification-code",
    // bing: "bing-verification-code",
  },
};

Page-Level Metadata

// app/pricing/page.tsx
import type { Metadata } from "next";

export const metadata: Metadata = {
  title: "Pricing", // Becomes "Pricing | Brand Name" via template
  description: "Simple, transparent pricing. Start free, upgrade when you need more.",
  openGraph: {
    title: "Pricing | Brand Name",
    description: "Simple, transparent pricing. Start free, upgrade when you need more.",
  },
};

export default function PricingPage() {
  // ...
}

Dynamic Metadata (generateMetadata)

// app/blog/[slug]/page.tsx
import type { Metadata } from "next";

type Props = {
  params: Promise<{ slug: string }>;
};

export async function generateMetadata({ params }: Props): Promise<Metadata> {
  const { slug } = await params;
  const post = await getPostBySlug(slug);

  if (!post) {
    return { title: "Post Not Found" };
  }

  return {
    title: post.title,
    description: post.excerpt,
    openGraph: {
      title: post.title,
      description: post.excerpt,
      type: "article",
      publishedTime: post.publishedAt,
      modifiedTime: post.updatedAt,
      authors: [post.author.name],
      images: [
        {
          url: post.coverImage,
          width: 1200,
          height: 630,
          alt: post.title,
        },
      ],
    },
    twitter: {
      card: "summary_large_image",
      title: post.title,
      description: post.excerpt,
      images: [post.coverImage],
    },
    alternates: {
      canonical: `${BASE_URL}/blog/${slug}`,
    },
  };
}

Part 4: Authentication Middleware Integration

When using auth (Clerk, NextAuth, etc.), add SEO routes to public matchers:

Clerk (proxy.ts or middleware.ts)

import { clerkMiddleware, createRouteMatcher } from "@clerk/nextjs/server";

const isPublicRoute = createRouteMatcher([
  "/",
  "/sign-in(.*)",
  "/sign-up(.*)",
  "/pricing",
  "/about",
  "/blog(.*)",
  "/terms",
  "/privacy",
  // SEO files - IMPORTANT!
  "/robots.txt",
  "/sitemap.xml",
  "/sitemap(.*).xml",
  // Icons
  "/icon(.*)",
  "/apple-icon(.*)",
  "/favicon.ico",
]);

export const proxy = clerkMiddleware(async (auth, request) => {
  if (!isPublicRoute(request)) {
    await auth.protect();
  }
});

export const config = {
  matcher: [
    "/((?!_next|[^?]*\\.(?:html?|css|js(?!on)|jpe?g|webp|png|gif|svg|ttf|woff2?|ico|csv|docx?|xlsx?|zip|webmanifest)).*)",
    "/(api|trpc)(.*)",
  ],
};

NextAuth

export { auth as middleware } from "@/auth";

export const config = {
  matcher: [
    // Exclude SEO files from auth
    "/((?!api|_next/static|_next/image|favicon.ico|robots.txt|sitemap.xml|sitemap.*\\.xml).*)",
  ],
};

Part 5: Environment Variables

Required environment variables for SEO:

# .env.local (development)
NEXT_PUBLIC_SITE_URL=http://localhost:3000

# .env.production (production)
NEXT_PUBLIC_SITE_URL=https://yourdomain.com

Quick Reference: File Locations

Implementation Checklist

Before implementing, verify:

1. NEXT_PUBLIC_SITE_URL environment variable is set
2. Auth middleware allows /robots.txt and /sitemap.xml
3. OG image exists at public/og-image.png (1200x630px)
4. All public pages have unique titles and descriptions
5. Canonical URLs point to preferred versions

After implementing, verify:

1. Visit /robots.txt - should show rules
2. Visit /sitemap.xml - should show URLs
3. Test with Google Rich Results Test
4. Test with Facebook Sharing Debugger
5. Submit sitemap to Google Search Console