Stop Scraping, Start Serving: A Guide to Model Context Protocol Next.js Integration

Tired of LLM bots scraping your Next.js site? Learn how to build a native Model Context Protocol (MCP) server inside your Next.js App Router to serve structured data directly to AI agents.

The web scraping war is over, and web developers lost. Every day, thousands of undocumented, aggressive LLM crawlers bypass your robots.txt, hammer your Next.js frontend, trigger hydration errors, and skew your product analytics. They aren't trying to steal your CSS; they just want your content to feed into an agentic workflow.

Instead of fighting a losing battle with rate-limiters and Cloudflare challenges, there is a better way: serve them exactly what they want on their own terms. By building a native Model Context Protocol Next.js integration, you can bypass the fragile HTML scraping cycle entirely. Instead of letting agents scrape your React components, you can expose a secure, schema-validated JSON-RPC interface that lets AI agents query your database, CMS, or internal tools directly.

In this guide, we will build a production-ready Model Context Protocol (MCP) server directly inside a Next.js App Router application using TypeScript.

Why Traditional Scraping Fails (and Why MCP Wins)

When an AI agent (like Claude 3.5 Sonnet or a custom GPT) visits your website, it uses a headless browser to scrape the DOM. This is incredibly inefficient:

Hydration and SPA Overhead: Modern Next.js apps rely on client-side hydration. Scraping bots often fail to wait for useEffect hooks or state transitions, resulting in incomplete data extraction.
Bandwidth Waste: You are paying for Vercel bandwidth to serve heavy JS bundles, images, and fonts to a machine that only cares about a single string of text.
No Semantic Guarantee: If you change a Tailwind class or refactor a <div> to a <section>, the scraper's parser breaks.

Model Context Protocol (MCP), open-sourced by Anthropic, solves this by defining an open standard for how LLM clients fetch data from servers. Instead of guessing your UI structure, the agent queries your MCP endpoint to discover available "Tools" (executable actions) and "Resources" (readable data).

By implementing a native Model Context Protocol Next.js integration, you transform your user-facing web app into a dual-purpose engine: highly optimized HTML for humans, and structured JSON-RPC for AI agents.

Designing a Model Context Protocol Next.js Integration

The MCP specification typically relies on two transport layers: stdio (for local CLI tools) and SSE (Server-Sent Events) (for remote web services).

For a Next.js application deployed to serverless environments like Vercel or AWS Amplify, long-lived SSE connections can be problematic due to execution timeouts and pricing models. Instead, we can build a stateless HTTP POST transport layer that adheres to the MCP JSON-RPC 2.0 specification. This allows our Next.js API routes to remain completely serverless, scale to zero, and respond in milliseconds.

Let's look at the architecture of our Next.js MCP endpoint:

[Claude Desktop / AI Agent] 
       │
       │ HTTP POST (JSON-RPC 2.0)
       ▼
[Next.js App Router Route Handler] (/api/mcp)
       │
       ├─► Auth & Rate Limiting Verification
       ├─► Router (Maps method to handler)
       │
       ├─► Method: "tools/list" ──► Returns JSON Schema of available actions
       └─► Method: "tools/call" ──► Executes DB query/CMS fetch & returns data

Step 1: Setting up the Next.js Route Handler

First, let's create our API endpoint. In the Next.js App Router, we will use a Route Handler. Create a file at app/api/mcp/route.ts.

We will start by defining our TypeScript interfaces for the MCP JSON-RPC payload. This ensures strict type safety across our tool definitions.

typescript

// app/api/mcp/types.ts
 
export interface JsonRpcRequest {
  jsonrpc: "2.0";
  id: string | number;
  method: string;
  params?: any;
}
 
export interface JsonRpcResponse {
  jsonrpc: "2.0";
  id: string | number;
  result?: any;
  error?: {
    code: number;
    message: string;
    data?: any;
  };
}
 
export interface MCPTool {
  name: string;
  description: string;
  inputSchema: {
    type: "object";
    properties: Record<string, any>;
    required?: string[];
  };
}

Now, let's implement the core POST handler in app/api/mcp/route.ts to process incoming JSON-RPC requests from LLM clients.

typescript

// app/api/mcp/route.ts
import { NextRequest, NextResponse } from "next/server";
import { JsonRpcRequest, JsonRpcResponse, MCPTool } from "./types";
 
// Define the tools our Next.js app will expose to AI agents
const AVAILABLE_TOOLS: MCPTool[] = [
  {
    name: "search_blog_posts",
    description: "Search the website's technical blog articles by keyword or tag.",
    inputSchema: {
      type: "object",
      properties: {
        query: {
          type: "string",
          description: "The search term or keyword to find relevant articles."
        },
        limit: {
          type: "number",
          description: "Maximum number of results to return (default 5)."
        }
      },
      required: ["query"]
    }
  },
  {
    name: "get_product_pricing",
    description: "Retrieve up-to-date pricing plans and feature matrices.",
    inputSchema: {
      type: "object",
      properties: {
        tier: {
          type: "string",
          enum: ["free", "pro", "enterprise"],
          description: "The specific tier to query."
        }
      }
    }
  }
];
 
export async function POST(request: NextRequest) {
  try {
    // 1. Authenticate the agent (essential for public-facing MCP endpoints)
    const apiKey = request.headers.get("x-mcp-api-key");
    if (apiKey !== process.env.MCP_API_KEY) {
      return NextResponse.json(
        { error: "Unauthorized. Invalid or missing x-mcp-api-key header." },
        { status: 401 }
      );
    }
 
    const body: JsonRpcRequest = await request.json();
 
    // Validate basic JSON-RPC structure
    if (body.jsonrpc !== "2.0" || !body.method || body.id === undefined) {
      return NextResponse.json(
        {
          jsonrpc: "2.0",
          id: body.id ?? null,
          error: { code: -32600, message: "Invalid Request" }
        },
        { status: 400 }
      );
    }
 
    // 2. Route the MCP method
    switch (body.method) {
      case "tools/list":
        return NextResponse.json(handleToolsList(body.id));
 
      case "tools/call":
        return NextResponse.json(await handleToolsCall(body.id, body.params));
 
      default:
        return NextResponse.json({
          jsonrpc: "2.0",
          id: body.id,
          error: { code: -32601, message: `Method not found: ${body.method}` }
        });
    }
  } catch (error: any) {
    return NextResponse.json(
      {
        jsonrpc: "2.0",
        id: null,
        error: { code: -32603, message: "Internal JSON-RPC error", data: error.message }
      },
      { status: 500 }
    );
  }
}

Step 2: Implementing MCP Tool Execution

Now that we have the routing backbone, we need to implement the business logic for tools/list and tools/call.

The tools/list method simply returns our array of schemas. The tools/call method executes the actual data fetching—whether that is querying your Prisma client, calling a CMS API, or reading local markdown files.

Add these helper functions to your route.ts file:

typescript

// app/api/mcp/route.ts (continued)
 
function handleToolsList(id: string | number): JsonRpcResponse {
  return {
    jsonrpc: "2.0",
    id,
    result: {
      tools: AVAILABLE_TOOLS
    }
  };
}
 
async function handleToolsCall(id: string | number, params: any): Promise<JsonRpcResponse> {
  const { name, arguments: args } = params || {};
 
  if (!name) {
    return {
      jsonrpc: "2.0",
      id,
      error: { code: -32602, message: "Missing tool name parameter" }
    };
  }
 
  try {
    switch (name) {
      case "search_blog_posts": {
        const query = args?.query as string;
        const limit = args?.limit || 5;
 
        // Replace this with your actual DB query or CMS fetch
        // e.g., const posts = await prisma.post.findMany({ where: { ... } })
        const mockPosts = [
          { title: "Mastering Next.js Route Handlers", slug: "mastering-route-handlers", tags: ["nextjs", "typescript"] },
          { title: "How to Secure Serverless Endpoints", slug: "securing-serverless-endpoints", tags: ["security", "nextjs"] },
          { title: "Introduction to Model Context Protocol", slug: "intro-to-mcp", tags: ["ai", "mcp"] }
        ];
 
        const filtered = mockPosts
          .filter(post => 
            post.title.toLowerCase().includes(query.toLowerCase()) || 
            post.tags.some(t => t.toLowerCase() === query.toLowerCase())
          )
          .slice(0, limit);
 
        return {
          jsonrpc: "2.0",
          id,
          result: {
            content: [
              {
                type: "text",
                text: JSON.stringify(filtered, null, 2)
              }
            ]
          }
        };
      }
 
      case "get_product_pricing": {
        const tier = args?.tier as string | undefined;
        
        const pricingData = {
          free: { price:

Stop Scraping, Start Serving: A Guide to Model Context Protocol Next.js Integration

Why Traditional Scraping Fails (and Why MCP Wins)

Designing a Model Context Protocol Next.js Integration

Step 1: Setting up the Next.js Route Handler

Step 2: Implementing MCP Tool Execution

Related Posts