Vercel Edge

Vercel Edge middleware allows you to intercept the requests to your application served by Vercel platform. We will create a simple Next.js application and send the traffic events to Upstash Kafka using the Vercel Edge functions.

Create a Next.js application:

npx create-next-app@latest --typescript

Create a middleware.ts (or .js) file in your app directory. Update the file as below:

app/middleware.ts

import { NextResponse } from "next/server";
import type { NextRequest, NextFetchEvent } from "next/server";
import { Kafka } from "@upstash/kafka";

// Trigger this middleware to run on the `/secret-page` route
export const config = {
  matcher: "/",
};

export async function middleware(req: NextRequest, event: NextFetchEvent) {
  // Extract country. Default to US if not found.
  console.log(req.url);

  const kafka = new Kafka({
    url: process.env.UPSTASH_KAFKA_REST_URL,
    username: process.env.UPSTASH_KAFKA_REST_USERNAME,
    password: process.env.UPSTASH_KAFKA_REST_PASSWORD,
  });

  let message = {
    country: req.geo?.country,
    city: req.geo?.city,
    region: req.geo?.region,
    url: req.url,
    ip: req.headers.get("x-real-ip"),
    mobile: req.headers.get("sec-ch-ua-mobile"),
    platform: req.headers.get("sec-ch-ua-platform"),
    useragent: req.headers.get("user-agent"),
  };
  const p = kafka.producer();
  const topic = "mytopic";

  event.waitUntil(p.produce(topic, JSON.stringify(message)));

  // Rewrite to URL
  return NextResponse.next();
}

Above, we simply parse the request object and send useful information to Kafka. You may add/remove information depending on your own requirements.

Configure Credentials

We’re going to store our credentials in .env file. You can copy/paste the necessary credentials from the Upstash Console.

.env

UPSTASH_KAFKA_REST_URL=<YOUR_URL>
UPSTASH_KAFKA_REST_USERNAME=<YOUR_USERNAME>
UPSTASH_KAFKA_REST_PASSWORD=<YOUR_PASSWORD>

You can test the function locally with npm run dev. Deploy your function to Vercel with vercel --prod

The endpoint of the function will be printed. You can check if logs are collected in Kafka by copying the curl expression from the console:

curl https://real-goldfish-14081-us1-rest-kafka.upstash.io/consume/GROUP_NAME/GROUP_INSTANCE_NAME/mytopic -H "Kafka-Auto-Offset-Reset: earliest" -u \
  REPLACE_HERE

Conclusion

We have successfully built to the pipeline to collect the traffic data from our web application to Upstash Kafka. In the remaining of the series, we will analyze the data in Kafka with different realtime analytics tool which are capable to read from Kafka.