Creating Server Side Pagination for a Next.js

Introduction

If you ever faced the warning "Warning: data for page '/' is XX which exceeds the threshold of 128 kB, this amount of data can reduce performance," serves as a critical indicator that a Next.js application is facing efficiency challenges and you are loading Large Page Data.

One case where you might encounter this issue is when you're loading your entire blog at once on the server and passing it to the props. You can read more about this in the GitHub discussion.

In this article, we will explore how you can use server-side pagination to resolve this issue. This solution applies to other use cases experiencing the same problem.

Step 1: Initialize Your Next.js Project

Ensure you have a Next.js project set up. If you're starting from scratch, create a new project with:

npx create-next-app@latest your-next-app
cd your-next-app

Step 2: Organize Your Markdown Content

Place your blog posts as markdown files in a directory, say posts/. Each file represents a single post and can include metadata in the front matter, such as the title, date, and author.

Step 3: Install Necessary Packages

Some npm packages are essential for parsing markdown and its metadata:

npm install gray-matter remark remark-html rc-pagination

gray-matter for parsing markdown front matter. remark and remark-html for converting markdown content to HTML.

Step 4: Develop a Post Fetching Utility

Create a utility function to read, parse, and return post data from markdown files. This utility should also handle sorting posts by date and implementing pagination.

Utility Function: Fetch and Paginate Posts

import fs from 'fs'
import path from 'path'
import matter from 'gray-matter'

const postsDirectory = path.join(process.cwd(), 'posts')

export async function geMDFilesFromFolder(
  folder: string,
  locale: string
): Promise<BlogPost[]> {
  const pages: BlogPost[] = []
  try {
    const filePath = path.join(process.cwd(), folder, locale)
    const mapFileToObject = async (filename: string) => {
      pages.push(await getPageSlug(filename, filePath))
    }

    await fromDirectory(filePath, /\.md$/, mapFileToObject)
    // Sort pages by 'createdAt' field in descending order
  } catch (error) {
    console.log(error)
  }
  return pages.sort((a, b) => {
    const dateA = new Date(a.createdAt)
    const dateB = new Date(b.createdAt)
    return dateB.getTime() - dateA.getTime()
  })
}

Helper Function: Recurse Through Directories

const fromDirectory = async (
  startPath: string,
  filter: RegExp,
  callback: any
): Promise<void> => {
  try {
    if (existsSync(startPath)) {
      const files = readdirSync(startPath)
      for (const file of files) {
        const filename = path.join(startPath, file)
        const stat = lstatSync(filename)
        if (stat.isDirectory()) {
          await fromDirectory(filename, filter, callback) // Recurse into directories
        } else if (filter.test(filename)) {
          await callback(filename.replace(startPath, ''))
        } else {
          console.log(
            `File does not match the specified filter criteria: ${filename}`
          )
        }
      }
    }
  } catch (error) {
    console.log(error)
  }
}

Function: Convert Filename to Post Object

export async function getPageSlug(
  fileName: string,
  filePath: string
): Promise<BlogPost> {
  const readFileAsync = util.promisify(fs.readFile)
  const markdownWithMetadata = await readFileAsync(
    path.join(`${filePath}`, fileName),
    'utf8'
  )

  const meta = matter(markdownWithMetadata)
  const showOnMenu = !!meta.data.showOnMenu

  return {
    slug: fileName.replace(/\.md$/, '').replace('/', ''),
    title: meta.data.title,
    authors: meta.data.authors,
    tags: meta.data.tags,
    description: meta.data.description,
    metaDescription: meta.data.metaDescription ?? null,
    createdAt: meta.data.createdAt,
    updatedAt: meta.data.updatedAt,
    images: meta.data.images ?? '',
    content: meta.content,
    isBlog: meta.data.isBlog,
    showOnMenu,
    publish: true
  }
}

Step 5: Server-Side Data Fetching with getServerSideProps

Utilize getServerSideProps in your blog page to fetch and display posts based on the current page number from the query parameters.

export const getServerSideProps: GetServerSideProps = async context => {
  const { locale, query } = context
  const page = Number.parseInt(query.page as string) || 1 // Current page
  const limit = 6 // Posts per page excluding the featured post
  const allPosts = await geMDFilesFromFolder(
    isProduction ? `/md_pages/blog` : `src/md_pages/blog`,
    locale ?? 'en'
  )

  // Assuming the first post is the featured post, adjust as needed
  const featuredPost = allPosts[0] // This could be determined differently
  const postsForPagination = allPosts.slice(1) // Exclude the featured post from pagination
  const totalPosts = postsForPagination.length
  const totalPages = Math.ceil(totalPosts / limit)
  const startIndex = (page - 1) * limit
  const endIndex = page * limit
  const posts = postsForPagination.slice(startIndex, endIndex)

  return {
    props: {
      posts,
      featuredPost, // Pass the featured post separately
      currentPage: page,
      totalPages,
      ...(await serverSideTranslations(locale ?? 'en', ['common', 'services']))
    }
  }
}

Step 6: Construct the Blog Page with Pagination Controls

Finally, build your blog page component to list the posts and include pagination controls, leveraging props for current page and total pages.

import {
  ChevronDoubleLeftIcon,
  ChevronDoubleRightIcon
} from '@heroicons/react/24/solid'
import dynamic from 'next/dynamic'
import Link from 'next/link'
import { ReactElement } from 'react'

import { BlogPost } from '@/@types/blog'
import FeaturedBlog from '@/components/blog/FeaturedBlog'
import Post from '@/components/blog/Post'
const Pagination = dynamic(() => import('rc-pagination'), {
  ssr: false
})

export const iconsProperties = {
  prevIcon: <ChevronDoubleLeftIcon className="w-5 h-5" />,
  nextIcon: <ChevronDoubleRightIcon className="w-5 h-5" />,
  jumpPrevIcon: (
    <button className="disabled p-2" type="button">
      .
    </button>
  ),
  jumpNextIcon: (
    <button className="disabled p-2" type="button">
      .
    </button>
  )
}
const Blog = ({
  posts,
  currentPage,
  totalPages,
  featuredPost,
  rootPath = '/blog' // Replace '/defaultPath' with your desired default path
}: {
  posts: BlogPost[]
  currentPage: number
  totalPages: number
  featuredPost?: BlogPost
  rootPath?: string // The '?' makes the rootPath optional
}): ReactElement => {
  return (
    <section>
      <div className="w-full py-10 mx-auto space-y-5 sm:py-8 md:py-12 sm:space-y-8 md:space-y-10 max-w-7xl 5xl:max-w-screen-4xl">
        {featuredPost && (
          <FeaturedBlog
            key={`${featuredPost.title}_featured`}
            post={featuredPost}
            rootPath={rootPath}
          />
        )}
        <div className="gap-y-5 sm:space-y-8 md:space-y-16">
          <div className="grid grid-cols-12 col-span-12 gap-y-16 px-4 mb-16 md:gap-x-4">
            {posts.map(post => {
              return <Post key={post.title} post={post} rootPath={rootPath} />
            })}
          </div>
          <div className="mt-4 text-center w-full">
            <Pagination
              current={currentPage}
              pageSize={6} // Assuming this matches the server-side limit
              total={totalPages * 6} // Total number of items (posts.length * pageSize)
              itemRender={(page, type, element) => {
                // Define href attribute for page links
                return type === 'page' ? (
                  <Link href={`${rootPath}?page=${page}`} passHref>
                    <>{page}</>
                  </Link>
                ) : type === 'prev' ? (
                  <>
                    <ChevronDoubleLeftIcon className="w-5 h-5" />
                  </>
                ) : (
                  <>
                    <ChevronDoubleRightIcon className="w-5 h-5" />
                  </>
                )
              }}
              {...iconsProperties}
            />
          </div>
        </div>
      </div>
    </section>
  )
}

export default Blog

Pagination Next.js with App Router

In the NextJS realm, with the advent of the new App Router, the game has slightly changed. Now, you'll want to step away from getServerSideProps and instead craft an API for fetching those blog posts on the server side. It's a straightforward swap, really. Whip up something like this, and then, simply call the API from your blog component. For incremental adoption refer to nextjs migration guide.

// app/api/blog.ts
import { geMDFilesFromFolder } from '../../../lib/ServerHelpers' // Adjust the import path as necessary

export default async function handler(req, res) {
  // Assuming `isProduction` is defined elsewhere or replace with your condition
  const isProduction = process.env.NODE_ENV === 'production'

  const { locale = 'en', query } = req
  const page = parseInt(query.page) || 1 // Current page
  const limit = 6 // Posts per page excluding the featured post, you can put this as global parameter

  try {
    const allPosts = await geMDFilesFromFolder(
      isProduction ? `/md_pages/blog` : `src/md_pages/blog`,
      locale
    )

    // Assuming the first post is the featured post; adjust as needed
    const featuredPost = allPosts[0] // This could be determined differently
    const postsForPagination = allPosts.slice(1) // Exclude the featured post from pagination
    const totalPosts = postsForPagination.length
    const totalPages = Math.ceil(totalPosts / limit)
    const startIndex = (page - 1) * limit
    const endIndex = page * limit
    const posts = postsForPagination.slice(startIndex, endIndex)

    // Return the paginated posts, total pages, and the featured post
    res.status(200).json({ posts, totalPages, featuredPost })
  } catch (error) {
    console.error('Failed to fetch paginated posts:', error)
    res.status(500).json({ error: 'Failed to fetch paginated posts' })
  }
}

FAQ

Why does loading large page data reduce performance in Next.js? Loading large page data exceeds the recommended threshold, impacting the application's performance due to increased load times and potential browser resource constraints.

How can server-side pagination solve large page data issues? Server-side pagination divides data into smaller chunks, loading only a portion of content per page request, thus reducing the initial load size and improving performance.

What packages are necessary for implementing server-side pagination in Next.js? Packages like gray-matter for parsing markdown, remark and remark-html for converting markdown to HTML, and rc-pagination for pagination controls are essential.

How do you fetch and paginate posts in Next.js? Develop a utility function to read and parse post data, including implementing sorting and pagination logic. Use getServerSideProps for server-side data fetching based on page numbers.

What changes with Next.js App Router in terms of data fetching? With the App Router, traditional data fetching functions like getServerSideProps are replaced by a new API, promoting dynamic server-side fetching and static generation within the app directory.

How does the App Router affect server-side pagination? The App Router enhances server-side pagination by allowing more dynamic and interactive data fetching patterns, improving application performance and user experience.

What are the benefits of using the new data fetching API with the App Router? Benefits include simplified project structure, improved performance through optimized fetching strategies, and enhanced developer experience with intuitive API usage.

Can server-side pagination be used with the new App Router? Yes, server-side pagination can be implemented seamlessly with the App Router's data fetching API, enabling efficient management of paginated data.

Conclusion

In wrapping up, we have explored a practical approach to overcoming a common issue that can hinder application performance—implementing pagination. We saw as well how to apply this to new Next.js with App Router.

By adopting pagination, you can streamline data handling, enhance your application's speed, and improve overall user experience, all while keeping your Application architecture in check.