600 File Reads to Serve One Article

Most file-backed content sites — a blog, a docs site, anything reading Markdown off disk — contain a function shaped like this:

export function getPublicDoc(dir: string, slug: string): Doc | null {
  return loadPublic(dir).find((d) => d.slug === slug) ?? null;
}

It reads every document in the collection, then discards all but one.

With three articles that is invisible. I measured what it costs at the scale the project was actually specified for, and the number is worse than the shape of the code suggests, because of a detail in how Next.js renders a page.

What the call actually does

loadPublic calls loadCollection, which is the honest version of the work:

export function loadCollection(dir: string): Doc[] {
  const full = path.join(CONTENT_ROOT, dir);
  if (!fs.existsSync(full)) return [];

  const files = fs.readdirSync(full).filter((e) => /\.mdx?$/.test(e));
  const results: Doc[] = [];

  for (const file of files) {
    const raw = fs.readFileSync(path.join(full, file), 'utf8');   // disk read
    const parsed = matter(raw);                                    // YAML parse
    const result = frontmatterSchema.safeParse(parsed.data);       // zod validation
    // ...
  }

  results.sort(/* by date, then title */);
  return results;
}

Per document: one synchronous disk read, one gray-matter parse, one zod validation. All of it to answer "give me the one whose slug is X".

The multiplier nobody notices

The route calls getPublicDoc twice per request. Once in generateMetadata, to build the title and description, and once in the component body to render the page:

export async function generateMetadata({ params }) {
  const doc = getPublicDoc('writing', slug);   // full collection read
  return { title: doc.title, description: doc.abstract };
}

export default async function ArticlePage({ params }) {
  const doc = getPublicDoc('writing', slug);   // full collection read, again
  // ...
}

Both are correct, idiomatic Next.js. Neither is aware of the other.

The measurement

I generated 300 synthetic publications — the scale the project was specified for — and instrumented fs.readFileSync to count real reads.

let reads = 0;
const realRead = fs.readFileSync;
fs.readFileSync = function (...a) {
  if (String(a[0]).includes('__scaletest__')) reads++;
  return realRead.apply(this, a);
};

Results:

Operation	Reads
getPublicDoc('doc-150') — before	300
Two calls, i.e. one real article request — before	600
getPublicDoc('doc-150') — after	1
Two calls — after	2
loadPublic() for the index page — unchanged	300

Six hundred synchronous file reads, six hundred YAML parses and six hundred schema validations, to serve one article. All of it blocking the event loop.

The index page still reads all 300, and that is correct — it genuinely needs every document to render a list.

The fix

Read the one file that was asked for:

export function getPublicDoc(dir: string, slug: string): Doc | null {
  if (!/^[A-Za-z0-9._-]+$/.test(slug) || slug.startsWith('.')) return null;

  const full = path.join(CONTENT_ROOT, dir);
  const file = ['.mdx', '.md']
    .map((ext) => `${slug}${ext}`)
    .find((f) => fs.existsSync(path.join(full, f)));
  if (!file) return null;

  const doc = parseDoc(full, file);
  return isPublishable(doc) ? doc : null;
}

Both paths — the single read and the whole-collection read — now share one parseDoc helper, so they cannot drift apart in how they validate frontmatter or compute reading time.

The part that matters more than the performance

Look again at the first two lines of that function.

The original implementation compared the slug against an already-loaded list. A slug of ../../etc/passwd simply matched nothing and returned null. Hostile input was harmless because it was never used to address the filesystem.

The optimised version uses the slug to construct a path. That is a genuinely different security posture, and the slug comes from the URL.

This is the thing I would most want another engineer to take from this: a performance fix changed the trust boundary. Nothing about "read one file instead of all of them" announces that, and a review focused on the read count would pass it. The optimisation was safe; the addressing was new.

Hence the allowlist, leading dots refused, and tests that assert it:

it('refuses path traversal in the slug', () => {
  expect(getPublicDoc('__fixtures__', '../../etc/passwd')).toBeNull();
  expect(getPublicDoc('__fixtures__', '../writing/hello-continuity')).toBeNull();
  expect(getPublicDoc('__fixtures__', '.hidden')).toBeNull();
});

The visibility rule also needed pinning. Previously the whole-collection filter was the only thing enforcing "unpublished documents are not returned". With a direct read, that filter is gone from the path, so a test now asserts it explicitly, and another asserts the single read agrees with the filter it replaced on slug, body and reading time.

When this is worth doing

Not always. With a dozen documents this is a micro-optimisation and the code you have is clearer.

It becomes worth it when:

• the collection will plausibly reach hundreds of documents,
• and the per-item route is rendered per request rather than fully static,
• or the same loader is called from several places per render, which is easy to miss precisely because each call site is individually reasonable.

That last condition is the one that turned 300 into 600 here, and it is the one that does not show up in the code you are looking at. Before optimising a loader, count how many times a single request calls it. Instrument it — the number is often not the one you would guess.