Index phase

The custom pipe gives each post a title and slug, but only knows about the document it is transforming. Now we want every post to include links to all the others, without maintaining that list by hand.

An index phase lets posts contribute metadata before any page needs to render the complete list. Keep the site from the previous chapter; we will replace its page pipe with an index-enabled posts pipe.

Prepare the post collection

falz add pipe posts --index

This scaffolds index(), transform(), and a contract. Unlike transform(), which produces output, index() prepares reusable data.

Replace the project's .falz/config.yaml:

output: dist
pipes: [md, posts]

The old page pipe directory can remain, but is no longer loaded. In .falz/schemes/posts/layout.xml, replace the content rule, keeping its image asset:

<Content name="article" src="*.md" pipe="posts" to="{ALIAS}/index.html">
  <Asset name="images" src="*.{png,jpg,jpeg,webp}" to="images/{BASENAME}"/>
</Content>

Replace .falz/pipes/posts/index.ts:

import { commonOptions, toHtml } from "md"
import { parse } from "yaml"

type Meta = { title: string, slug: string, source: string }

export const contract: Contract = {
  provides: ["posts"],
  consumes: ["posts"],
}

function splitDocument(input: Buffer): { meta: Record<string, unknown>, body: string } {
  const text = input.toString("utf8")
  const match = text.match(/^---\s*\n([\s\S]*?)\n---\s*\n?/)
  if (!match) return { meta: {}, body: text }
  return { meta: parse(match[1]) ?? {}, body: text.slice(match[0].length) }
}

export function index(this: iCtx, input: Buffer): Meta {
  const { meta } = splitDocument(input)
  const post: Meta = {
    title: String(meta.title ?? this.file.stem),
    slug: String(meta.slug ?? this.file.stem),
    source: this.file.path,
  }
  this.kv.set(`posts:${this.file.path}`, post)
  return post
}

function escapeHTML(value: string): string {
  return value.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;")
    .replace(/"/g, "&quot;").replace(/'/g, "&#39;")
}

export function transform(this: tCtx, input: Buffer): TransformResult {
  const current = this.blob.get() as Meta | null
  const { body } = splitDocument(input)
  const links = this.kv.keys("posts:")
    .map(key => this.kv.get(key) as Meta)
    .map(post => `<li><a href="../${encodeURIComponent(post.slug)}/">${escapeHTML(post.title)}</a></li>`)
    .join("")

  return {
    content: `${toHtml(body, commonOptions)}<aside><h2>All posts</h2><ul>${links}</ul></aside>`,
    alias: current?.slug ?? this.file.stem,
  }
}

The Markdown conversion is familiar. The new part is the preparation: each post publishes metadata under posts:<source path>. The transform then reads all published metadata to make its list.

See the collection on every post

Keep posts/hello.md from the previous chapter. Add posts/second.md:

---
title: Second post
slug: second
---
# Another post

Now there is more than one post to discover.

For this site, use unique, single-segment slugs such as first and second; the links above assume all posts share the posts/ directory.

falz index
falz build

Open /posts/first/ and /posts/second/: both show the same list. Change the second post's title, then run index and build again. The first page updates even though its own source did not change.

The indexed-site checkpoint contains the complete project. Watch mode coordinates indexing before the initial build and subsequent rebuilds. For one-shot use, run both commands: falz build reads existing index results but does not update them.

The contract makes sharing explicit

provides: ["posts"] declares the namespace this pipe writes during index(). A key has the form namespace:key, so posts:posts/hello.md belongs to the posts namespace. Using the full source path distinguishes contributing files.

consumes: ["posts"] makes that namespace visible during transform() and declares a dependency on it. When the merged collection changes, every entry using this consuming pipe becomes build-dirty. That is why editing one title updates both pages.

Falz validates writes against provides. Different pipes can split the responsibilities: a posts pipe might provide metadata while a separate feed pipe consumes it. A pipe can also provide and consume the same namespace, as ours does.

this.kv.get(key) returns the decoded JSON value or null. this.kv.keys(prefix) returns matching full keys in sorted order, giving deterministic output. Keep namespaces focused: every change in a namespace invalidates all its consumers.

Shared data and private data are different

Our index() uses two channels intentionally:

• this.kv.set() publishes the metadata for other entries to read.
• Returning post stores a blob belonging only to this entry. Its transform reads that cached metadata with this.blob.get() to obtain its own slug.

Blobs need no contract because they never cross entry boundaries. They are useful independently of KV, especially for expensive per-file work:

export function index(this: iCtx, input: Buffer) {
  return expensiveAnalysis(input)
}

export function transform(this: tCtx, input: Buffer): string {
  const analysis = this.blob.get()
  return renderResult(input, analysis)
}

This is a schematic example: expensiveAnalysis and renderResult are your own functions. If the index entry remains current, rebuilding output reuses the analysis instead of recomputing it. HTTP fetching, image analysis, or an AI call can fit here when cached freshness is acceptable. Prefer network calls here over transform(), where they repeat whenever output rebuilds.

Both blobs and KV values must be JSON-serializable. this.blob.get() returns null when no stored result exists; returning null or undefined from index() leaves no useful blob. A transform may provide a fallback, but an indexed project's normal one-shot workflow should index first.

Cheap operations do not automatically belong in index(). We still parse the body in transform() to render it. Sharing the metadata is the reason to index this site, not the cost of front-matter parsing.

When preparation runs again

Indexing discovers the same entries as building, calls index() for dirty entries, stores their blobs, and merges their KV contributions. It then compares namespaces with their previous state and marks consumers of changed data build-dirty.

An index entry is dirty when it has no recorded result or one of these inputs changes:

A layout template change does not make the index dirty: templates are not inputs to index(). It can still make output dirty. In watch mode, observed source removals also remove the entry's recorded KV contribution.

Blobs and namespace data live in the user cache directory. dist/.manifest.json records references and namespace hashes. Use falz index --force to recompute entries, or falz clean index to discard index state. The exact methods are documented in the JavaScript API.

The site now has shared presentation, custom conversion, and shared data. In Build phase, we will use these same files to understand why some changes rebuild one page and others rebuild the whole site.