gomail/.claude/iterative-build-discipline-SKILL.md

---
name: iterative-build-discipline
description: Practices for building large, multi-phase software projects (servers, protocol implementations, hand-rolled clients, multi-session builds) where correctness and honest progress tracking matter more than speed. Use this whenever a project spans multiple sessions or phases, involves implementing a network protocol or parser from scratch, needs to prove correctness without access to the real external service it talks to, or risks losing state between sessions (sandbox resets, context limits, long-running builds). Also use when maintaining living planning/handover documents across many edits, or when deciding whether to test something before believing it works. Trigger this proactively for phased builds even if the user doesn't ask for "best practices" — the value is in not needing to be asked.
---

# Iterative Build Discipline

Distilled from building GoMail, a ~14,000-line self-hosted email server
implemented phase-by-phase across many sessions (SMTP/IMAP/JMAP/CalDAV/
ManageSieve/OAuth2/ACME, all hand-rolled on stdlib). The patterns below
caught real bugs and prevented real losses, repeatedly, across nearly
every phase. This isn't theoretical advice — every rule here has a
"here's the actual bug this caught" behind it.

## Core principle

**"It compiles" and "the happy path returned 200" are not evidence of
correctness.** Every phase of GoMail had at least one real bug that a
build-only check would have missed entirely and shipped silently. The
single highest-leverage habit is: after writing a chunk of new
functionality, write a real test that exercises it as a real client
would, including the paths where it should fail. Do this before
declaring the phase done, not as an afterthought.

## Testing philosophy

### Write disposable, real end-to-end tests — not unit-test theater
For each meaningful chunk of work, write a throwaway test program that:
1. Boots the real server(s) as goroutines/processes in the same test run
2. Acts as a **real client** over the **real wire protocol** — raw TCP for
   custom protocols, real `net/http` for HTTP, not an in-process function
   call that skips the actual serialization/parsing
3. Asserts against real downstream state (a database row, a file on disk),
   not just a status code
4. Covers negative paths deliberately: wrong auth, malformed input,
   cross-user/cross-tenant access attempts, replay of a one-time token,
   unreachable dependency
5. Gets deleted once it passes — it was scaffolding, not the deliverable

This pattern found a real bug in nearly every phase of a 15-phase project:
a NULL-scan panic from a naively-wrapped nullable column, a boolean check
that covered only one of two equivalent session kinds (silently dropping
an entire code path for one server mode), a regex missing the `(?s)`
DOTALL flag that made an entire class of multi-line responses parse as
empty, a wire command sent with the wrong syntax that "worked" by
returning an empty result instead of an error, a struct with no JSON tags
that would have silently shipped `TotalCount` instead of `total_count` on
the wire, a SQLite connection-pool deadlock from calling `Query` then
`Exec` in the same function without closing `rows` first, and a fake
protocol test server whose oversized buffered `Read()` silently over-read
into subsequent protocol bytes and desynced its own parser (looked
exactly like a real product bug until traced).

**None of these would have been caught by "the code compiles and looks
right."** All of them were caught by writing a client that actually spoke
the protocol and checking what actually happened downstream.

### When you can't test against the real external service, build a fake
one that actually enforces the protocol — not a rubber stamp
For OAuth2, ACME, ClamAV, rspamd, and an LLM server, there was no real
instance available in the build environment (no real Google/Microsoft app
credentials, no public domain to satisfy Let's Encrypt's HTTP-01
validator, no ClamAV/rspamd/llama.cpp installed). The fix was never to
skip testing — it was to build a **from-scratch fake server that
genuinely implements the documented protocol**, then prove the real
client code against it:
- A fake ACME server that does real ES256 JWS signature verification and
  real RFC 7638 JWK thumbprint recomputation, and only issues a
  certificate if a real HTTP callback to the challenge responder actually
  succeeds — not a server that just returns "OK" to whatever it's sent.
- A fake ClamAV that reassembles the streamed chunks and inspects the
  bytes for the real EICAR test signature, rather than returning a canned
  "clean" response regardless of input.
- A fake OAuth2 authorization server implementing the real
  authorization-code + refresh-token grant flows, including one-time-use
  codes and CSRF state validation.

This is real verification of protocol correctness — not a substitute for
eventually testing against the real service, but a much stronger check
than "it compiles" and often catches the exact class of bug that would
otherwise only surface against production. **Explicitly flag, in both
code comments and any handoff docs, that this is fake-server verification
and that live verification against the real service is still an open
item** — don't let a passing fake-server test quietly become "this is
proven to work."

### When a real published test vector exists, use it
For anything implementing a spec with official test vectors (RFC 6238's
TOTP Appendix B, for instance), verify against the actual published
numbers rather than only checking internal self-consistency. This is a
strictly stronger correctness check: internal self-consistency can pass
while the implementation is confidently wrong in a way that happens to be
symmetric; matching an external published vector rules that out.

### A negative test is not optional polish
Auth-failure paths, permission-boundary checks, tampered-input rejection,
and replay-prevention are not "nice to have if there's time" — in this
project, testing them found real security-relevant gaps (a privilege
escalation attempt that wasn't actually blocked until tested, a CSRF
state that could be replayed until a check was added, a signature
verification path that would have accepted a forged input). Budget time
for these as part of "done," not after it.

## Session and state continuity

### Checkpoint working state as an artifact, not just as conversation history
In any environment where the working directory or container can reset
(sandbox limits, context window limits, session boundaries), the actual
recovery mechanism is a **packaged artifact saved outside the working
directory** — a tarball, a committed file, whatever the environment's
durable output channel is. Do this after every meaningful increment, not
just at the end. Conversation history describing what you did is not a
substitute for the file existing somewhere durable.

### Before editing a living document, verify its current state — don't
assume your last edit landed
Long sessions with many sequential edits to the same planning/handover
document are exactly where partial failures compound silently: an
aborted tool call, a `str_replace` that matched the wrong location, or an
edit applied against stale context can leave a document with duplicated
or contradictory sections (a "done" entry sitting right next to a stale
"not started" stub for the same item). Before trusting that a previous
edit succeeded, check: grep for the section header count (should be
exactly 1), read the actual current content of the section you're about
to touch, and only then edit. After editing, verify again — don't chain
edits on documents based on what you intended to have written rather than
what's actually there.

### Keep a living plan document AND a separate onboarding/handover document
The plan document is the source of truth for **what's done, what's
tested, what's deferred and why** — organized by phase/milestone, updated
every time scope changes. The handover document is written for **a fresh
reader with zero context** — environment setup that differs between the
build environment and a real deployment target, decisions not to
re-litigate, and a prioritized "what to do next" list. These serve
different readers and get stale in different ways; don't collapse them
into one document, but do keep them in sync with each other (a completed
phase should be reflected in both).

## Scoping and honesty

### When a sub-feature is genuinely out of scope for the current pass, say so explicitly — don't half-build it
Real examples from this project: WebAuthn/passkeys were skipped entirely
because they require CBOR decoding, which isn't in Go's standard library
and is a large sub-project in its own right — attempting a partial
implementation would have produced something that looked done but wasn't
trustworthy. QR code rendering for TOTP setup was skipped in favor of the
plain-text provisioning URI (which every real authenticator app accepts
as manual entry) because real QR encoding is a separate, non-trivial
piece of work. In both cases, the deferral was written down explicitly —
in code comments and in the plan document — with the reason, rather than
silently leaving a stub or, worse, a fake implementation that doesn't
actually work.

### Prioritize by what actually reduces risk, not strictly by original task order
When a later-numbered piece of work would close a security gap that
affects everything already built (in this project: real TLS certificate
issuance, since every existing self-signed-cert fallback was a flagged
weakness), it's worth pulling that forward ahead of strict sequential
order — and saying so explicitly, with the reasoning, rather than
silently reordering.

### A fact stated in a handoff document should be checked, not assumed durable
Version numbers, "as of" dates, and "extract this tarball" instructions
written early in a long session go stale as the session continues. Before
finalizing a handoff document, re-read it as if you were the person
receiving it fresh — do the setup instructions still point at the right
artifact? Does the "current phase" claim match what's actually true now?

## Technical patterns worth remembering generically

- **Raw TCP protocol work (client or test server)**: never assume one
  `Read()` call aligns with the sender's `Write()` call boundaries — TCP
  doesn't preserve them. Use a `bufio.Reader` with exact-length
  `io.ReadFull` reads for anything with a defined wire format (a command
  prefix, a length-prefixed chunk), not a single oversized buffered
  `Read()`.
- **A database connection pool capped to one connection** (common for
  embedded databases like SQLite in a single-process server): never call
  a second query/exec while an earlier `*sql.Rows` from the same
  connection is still open — close it explicitly first, don't rely on a
  deferred close if you need the connection again before the function
  returns.
- **Wire-format structs**: always give them explicit serialization tags.
  A struct with no tags will round-trip fine in same-language tests
  (Go-to-Go) while silently breaking the actual wire contract expected by
  any other client.
- **When verifying a hand-rolled crypto/signing implementation**, prefer
  checking against a real published test vector over only checking
  internal round-trip consistency, wherever the spec provides one.

## What this skill is not

This isn't a checklist to run through mechanically per line — it's a set
of habits earned from what actually went wrong across a large real
project. The common thread: **verify against reality (real protocol,
real published vectors, real negative paths, real current document
state) instead of trusting that code which compiles or looks right is
therefore correct.**