gomail/GOMAIL_HANDOVER.md

# GoMail — Handover Document
**For: continuing this project in Claude Code / Claude CLI on a real machine**
**As of: end of Phase 15 (all 15 phases in the plan now touched — see the action plan for exact per-phase status)**

If you're a fresh Claude instance reading this with no other context: this
document plus `gomail-action-plan-v4.md` (in the same delivery) are
everything you need. Read both before writing any code.

---

## What this project is

**GoMail** — a complete self-hosted email server in a single Go binary,
replacing postfix + dovecot + roundcube + spamassassin + radicale. Built
from scratch across 9 completed phases, dependency-minimal (stdlib-first —
almost everything is hand-rolled: SMTP, IMAP, POP3, JMAP, CalDAV/CardDAV,
JWT, DKIM, SPF, DMARC parsers/servers/clients, all written against Go's
standard library rather than third-party protocol libraries).

The person you're working with (their GitHub handle: `ghostersk`) wants a
production-grade, secure, self-hosted mail server they fully understand and
control — that's *why* everything is hand-rolled instead of using
`emersion/go-imap` or similar: fewer dependencies, fewer supply-chain
trust points, full auditability.

## Immediate first steps on a real machine

```bash
# 1. Extract whichever phase tarball is newest (gomail-phase15.tar.gz as of
#    this handover — check /mnt/user-data/outputs or wherever it was saved)
tar xzf gomail-phase15.tar.gz
cd gomail

# 2. Use the CLEAN go.mod (no local replace directives — those were only
#    needed in the sandbox's network-restricted environment)
# go.mod.production is already correct; if a go.mod with /tmp/ replace
# directives exists instead, replace it:
cp go.mod.production go.mod  # only if go.mod still has replace directives

# 3. Normal Go workflow from here — real internet access means this just works
go mod tidy
go build ./cmd/gomail/

# 4. Confirm the Go version matches what's declared
go version   # should be able to satisfy "go 1.25.6" in go.mod
```

If `go mod tidy` fails on anything, it's almost certainly because the
`go.mod` still has a stray `replace (...)` block from sandbox development —
delete that block entirely and re-run `go mod tidy`.

**For a real system install** (not just a local build), use `install.sh`
instead of running the binary by hand — it creates a dedicated system
user, sets up `/etc/gomail`/`/var/lib/gomail`/`/var/log/gomail` with
correct permissions, generates the required secrets, and installs
`gomail.service` (a systemd unit with real hardening directives —
verified with `systemd-analyze verify` in the sandbox this was built in).
`README.md` has the full quick-start plus a DNS setup section (MX, SPF,
DKIM, DMARC record examples) for whoever's actually standing this up.

## How to verify the current state actually works

Don't trust that phase N's code is correct just because it's here — the
sandbox proved things via disposable E2E test programs that get deleted
after passing (see "Testing pattern" in the action plan doc). Recreate that
habit:

```bash
# Generate a master key and JWT secret for local testing
go run ./cmd/gomail/ -gen-master-key   # prints GOMAIL_MASTER_KEY=...

export GOMAIL_MASTER_KEY=<paste from above>
export GOMAIL_JWT_SECRET=$(openssl rand -hex 32)
export GOMAIL_ADMIN_INIT_PASSWORD="ChangeMe123!"

# First run auto-generates gomail.yaml — edit server.hostname to something
# real before testing against actual mail flow, or leave as-is for
# loopback-only smoke testing.
go run ./cmd/gomail/ -config /tmp/test-gomail.yaml -debug
```

Then write a throwaway `cmd/e2etestX/main.go` that dials the relevant
port(s) as a real client and asserts against real server responses **and**
real database state — this is the pattern that caught a genuine bug in
every single phase so far (see the bug list in the action plan). Delete the
test program once it passes; it's not meant to ship.

## What NOT to do

- **Don't add third-party protocol libraries** (no `emersion/go-imap`, no
  `emersion/go-smtp`, no JWT libraries, no CalDAV/CardDAV libraries) unless
  explicitly asked. The entire point of this project, per the person's
  explicit instruction at the start, is minimal dependencies and full
  control. The only allowed exceptions so far: `mattn/go-sqlite3` (CGO
  SQLite driver — no way around needing *some* driver), `golang.org/x/crypto`
  (bcrypt + hkdf — no stdlib equivalent), `google/uuid`, `gopkg.in/yaml.v3`.
- **Don't skip the negative-path tests.** "It compiles" and "the happy path
  returned 200" are not sufficient — every phase's real bug was caught by
  testing auth failures, cross-user access attempts, malformed input, or
  literal/encoding edge cases.
- **Don't silently narrow scope.** If something from the plan turns out to
  be too big for one pass (this happened repeatedly — DAV filtering, IMAP
  IDLE, JMAP Email/set), say so explicitly in code comments and in your
  response, the way every deferred item in the action plan is flagged. The
  person values honesty about gaps over an illusion of completeness.
- **Don't regenerate work that already exists and passed its tests.**
  Read the action plan's "what's built" table first. Phases 1–9 are done;
  start at Phase 9.5 (ManageSieve, currently a full gap) or Phase 10
  (OAuth2/Gmail/M365) per the person's stated priority.

## Design decisions you should NOT re-litigate

These were explicitly decided across a long planning conversation before
any code was written — don't re-ask or second-guess them:

1. **SQLite via `mattn/go-sqlite3` (CGO)**, not the pure-Go alternative —
   chosen for performance/maturity. Postgres/MySQL support planned via
   build tags (`-tags postgres`), not yet implemented.
2. **Message storage**: Maildir++ on disk, root path configurable, **every
   message encrypted at rest** (AES-256-GCM, HKDF-derived per-record key
   from a single master key — see `internal/crypto`). This applies to
   messages, contacts, and calendar events uniformly.
3. **Multi-tenancy**: all tenants share the same ports; tenants/domains are
   managed via the (not-yet-built) admin portal, not separate listeners.
4. **Auth**: strong password + MFA/passkey are the target for human login;
   app passwords (multiple, named, optional expiry) are for mail clients.
   Already fully working server-side (`app_passwords` table +
   `auth.Authenticate`'s scope checking) — just needs a management UI.
5. **Addressbooks/calendars**: both per-user AND per-tenant (shared) scopes
   supported — see `db.OwnerType` (`user` | `tenant`).
6. **CalDAV/CardDAV**: in scope, built (Phase 7), core operations only.
7. **Webmail**: pure Go `html`/JS served via `net/http` + Tailwind CDN, no
   frontend build toolchain, no JS framework. Reference inspiration was
   `github.com/ghostersk/gowebmail` (the person's own earlier project) for
   UI patterns — dark theme, folder sidebar, message list/view, compose.
8. **Push notifications**: SSE only when the browser tab is open, no
   WebPush. Already implemented in Phase 8 (`webmail.sseEvents`).
9. **POP3**: included, off by default, admin-toggled in config
   (`pop3.enabled: false`). Built in Phase 5.
10. **OAuth**: self-hosted operators register their own Google/Microsoft
    apps (Cloud Console / Azure AD), paste Client ID/Secret into config —
    no shared GoMail-branded OAuth app. Config keys already exist
    (`oauth.google`/`oauth.microsoft` in `config.go`), unused until Phase 10.
11. **JMAP external exposure**: off by default (`jmap.external_enabled:
    false`), full spec compliance is still built either way since the
    webmail's *internal* use needs it eventually. Currently JMAP is mounted
    on the webmail's own address always (Phase 9); a second standalone
    listener binds when the config toggle is on.
12. **Linked-account message caching** (for Gmail/M365 accounts once Phase
    10 lands): time-based retention, default 90 days, user-adjustable
    (days/months/years), operator ceiling via `max_cache_retention` in
    config. Schema (`linked_accounts.cache_retention_days`) already exists.
13. **Go version**: real target is 1.25.6. The sandbox this was built in
    could only get 1.22/1.23 via apt (no network path to the real Go
    toolchain distribution) — this is a **sandbox limitation, not a
    project decision**. On a real machine, just use 1.25.6 directly, no
    workaround needed.

## Where the two-tier plan documents are

- `gomail-action-plan-v4.md` — the authoritative current-state doc:
  phase-by-phase status, deferred items within completed phases, and full
  detail on Phases 9.5–15 (not yet started). **Read this before writing
  any code** — it tells you exactly what exists, what's deliberately
  incomplete, and why.
- This document (`HANDOVER.md`) — onboarding for a fresh Claude
  instance: environment setup, testing philosophy, decisions not to
  re-litigate.

Earlier plan versions (v1 through v3, if you find them referenced anywhere)
are superseded — v4 is authoritative for current state; v3 is still useful
for the *reasoning* behind decisions that haven't changed (full Part
A/B/C/D/E breakdown of the original Stalwart-inspired design), but don't
treat its phase-completion claims as current.

## Suggested next session's scope

All 15 phases from the original plan have now been worked through — 13
reached a fully complete, tested state; Phases 10 and 11 are "partially"
or "core" complete with named remaining pieces. There is no phase left
that hasn't been touched. What's left is: verifying the protocol-correct
implementations against real external services, and finishing the
named remaining pieces within otherwise-complete phases.

**First thing next session — verify against real external services.**
Three separate pieces of this project were built and proven correct
against genuinely-behaving fake servers (not rubber-stamp mocks — real
protocol verification: real JWS signatures, real RFC 7638 thumbprints,
real EICAR byte-level detection, etc.) but never against the real thing,
because this sandbox has no public domain, no real OAuth app credentials,
and no real ClamAV/rspamd/llama.cpp installed:
- **OAuth2** (Phase 10): register a real Google Cloud OAuth app (or ask
  the person for one) and link a real Gmail account end-to-end.
- **ACME** (Phase 13): run one live issuance against **Let's Encrypt
  staging** (`https://acme-staging-v02.api.letsencrypt.org/directory` —
  set via `tls.acme_directory_url`; never test against production first)
  with a real domain you control.
- **ClamAV/Rspamd/LLM** (Phase 14): if the operator has any of these
  installed, point `pipeline.clamav_socket` / `pipeline.rspamd_url` /
  `pipeline.llm_url` at them and send one real EICAR-string test message.

**Remaining named pieces within already-complete phases** (full detail in
the action plan):
1. Phase 10: native Gmail/Graph API for calendar/contacts, webmail
   account-switcher/unified-inbox UI, real userinfo lookup instead of the
   `?email=` stand-in at OAuth callback time.
2. Phase 11: aliases CRUD, per-tenant pipeline settings UI, live log
   viewer, and actually enforcing `admin_ip_allowlist` at the HTTP layer
   (currently inert).
3. Phase 13: DANE, MTA-STS, TLS-RPT, and DNS-01 challenge support are all
   fully unstarted, not just untested.
4. Phase 12: passkeys/WebAuthn are fully unstarted (needs CBOR decoding,
   not in Go's stdlib — a sub-project on the scale of Phase 13's JWS/ACME
   work), QR rendering for TOTP setup isn't built, no webmail SPA page yet.
5. Phase 14: Gmail API push / Microsoft Graph delta webhooks to replace
   IMAP polling — not started. No admin UI toggle for ClamAV/Rspamd/LLM —
   config-file only.
6. Phase 15: `staticcheck` and `shellcheck` couldn't run in this sandbox
   (toolchain/network restrictions, not a decision) — run both on a real
   machine, it's a quick win. No log rotation, metrics endpoint, or backup
   tooling for the SQLite database / Maildir tree.

**Two bugs worth remembering as general patterns**, both caught by
actually testing rather than assuming code was correct:
- A protocol test's fake server hung its client with what looked exactly
  like a real product bug — an i/o timeout. The actual cause: the fake
  server read an initial command with a generic buffered `Read()` into an
  oversized buffer, which silently over-read into the *next* protocol
  bytes (TCP doesn't preserve write-call boundaries), desyncing the whole
  parser downstream. Fixed with `bufio.Reader` + exact-length
  `io.ReadFull`. Applies to any future raw-TCP protocol work, product or
  test code alike.
- A function that called `db.Query` then `db.Exec` in the same call
  genuinely deadlocked in production code (not a test), because this
  database is capped to a single open SQLite connection
  (`SetMaxOpenConns(1)`) and the `Query`'s `rows` weren't closed before
  the `Exec` tried to grab that same connection. If you add a new
  function mixing `Query` and `Exec`, close `rows` explicitly first —
  don't rely on a deferred close.

Don't feel obligated to do any of this in the order listed if the person
asks for something else first — this is a suggestion based on stated
priority, not a hard requirement.

## One more thing

This was built through genuinely iterative, tested development — not
"write code, assume it works." Every phase had real bugs caught by real
testing, documented honestly rather than glossed over. If you find
yourself tempted to skip writing a test because "this part is simple" —
don't. The simplest-looking parts of this codebase (a regex missing a
DOTALL flag, a struct missing JSON tags, a boolean check covering only one
of two equivalent cases) are exactly where the real bugs were.