nahakubuilder/gomail
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
gomail/gomail-action-plan-v4.md
T
nahakubuilder d7ca591b76 first commit
2026-08-09 18:03:09 +01:00

673 lines
42 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# GoMail — Action Plan v4 (post-Phase 9)
**Status: Phases 1–9 complete and verified. Phases 10–15 remain.**
This supersedes v3 for anything marked complete below. Where v3's design
held up under actual implementation, it's summarized, not repeated — see
v3 for full rationale on decisions that didn't change (encryption scheme,
POP3 off-by-default, OAuth self-registration, time-based linked-account
caching, etc.).
---
## What's built (Phases 1–9)
| Phase | Package(s) | What it does | Status |
|---|---|---|---|
| 1 | `config`, `crypto`, `db` | YAML config + env secrets, HKDF-per-record encryption, SQLite schema/migrations, bootstrap | ✅ verified |
| 2 | `smtp`, `mailstore`, `tlsutil` | Inbound MTA (:25/:587/:465), STARTTLS, SASL PLAIN/LOGIN, encrypted Maildir delivery | ✅ verified |
| 3 | `dkim`, `queue` | DKIM sign/verify (stdlib crypto only), outbound retry queue, MX delivery via `net/smtp`, bounce DSN | ✅ verified |
| 4 | `pipeline` | SPF/DKIM/DMARC/header/URL checks, quarantine storage, verdict scoring | ✅ verified against live DNS |
| 5 | `imap`, `pop3`, `auth` | Full IMAP core command set, POP3 (off by default), shared credential verification | ✅ verified |
| 6 | `accounts`, `imapclient` | `MailProvider` interface, local + generic-IMAP backends, hand-rolled IMAP client | ✅ verified (dogfoods own server) |
| 7 | `vcard`, `ical`, `dav` | CalDAV/CardDAV server (PROPFIND/REPORT/PUT/GET/DELETE), encrypted contacts/events | ✅ verified |
| 8 | `webtoken`, `webmail` | Hand-rolled JWT, REST API + dark Tailwind SPA (folders/messages/compose/quarantine) | ✅ verified |
| 9 | `jmap` | JMAP Core/Mail subset (session, Core/echo, Mailbox/get, Email/query, Email/get) | ✅ verified |
| 9.5 | `sieve`, `managesieve` | Sieve interpreter (RFC 5228 subset) + ManageSieve server (RFC 5804), wired into inbound delivery | ✅ verified |
| 10 | `oauth2`, extended `accounts`/`imapclient`/`webmail` | Hand-rolled OAuth2 (auth code grant + refresh), XOAUTH2 IMAP login, webmail account-linking endpoints (CSRF-protected) | ✅ verified against fake OAuth2 server + fake XOAUTH2 IMAP stub — see note below |
| 11 | `admin` | Admin portal: RBAC-gated REST API (global_admin/tenant_admin) + dark Tailwind SPA — tenants, domains (+DKIM keygen/rotation), users, list rules, outbound queue, global quarantine, dashboard stats | ✅ verified — full RBAC enforcement, real DKIM key generation/rotation checked cryptographically |
| 13 | `acme`, `tlsutil` (ACMEManager) | Real ACME v2 client (RFC 8555): JWS/ES256 signing, HTTP-01 challenge, SNI-aware cert serving, encrypted storage, auto-renewal loop. Pulled forward from phase order — see rationale below. | ✅ verified against a from-scratch fake ACME server with real JWS/thumbprint verification — see note below |
| 12 | `totp`, extended `webmail`/`db` | Auth hardening: TOTP MFA (RFC 6238) with two-step login, backup codes, app-password self-service UI, recovery-email password reset. Completed after Phase 13 since 13 was pulled forward — see Phase 13's entry above. | ✅ verified — TOTP checked against 5 real RFC 6238 published test vectors, full two-step login/backup-code/app-password/reset flows over real HTTP |
| 14 | extended `pipeline` (ClamAV/Rspamd/LLM stages) | Optional external security services: real clamd INSTREAM protocol, real rspamd `/checkv2` HTTP API, OpenAI-compatible LLM classification (llama.cpp server) — all off by default, only added to the pipeline when configured | ✅ verified — genuine EICAR malware detection (byte-level, not canned), real rspamd score pass-through, LLM score capping, graceful negative-path handling for unreachable services |
| 15 | `ratelimit`, fuzz tests, `install.sh`/`gomail.service`/`README.md` | Hardening + deploy: per-IP token-bucket rate limiting on every listener, fuzz tests on the 4 riskiest hand-written parsers, `go vet` pass, systemd install script + hardened unit, full README with DNS setup guide | ✅ verified — rate limiting proven under the race detector and over real sockets/HTTP, ~1.5M fuzz executions with zero panics, systemd unit verified with `systemd-analyze verify` |
**~14,400 lines of Go**, one third-party runtime dependency beyond
`mattn/go-sqlite3`/`golang.org/x/crypto`/`google/uuid`/`gopkg.in/yaml.v3` —
everything else (SMTP, IMAP, POP3, JMAP, CalDAV/CardDAV, JWT, DKIM, SPF,
DMARC) is hand-rolled on the Go standard library, per the original
dependency-minimal requirement.
### Deferred within completed phases (honest gaps, not hidden)
- **DAV**: REPORT filter/time-range queries return the full collection, not
a filtered subset. MKCALENDAR/MKCOL not implemented — default collection
auto-created per user instead.
- **IMAP**: no IDLE, CONDSTORE/QRESYNC, SORT/THREAD, or mailbox
CREATE/DELETE/RENAME. Core command set only (see Phase 5 code comments).
- **JMAP**: Email/set (flags/delete via JMAP), Email/import (send via
JMAP), push (EventSource), and Mailbox/set are not implemented — the
webmail SPA still uses Phase 8's direct REST API for all mutations.
ManageSieve (RFC 5804) was paired with JMAP in the original plan and
**was not started this phase at all** — full gap, see Phase 9.5 below.
- **accounts.IMAPProvider.Move**: returns "not implemented" — needs IMAP
APPEND, which isn't in the imapclient package yet.
- **TLS**: self-signed cert generation only (`tlsutil`), no ACME. Every
linked IMAP/SMTP connection uses `InsecureSkipVerify: true` as a known,
commented gap — real cert trust (pinning or ACME) is Phase 13.
---
## Remaining phases
### Phase 9.5 — ManageSieve + Sieve interpreter — ✅ COMPLETE
`internal/sieve`: hand-written recursive-descent interpreter covering
`if`/`elsif`/`else`, `header :contains`/`:is` tests, `fileinto`/`discard`/
`keep`/`stop` actions. Verified: correct short-circuit on `stop`, correct
RFC 5228 §4.4 "explicit keep after discard still delivers" semantics,
malformed scripts rejected at parse time.
`internal/managesieve`: RFC 5804 server on `:4190` — CAPABILITY, STARTTLS
(auth refused before TLS, verified), AUTHENTICATE PLAIN, PUTSCRIPT (rejects
syntactically invalid scripts at upload time via `sieve.Parse`), GETSCRIPT,
LISTSCRIPTS, SETACTIVE (enforces exactly one active script per user),
DELETESCRIPT, LOGOUT.
Wired into `internal/smtp/session.go`'s local-delivery path: after the
security pipeline clears a message (clean/flagged), the recipient's active
Sieve script (if any) runs against the message headers and determines the
destination folder or discard — verified end-to-end with a real SMTP
delivery routed by an uploaded rule.
**Deferred within this phase**: only `header` tests are supported (no
`address`, `envelope`, `size`, `exists` tests); no `anyof`/`allof` boolean
combinators (only single-condition if/elsif); no Sieve extensions
(`vacation`, `reject`, `notify`, `imap4flags`); webmail visual rule builder
not built — raw script upload via the ManageSieve protocol is the only
interface (testable with `sieve-connect` or similar real clients).
### Phase 10 — Multi-account webmail: OAuth2 + Gmail/M365 — ✅ PARTIALLY COMPLETE
**What's done and verified:**
- `internal/oauth2`: hand-rolled OAuth2 authorization-code grant + refresh
(RFC 6749 §4.1, §6), ~150 lines, stdlib `net/http`+`encoding/json` only.
`WellKnownEndpoints("google"|"microsoft")` returns the real fixed
endpoint URLs — not operator-configurable, matching how real integrations
work (only Client ID/Secret are operator-supplied).
- `imapclient.LoginXOAUTH2`: SASL XOAUTH2 mechanism for connecting to
Gmail/M365 with a bearer token instead of a password.
- `accounts.IMAPProvider` extended: branches on `AuthType` — password
accounts unchanged, OAuth2 accounts decrypt a stored token, transparently
refresh it if expired (persisting the new token to avoid re-refreshing
on every call), and authenticate via XOAUTH2.
- `accounts.LinkOAuth2Account` / `wellKnownIMAPHost`: stores encrypted
OAuth2 tokens, points Gmail/M365 accounts at their real documented IMAP
hosts (`imap.gmail.com:993`, `outlook.office365.com:993`).
- `webmail` API: `/api/accounts` (list), `/api/accounts/{id}` (unlink),
`/api/accounts/oauth/{provider}/start` (builds auth URL, stores
CSRF state server-side keyed to the authenticated user),
`/api/accounts/oauth/{provider}/callback` (validates state — rejects
unknown AND replayed state, verified — exchanges code, links account).
- **Verified** with a from-scratch fake OAuth2 authorization server
(`/authorize` + `/token`, both grant types) and a fake XOAUTH2-accepting
IMAP stub — full round trip: start → redirect → callback → encrypted
token storage → real XOAUTH2 IMAP login → forced token expiry → transparent
refresh → persisted new token. This is real protocol-correctness
verification; it does **not** touch real Google/Microsoft infrastructure,
since no real app credentials exist in the dev environment this was
built in. **The next session should register a real Google Cloud OAuth
app (or ask the person for one) and do one live end-to-end test against
actual Gmail before considering this phase fully closed** — the protocol
layer is proven, but "does real Gmail's token endpoint actually behave
the way RFC 6749 says it should" has not been checked against the real
service.
**Deferred within this phase:**
- Native Gmail API / Microsoft Graph API providers (`GmailProvider`,
`M365Provider` as distinct from the IMAP+OAuth2 path) — per the locked
decision, mail goes through `IMAPProvider` with OAuth2 credentials for
now; native API push (Gmail watch+Pub/Sub, Graph delta webhooks) is
Phase 14 per the original plan.
- Calendar/contacts via native Graph API / Gmail Calendar+People API — not
started. No IMAP fallback exists for these, so this is still fully
greenfield work reusing the OAuth2 token store just built.
- Webmail UI: account switcher, unified "All Inboxes" view, per-account
"send as" in compose, federated search — the backend
(`/api/accounts`) exists but the SPA has no UI for any of this yet.
- Real userinfo/profile API call to learn the linked account's actual
email address at callback time — currently accepts an `?email=` query
param as a stand-in (noted in code); a real implementation calls
Google's `userinfo` endpoint or Microsoft Graph's `/me` with the fresh
access token.
- `IMAPProvider.Move` still returns "not implemented" (needs IMAP APPEND,
unchanged from Phase 6).
### Phase 11 — Admin portal — ✅ COMPLETE (core CRUD)
**What's done and verified:**
- `internal/admin`: REST API + embedded dark Tailwind SPA, JWT sessions
(same `webtoken` scheme as webmail), role checked at auth time AND
re-checked against the live DB row on every request (a demoted admin's
existing token stops working immediately, not just after expiry).
- **Tenants**: list/create, global_admin only (verified: tenant_admin gets
403).
- **Domains**: list/create/delete, real DKIM key generation on create
(verified: stored key decrypts and parses as a genuine RSA private key,
not just "a create call succeeded"), DKIM rotation (verified: key
material genuinely changes, not a no-op). global_admin defaults to their
own tenant if none specified; tenant_admin forced to their own tenant
regardless of what they send.
- **Users**: list/create/suspend/activate/reset-password/delete.
tenant_admin scoped to their own tenant and blocked from creating or
modifying admin-role accounts (verified: 403 on a privilege-escalation
attempt).
- **List rules**: list/create/delete, tenant-scoped.
- **Outbound queue**: list, retry-now (verified: `next_attempt_at`
genuinely rescheduled to immediate, not just a 200 response), cancel.
- **Global quarantine**: list, discard (verified: status genuinely
transitions to `deleted` in the DB). Per-user *release* (as opposed to
discard) already existed in webmail from Phase 8 — this is the
admin-wide review/cleanup view, not a duplicate of that.
- **Dashboard**: user/domain counts, 24h message count, queue depth,
quarantine-held count.
**Two real bugs caught by testing, both fixed:**
1. `global_admin` has no *forced* tenant (unlike `tenant_admin`), but every
domain still needs one — domain creation 400'd with "tenant_id is
required" until a fallback to the admin's own tenant was added.
2. User creation requires a valid `domain_id` (a mailbox must belong to a
domain) — neither the test nor the admin SPA's "Add User" form supplied
one at first. Fixed in both: the SPA now has a domain ID field, and the
Domains page shows each domain's ID (click to copy) so there's an actual
way to get that value into the form.
**Deferred within this phase:**
- Aliases CRUD (table exists, no admin UI yet).
- Pipeline settings UI (per-tenant score thresholds / check toggles) — the
`tenants.settings_json` column exists but nothing reads or writes it yet;
pipeline behavior is still entirely driven by the global `config.Pipeline`
values.
- TLS cert status view, DMARC/TLS-RPT report viewer — both depend on
Phase 13 (ACME/DANE/MTA-STS) existing first.
- Live log viewer (SSE) — webmail's `sseEvents` pattern from Phase 8 is
directly reusable here, just not wired up yet.
- Admin IP allowlist (`admin_ip_allowlist` in config) is not enforced at
the HTTP layer — the admin server binds to `127.0.0.1` by default, which
covers the common case, but the config value itself is currently inert.
### Phase 13 — TLS + ACME + DANE/MTA-STS — ✅ CORE COMPLETE (pulled forward)
Pulled ahead of strict phase order per the reasoning that closing the
`InsecureSkipVerify` gap improves the security posture of everything
already built more than most net-new features would — see the handover
doc for the fuller rationale.
**What's done and verified:**
- `internal/acme`: hand-rolled ACME v2 client (RFC 8555) — no third-party
ACME/JOSE library. `jws.go` implements JWS signing (ES256, flattened
JSON serialization per RFC 7515) and RFC 7638 JWK thumbprint computation
from scratch. `client.go` implements the full protocol: directory fetch,
nonce management, account registration, order creation, authorization
polling, HTTP-01 challenge response, CSR building, finalize, and
certificate download. `challenge.go` is the HTTP-01 responder
(`/.well-known/acme-challenge/{token}`). `obtain.go` is the high-level
orchestration function tying it together.
- `internal/tlsutil/acme_manager.go`: `ACMEManager` provides SNI-aware
certificate serving (`tls.Config.GetCertificate`), encrypted-at-rest
storage (same HKDF-per-record scheme as messages/contacts/DKIM keys —
new `tls_certs` table, migration 0010), in-memory caching, and a
background renewal loop (checks daily by default, renews within 30 days
of expiry).
- Wired into `cmd/gomail/main.go`: when `tls.mode: acme` and
`tls.acme_domains` are configured, a real `ACMEManager` drives cert
serving instead of the self-signed fallback, and a plain `:80` listener
serves the HTTP-01 challenge responder. Falls back to self-signed with a
now-accurate warning message when ACME mode is set but no domains are
configured, or when mode is `off`.
- **Verified** with a from-scratch fake ACME server (`cmd/e2etest13`,
deleted after passing per the established pattern) that does *real*
protocol-level verification, not a rubber stamp: ES256 JWS signature
verification against the account's actual public key, RFC 7638
thumbprint recomputation from that key, and a genuine HTTP callback to
the challenge responder to check the key authorization matches — exactly
what a real CA does. Confirmed: full obtain flow succeeds; the issued
certificate works in an actual `tls.Dial` handshake with the correct
`CommonName`; the certificate is encrypted at rest (checked the raw DB
column, then decrypted and confirmed it's a real PEM cert); a second
call hits the cache instead of re-issuing; a tampered JWS signature is
correctly rejected; and — importantly — finalizing an order without
completing domain validation is correctly refused with 403, proving the
fake CA (and by extension, the real protocol logic our client drives)
won't issue a certificate without genuine authorization.
- Two stale/inaccurate log messages caught and fixed while wiring this in:
the self-signed fallback's warning previously claimed "ACME is not yet
implemented (Phase 13)" — no longer true, so the message is now
context-aware (distinguishes "acme mode but no domains configured" from
"mode is off entirely"). A comment in `accounts.IMAPProvider` about
`InsecureSkipVerify` was similarly updated to reflect that the gap is
now narrower (GoMail's own server can get a real cert; a *linked*
external account's server is still outside our control regardless).
**Deferred within this phase** (genuinely not started, not just
untested):
- DANE (TLSA record lookup + verification for outbound delivery) — not
implemented.
- MTA-STS (fetch/cache `.well-known/mta-sts.txt`, enforce TLS to declared
MX hosts) — not implemented.
- TLS-RPT (aggregate report generation/sending) — not implemented.
- ACME DNS-01 challenge type (only HTTP-01 is implemented) — DNS-01 is
necessary for wildcard certs, which GoMail doesn't currently need, but
worth naming as a real gap rather than assuming HTTP-01 is sufficient
forever.
- No live test against a real CA (Let's Encrypt staging or production) —
same category of limitation as Phase 10's OAuth2: no real public
domain/DNS control in this sandbox to satisfy HTTP-01 validation from
the actual internet. **The protocol implementation is proven correct
against a genuinely-verifying fake server; a live run against Let's
Encrypt staging is the natural first action for whoever continues this,
same recommendation as Phase 10's OAuth caveat.**
- Every `InsecureSkipVerify: true` in the codebase (linked IMAP accounts,
the ACME manager's own... actually the ACME manager doesn't skip
verification, only the *generic IMAP/OAuth2 linked account* paths in
`internal/accounts` still do, since those connect to servers outside
GoMail's control) is not yet an argument for removal — search the
codebase for that string to find remaining call sites.
### Phase 12 — Auth hardening — ✅ CORE COMPLETE (TOTP + app passwords + reset)
Completed after Phase 13 in this session's actual order (13 was pulled
forward for its security-posture impact; this closed the gap in numeric
order afterward).
**What's done and verified:**
- `internal/totp`: hand-rolled RFC 6238 TOTP (HOTP RFC 4226 underneath) —
no third-party OTP library. **Verified against 5 of RFC 6238's own
published Appendix B test vectors** (not just internal self-consistency):
our 6-digit output exactly matches the last 6 digits of each published
8-digit reference vector, which is mathematically guaranteed to hold only
if the HMAC-SHA1 counter derivation and truncation are byte-for-byte
correct. Also verified: ±30s clock-skew tolerance works and correctly has
a boundary (120s does NOT validate), and the `otpauth://` provisioning
URI is well-formed.
- Two-step login: `POST /api/auth/login` returns `{mfa_required: true,
mfa_token: <5-min pre-auth token>}` instead of a session when
`user.MFAEnabled` — `POST /api/auth/mfa-verify` redeems that token plus
either a valid TOTP code or an unused backup code for a real session.
The pre-auth token is a genuinely distinct, narrowly-scoped token type
(`webtoken.Claims.Purpose = "mfa_pending"`), not just a normal session
token handed out early.
- MFA setup/confirm/disable: `POST /api/me/mfa/setup` generates a secret
and stores it encrypted but **not yet enabled** — `POST
/api/me/mfa/confirm` requires one valid code before MFA actually takes
effect and backup codes are issued, so an abandoned setup never locks
anyone out. 8 backup codes (SHA-256 hashed, one-time use, verified) are
shown to the user exactly once, matching the app-password pattern.
`POST /api/me/mfa/disable` re-requires the password (a session token
alone isn't enough to turn MFA off).
- App passwords: full self-service CRUD
(`GET/POST /api/me/app-passwords`, `DELETE
/api/me/app-passwords/{id}`) — the server-side scope-checking machinery
already existed from Phase 5, this phase added the UI-facing API.
Verified: the raw token is returned exactly once at creation and never
re-appears in the listing response.
- Password reset: `POST /api/auth/forgot-password` +
`POST /api/auth/reset-password`, gated on a user-configured
`recovery_email` (new `users.recovery_email` column) rather than the
account's own mailbox — deliberately avoids the chicken-and-egg problem
of emailing a reset link to a mailbox the person is locked out of.
Verified: a genuine outbound-queue entry is created addressed to the
recovery email (real delivery infrastructure, not a stub), and the
endpoint returns the identical response whether or not the account
exists (no account-existence leak via response differences).
**Two real bugs found and fixed while testing, both worth remembering:**
1. `db.GetUser` was built in Phase 11 for admin's minimal display needs
(id/email/role/active) and never extended when Phase 12 code started
depending on it for MFA fields — `mfa_confirm` failed with "no pending
MFA setup" even though setup had just succeeded, because the fetch
silently dropped `totp_secret_enc`. Fixed by making `GetUser` the one
canonical full-row fetch rather than fragmenting into
admin-flavored/auth-flavored variants.
2. **A genuine SQLite connection-pool deadlock** in `ConsumeBackupCode`:
it ran a `db.Query` (find the matching backup code) followed by a
`db.Exec` (mark it used) in the same function, but this database is
capped to a single open connection (`SetMaxOpenConns(1)`, see
`internal/db/db.go`) — calling `Exec` while the `Query`'s `rows` was
still open (only deferred-closed, not yet executed) caused the request
to hang forever waiting for a connection that could only free up
*after* the function returned. Fixed by closing `rows` explicitly
before the `Exec`. **The codebase was then scanned for the same
pattern** (any function mixing `Query` and `Exec`) and this was
confirmed isolated — worth doing that scan again after any future
change that mixes both in one function, given SQLite's single-connection
constraint makes this an easy trap to fall into.
**Deferred within this phase:**
- **Passkey/WebAuthn — not started at all**, and deliberately so: it
requires CBOR decoding for attestation/assertion objects, which is not
in Go's standard library and would be a meaningfully sized sub-project
of its own (on the order of the JWS/ACME work in Phase 13), with little
shared surface with anything else in this codebase. `users.
passkey_credentials_json` exists in the schema as a placeholder for
when this is picked up.
- QR code rendering for TOTP setup is not implemented — `totp.
ProvisioningURI` returns the `otpauth://` URI as plain text, which every
mainstream authenticator app accepts as manual entry; actual QR
rendering (Reed-Solomon error correction, matrix placement) is real
standalone work, noted in `totp.go`'s doc comment rather than silently
skipped.
- No webmail SPA pages for any of this yet — MFA setup/confirm, app
password management, and recovery-email configuration are all
functional, tested REST endpoints with no frontend UI. The existing
`internal/webmail/static/index.html` pattern is directly reusable, just
not built out for these yet.
### Phase 14 — Optional external services — ✅ CORE COMPLETE (ClamAV/Rspamd/LLM)
**What's done and verified:**
- `internal/pipeline/stage_clamav.go`: `ClamAVStage` speaks clamd's real
documented INSTREAM binary protocol by hand — no third-party clamd
client library. Send `"zINSTREAM\0"`, then the message in 4-byte-
big-endian-length-prefixed chunks terminated by a zero-length chunk,
then read the single-line response. Accepts `unix:` or `tcp:` prefixed
addresses. A detected match is always a hard block (score 100) rather
than a scored contribution, since malware detection isn't a "maybe."
**Verified with a genuine EICAR test-string detection** — the fake
clamd test server actually reassembles the streamed chunks and inspects
the bytes for the real industry-standard EICAR signature, the same way
real clamd does, rather than returning a canned response; a clean
message correctly passes, and an unreachable clamd is handled as
`CheckError` rather than crashing or hanging.
- `internal/pipeline/stage_rspamd.go`: `RspamdStage` POSTs the raw
RFC 5322 message to rspamd's real documented `/checkv2` HTTP endpoint
and maps its JSON response (`score`, `action`, `symbols`) onto this
pipeline's result model — `reject` → `CheckFail`, `add header`/
`rewrite subject`/`greylist` → `CheckWarn`, anything else → `CheckPass`.
rspamd's own score is passed through directly rather than being
rescaled, since both scales are already meant to be compared against
configurable thresholds the same way. Verified: score pass-through is
exact, action-to-result mapping is correct for all three cases tested.
- `internal/pipeline/stage_llm.go`: `LLMStage` calls the OpenAI-compatible
`/v1/chat/completions` endpoint that llama.cpp's server (and most other
local-inference servers) expose — no llama.cpp-specific protocol
needed. The model is asked for a single 0-100 integer; parsing extracts
the first run of digits rather than requiring an exact match, since
local models don't always follow format instructions exactly (verified
with a deliberately messy response — extra whitespace and trailing
commentary — that still parses correctly). LLM output is capped at 30
points of total score contribution regardless of what the model
returns, since non-deterministic model output shouldn't singlehandedly
quarantine mail the way a deterministic SPF/DKIM/DMARC failure can —
verified the cap applies correctly (raw score 95/100 → capped
contribution 28.5, not 95).
- `pipeline.StagesFromConfig`: builds `DefaultStages()` (the always-on
deterministic set) plus any of the three optional stages whose config
field (`clamav_socket` / `rspamd_url` / `llm_url`) is non-empty — an
unconfigured optional service is genuinely **absent** from the pipeline,
not merely present-but-disabled, so a misconfigured or unreachable
service that was never meant to be used can't accidentally affect
delivery. Verified: 5 stages with nothing configured, 8 with all three
configured. Wired into `cmd/gomail/main.go` in place of the old
`DefaultStages()` call.
**One real bug found and fixed while testing — in the test's fake clamd
server, not the product code**: the fake server's initial read of the
`"zINSTREAM\0"` command used a generic buffered `conn.Read()` call with a
32-byte buffer, which could over-read into the *next* protocol bytes (TCP
doesn't preserve write-call boundaries, so a client's separate `Write()`
calls can arrive coalesced in one read). That silently desynced the fake
server's chunk-length parsing loop, which then hung waiting for bytes that
had already been consumed — manifesting as an i/o timeout on the *client*
side, initially indistinguishable from a real product bug. Fixed by
switching the fake server to a `bufio.Reader` with exact-length
`io.ReadFull` reads for both the command and every subsequent chunk.
Worth remembering as a general pattern for any future protocol work over
raw TCP: never assume one `Read()` call aligns with the sender's `Write()`
call boundaries.
**Deferred within this phase:**
- Gmail API push (watch+Pub/Sub) and Microsoft Graph delta webhooks, to
replace Phase 10's IMAP polling for linked accounts — not started.
- No live test against real ClamAV, rspamd, or a real llama.cpp server —
same category of limitation as Phase 10's OAuth2 and Phase 13's ACME:
the protocol implementations are proven correct against genuinely-
behaving fake servers, but never against the real services. If the
operator has any of these installed, pointing `clamav_socket` /
`rspamd_url` / `llm_url` at them and sending one real test message
(the EICAR string is safe and appropriate for this) is the natural
first live-test.
- No admin UI toggle for these — they're config-file only
(`pipeline.clamav_socket`, `pipeline.rspamd_url`, `pipeline.llm_url`/
`llm_model` in `config.yaml`), consistent with how they were already
scaffolded as plain fields back in Phase 1.
### Phase 15 — Hardening + deploy — ✅ CORE COMPLETE (last phase in the plan)
**What's done and verified:**
- `internal/ratelimit`: hand-rolled per-key token-bucket limiter (no
third-party rate-limiting library) — a token bucket rather than a fixed
window deliberately, to avoid the classic "burst allowed at both edges
of a reset boundary" flaw a naive counter has. `rate=0` cleanly disables
limiting for a given listener rather than blocking everything, which is
how an unset config value opts out. **Verified under Go's race
detector** with 500 concurrent goroutines against a rate=100 limiter —
exactly 100 allowed, zero drift, proving the mutex genuinely serializes
access rather than merely looking correct single-threaded. Then proven
over real sockets and real HTTP: a real TCP listener genuinely
rejecting connections past the limit, a real `httptest` server
returning genuine 429s past the limit, and a specific check that
`X-Forwarded-For` trust correctly keys the limiter on the forwarded IP
rather than the raw socket peer (only trusted when
`server.real_ip_header` is explicitly configured — untrusted otherwise,
since blindly trusting it would let any client spoof their rate-limit
identity).
- Wired into every listener: SMTP (per-IP connection limit at accept
time, plus a separate limiter on AUTH attempts specifically, checked
before any credential parsing happens), IMAP (per-IP connection limit
at accept time), and a single **shared** limiter across DAV, webmail,
admin, and external JMAP — deliberately one combined per-IP budget
across all of them, not one each, since a client hitting its limit on
one HTTP surface shouldn't get a fresh budget by switching to another.
- Fuzz tests (`go test -fuzz`, real `*_fuzz_test.go` files — the first
standard Go test files in this repo; everything before this used
disposable `cmd/e2etestN` directories) on the four hand-written parsers
most exposed to untrusted network input: `vcard.Parse`, `ical.Parse`,
`sieve.Parse` (explicitly flagged as the highest-stakes target, since
every ManageSieve `PUTSCRIPT` runs through it), and IMAP's `tokenize`
(runs on every line a connected client sends, before authentication
necessarily succeeds). Each actually run for 15s, not just written:
~400K–590K executions per target, ~1.5M executions total, zero panics.
- `go vet ./...` — clean across the entire codebase.
**`staticcheck` was attempted but is not runnable in this sandbox** — it
requires a newer Go toolchain than this network-restricted environment
can fetch (same class of limitation documented elsewhere in this plan
for the real Go 1.25.6 toolchain). This is a real, disclosed gap: `go
vet` provides real but narrower coverage than `staticcheck` would.
Running `staticcheck ./...` on a real machine with normal internet
access is a legitimate quick win for whoever continues this.
- `install.sh`: creates a dedicated system user (no login shell), sets up
`/etc/gomail`, `/var/lib/gomail`, `/var/log/gomail` with correct
ownership, builds and installs the binary, generates
`GOMAIL_MASTER_KEY`/`GOMAIL_JWT_SECRET` into a mode-600 env file if none
exist, installs and enables (but does not start) the systemd unit.
Syntax-checked with `bash -n` (clean); `shellcheck` isn't available in
this sandbox either, so deeper static analysis of the script itself is
another real machine follow-up.
- `gomail.service`: systemd unit with genuine hardening — capability-based
privileged-port binding (`AmbientCapabilities=CAP_NET_BIND_SERVICE`)
instead of running as root, plus `ProtectSystem=strict`,
`ProtectHome`, `PrivateTmp`, `PrivateDevices`, kernel/clock/hostname
protection, namespace/SUID/realtime restrictions,
`MemoryDenyWriteExecute` (with an inline comment flagging it as the
first thing to try removing if the CGO-based binary fails to start
under it — `mattn/go-sqlite3` requires CGO), and a syscall filter.
**Actually verified with `systemd-analyze verify`** (available in this
sandbox, unlike `staticcheck`/`shellcheck`) — returned clean against a
dummy executable standing in for the real binary.
- `README.md`: quick start, TLS/ACME guidance (including the
staging-before-production reminder from Phase 13's own caveat), and a
full DNS setup section — MX, SPF, DKIM (with the exact record shape
GoMail logs at bootstrap), DMARC, all with real example values and
guidance on starting permissive (`~all`, `p=quarantine`) before
tightening. MTA-STS/TLS-RPT sections are present but honestly marked
not-yet-implemented rather than filled with placeholder content.
**Deferred within this phase:**
- `staticcheck` and `shellcheck` — both blocked by sandbox network/
toolchain restrictions, not by any decision; run both on a real machine.
- No fuzz corpus regression files exist yet (expected — Go only persists
a corpus entry to `testdata/fuzz/` on an actual crash, and none of the
15s runs found one; longer fuzzing runs on a real machine, especially
with `-fuzztime` measured in hours rather than seconds, stand a better
chance of finding something a 15-second burst didn't).
- No fuzz targets for the SMTP command parser or JMAP JSON handling
specifically — the four targets chosen were judged highest-value for
the time available; SMTP's command parsing is somewhat covered
indirectly by the SMTP E2E tests from earlier phases but not fuzzed
directly, and JMAP's JSON handling rides on `encoding/json`, which is
already extensively fuzzed upstream in the Go project itself.
- No log rotation configuration, no monitoring/metrics endpoint, no
backup tooling for the SQLite database or Maildir tree — genuinely
unstarted, not scoped in the original Phase 15 description either.
---
## Project status: all 15 phases from the original plan have now been
## touched, 13 of them completed and verified
Phase 10 (native Gmail/Graph API, webmail account-switcher UI) and Phase
11 (aliases CRUD, per-tenant pipeline settings UI) remain "partially" or
"core" complete with named remaining pieces — see their entries above.
Every other phase reached a genuinely complete, tested state. This does
**not** mean the project is finished in an absolute sense — every phase's
entry above lists real deferred items, and the live-service verification
caveats (OAuth2 against real Google, ACME against real Let's Encrypt
staging) are still open — but the plan as originally scoped has been
worked through in full, not just partially sampled.
---
## Architecture notes that matter for continuation
### Dependency management in a network-restricted sandbox
The Anthropic sandbox this was built in blocks `proxy.golang.org` and
`sum.golang.org` — only `github.com` (plus npm/pypi, irrelevant here) is
reachable. Every session that needed a new dependency had to:
```bash
git clone --depth=1 --branch vX.Y.Z https://github.com/OWNER/REPO.git /tmp/REPO
```
then add a `replace` directive in `go.mod` pointing at `/tmp/REPO`. **This
does not apply on a real machine with normal internet access** — `go.mod`
ships with real version requires and no replace directives; `go mod tidy`
will just work. The `go.mod.production` file in each phase's tarball is
already the clean version with no local paths.
### Go version
Sandbox only had Go 1.22/1.23 available via `apt`; the user's real target
is **1.25.6**. Every phase's shippable `go.mod` declares `go 1.25.6`; all
sandbox development/testing happened against 1.22 syntax (no
version-specific features used, so this is a non-issue functionally) with
the directive temporarily lowered for local builds, then restored before
packaging. **On the real 1.25.6 toolchain, nothing needs to change.**
### Sandbox environment quirks discovered (informational, not GoMail bugs)
- The sandbox's default shell is `/bin/sh`, not bash — brace expansion
(`mkdir -p {a,b}`) silently creates a literal directory named `{a,b}`
instead of two directories. Always use explicit `bash -c` or separate
`mkdir` calls.
- The sandbox container has reset entirely mid-session at least once
(Go toolchain, `/tmp`, and the whole working directory vanished) and
resets `/etc/hosts` between *every* tool call. Backgrounded long-running
processes (`&` + later `kill`) don't reliably survive across tool call
boundaries either. **The working pattern that survived all of this**:
write E2E tests as a single self-contained `go run` invocation that
starts the server in-process (as a goroutine) and exercises it as a
client in the same process — never rely on a detached background
process still being alive in a later tool call.
- Because of the above, **always keep a tarball of the last-known-good
state in `/mnt/user-data/outputs`** — that survived every reset and was
the actual recovery mechanism used once.
### Testing pattern used throughout
Every phase has (had, before cleanup) a `cmd/e2etestN/main.go` that:
1. Boots the real server(s) for that phase as goroutines in the same process
2. Acts as a real client over the real wire protocol (raw TCP for
SMTP/IMAP/POP3, real `net/http` for DAV/webmail/JMAP)
3. Asserts against actual database state, not just protocol response codes
4. Is deleted after the phase's tests pass — **not shipped** in the
tarballs, since it's sandbox-only scaffolding, not part of the product
**When continuing this project, recreate this pattern per phase** — it
caught a genuine bug in nearly every phase (see "Bugs actually found"
below) that a build-only check would have missed entirely.
### Bugs actually found by this testing pattern (worth remembering the *pattern*, not the specific fixes)
- NULL-scan panics from `sql.NullString`-shaped columns not being wrapped
correctly (Phase 2, Phase 3)
- A whole SMTP port (`:465` implicit TLS) silently not receiving the
outbound-relay code path because a check only looked for one of two
equivalent session kinds (Phase 2/3)
- Foreign-key ordering: child rows inserted before their parent row
existed (Phase 4)
- Regex `.` not matching embedded `\r\n` from IMAP literals — an entire
class of FETCH responses silently parsed as empty (Phase 6)
- Wrong wire command entirely (`FETCH UID 1` sent instead of `UID FETCH
1`) — server correctly returned nothing, client incorrectly assumed
"not found" (Phase 6)
- Missing `json` struct tags meant to the wire API would have shipped
`TotalCount` instead of `total_count` (Phase 8) — caught before it ever
ran, not after
**Takeaway for whoever continues this: don't trust "it compiles" or even
"the happy-path test passed once." Write the negative-path tests (wrong
auth, cross-user access, malformed input) every time — they found real
bugs in every single phase.**
---
## File map (as of end of Phase 15 — the full plan)
```
gomail/
├── cmd/gomail/main.go # wires everything together
├── go.mod / go.mod.production # sandbox-dev vs real-deploy versions
├── install.sh # system user, directories, secrets, systemd install
├── gomail.service # hardened systemd unit — verified with systemd-analyze
├── README.md # quick start, TLS/ACME guidance, DNS setup guide
├── internal/
│ ├── accounts/ # MailProvider interface + local/IMAP backends
│ ├── auth/ # shared password + app-password verification
│ ├── config/ # YAML + env config loader
│ ├── crypto/ # HKDF-per-record AES-256-GCM encryption
│ ├── db/ # schema, migrations, all queries
│ ├── dav/ # CalDAV/CardDAV HTTP server
│ ├── dkim/ # DKIM sign + verify (stdlib crypto)
│ ├── ical/ # minimal RFC 5545 parser/builder
│ ├── imap/ # IMAP server (core command set)
│ ├── imapclient/ # hand-rolled IMAP client
│ ├── jmap/ # JMAP Core/Mail subset
│ ├── acme/ # ACME v2 client (RFC 8555) — JWS signing, full protocol flow
│ ├── admin/ # admin portal REST API + embedded dark Tailwind SPA
│ ├── mailstore/ # encrypted Maildir storage
│ ├── managesieve/ # RFC 5804 ManageSieve server
│ ├── oauth2/ # hand-rolled OAuth2 client (auth code grant + refresh)
│ ├── pipeline/ # SPF/DKIM/DMARC/header/URL security checks
│ ├── pop3/ # POP3 server (off by default)
│ ├── ratelimit/ # per-key token-bucket rate limiter (SMTP/IMAP/HTTP)
│ ├── sieve/ # RFC 5228 Sieve interpreter (lexer/parser/exec)
│ ├── queue/ # outbound retry queue + MX delivery
│ ├── smtp/ # SMTP server (inbound MTA + submission)
│ ├── tlsutil/ # self-signed cert generation + ACMEManager (SNI-aware serving, renewal)
│ ├── totp/ # RFC 6238 TOTP MFA — verified against real published test vectors
│ ├── vcard/ # minimal RFC 6350 parser/builder
│ ├── webmail/ # REST API + embedded Tailwind SPA
│ └── webtoken/ # hand-rolled JWT (HS256, plus scoped Purpose tokens for MFA/reset)
```
Native Gmail/Graph API providers (as opposed to the IMAP+OAuth2 path
already built) are not yet in `internal/accounts` — see Phase 10's
deferred-items list above. `internal/dane`, `internal/mtasts`, and
`internal/tlsrpt` don't exist yet — see Phase 13's deferred-items list.
No `internal/webauthn` (passkeys) — see Phase 12's deferred-items list;
`internal/totp` exists and is fully wired. That's the complete list of
genuinely unstarted packages as of the end of Phase 15 — every other
phase in the original plan has at least a core-complete, tested
implementation.
Reference in New Issue View Git Blame Copy Permalink