Form Config v1

Plan for expanding and standardizing the webform configuration format, delivered as phased, Opus-orchestrated red/green/verify cycles (see the red-green-verify skill). This reconciles the Form Definition Format — Specification Draft v0.1 with the relay that already exists: a stateless Go Lambda that loads a YAML config from S3, validates a submission, maps fields, and fans out to output targets.

There is no new module or binary. Every phase evolves the existing packages (internal/config, internal/form, internal/mapping, internal/relay, internal/handler, cmd/cli) in place.

Reconciliation: how the spec bends to the code

The spec was drafted against an assumed static-site build model (HTML + a client-side JS bundle + a manifest, handlers running off-host). The project is the opposite: a runtime relay. Where the two disagree, the runtime model wins, because it is what ships today and what the design goals commit to (minimal, stateless, cheap, hot-reloaded from S3). Concretely:

Kept — the spec adapts to the existing shape:

• Runtime relay, not a static build. No emitted client JS, no WASM, no transpiled evaluator. All validation and templating run server-side in the Lambda. This retires spec §7.4 (client/server parity) and Q5a (shared evaluator) entirely.
• forms: stays a map keyed by name. The key is the form id and the route (POST /api/v1/submit/{form}). No per-form version/id/title keys, no one-file-per-form. Multiple forms in one S3 file is how routing and hot-reload work.
• cache_ttl + S3 hot reload preserved. A build step that compiles forms to static artifacts would destroy this; we do not add one.
• captcha: stays a form-level block (internal/captcha already verifies reCAPTCHA v2/v3, hCaptcha, Turnstile server-side). The spec’s captcha field type is dropped.
• outputs: keeps its name (not renamed to handlers). Types stay http_post / email / salesforce_web2lead / smtp. The spec’s richer handler execution model (on_error, retry, per-handler conditions.run) is deferred.
• application/x-www-form-urlencoded only. No multipart, no file uploads.

Adopted — the code grows toward the spec:

• Richer field objects: label, help, hint, hint_position, placeholder, default, required, disabled, readonly, hidden, autocomplete, attributes (S2).
• Expanded field-type catalogue: text, textarea, email, url, tel, number, date, select, radios, checkboxes, checkbox, hidden (S2).
• Options in shorthand and explicit spellings (S3).
• A server-side validation constraint vocabulary with per-constraint messages (S4).
• A multi-value submission model so checkboxes and multi-select survive end to end (S5).
• A restricted server-side template engine for string-valued positions, which also standardizes secret resolution via {{ env.NAME }} (S6). This is a real gap today — the current mapping does rename + static injection only, yet examples already imply templating.
• Strict unknown-key parsing and a dedicated structural-validation pass (S8).
• A generated JSON Schema for editor completion (S9).

Deferred — explicit non-goals for v1 (each is a future plan):

conditions (§8), containers group/columns/details/page (§4.2, §5), composites and parts (§10), repeat (§9), computed fields, file / signature uploads, multi-page forms, i18n (§Q13b), grouped options and options_from (§6), the settings: block (§12), and the expanded handler execution semantics (§13.4, §Q8).

Hard constraints / non-goals

• Go 1.23, container-only tooling (nothing runs on the host but Docker + Task).
• Sole YAML dependency stays gopkg.in/yaml.v3. No new runtime dependencies; templating uses the standard library text/template with a locked function map.
• Backward compatible. Every config valid today must stay valid, except that unknown keys become errors (S8) — a deliberate break that turns silent typos into build failures. Call this out in release notes.
• Stateless: no per-submitter state, sessions, drafts, or persistence beyond the existing outputs.
• Templating never reaches request context, the environment beyond an allowed secret list, arbitrary functions, or the filesystem (S6).
• The Taskfile is the only tooling interface; go/hugo are never invoked on the host.

Repository layout

internal/
  config/
    config.go          # enriched structs, strict parse (Phase 1)
    options.go         # option normalization (Phase 2)
    validate_config.go # structural validation pass (Phase 1)
    schema.go          # JSON Schema emitter (Phase 7)
    testdata/          # valid + invalid config fixtures (owned by Phase 0)
  form/
    validate.go        # constraint enforcement (Phase 4)
    render.go          # enriched HTML renderer (Phase 6)
    testdata/golden/   # golden HTML fixtures (owned by Phase 0/6)
  submission/
    submission.go      # multi-value input model (Phase 3)
  template/
    template.go        # restricted server-side engine (Phase 5)
  mapping/mapper.go    # template-aware mapping (Phase 5)
  relay/               # targets consume templated values (Phase 5)
  handler/handler.go   # parses multi-value body (Phase 3)
cmd/cli/main.go        # config validate / form html / form schema (Phase 7)

Execution model — phased red/green/verify

Each phase is one Opus-orchestrated red/green/verify cycle. Do not attempt the whole spec in one pass. The mechanics live in the red-green-verify skill; this plan supplies each phase’s red brief, green brief, and done-when.

Roles, per phase. An Opus orchestrator drives the phase: it fires the workers, then reviews the result adversarially and restarts until it holds up. Red = Opus writes failing tests. Green = Sonnet implements. Verify = Haiku re-runs independently and checks green didn’t cheat.

Shared commands (containerized):

• task test — run all Go unit tests.
• task lint — go vet + gofmt check.
• task check — the quality gate: test then lint (added in Phase 0).
• task validate:config CONFIG=<path> — domain validation of a config file via the CLI (added in Phase 0; named to avoid colliding with check).

Global rules for every phase (check the parent box only once every child box is checked, including the docs update):

• Red fails at runtime, never at compile. Zero-logic stubs to make tests compile are expected.
• Off-limits to every green agent: internal/*/testdata/, every *_test.go, this plan and the user’s spec draft, and go.mod/go.sum (no new deps).
• Verify, every phase: re-run task check; git diff the tests and fixtures to prove green edited neither.
• A phase is not done at green. After verify passes, the orchestrator runs the skill’s assume-bad review (fresh reviewer: hunt regressions, missed edge cases, incompleteness). The Done when is the green target; the phase closes only when that review finds nothing to send back.
• Restart test-first. Every review finding goes back to red as a new failing test, then green, then verify. If two restarts don’t converge, stop and surface the sticking point.
• A phase’s red tests must stay green in all later phases — later phases only add.
• Close each phase by updating docs (README + docs/), rebuilt with task build:docs.

Phase 0 — Bootstrap (setup, not a red/green cycle)

• Phase 0 complete
  • Create internal/config/testdata/ fixtures: valid_full.yaml (exercises every new field type, options spelling, and constraint), valid_legacy.yaml (a config in today’s shape, to prove back-compat), and invalid_*.yaml (unknown key, options on a text field, number props on a select, reserved field name, missing options on a choice type). These are owned by bootstrap and off-limits to green thereafter.
  • Create internal/form/testdata/golden/ with an empty README; golden HTML lands in Phase 6.
  • Add stub types/fields so future red tests compile: new fields on config.Field, an Option struct, Constraints, submission.Values, template.Render, config.Schema — all zero-logic. (The enriched Field became non-comparable, so form_test.go’s != comparison was moved to reflect.DeepEqual — a behavior-preserving bootstrap fix.)
  • Taskfile: add test/lint aliases, check (test+lint gate), validate:config, and generate:schema tasks (S9); confirm the required-set naming (taskfile-conventions).
  • Confirm go test -race works in the golang:1.23 image (Debian base has a C toolchain; do not set CGO_ENABLED=0). Verified: full suite passes with -race.
  • Update docs: this plan lives in the Plans section (docs/content/plans/).

Phase map

Phase 1 — Enriched field model + strict parse

• Phase 1 complete

  • Red — parse_phase1_test.go: parses valid_full.yaml asserting the new common properties (S2.1) decode; accepts every new field type (S2.2); rejects invalid_unknown_key.yaml via KnownFields(true); rejects invalid_reserved_name.yaml; rejects type↔property mismatches (min/options/multiple, asserting on the offending-property substring so they can’t pass on the old “invalid field type” error); parses valid_legacy.yaml unchanged. Restart added hidden-requires-default and case-insensitive reserved-name tests, plus a non-string-default regression lock.
  • Green — rawField/Field grown with the S2 properties (min/max/ step decoded via yaml.Node to accept int/date scalars as strings); validFieldTypes extended to the 12 v1 types; decoder switched to strict KnownFields(true); validate_config.go runs the S8 structural pass.
  • Verify + review — full suite + vet + gofmt green (independent Haiku run); fixtures and red tests confirmed un-weakened; Opus adversarial review found 3 items — 2 fixed test-first, 1 (non-string default) empirically disproven. Two stale pre-existing tests reconciled (checkbox→bogus; hidden field given a default).
  • Done when: valid_full.yaml and valid_legacy.yaml parse; all five invalid_*.yaml fail with a message naming the offending key/field. Met.
  • Update docs: new docs/content/reference/field-model.md; strict-parse breaking change noted in README + CHANGELOG.md; docs site rebuilds.

  Deliberately out of Phase 1 scope (candidate for a later validation phase): the field-name pattern ^[a-z][a-z0-9_]{0,63}$ (spec §3) is not yet enforced — only reserved names, uniqueness, and type/property compatibility are. Also deferred: attributes map values must currently be strings.

Phase 2 — Options normalization

• Phase 2 complete

  • Red — options_phase2_test.go locks shorthand + explicit normalization (with selected/disabled and label-defaults-to-value), mixed spellings, and rejection of grouped options, options_from, duplicate values, and unknown option keys. Restart added options_robustness_test.go: empty/null/non-scalar option entries.
  • Green — buildOptions reworked to (form, field, nodes) → ([]Option, error): a three-way node switch (scalar / mapping / invalid), label-defaults-to-value, duplicate detection, strict per-option keys, grouped + options_from rejection, and empty/null/non-scalar rejection.
  • Verify + review — full suite + vet + gofmt green (independent Haiku); fixtures untouched. Opus review confirmed normalization, all rejections, proper bool decoding, cross-spelling dedup, and order preservation correct; it surfaced two robustness holes (empty/non-scalar options admitted silently), both fixed test-first in the restart.
  • Done when: both spellings normalize identically where equivalent; grouped options and options_from fail with “not supported in v1”. Met.
  • Update docs: Options section added to field-model.md (both spellings
    • deferred/rejected forms flagged); CHANGELOG.md bullet; docs site rebuilds.

  Deferred minor (noted by review): when a non-choice field carries a malformed options entry, the per-option error (“unknown key”) can surface before the cleaner “options not allowed on type” placement error. Still a valid error; message-ordering polish only.

Phase 3 — Multi-value submission model

• Phase 3 complete
  • Red — a large refactor-with-tests pass: threaded submission.Values through every signature and added multi-value locks for submission (accessors, clone-on-New, copy-on-All, zero-value safety), handler (repeated key reaches the relayer), mapping (multi-value preserved + static override), relay (email/smtp two-line body, http Encode), and form.Validate. Restart added an email-format-on-non-blank-value test.
  • Green — implemented internal/submission (wraps url.Values, deep-clones); reworked mapping.Apply, form.Validate, the 3 relay targets (postForm posts data.Encode(); email/smtp render one key=value line per value), and relay.Relay — recovering the original SES/SMTP/HTTP logic from git HEAD and re-threading it. handler.go no longer collapses to v[0].
  • Verify + review — full suite + vet + gofmt green (independent Haiku). Opus review diffed every target against git HEAD: no regressions (SMTP TLS/STARTTLS/auth, stripCRLF, SES charset, postForm guards, fan-out, handler ordering all preserved) and every multi-value edge handled. Two gaps closed test-first: the missing submission unit-test file, and the email-format check reading only the first (possibly blank) value.
  • Done when: a two-value checkbox submission arrives intact at a fake target; all prior-phase and existing relay tests stay green. Met.
  • Update docs: “Multi-value fields” section in field-model.md + per-output serialization table in the reference; CHANGELOG.md bullet; docs site rebuilds.

Phase 4 — Validation constraints + messages

• Phase 4 complete

  • Red — constraints_phase4_test.go: table tests for all 11 S4 constraints, each pass/fail, with custom-messages (verbatim) and default-substring assertions; blank-optional-skips and required-promotion cases; pattern_phase4_test.go locks parse-time rejection of a bad regex. Restarts added constraints_edge_test.go (non-numeric under numeric constraint, min_selected on optional-blank, messages.required).
  • Green — validate.go enforces every constraint against submission.Values (rune length; numeric vs ISO-lexical min/max; step; anchored RE2 pattern; one_of; selection counts; matches/differs); required and all 12 keys honor Messages[...] verbatim via one helper. validate_config.go compiles pattern at parse time (bad regex → error).
  • Verify + review — full suite + vet + gofmt green (independent Haiku). Opus review confirmed inclusive boundaries, step epsilon, pattern anchoring, message keys, cross-field checks correct; three real gaps closed test-first: non-numeric value silently passing a numeric constraint, min_selected being a no-op on an optional-blank field, and messages.required being ignored.
  • Done when: every constraint’s pass/fail case and custom/default message assertion is green. Met.
  • Update docs: Validation section (constraint table + messages mapping + assumptions) in field-model.md; CHANGELOG.md bullet; rebuilt.

  Accepted v1 limitations (spec-silent): date/time min/max assume zero-padded ISO values (which HTML date/time inputs produce); step is based at 0, not at the field’s min.

Phase 5 — Restricted server-side template engine

• Phase 5 complete

  • Red — template_test.go locks the engine: .fields.<name> (joined string), .form.id, .submitted_at, env "NAME" secrets, {{ .env }}→ empty (never dumpable), the function whitelist, unknown-func error, and no-HTML-escaping; Check valid/syntax/unknown-func. Plus config parse-time template rejection, a handler render test (subject/to/static/ header render from fields + env), and submission.Map. Restart added string-funcs-on-multi-value-field and values tests.
  • Green — internal/template over text/template (no HTML escaping): .fields.<name> is always a joined string (so string funcs never fail on cardinality), values/join reach raw multi-values, secrets via an env "NAME" function (environment never a value). Check runs on every templated output position at parse (bad template → load error). The handler renders subject/to/static/headers/creds into a fresh copy of the outputs — never mutating the TTL-cached config. relay/targets/ mapping untouched.
  • Verify + review — full suite + vet + gofmt green (independent Haiku). Opus review confirmed the env is non-dumpable, no-HTML-escape, and config not mutated; it caught a data-dependent 500 (string funcs crashed on multi-value fields under cardinality-typed .fields) — fixed test-first by making .fields always a joined string and adding values, which also deleted the AST-walk hack.
  • Done when: the allowed scope renders, bad templates error at load, and secrets resolve only via env "NAME". Met.
  • Update docs: Templating section in the config reference (positions, scope, function whitelist, env secret convention, no-HTML-escape); CHANGELOG.md bullet; docs rebuild.

  Deviations from the draft spec (documented): secrets use the function {{ env "NAME" }}, not {{ env.NAME }}; {{ .env }} renders empty rather than being a parse error (the environment is simply never a value in scope, which meets the no-dump security goal). Minor: env "TYPO" on an unknown name renders empty. url and mapping are intentionally not templated.

Phase 6 — Enriched, accessible HTML renderer

• Phase 6 complete

  • Red — render_golden_test.go + 7 hand-authored goldens (text_full, select, radios, checkboxes, checkbox, hidden, all_types) byte-comparing RenderHTML, covering every field type, the label for/id + aria-describedby wiring, groups’ fieldset/legend, hidden emitting default, sorted custom attributes, escaping, and determinism (re-render identical). Restart added attribute-key and empty-label tests.
  • Green — rewrote render.go with a strings.Builder + html.EscapeString (dropped html/template) to reproduce the goldens byte-for-byte; fixed attribute-name-position escaping and empty label/legend omission.
  • Verify + review — full suite + vet + gofmt + 7 golden cases green (independent Haiku). Opus review confirmed all values escaped, name/id breakout closed, hidden interplay and determinism correct; three low-severity gaps handled: attribute-KEY injection (now validated at parse), empty <label>/<legend> (now omitted), and group required (documented as server-side).
  • Done when: golden bytes match for every field type across two runs. Met.
  • Update docs: Rendered HTML section (markup per category + a11y wiring
    • escaping/determinism + limitations) in field-model.md; CHANGELOG.md bullet; docs rebuild.

  Notes: group required (radios/checkboxes) is enforced server-side (required/min_selected), not via a client-side required attribute — checkbox-group “at least one” has no clean single-attribute HTML form. Custom attributes keys are validated to ^[A-Za-z][A-Za-z0-9_.:-]*$ at parse.

Phase 7 — CLI, JSON Schema, docs (final; not a red/green cycle)

• Phase 7 complete

  • config.Schema emits a draft-07 JSON Schema from the Go types (S9), with $defs/$ref, additionalProperties:false, type/output/provider enums, and a reserved-name-excluding name pattern; schema_test.go byte-compares it to a golden and checks determinism.
  • form schema CLI command added; generate:schema task writes docs/static/config.schema.json. config validate uses the strict parser (yaml decode errors already carry line numbers). Deferred: precise line/column on structural errors — capturing per-node positions conflicts with KnownFields strictness; structural errors name the form/field/property instead. Documented nicety, not shipped.
  • Refreshed openapi.yaml (status codes 200/400/404/422/500, multi-value body, full 422 vocabulary), README (form schema), the Plans landing page (corrected its false “no validation / no relay targets” gaps into an accurate shipped/deferred split), and the how-to guides (fixed a real --config placement bug Go’s flag silently ignored). generate:all + build:docs succeed (32 pages).
  • Verify by review: independent Opus reviewer (using the tools image’s Python jsonschema) confirmed the schema ACCEPTS valid_*.yaml and REJECTS the unknown-key and reserved-name fixtures; the type↔property fixtures are accepted at the schema level and rejected by the authoritative Go validator (documented division). All invalid_*.yaml rejected by config validate.

  Scope notes: the JSON Schema does shape validation (editor completion); the Go config validate remains authoritative for type↔property rules, reserved names, choice/options, template syntax, and per-output requirements.

Specification sections

S1 — Document shape (unchanged surface)

Top level stays cache_ttl + forms: (a map keyed by form name = route). Each form keeps captcha:, fields:, outputs:. No version, id, title, settings, or handlers keys are introduced in v1.

S2 — Field object

S2.1 Common properties (all types): name (required), type (required), label, label_hidden, help, hint, hint_position (before|after), placeholder, default, required, disabled, readonly, hidden, autocomplete, attributes (map merged onto the input). Unlisted spec properties (admin_label, internal, wrapper_attributes, notes, repeat, conditions) are not in v1.

S2.2 Types and type-specific properties:

A type-specific property on the wrong type is a structural error (S8).

S3 — Options

Two spellings, normalized to []Option{Value, Label, Selected, Disabled}:

options: [small, medium, large]        # shorthand: value == label
options:
  - { value: md, label: Medium, selected: true }
  - { value: xl, label: Extra large, disabled: true }

Grouped options and options_from are rejected with an explicit “not supported in v1” error. Choice types (select/radios/checkboxes) require non-empty options; every other type rejects an options key.

S4 — Validation

Constraints live under validate:; messages under messages:, keyed by constraint name. required stays a top-level property; its message stays in messages.

All enforcement is server-side in form.Validate. Anything needing a lookup, external call, or cross-submission uniqueness is out of scope (spec §7.4).

S5 — Multi-value submission model

Runtime input moves from map[string]string to internal/submission.Values (wrapping url.Values). Accessors: First(name), All(name), Has(name). handler.go stops collapsing repeated keys to the first value. Single-value targets and mappings read First; min_selected/max_selected and multi- select/checkboxes read All.

S6 — Templating

Engine: standard-library text/template with a locked FuncMap. Data scope: .fields.<name> (submitted values), .form.id, .submitted_at, and env.<NAME> (resolved only from an allow-listed set of Lambda env vars — the canonical secret mechanism; spec’s { secret: NAME } is not adopted, keeping one spelling). Allowed functions: a small set — trim, lower, upper, default, join. No method calls, no arbitrary functions, no filesystem, no full environment. Every templated string is parse-checked at config load, so a bad template fails the build, not a request. Allowed positions: output subject, output static values, output headers values, and SMTP/SES credentials. Values are substituted as raw text (targets are email/webhook sinks, not HTML) — documented explicitly.

S7 — HTML rendering

Server-rendered only (no client JS in v1). Labels wire to inputs via for/id; help/hint wire via aria-describedby; required, disabled, readonly render as attributes; hidden emits its default; choice types render native select/radio/checkbox groups from normalized options. Output is deterministic (stable ordering, all values escaped through html/template). Target: WCAG 2.2 AA label/description patterns.

S8 — Strictness and structural validation

yaml.Decoder.KnownFields(true) on both decode passes → unknown keys are errors. A dedicated structural pass after decode enforces: unique field names (exists today); reserved names rejected (id, form_id, submitted_at, handler); type↔property compatibility (S2.2); choice/options rules (S3); RE2 pattern compilation; template parse-checks (S6). Decode never validates; errors carry yaml.Node line/column where available.

S9 — JSON Schema

config.Schema emits a JSON Schema from the Go types for editor completion and inline errors. A golden test locks its bytes; form schema prints it; it must accept every valid_*.yaml fixture and reject every invalid_*.yaml.

Taskfile

Reconciled with taskfile-conventions and containerized-tooling. Existing tasks (build:*, test:unit, test:e2e, lint:*, fmt:code, generate:*, build:docs, deploy:*) are kept. Additions:

• test — alias to test:unit (the plan’s shared command name).
• lint — alias to lint:all.
• check — the quality gate: runs test then lint. (New; no prior check task exists, so no collision.)
• validate:config — docker run … go run ./cmd/cli --config {{.CONFIG}} config validate. Named validate:config, not check, to keep the domain validation distinct from the quality gate.
• generate:schema — emit JSON Schema via cmd/cli form schema into docs/static/ for the docs site; folded into generate:all.

Everything runs in the golang:1.23 container. -race (if added to test) needs the C toolchain the Debian-based image already provides — do not set CGO_ENABLED=0.

Definition of done

• Every config valid before this work still parses, except configs with unknown keys, which now error (documented break).
• All new field types, options spellings, and constraints from S2–S4 parse, validate, and render.
• Multi-value submissions survive end to end (S5).
• Templating renders only the allowed scope and functions; bad templates fail at config load; secrets resolve only via env "NAME" from env vars (S6).
• Rendered HTML is deterministic, escaped, and accessibility-wired (S7); golden files locked.
• JSON Schema emitted and validated against the fixtures (S9); the Go validator remains authoritative for the rules JSON Schema can’t express.
• README, docs/, openapi.yaml, and CHANGELOG.md reflect shipped behavior; task build:docs succeeds (32 pages).
• Every phase landed via an Opus-orchestrated red/green/verify cycle whose assume-bad review found nothing left to send back.