Production Hardening (0.4.x+ backlog)

The full backlog of work required before webform-relay is a fully durable, abuse-resistant public endpoint. The Form Config v1 work (0.2.x) made the form/config engine production-quality (validation, templating, rendering, ~82% tested with -race); the gaps here are operational and concentrate in three areas: abuse control, submission durability, and CORS / post-submit UX. For an internal, trusted-network deployment relaying to a webhook, most of this is optional.

The 0.3.x slice has been split out. Four small, no-ongoing-cost items from this backlog — CORS (3a), stage throttling (1a), per-output on_error (2a), and the honeypot (1b) — are the Public Endpoint MVP (0.3.x) milestone. They are struck through below and cross-referenced; this document is now the 0.4.x-and-later remainder: durability, WAF, min-fill token, success redirect, and the operational should-fix items.

Two related items are already done and are not repeated here: the Lambda / relay-client timeout alignment (Timeout: 25 under API Gateway’s 30s cap, with 10s per-output relay timeouts) and the CI pages branch fix.

Context / constraints

  • Stateless Go Lambda behind API Gateway HTTP API, deployed with SAM. No database, no session.
  • API Gateway → Lambda is synchronous (request/response), so Lambda async destinations / function DLQs do not catch relay failures — durability needs an explicit design (below).
  • Keep the “no new runtime dependency beyond yaml.v3” rule where practical; AWS SDK clients (SQS/S3) are already available.
  • Each workstream marks what is unit-testable (red/green/verify per the red-green-verify skill) vs. infra-verified (SAM template + a deploy + a curl/probe check).

Workstream 1 — Abuse control (rate limiting + spam)

Problem. POST /api/v1/submit/{form} is public and unauthenticated with no throttling, usage plan, or WAF. A single client can drive unbounded Lambda, SES, and downstream cost. Captcha is optional per-form and only stops unsolved bots — it does not cap volume.

Decisions to make:

  • Global throttle only, or per-IP? (Per-IP needs AWS WAF — HTTP API has no usage plans; those are REST-API only.)
  • Add the deferred in-app spam controls (honeypot, min-fill time) from spec §12?

Approach (layered, cheapest first):

  1. Stage throttling (infra) — set DefaultRouteSettings (ThrottlingRateLimit / ThrottlingBurstLimit) on the HTTP API in template.yaml, exposed as SAM parameters. Caps total volume immediately.
  2. AWS WAF rate-based rule (infra) — a WebACL with a per-IP rate rule associated with the API stage; stops single-IP floods. More setup; add if the endpoint is high-value.
  3. Honeypot + min-fill (unit-testable) — a per-form spam: block (honeypot: <field>, min_fill_seconds: N). The handler silently returns 200 without relaying when the honeypot field is non-empty; min-fill needs a server-issued timestamp embedded at render + verified on submit (a small signed token) — scope this second, it is more involved than the honeypot.
  • 1a — Stage throttling → moved to 0.3.x (Phase 2).
  • 1b — Honeypot → moved to 0.3.x (Phase 4).
  • 1c — WAF rate rule (infra, optional): WebACL + association; verify per-IP throttling. Ongoing cost: AWS WAF is not free-tier — a WebACL is ~$5/month + ~$1/rule/month + per-request charges. Add only when a per-IP cap is genuinely needed; the 0.3.x stage throttle already caps total volume at no cost.
  • 1d — min-fill token (red/green, optional): signed render-time timestamp + submit-time check. No new AWS resource (HMAC in-process), so no ongoing cost. Fully specced in Min-Fill Signed Token — including the one architectural decision it forces (a per-request token issuance path, since nothing renders forms at request time today).

Workstream 2 — Submission durability (highest priority for 0.4.x)

Problem. relay.Send attempts all outputs and returns an error if any fails → the handler returns 500 and the submission is lost. No retry, no queue, no persistence. A transient SES throttle or a laggy CRM drops real user data — the worst failure for a relay.

The all-or-nothing-on-any-failure half of this is addressed by 0.3.x per-output on_error (2a), so one failing optional output no longer 500s the user. What remains for 0.4.x is durability of a required output that hard-fails — retry and persistence — below.

Design options (this is the key decision for 0.4.x):

  • A. Sync-validate, async-relay (recommended). The API Lambda validates synchronously (keeps the 422-on-invalid UX), then enqueues the valid submission to SQS and returns 200. A separate worker Lambda consumes the queue and relays, with SQS’s built-in retries + DLQ. Durable, retried, and keeps synchronous validation. Cost: a second function + queue; the 200 means “accepted”, not “delivered”.
  • B. Sync relay + durable spool. Keep the current synchronous relay, but on failure write the submission to an S3 “failed” prefix (the deferred store handler) so nothing is lost; a scheduled retrier drains it. Simpler, no queue, but retries are DIY and delivery is still best-effort within the request.
  • C. Per-output on_error semantics only. — shipped in 0.3.x; listed here for completeness. Smallest change; still loses hard-failed required outputs unless combined with A or B.

SAM compatibility & ongoing cost (checked against the “no maintenance cost” goal). Option A is fully SAM-native — AWS::SQS::Queue for the queue and DLQ (via RedrivePolicy), an Events: { Type: SQS } source on the worker function (SAM generates the event-source mapping + IAM), and an AWS::CloudWatch::Alarm on DLQ depth. Nothing is provisioned, so there is no fixed monthly charge and nothing to patch — it scales to zero. Against AWS’s perpetual free tier (1M SQS requests, 1M Lambda requests, 10 CloudWatch alarms per month) a form relay’s volume costs ~$0/month. Two honest nuances: a DLQ alarm is $0.10/mo beyond the 10 free alarms, and the SQS→Lambda event-source mapping keeps long-poll connections open continuously, so it emits a low steady trickle of ReceiveMessage requests even at idle — well within the free tier, but not literally zero API activity. Option B’s S3 spool avoids the idle poll (S3 has no poller) but, if its retrier is a cron-scheduled Lambda, trades it for scheduled idle invocations. Conclusion: A meets the no-maintenance-cost constraint; its real cost is architectural (a second function + queue + DLQ to reason about), which is why it was held out of the 0.3.x MVP rather than ruled out.

Recommendation: A for a real public deployment (true durability), or B if you want a smaller step that still stops data loss.

Testability: the enqueue/spool decision logic is unit-testable (fake queue/store); the SQS/S3 wiring is infra-verified.

  • 2a — Per-output on_error → shipped in 0.3.x (Phase 3).
  • 2b — Durable path (red/green + infra): implement the chosen option (A: SQS enqueue + worker; or B: S3 spool). Unit-test the handler’s enqueue/spool behavior against a fake; wire the real queue/bucket in SAM. Resolve the A-vs-B decision (above) before starting.
  • 2c — DLQ + alarm (infra): dead-letter queue + a CloudWatch alarm on DLQ depth / relay-failure rate.

Workstream 3 — CORS and post-submit UX

Problem. No CORS on the HTTP API. A full-page <form> POST works but navigates the user to raw JSON ({"message":"ok"}); any fetch/XHR embed cannot read the response without Access-Control-Allow-Origin, and there is no origin allowlist restricting who may POST.

Approach:

  1. CORS (infra) — add CorsConfiguration to the HTTP API (AllowOrigins as a SAM param allowlist, AllowMethods: [POST], AllowHeaders: [content-type]). API Gateway answers OPTIONS preflight automatically. Enables fetch-based submission and restricts origins.
  2. Success redirect (red/green, optional) — an optional per-form confirmation.redirect URL (spec §12): on 200, respond 303 to that URL so full-page POST forms land somewhere sensible. Handler-level, testable.
  • 3a — CORS config → moved to 0.3.x (Phase 1).
  • 3b — Success redirect (red/green, optional): confirmation.redirect in config; handler returns 303 on success when set. No new AWS resource.

CSRF — not applicable (deliberate, spec Q12a)

CSRF tokens are intentionally not used, and this is a reasoned decision, not a missing control. CSRF defends state-changing actions taken with a victim’s ambient authority (session cookie / auth). This endpoint is stateless, anonymous, and public — there is no session, no auth, and nothing done “on behalf of” a user, so a forged cross-site POST achieves nothing a direct curl POST doesn’t. A token guarding a non-existent session is theater, and on a static-render model there is no per-request server render to mint one anyway.

Note the common conflation: CORS is not CSRF protection here. A application/x-www-form-urlencoded POST is a CORS “simple request” (no preflight), so CORS only restricts who can read the response from a browser — it does not block the submission (curl/bots ignore it). CORS is an origin allowlist (workstream 3), not a submission gate.

The real threat on this endpoint is volume/abuse, handled by rate limiting + WAF (workstream 1), captcha (shipped), and the honeypot/min-fill token (workstream 1) — whose signed render-time token provides weak anti-automation provenance, not session protection. CSRF would only become relevant if the endpoint ever gained authentication or per-user state (explicitly out of scope).

Should-fix (independent of the three workstreams)

  • Secrets — captcha secret and SMTP password live in the S3 config, and the {{ env "NAME" }} template path covers only output positions (not the captcha secret). Encrypt + lock down the config bucket, and/or resolve these from SSM Parameter Store / Secrets Manager. Confirm no plaintext secrets in the SAM template’s Environment.Variables.
  • SES production access — SES starts in sandbox (verified recipients only, low quota). Request production access before relying on the email output. Operational, lead-time.
  • Gate security scans — gosec / govulncheck are allow_failure: true in CI. Establish a clean baseline and make them blocking (the CI comment already says so).
  • Validate config on deploy — a bad S3 config only surfaces at request time as 500 "failed to load config". Run webform-relay config validate in CI / a pre-deploy step, and consider a lightweight health route.

Suggested order (post-0.3.x)

  1. Durability (2b/2c) — the largest and highest-priority piece; decide option A vs B first (see the SAM/cost analysis above).
  2. Should-fix items — secrets, SES production access, scan gating, and deploy-time config validation, in parallel as operational tasks.
  3. Optional polish — success redirect (3b), min-fill token (1d), and the WAF per-IP rule (1c) only if the endpoint is high-value enough to justify WAF’s ongoing cost.