Phase 4 of the deploy-hardening I master bundle. The canonical NGINX
implementation that Phases 5-9 model on. Replaces the historical
os.WriteFile flow at internal/connector/target/nginx/nginx.go:99
with deploy.Apply() and adds three production-grade competitor-gap
features: atomic deploy with rollback, post-deploy TLS verify, file
ownership preservation.
NGINX connector — internal/connector/target/nginx/nginx.go:
- DeployCertificate now wires deploy.Apply with PreCommit running
the operator's ValidateCommand (e.g. `nginx -t`), PostCommit
running ReloadCommand (e.g. `nginx -s reload`), and an explicit
post-deploy TLS verify step that dials the configured endpoint,
pulls the leaf cert SHA-256, and compares against what was just
deployed. SHA-256 mismatch (wrong vhost / cached cert / NGINX
still serving stale) triggers automatic rollback: backup files
are restored + reload fired again. Failed-second-reload returns
ErrRollbackFailed (operator-actionable; loud audit + alert).
- ValidateOnly replaces the Phase 3 stub: runs the operator's
ValidateCommand without touching the live cert. V2 contract is
syntax-only validation (full pre-deploy temp-config validation
is V3-Pro). Returns ErrValidateOnlyNotSupported when no
ValidateCommand is configured.
- New per-target Config fields: PostDeployVerify (frozen-decision-
0.3 default ON), PostDeployVerifyAttempts (default 3 — defends
against load-balanced targets where the verify might hit a
different pod that hasn't picked up the new cert yet),
PostDeployVerifyBackoff (default 2s exponential), per-file
Mode/Owner/Group overrides (KeyFileMode, CertFileMode,
KeyFileOwner, etc.), and BackupRetention (default 3, -1 to
disable backups entirely — documented foot-gun).
- buildPlan honors per-distro nginx user (Debian: www-data,
Alpine: nginx, Red Hat: nginx) by checking the local user
database; falls back to no-chown when neither exists. Means
the connector is portable across distros without operator
config.
Deploy package — internal/deploy/ownership.go:
- applyOwnership now silently swallows chown failures when the
agent isn't running as root. Production agents always run as
root and chown failures are real bugs; dev / CI runs as a
regular user where chown to a different uid will always fail
with EPERM (or EINVAL on some tmpfs configs) and would
otherwise force every test to run with sudo. Production-grade
contract preserved (uid 0 still hard-fails on chown errors).
Test suite — internal/connector/target/nginx/nginx_atomic_test.go
ships 42 new named tests (NGINX total: 17 pre-existing + 42 new = 59,
above the prompt's >=40 bar; matches the IIS depth bar of 41):
- Atomic-deploy invariants (cert+chain+key all-or-nothing,
validate-fails-no-files-changed, reload-fails-rollback,
rollback-also-fails-escalation)
- SHA-256 idempotency (full match skips, partial match deploys all)
- Post-deploy TLS verify (fingerprint-match-success,
SHA256-mismatch-rollback, dial-timeout-rollback, retries-until-
match, retries-exhausted-rollback, no-endpoint-skips,
disabled-skips-entirely, default-10s-timeout, endpoint-forwarded)
- Ownership / mode preservation (existing-mode-preserved, override-
wins, KeyFileMode override applied)
- Backup retention (keeps-last-N, disabled-creates-no-backups,
fresh-deploy-creates-backup)
- Concurrency (same-paths-serialize via deploy package's file mutex,
different-paths-parallelize)
- ValidateOnly (happy-path-nil, command-fails-wrapped-error,
no-config-returns-sentinel, ctx-cancelled, stderr-in-message)
- Edge cases (no-chain, no-key, no-chain-path, empty-cert-PEM,
ctx-cancelled, all-four-one-apply)
- Result.Metadata + DeploymentID shape contracts
Coverage: NGINX 91.0% (above the >=85% prompt bar). Race detector
clean. golangci-lint v2.11.4 clean. Existing 17 tests still all pass
(no behavior change in the legacy paths exercised there).
Phase 5 next: mirror this implementation for Apache + lift its
test count from 3 to >=30. Same template applies through Phases
6-9 for the remaining 11 connectors.
Phase 1 of the deploy-hardening I master bundle. Closes the load-bearing
prerequisite for the seven Bundle I items by extracting one canonical
atomic-deploy primitive at internal/deploy/ that all 13 target connectors
will consume in Phases 4-9.
The package ships:
- Plan + Apply API: write all File entries to sibling .certctl-tmp.<nanos>
in the destination directory (same-filesystem guarantees os.Rename atomicity),
call PreCommit (validate-with-the-target), atomic-rename all temps to final,
call PostCommit (reload). On PostCommit failure, restore from pre-deploy
backups + re-call PostCommit. If second PostCommit also fails, return
ErrRollbackFailed (operator-actionable; documented loud).
- AtomicWriteFile lower-level entry for connectors that don't fit the Plan
model (F5, K8s — they ship bytes through APIs, not local files).
- SHA-256 idempotency: every Apply short-circuits when all File destinations
already match SHA-256 of new bytes. Defends against agent-restart retry
storms hammering targets with no-op reloads.
- Ownership + mode preservation: existing nginx:nginx 0640 stays
nginx:nginx 0640 across renewals. Per-target FileDefaults applies for
first-deploy. Per-File explicit Mode/Owner/Group overrides win over both.
Closes the silent-failure mode where os.WriteFile(path, bytes, 0600) at
apache.go:119 (et al.) clobbered worker access.
- Backup retention janitor: pre-deploy backup at <path>.certctl-bak.<nanos>;
default keeps last 3 (DefaultBackupRetention); BackupRetention=-1 disables
backups (rollback impossible — documented foot-gun).
- File-level mutex via sync.Map: two concurrent Apply calls touching the
same destination serialize. Per-target serialization (Phase 2) is finer-
grained at the agent dispatch layer; this is the file-level guard.
- Sentinel errors for connector errors.Is checks:
ErrPlanInvalid, ErrValidateFailed, ErrReloadFailed, ErrRollbackFailed.
Tests (37 named cases across deploy_test.go + coverage_test.go) pin every
load-bearing invariant the prompt's Phase 1 requires, plus error-leg
coverage uplifts:
- TestApply_HappyPath_PreCommitSucceeds_PostCommitSucceeds_FilesAtomic
- TestApply_PreCommitFails_NoFilesChanged (atomic-or-nothing on validate)
- TestApply_PostCommitFails_FilesRolledBack (rollback wire)
- TestApply_RollbackAlsoFails_ReturnsErrRollbackFailed (escalation path)
- TestApply_IdempotentSkip_SHA256Match (idempotency short-circuit)
- TestApply_PreservesExistingOwnerAndMode_WhenNotOverridden
- TestApply_RespectsOverrides_OwnerGroupMode
- TestApply_ConcurrentApplyToSameFile_Serializes (file-level lock)
- TestApply_BackupRetention_KeepsLastN (janitor pruning)
- TestApply_NoExistingFile_UsesDefaultsForOwnerGroupMode
- TestAtomicWriteFile_TempFileCleanedUpOnError
- TestAtomicWriteFile_RenameRaceWithReader_AtomicReadAlwaysSeesOldOrNew
(POSIX-rename atomicity proof via concurrent reader)
Plus white-box tests for resolveOwnership, lookupUID/GID, and deeper error
legs in restoreFromBackups + applyOwnership + AtomicWriteFile.
Coverage 87.3% — practical ceiling without injecting a fault-aware FS
abstraction (Write/Sync/Close OS errors are unreachable from go test
without sudo'd disk-fill or a custom interface seam). Above the existing
service-layer 70% floor; Phases 4-9 will lift this further as they exercise
the package through real-connector use.
Race detector clean; gofmt + go vet + golangci-lint v2.11.4 all 0 issues.
The package is the load-bearing prerequisite for Phases 4-9. Phase 2 next:
per-target deploy mutex in cmd/agent/main.go.
Spec: cowork/deploy-hardening-i-prompt.md
Baseline + recon: cowork/deploy-hardening-i/baseline.md
