otche/otche-deploy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Force exact image recreation and preserve admission fences during recovery

Browse Source
This commit is contained in:
omar
2026-09-24 19:04:22 +03:00
parent ffc462cf28
commit df3eed0c1b
2 changed files with 21 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+2
View File
@@ -250,6 +250,8 @@ Use literal reviewed 40-character commits, not the uppercase explanatory placeho
4. Only after zero active work, recheck checkout cleanliness/ancestry, fast-forward the three reviewed commits, and promote the captured built image IDs to the exact runtime tags resolved by target Compose (normally `PROJECT-api`, `PROJECT-migrate`, `PROJECT-worker`, `PROJECT-watchdog`, `PROJECT-web`, all `:latest`). No rebuild is needed: the temporary worktrees used those exact commits. Start with **`--profile execution up --no-build --wait`**, including worker/watchdog; migration must complete before the dependent API/executors start. This deliberately activates execution containers, though all qualification/isolation gates still apply.
5. Explicitly recreate web and wait for health again. Compare each built service container's actual image ID with the captured build ID, require healthy/running services and a successful exited migration, then atomically write `approved-commits.env`. Missing/mismatched images or unhealthy services never approve pins. A failure after the fence may leave a partial rollout and advanced checkouts; the script prints the old container image IDs for **manual** recovery and never deletes old images or resets source. Retain the API admission fence until the operator has inspected migration compatibility and established a consistent recovery state; do not blindly downgrade a migrated database.
Temporary build tags include a digest of all three selected commits, not only the deployment revision. Activation force-recreates every built service so Compose cannot silently retain an older image after a retag. For recovery after a recorded partial rollout, the existing API may already be intentionally stopped: its artifact-volume identity is still checked and it remains stopped if validation or quiescence fails. A previously running API is restored on pre-activation refusal; an already fenced API is never reopened automatically. Inspect all checkout/container/image identities before selecting the exact recovery commits; do not edit approved pins to hide a mismatch.
Coordinate the maintenance window and prohibit concurrent deployments/config edits. If upgrading from the old updater, do not first merge its deploy commit: after fetching and reviewing it, execute the reviewed script directly from Git while staying in the deploy checkout, e.g. in Bash with `set -o pipefail`: `git show APPROVED_DEPLOY_COMMIT:update.py | python3 - --backend APPROVED_BACKEND_COMMIT --frontend APPROVED_FRONTEND_COMMIT --deploy APPROVED_DEPLOY_COMMIT --env-file /srv/otche/deployment/deploy.env --override /srv/otche/deployment/compose.production.yaml --project EXISTING_PROJECT` (append the two activation flags for the second invocation). This uses the current directory to locate sibling checkouts without advancing them.
Keep systemd's WorkingDirectory and explicit `--env-file`, `-p`, `-f` arguments aligned with these same clones/private files. After activation compare actual HEADs with approved public commits, verify authenticated health and prior accounts/uploads/jobs/artifacts/seals/held evidence, and perform real acceptance before reopening submissions. Logs and the exact host-specific rollout record remain private; do not put credentials or live inventory into commit messages or this public guide.
update.py
+19 -9
View File
@@ -2,6 +2,7 @@
"""Update reviewed sibling Git commits, build, and optionally activate a quiet deployment."""
import argparse
from contextlib import contextmanager
import hashlib
import json
import os
from pathlib import Path
@@ -50,10 +51,15 @@ def check_volumes(parser, compose, config, project, running):
if running:
for service, destination, logical in (('postgres', '/var/lib/postgresql/data', 'postgres-data'),
('api', '/var/lib/otche', 'artifacts')):
container = run(compose + ['ps', '-q', service])
if not container:
parser.error('Expected running ' + service + '; inspect existing deployment before updating')
mounts = json.loads(run(['docker', 'inspect', container]))[0]['Mounts']
containers = run(compose + ['ps', '-a', '-q', service]).splitlines()
if len(containers) != 1:
parser.error('Expected exactly one existing ' + service + ' container; inspect deployment before updating')
actual = json.loads(run(['docker', 'inspect', containers[0]]))[0]
if service == 'postgres' and not actual['State']['Running']:
parser.error('Existing database must be running')
if service == 'api' and actual['State']['Status'] not in ('running', 'exited'):
parser.error('API must be running or intentionally fenced (exited)')
mounts = actual['Mounts']
if not any(m.get('Name') == config['volumes'][logical]['name'] and m['Destination'] == destination for m in mounts):
parser.error('Persistent volume identity would change for ' + service)
@@ -140,7 +146,8 @@ def main():
'--profile', 'execution']
target_config = json.loads(run(staged_compose + ['config', '--format', 'json']))
check_volumes(parser, compose, target_config, args.project, True)
built = {service: {'image': 'otche-update-' + args.project + '-' + service + ':' + args.deploy}
build_key = hashlib.sha256((args.backend + args.frontend + args.deploy).encode()).hexdigest()
built = {service: {'image': 'otche-update-' + args.project + '-' + service + ':' + build_key}
for service, settings in target_config['services'].items() if 'build' in settings}
for service, settings in built.items():
if service not in ('api', 'migrate', 'worker', 'watchdog', 'web'):
@@ -160,8 +167,10 @@ def main():
old_images[service] = [json.loads(run(['docker', 'inspect', container]))[0]['Image']
for container in containers]
# Stop only API admission, after build; workers keep their existing containers.
old_api = run(compose + ['ps', '-q', 'api'])
run(['docker', 'stop', '--time', '30', old_api], capture=False)
old_api = run(compose + ['ps', '-a', '-q', 'api'])
api_was_running = json.loads(run(['docker', 'inspect', old_api]))[0]['State']['Running']
if api_was_running:
run(['docker', 'stop', '--time', '30', old_api], capture=False)
query = "SELECT (SELECT count(*) FROM jobs WHERE status IN ('queued','running')) + (SELECT count(*) FROM qualifications WHERE status IN ('queued','running'));"
try:
active = run(compose + ['exec', '-T', 'postgres', 'psql', '-U', 'otche', '-d', 'otche', '-At', '-c', query])
@@ -172,7 +181,8 @@ def main():
parser.error(str(directory) + ' changed during build; refusing activation')
run(['git', 'merge-base', '--is-ancestor', 'HEAD', revision], directory)
except BaseException:
run(['docker', 'start', old_api], capture=False)
if api_was_running:
run(['docker', 'start', old_api], capture=False)
raise
try:
# Only the successful fence permits advancing source and runtime image tags.
@@ -180,7 +190,7 @@ def main():
run(['git', 'merge', '--ff-only', revision], directory, False)
for service, image in images.items():
run(['docker', 'image', 'tag', image, image_name(target_config, service)], capture=False)
run(compose + ['up', '-d', '--no-build', '--wait', '--wait-timeout', '120'], capture=False)
run(compose + ['up', '-d', '--no-build', '--force-recreate', '--wait', '--wait-timeout', '120'] + sorted(images), capture=False)
# Recreate web explicitly even when its image is unchanged, then await health.
run(compose + ['up', '-d', '--no-deps', '--no-build', '--force-recreate', '--wait', '--wait-timeout', '120', 'web'], capture=False)
verify_images(parser, compose, images)