Upgrades¶
Status
Preflight, Fence, Migrate, Verify, and the roll forward guard are implemented in the Foreman controller. The upgrade E2E test installs the released 0.1.0 chart and upgrades it to the current images. The Release CRD for compatible image sets is target design.
Until Release exists, an upgrade is a new image in Foreman.spec.image, or a new operator version whose default image tag changed. A change of spec.plugins runs the same flow.
stateDiagram-v2
[*] --> Preflight
Preflight --> Blocked: unsupported path or plugin removal
Preflight --> Preflight: tasks active
Preflight --> Fence: checks passed
Fence --> Migrate: writers stopped
Migrate --> Rollout: Job completed
Migrate --> RollForwardRequired: Job failed
Rollout --> Verify: web and orchestrator available
Verify --> Complete: smoke checks pass
Complete --> [*]
Preflight¶
The operator computes a hash over the image reference and the sorted plugin list. When it differs from status.appliedMigrationHash, a preflight runs while the previous version keeps serving:
- The Job
<name>-preflight-<hash>starts the new image and reads/usr/share/foreman/VERSION. - On a fresh install, the checks below are skipped.
- Disabling
foreman-tasks,foreman_remote_execution, orforeman_ansibleis refused with phaseBlocked. These plugins schedule Dynflow tasks that reference their code, andforeman-tasksalways schedules recurring tasks, so the Dynflow orchestrator would crash on the next start. - The version path from
status.foremanVersionto the new version must be one release at a time: a rebuild or patch release, the next minor release, or the first minor release of a later major version, for example3.19to5.0. Skipped releases and downgrades are refused with phaseBlocked. For installations from an operator version that did not recordstatus.foremanVersion, the running Foreman is asked through/api/v2/status. - With
foreman-tasksenabled, the operator waits until no task isplanning,running, orpaused. Resume or cancel paused tasks. Scheduled tasks, such as recurring cleanups, survive the upgrade.
A blocked preflight is checked again every minute. A task can still start between the check and the stop of the writers.
Fence and migrate¶
- The recurring task CronJobs are suspended, and all web and Dynflow Deployments are scaled to zero. The frontend runs in the web pods, so Foreman is unreachable until the rollout.
- The controller waits until no Foreman pods remain, including running recurring tasks. Finished pods and the preflight and migration pods do not count.
- Older migration Jobs are deleted, and the Job
<name>-migrate-<hash>runsdb:migrate,db:seed, andupgrade:run. Beforedb:seed, it resets the stored seed hash: Foreman skips the seeds while its seed files and templates are unchanged, but templates shipped with Foreman that require a plugin, such as those offoreman_bootdisk, are imported only once the plugin is enabled.upgrade:runexecutes one-time data tasks of a release and refuses to run while migrations or seeds are pending. - After the Job completes, the hash, the version, and the plugin list are recorded and the Deployments roll out with the new image.
Roll forward¶
There is no automatic rollback: once the migration Job exists, the database may be partly migrated for the new version.
- A failed Job sets the phase
RollForwardRequired. Read the Job logs. Delete the Job to retry with the same image, or set a fixed image to roll forward. The preflight of the fixed image checks only the version path, since Foreman is stopped. - Setting the previous image again after the migration Job started is refused with phase
RollForwardRequired, and Foreman stays stopped. Going back is only safe after restoring the database from a backup taken before the upgrade. - Going back before the migration Job started, for example while the writers stop, is allowed.
Verify¶
After the rollout, the Ready condition becomes True, so the Smart Proxies refresh their features. The condition Verified stays False until:
/api/v2/statusanswers with the version instatus.foremanVersion, and- every
SmartProxyof this Foreman is ready and refreshed its features since the migration (status.foremanMigrationHash).
Then status.verifiedMigrationHash is set and Verified becomes True with reason SmokeChecksPassed.
Overriding the checks¶
The annotation platform.theforeman.org/skip-upgrade-checks: "true" skips the preflight checks and the roll forward guard. Use it only to recover, for example when the running Foreman is broken and cannot answer the preflight, or after restoring the database. Remove it afterwards.
Smart Proxies are upgraded after Foreman. DHCP and TFTP keep serving during a Foreman upgrade, but Foreman cannot change reservations while it is fenced.
Health¶
Web startup, readiness, and liveness probes use the HTTP status of /api/v2/ping only, with the Host header set to spec.fqdn. The response body includes dependency status, which must not remove all web pods when a dependency fails.
The Smart Proxy probes complete a TLS handshake with curl against /version and accept any HTTP status. A plain TCP check would make WEBrick log an SSL error on every probe.
Backup and restore¶
Take a backup of the database and <name>-secrets before an upgrade: after the migration Job started, a backup is the only way back. See Backup and Restore.