Skip to main content

Hotfix Flow

Hotfixes are urgent production fixes that bypass the normal develop -> dev -> qa -> uat -> staging -> prod promotion chain. They merge directly to main and deploy to production, then automatically backport to develop.

When to Use

Use the hotfix flow when:

  • A critical bug is in production and needs an immediate fix
  • The normal promotion chain would take too long
  • The fix is scoped and low-risk (small diff, targeted change)

For non-urgent fixes, use the standard flow through develop.

Hotfix Pipeline

Step-by-Step

1. Create and Merge the Hotfix MR

Create a branch from main, make the fix, and open an MR targeting main. The normal MR pipeline (lint, trivy, build, coverage, AI review) runs on this MR.

2. Deploy to Production (Manual)

The deploy-prod-hotfix job is manual with allow_failure: false. It requires protected environment approval from both the tech lead and PM before executing. All 3 production apps (portal, admin, api-gateway) are deployed every time.

The deploy uses the same .promote-app flow as the normal pipeline -- pull from GitLab Container Registry, retag for Heroku, release via Platform API, and health check.

3. Automatic Backport

After the production deploy succeeds, the hotfix-backport job automatically creates a merge request to bring the hotfix into develop:

  1. Creates a new branch: backport/develop-<SHORT_SHA>
  2. Fetches both develop and main
  3. Cherry-picks the hotfix commit onto a branch from develop
  4. Handles merge commits by using cherry-pick -m 1
  5. Pushes the branch and creates an MR via the GitLab API

Conflict Handling

If the cherry-pick has conflicts:

ScenarioResult
Clean cherry-pickMR created with labels: backport, auto-backport
Cherry-pick conflictsMR created with labels: backport, conflicts + warning in description

When conflicts occur, the backport MR is created with an empty commit and a clear warning that manual resolution is needed. A developer must resolve the conflicts and update the MR before merging.

Differences from Standard Deploy

AspectStandard (develop)Hotfix (main)
Source branchdevelopmain
Environmentsdev -> qa -> uat -> staging -> prodprod only
Auto-deployDev is automaticNone (manual only)
Apps deployed3 core apps through the promotion lane3 production apps only
Post-deploysync-to-mainhotfix-backport MR to develop
E2E testsAfter dev and QA deploysNone
MigrationsAuto at dev and prod when eligibleAuto at prod when eligible

Required Approvals

The deploy-prod-hotfix job requires the same production environment approval as a normal production deploy:

  • Tech lead approval
  • PM approval

Both must approve in GitLab before the job executes.

Post-Hotfix Checklist

After a hotfix is deployed and backported:

  1. Verify the fix in production
  2. Review and merge the backport MR to develop
  3. If the backport had conflicts, resolve them before merging
  4. If database migrations were part of the hotfix, verify run-migrations-prod completed or trigger it from the pipeline if needed