release-checklist

name	release-checklist
description	Drive the MEGA Android Secure Release Checklist as Release Captain (RC). Orchestrates the full release train: create the RC tracking ticket, prepare release notes, request SDK/MEGAChat builds, cut the code-freeze/Alpha build, run QA rounds, promote through Beta → 25/50/100% Production, and run hotfixes. Encodes the flow, gradle/MR commands, and date rules so each step is executable, not just described. Internal identifiers (Slack channel / user-group IDs, Jira epic + fields, Confluence/TestRail ids) are referenced by NAME and resolved from a gitignored `local-constants.md` — never hardcoded here. Always defer to the latest Confluence page for the version.
triggers	["/release-checklist","release captain","release checklist","start release","code freeze","promote release"]

Android Secure Release Checklist (Release Captain)

Orchestration guide for running an Android release end-to-end. The source of truth is the Confluence page for the version: Android Secure Release Checklist - v<X.Y> (space MOB). This skill mirrors that flow and pre-fills the constants RC work needs, but the checklist wins on any conflict — always re-read the latest Confluence page; do not rely on memory.

🆕 At the start of every release (NOT a hotfix): ensure the per-version page exists first. If Android Secure Release Checklist - v<X.Y> doesn't exist yet, clone it from the master template Android Secure Release Checklist (https://confluence.developers.mega.co.nz/display/MOB/Android+Secure+Release+Checklist) and rename the copy with the exact version in the title — use confluence_copy_page (copy the template page → set the new title to Android Secure Release Checklist - v<X.Y>). Then work from that copy (e.g. update the Prerequisite release-ticket link on it). Hotfixes reuse the base version's page — don't create a new checklist page for a hotfix.

⚠️ The checklist order matters. Do not jump between steps. Any ticket add/remove after code freeze needs a mobile leader's approval.

How to use

1. Determine the current state (which section are we in) before doing anything.
2. Work only the steps due for today unless told otherwise. The schedule is day-anchored (see Schedule below).
3. Do the read/analysis yourself; draft every outward message (Slack, Jira) and confirm before sending — these are outward-facing and broad-mention.
4. Track multi-step work with the task tools.

🔐 Local constants (resolve IDs by name — never hardcode here)

This repo mirrors publicly, so MEGA-internal identifiers are not stored in this file. They live in local-constants.md (same folder, gitignored). Everywhere below refers to things by name — #devops-cicd, @eu-mobile-release, "RC ticket Epic", "Release Plan page id", etc. At runtime, read local-constants.md and resolve the name to its real value.

• First use / file missing → copy local-constants.example.md → local-constants.md and fill it in (each value's source is noted in the template).
• Slack pings: message templates below use the readable @group form. When sending via the Slack API, expand each @group to its <!subteam^ID> (ID from local-constants.md) — plain text @group does not notify. Post to channels by the C… id from local-constants.md.
• ⚠️ Channel-search is unreliable — slack_search_channels "mobile-dev-team" returned no results even though the channel exists. Prefer the id in local-constants.md; don't conclude a channel is gone just because search misses it.

Constants (non-sensitive; re-confirm if stale)

Thing	Value
RC ticket title	Release management - Android <X.Y> (type Story)
RC ticket required fields	Expense Product = Cloud/Default; Expense Type = OPEX; Epic Link = RC ticket Epic (field ids + Epic in local-constants.md)
Cadence	Every 2 weeks; code freeze = Wednesday
Jira projects in a release	AND, AP, BAC, CC, CU, MEET, TRAN, SHR, FM, SAT, SAO

Slack channels / user groups — role reference (IDs in local-constants.md)

Handle	Used for
#mobile-platform	release-notes request (1.1); SDK Release Train is announced here
#mobile-dev-team	code-freeze 30-min warning + "freeze started" (1.4); crash/ANR monitoring reminders
#android-dev-team	feature-flag request (1.3); next-RC reminder; QA Failed/Feedback reports (1.5)
#sdk	post target version in Release Train thread (1.3); SDK RC announcements
#sdk-android-pipeline	prebuilt-SDK build results (build sdk-aar) post here — watch this to get the …-rel version without a GitLab token; each post names its source MR + SDK/chat branch
#megachat_native	MEGAChat version announcements (1.3)
#devops-cicd	ask DevOps to promote Beta/Production (2.x)
#android, #release, #app_release_updates	build reports / release thread / release announcements
@productmanagers (user group)	release-notes highlights
@androiddevs-urgent (user group)	dev-facing freeze / feature-flag pings
@eu-mobile-release (user group)	DevOps promotion approvers (Beta/Prod) — always ping
@nz-mobile-release (user group)	DevOps promotion approvers — ping only 7am–7pm NZ time

Date rules

• "Previous Friday" = the Friday of the week before code-freeze week (code freeze is the following Wednesday). This is release-notes prep day (1.1).
• Next code freeze = current code freeze + 14 days.
• Next-RC reminder fires the Friday before the next code-freeze week (i.e. next-freeze-Wednesday − 5 days).
• Get the version's code-freeze date from Jira: it's the version startDate for Android <X.Y> (jira_get_project_versions on AND).

Version type

• X.Y where Y != 0 → minor. Skip the major-only steps (Jira version bumps across all projects + the major-version #android announcement).
• X.0 → major. Do the major-only steps and prepare real release notes.
• If unsure, check the latest Release Train announcement in #mobile-platform.

---

Prerequisite — create the RC tracking ticket

Create once at the start of being RC. Log all RC time against it.

• Search the RC ticket Epic (see local-constants.md) children for an existing Release management - Android <X.Y>. (Server uses Epic Link, not parent: JQL "Epic Link" = <RC-Epic>.)
• If none, create it (mirror the most recent one):
  • project AND, type Story, assignee = you,
  • additional_fields: set epic_link = RC ticket Epic, the Expense Product field = {"value":"Cloud/Default"}, and the Expense Type field = {"value":"OPEX"} (Epic + both customfield_* ids in local-constants.md).
  • then transition with the Start Progress id (in local-constants.md) → In Progress.
• Confluence: the per-version page is a copy of the template; update the placeholder release-ticket link in the Prerequisite step to the new ticket. Note: the Confluence integration only does whole-page replace (confluence_update_page) — there is no line/section edit, and these pages are ~130k chars, so this single link swap is best done manually in the UI (or recorded via confluence_add_comment).

---

Schedule (Code Freeze & Alpha)

1.1 Release-notes prep — previous Friday

• (major only) Bump the Jira version number across all 11 projects; announce the major release plan in #android.
• Draft + post the release-notes request in #mobile-platform, tagging both user groups, deadline Monday:

  Hi team (@productmanagers & @androiddevs-urgent), we are preparing release notes for the upcoming Android release v<X.Y> on . If you have highlight features to announce to our end users in your responsible area, please reply in this thread at the latest on Monday. Thank you.

• Review tickets in fixVersion = "Android <X.Y>" for highlight features; confirm with devs whether to mention them. No highlights → default notes (- Bug fixes and performance improvements).
• Set the next-RC reminder. This MUST be typed by the RC in their own Slack client — there is no automation for it:

  /remind #android-dev-team that @<next-RC> please start requesting release notes for next release at 9am on <Friday before next code-freeze week>.

  • Fill in the actual next RC (e.g. @kg) and the computed date, not a placeholder.
  • ⛔ Do NOT send the /remind text via the Slack API and do NOT substitute slack_schedule_message. A slash command sent through the API posts as literal text in the channel and creates no reminder; only typing /remind in the Slack message box registers it. Just hand the RC the ready-to-paste command above.

1.2 Prepare code freeze — Monday

• Add the new version row to Android Release plan - <year> (Confluence, Release Plan page id in local-constants.md): fill Version + Package link only. Leave Code Freeze Date (filled Wed, step 2321), Release Date, and — unless confirmed — Release Captain blank. Newest row goes at the top of the data table (above the previous version).
  • Package link format: …/issues/?jql=fixVersion %3D "Android <X.Y>" ORDER BY priority DESC, updated DESC (URL-encoded).
  • This page is small (~12 KB) so it CAN be updated via confluence_update_page with content_format='storage': fetch raw storage, insert one <tr>…</tr> above the latest row, push it back, then re-fetch and confirm no historical row changed. Copy RC ri:user keys verbatim from the existing rows / the schedule table — never hand-type them. (Contrast: the checklist page is ~130 KB → manual.)

1.3 Request SDK & MEGAChat — Tuesday

• In #sdk, reply in the Release Train thread with the target Android version + Jira package link.

  Hi team, <jira-package-link|Android <X.Y>> will be our next release. Thanks!

  • ⚠️ The Release Train thread rotates every cycle. A fresh Slackbot reminder ("A new Release Train is coming :steam_locomotive:… post the target version in the thread") is posted (~Mon/Tue) as a new thread parent. Reply to the newest reminder, NOT the long-running prior thread — the previous cycle's thread keeps getting SDK-bot build/hotfix posts (e.g. last cycle's v10.15.0a targeting only the previous app versions), so it looks active but is stale.
  • Find it: search #sdk for the Slackbot reminder, sort by timestamp, take the latest; confirm its parent has no/our-cycle replies before posting.
• Watch #sdk and #megachat_native for the new SDK / MEGAChat releases targeted at our version. A published SDK that lists only other apps under "Target apps" (e.g. iOS only) is NOT ours — the Android-targeted build follows after we post. You need both the SDK and MEGAChat tags for preRelease --sdk … --chat … (1.4).
• Verify all tickets are QA/Resolved, and none are missing fixVersion (JQL: project in (AND,CC,TRAN,MEET,AP,BAC,CU,SHR,FM,SAO,SAT) AND status in (Resolved,QA) AND fixVersion = EMPTY AND assignee in (membersOf(android))).
  • Also confirm none of fixVersion = "Android <X.Y>" are still un-QA'd: … AND status not in (Resolved, QA, Closed) should return 0.
• Collect remote feature flags to enable — post in #android-dev-team:

  Hi @androiddevs-urgent, if your feature is ready and you are going to enable your remote feature flag in this release v<X.Y>, please reply to this message. Thank you.

1.4 Create the code-freeze build — Wednesday

Get the SDK + MEGAChat tags first. From the #sdk and #megachat_native bot posts ("New SDK version → vX", "New MEGAchat version → vY"), take the rc tags whose "Target apps" lists Android <X.Y> (ignore posts targeting only iOS/MEGAsync). e.g. SDK v10.16.0-rc.1, MEGAChat v9.3.7-rc.1.

Pre-flight — get a clean develop with the rc tags reachable:

1. git checkout develop && git pull --ff-only.
2. Apply step 2669 (GitLab redirect) — mandatory, not optional. The submodules' origin is GitHub, but brand-new develop commits + the rc tags live only on GitLab (GitHub mirror lags), so a plain submodule update fails with fatal: ... not our ref <sha>. Run the git config --file=.gitmodules … block (both submodules → GitLab [e-mail]:…, branch develop), then git submodule sync && git submodule update --init --recursive --remote.
3. Verify the rc tags resolve: git -C sdk/src/main/jni/mega/sdk tag -l <SDK_TAG> and the megachat one; both submodule origin should now be GitLab.

⚠️ Beware git add -A + branch switches eating untracked files. preRelease runs git add -A; later git checkout -B develop … can drop untracked files (e.g. this skill folder) — they may end up in a stash/commit. If something vanishes, recover with git log --all -- <path> / git show <sha>:<path>, don't rewrite from memory.

• Upload approved release notes to WebLate (skip if default notes). Use the weblate skill. Get content team approval in #android.

• preRelease: ./gradlew preRelease --rv "<X.Y>" --sdk "<SDK_TAG>" --chat "<CHAT_TAG>" (~3 min). Creates branch task/pre-release/v<X.Y> + MR, already assigned to you, squash ON (correct for the pre-release MR), 3 commits: Update SDKs / Update App version / Update strings. The bot's "Code Review Failed — Diff too large" note is expected (translations bloat the diff) and non-blocking.

• Pre-built SDK (MR comment): build sdk-aar --lib-type=rel --sdk-branch=<SDK_TAG> --chat-branch=<CHAT_TAG>. The result does NOT reply on the MR — it posts (~15–60 min; ~13 min observed) to #sdk-android-pipeline as :rocket: Prebuilt SDK is published to Artifactory Successfully! with SDK Branch, Chat SDK Branch, Version: <YYYYMMDD.HHMMSS-rel>, an AAR download link, and Triggered from: <this MR> (match the MR to be sure it's your build). Set extra["megaSdkVersion"] in root build.gradle.kts to that -rel value (replacing the prior -dev/-rel), commit only build.gradle.kts as "Update prebuilt SDK version", push.

• Next Jira version (MR comment): create_jira_version -rv "<NEXT_X.Y>" -rd "<NEXT_FREEZE_DATE>" — the version 2 weeks out (e.g. 16.9 → 16.10, freeze +14d). Bot replies "Create Jira Version succeeded"; verify in AND (jira_get_project_versions).

• Add reviewers; needs 2 approvals, then merge the pre-release MR → develop (squash).

• Re-sync local develop after the squash-merge (gotcha): preRelease leaves a local-only "Update SDKs" commit on develop, but the squash-merge created ONE new commit on origin/develop → they diverge and git pull --ff-only aborts. The destructive-git guard blocks git reset --hard, so realign with (tree must be clean) then . — otherwise cuts from the old version.

1.5 QA tests

• Monitor the Android TestRail project (id in local-constants.md) run Android ALPHA release v<X.Y>(NNNNNNNNN) daily.
• Report Failed/Feedback TCs every morning to #android-dev-team — use the summary_release_testrail skill (e.g. /summary_release_testrail v<X.Y> to #android-dev-team).
• Monitor Play Console for Google approval; watch Crashlytics for new crashes/ANRs.

1.6 New Release Candidate builds

Each round, devs cherry-pick TC fixes to release/v<X.Y> (RC verifies). To roll a new RC:

• If there's a new SDK/MEGAChat RC (check #sdk / #megachat_native thread for the new tag targeting Android <X.Y> — often only one of them bumps; pass BOTH tags anyway):
  1. Re-apply the GitLab redirect (step 2669) — checkout -B/branch switches reset .gitmodules back to GitHub, so the new rc tag won't fetch until you redo it + submodule sync && submodule update --remote. Verify git -C …/mega/sdk tag -l <NEW_TAG> resolves.
  2. On develop: ./gradlew preRelease --rv "<X.Y>" --sdk "<NEW_SDK_TAG>" --chat "<CHAT_TAG>" → new pre-release MR (strings + SDK submodule bump). Note: develop's appVersion stays <X.Y> through QA (only the Jira fixVersion moved to NEXT), so --rv "<X.Y>" is right.
  3. build sdk-aar … in the PRE-RELEASE MR (NOT the release MR) — the result posts to #sdk-android-pipeline with Version: <…-rel>. Read it there (no GitLab token needed).
  4. Set megaSdkVersion to that -rel value in the pre-release MR (commit only build.gradle.kts), push.
  5. Get the pre-release MR 2 approvals + green pipeline → merge to develop (squash), then cherry-pick the squashed merge commit onto release/v<X.Y> (next bullet).
• Monitoring the merge without a GitLab token (the merge is reviewer-gated; you wait on it): background-poll the remote over SSH (no token needed) and auto-continue when it fires. Two definitive signals — the source branch is deleted on merge (force_remove_source_branch), and the squash lands on develop:

  for i in $(seq 1 120); do
    git fetch origin develop -q 2>&1 | grep -viE 'submodule|not our ref' || true
    [ -z "$(git ls-remote --heads origin task/pre-release/v<X.Y>)" ] && { echo "MERGED (branch deleted)"; exit 0; }
    case "$(git log --oneline -1 origin/develop)" in *"Pre-release - v<X.Y>"*) echo "MERGED"; exit 0;; esac
    sleep 60
  done
  

  Run it with run_in_background: true; you're notified on exit, then continue with the cherry-pick. (Avoid /dev/null redirects — the sandbox blocks them.)
• Strings-only round (no SDK bump): ./gradlew preRelease --rv "<X.Y>" → merge to develop → cherry-pick strings commit to release.

---

2. Release (Beta → Production)

Trigger: QA signs off in the #android release thread.

🛑 VERIFY THE EXACT BUILD VERSION CODE IMMEDIATELY BEFORE EVERY PROMOTION REQUEST (Beta, and each of Production 25/50/100). Do NOT reuse the version code of the Alpha you personally built — the release/v<X.Y> branch keeps moving (dev hotfixes, new SDK RCs, extra deliver_appStore runs), so the current build is often newer than your last Alpha. Real mistake (v16.9): posted a #devops-cicd Production-25% request for 16.9(261880606) (my round-4 Alpha) when the actual latest build was 16.9(261881039) (3c7e30562a, a hotfix + SDK v10.16.0-rc.3 had landed after). Slack has no edit/delete → a correction reply is the only fix and it's messy/embarrassing. How to get the right code every time: take the newest entry from any of:

• #android — the deliver/build reports post here; grab the most recent one for the current release (e.g. 16.9(...)). ⚠️ Filter by version — #android also carries builds/hotfixes for other live versions (e.g. 16.8.x); never grab a code that isn't 16.9(...).
• release/v<X.Y>/ on Artifactory — the highest <code>_<sha>/ folder; or
• the release MR's latest :rocket: … uploaded to Google Play / deliver report. Cross-check: confirm the report's <sha> matches git rev-parse origin/release/v<X.Y> before posting. Promote that <X.Y>(<code>).

2.1 Beta

• Feature flags → PUBLISH: post the reminder in #android-dev-team, tagging @androiddevs-urgent (even if the 1.3 request got no replies — send it anyway):

  Hi @androiddevs-urgent, v<X.Y> has been signed off by QA and is moving to Beta. If you have a remote feature flag to enable in this release, please set its status to PUBLISH in https://featureflags.tools.mega.co.nz/ and set Min Android (rules.av) to <X.Y>. Also please update your flag status in the Android Remote Feature Flags Tracking doc. Thank you.

• Ask #devops-cicd to promote to Beta, 100% — exact template (ping @eu-mobile-release always; add @nz-mobile-release only 7am–7pm NZ time — you can't read the wall clock, so confirm the NZ hour with the user; app build timestamps are +12 NZ):

  Hello @eu-mobile-release team, the MEGA Android app <X.Y>() is now ready for promotion to BETA channel with a 100% rollout. Job: https://controller.cibuild.mega.co.nz/job/Android-Promote • Application: MEGA • Version: <X.Y>() • Target channel: Beta • Percentage: 100% Thank you (Add @nz-mobile-release after @eu-mobile-release when within NZ hours.)

• DevOps runs the Android-Promote job and replies "done" in-thread; it auto-notifies the #android release thread.

2.2 Production (phased: 25% → 50% → 100%)

• Ensure the previous version is at 100% first.
• #devops-cicd promote to Production 25% (same template as 2.1, Target channel: Production, Percentage: 25%); wait 1–2 days at each stage.
• Upload Crashlytics symbols: upload_symbol on the release MR.
• postRelease -rv <X.Y> on the release MR → sets cross-project releases to Released, closes tickets, publishes GitLab/GitHub tags, merges release→master, pushes to GitHub.
• Publish GitLab + GitHub releases (auto via postRelease; manual steps in Confluence). Publish Core-UI to GitHub.
• Upload release-notes translations: ./gradlew readReleaseNotes --app-version "<X.Y>" --xml release.xml → paste into Play Console (skip if default notes / already uploaded).
• Promote to 50%, then 100% (#devops-cicd templates).
• Notify #sdk (reply in thread), #release, #app_release_updates that the build is signed off. Update the Release plan doc.
• Set Crashlytics reminders (/remind #mobile-dev-team … in 4 hours / … in 12 hours).

2.3 After release

• Watch Play Console ANR/crash, Crashlytics, Play reviews, and App Content section (notify Android TLs in #android).
• Close the RC ticket — do NOT add a fix version to it.

---

3. Hotfix

For TL-approved crashes/bugs only; minimal code; QA usually skipped so test carefully. Worked example below = 16.9.1 for AND-24225 (IllegalStateException — LiveData.setValue off the main thread in AudioPlayerServiceViewModel.onPlayerError; fixed by wrapping in withContext(mainDispatcher)).

1. Analyse — read the crash ticket (jira_get_issue); it links a Crashlytics issue id, so you can pull the stack trace from Firebase too. Get root cause + impact (users/events).
2. Is the fix already on develop? Often a dev has already merged it. Check: git merge-base --is-ancestor <fix-sha> v<X.Y> → not on the tag means it needs cherry-picking (and means no new code to write — just cherry-pick). If not fixed yet, follow the fix flow: develop+release → fix on develop → merge → cherry-pick to release; release-only → fix on release.