Skip to main content

Release process

8 weeks prior​

#MajorMinorTaskResponsibility
1.✓Recurring release epics review:
  • Review how previous deprecation epics are going and share about them.
  • Review how previous library review epics are going and share about them.
Integration Team
2.✓Ensure we are running latest/good behat, phpunit and nodejs (npm) versions.Integration Team
3.✓Confirm external testers daily availability from -7w to release (internally with dev manager and then, externally, with contractors). They start in just 1 week!Integration Team
4.✓
  • Move QA tests for new features in old release with automated tests from MDLQA-1 to MDLQA-5249.
  • Remind devs:
    • To label issues qa_test_required when a new feature/improvement is landing without automated tests. Also add a comment advising exactly what should be covered in the QA test (for example, steps 6-10 in testing instructions).
    • The best possible lang strings land at first attempt. Given the proximity of the major release and to facilitate the work of translators and others... the sooner, the better, instead of relying on (late) en_fixes too much.
Testing Maintainer
5.✓Integration Team

7 weeks prior​

#MajorMinorTaskResponsibility
1.✓The Continuous Integration period begins. Warn about it everywhere (telegram, exposed posts...).
Rolling (on demand, beta, rc... all them together with stable weeklies) happens often (Tuesday & Friday are the usual days). All the team is on integration 100% since this week and until the end of continuous.		Integration Team
2.✓Full demo of new code and sign-off for internal HQ projects. Decide which projects will be completed by the code freeze.Platform manager / Head of educational solutions
3.✓Testing Maintainer
4.✓Warn external developers about the impending code freeze in a post to the General developer forum. (example)Platform Manager

6 weeks prior​

#MajorMinorTaskResponsibility
1.✓Confirm the code freeze by replying to the previous warning in the General developer forum.Platform manager
2.✓This week, once the freeze has happened at 11:00 UTC:
  • The Manage queues on normal job has to be disabled.
  • Immediately after that, the Move awaiting issues to current integration job must be enabled and executed, only once, and all the candidate issues already @ integration will come to the current queue. All them did the freeze.
  • Once run, disable the job again to avoid executing it by mistake.
  • Once the previous job has ended, the Manage queues on continuous job, in charge of managing the integration queues after freeze, will be enabled (until the end of on-sync). It performs some tasks like moving important (must fix, qa, security...) issues to the current integration queue and also keep if fed by priority when there are few issues remaining. Don't forget to configure the releasedate date to get it automatically holding issues the last week (instead of moving them to current integration) and also processing issues over the on-sync (after release) period.
Irrespectively from any automation, issues can, also, be manually picked for integration until the post-release On Sync period ends, usually 2 weeks after release.

Review all the new features and improvements so, everything "unrelated" with the release, is given the integration_held label. That means that will be ignored unless there is an unhold request process started on them.

Dev leaders promptly vote on held issues labelled with unhold_requested label and decide whether they can be allowed to break the freeze.		Integration Team
3.✓Prepare a fresh installation of the QA site based on the latest weekly/on-demand release code to make sure the site is not affected by incremental upgrade steps during the development cycle.QA Master Site Maintainer
4.✓Create the next "X.Y+1" (next dev) version in the Tracker (MDL and CONTRIB), so people can move delayed stuff to next major release if needed.Integration Team

5 weeks prior​

#MajorMinorTaskResponsibility
1.✓Create new QA test cycle and post in moodle.org front page news forum about QA testing.Testing Maintainer
2.✓Invite community volunteers to start QA testing.Community Manager
3.✓Modify Current QA cycle filter to link to the new QA test cycleTesting Maintainer
4.✓Monitor QA fails. Check each fail is real and if so ensure an MDL issue has been created and correctly linked and labelled.Testing Maintainer
5.✓Monitor MDL issues created for QA fails. Add them to the "Must fix for X.Y" list and get a developer to work on the issue immediately.Product delivery lead
6.✓Begin reviewing new and changed English language strings ready for en_fix to be merged 2 weeks prior.English fixes lang pack maintainer
7.✓Review standards certifications (Open Badges, LTI, etc) and schedule in recertification to be performed.Product delivery lead
8.✓Bump the priority of all the minor new features and improvements being must-fix or already under IR/CLR with X.Y as only affected version. Use this search to find them (note that it will need to be adjusted for current X.Y version). Standard message will be used to explain the priority raise.Integration Team

4 weeks prior​

#MajorMinorTaskResponsibility
1.✓Start reviewing fixed issues and add the release_notes label to all issues to be listed in the release notes.Product manager / Head of educational solutions
2.✓Organise a meeting to review requirements for adding user tours for new features in the upcoming release (clone of MDL-72783). Review existing tours and remove very old ones (clone of MDL-72781)Product delivery lead
3.✓Beta release (The Moodle release tool - mdlrelease - manages it): Release normal weeklies but with these changes in the master branch being considered all the time, until we are feature complete (can happen @ -4w - ideally - but also later):
  • version.php: Move to $maturity = MATURITY_BETA and $release = 'X.Ybeta (Build:xxxxxxxx)'
  • tag git repo with: vX.Y.0-beta (and MOODLE_XY_BETA as description and Git tag)
  • Automatic: Confirm packages are available under download.moodle.org (and windows).
  • Once the beta is available create the testing matrix for X.Y.0 and share it @ HQ & Dev. chat.
  • (Optionally) Announce Beta release in forums (ideally once packages are available).
Integration Team
4.✓Clone MDL-77758 for the X.Y release, adding there all the debugging / PHP notices happening in the web server logs while running tests.Integration Team
5.✓Release candidates release (The Moodle release tool - mdlrelease - manages it): At some points (between beta to final release) produce release candidates (Z = 1, 2, 3..), which are normal builds with the following changes:
  • version.php: Move to $maturity = MATURITY_RC and $release = 'X.YrcZ (Build:xxxxxxxx)'
  • tag git repo with: vX.Y.0-rcZ (and MOODLE_XY_RCZ as description and Git tag)
  • Automatic: Confirm packages are available under download.moodle.org (and windows).
  • Upgrade all the Moodle CI sites to recent release candidate by configuring the "Moodle CI Auto Upgrade" job in all them.
Integration Team
6.✓Add a new version in the Plugins Directory with the version's name and the beta release version build number (https://moodle.org/plugins/admin/softwareversions.php).Plugins Liaison
7.✓Push any plugins to the plugins database which were previously a part of core.Plugins Liaison
8.✓Tidy up current latest en version of Moodle Docs prior to copying it to create new version wiki as described in New docs version process.Community Manager
9.✓Review the list of features and improvements submitted before code freeze which are awaiting integration review and decide on related QA tests to be put on hold. This should be done in the same meeting in point 2.Product delivery lead

3 weeks prior​

#MajorMinorTaskResponsibility
1.✓Ask developers to begin QA tests marked external_skipped then test_server_required tests and credentials_required tests.Product delivery lead
2.✓Create new en and de Moodle Docs version wikis.Moodle Docs Maintainer
3.✓Go through all points listed under 3 weeks prior in New docs version process.Community Manager
4.✓Review issues with labelled with dev_docs_required, prompting assigned developers to create/update this documentation and remove this label when relevant docs are updated.Product delivery lead
5.✓✓Organise a meeting to verify how everything is going and, before the end of the week, decide (dev leaders & integrators) if there are real reasons for delaying any release. Whenever a delay is agreed, run the "Release delaying process" (doc) actions ASAP.

Important note: ASAP stands for as soon as possible ;-), the delay needs to happen before the last week before release begins (there are flows changing that week requiring the decision and actions to be taken and applied earlier)		Product delivery lead

2 weeks prior​

#MajorMinorTaskResponsibility
1.✓Check the number of open QA tests. Depending on the amount of work involved, encourage HQ developers to assist from the start of the week or later in the week in order to achieve the target of 100% pass rate by the end of the week.Product delivery lead
2.✓Create MDLSITE issue to ensure all prototype.moodle.net sites which are no longer relevant (part of release) are removed from prototype.moodle.netPlatform manager / Head of educational solutions
3.✓✓Merge fixes from en_fix pack and then integrate them.AMOS Maintainer
4.✓Give Partners complete branded Marketing pack for release including list and description of major new features.Marketing Manager
5.✓Label (ui_change) in the Tracker all pending issues having noticeable UI modifications and track them in the "Late landing UI changes" sheet, in order to have all the associated docs, screenshots, videos under control. The Integration and Documentation people will try to keep the sheet updated until release.Integration Team
6.✓✓For major releases, use (clone if needed) MDL-75906 to keep the security.txt files in all supported branches updated. For minor releases, also check the security.txt expiry dates in all security supported branches in case known release delays will result in any security.txt expiring before the next release - in that case relevant branches should also be updated.Security Officer
7.✓✓Identify issues that may block the release (for example, security, must-fix issues, etc.) and confirm with team leads. This is to ensure that these issues are prioritised and allow us to be able to land them as early as possible and ensure a timely release schedule.Integration Lead

1 week prior​

#MajorMinorTaskResponsibility
1.✓Clone as many filters as needed in the Tracker, modifying them to point to the new, upcoming, branch (keeping same perms, title...).Integration Team
2.✓✓Create new minor version X.Y.Z+1 in the Tracker (MDL and CONTRIB). Archive any version > 6 months old, verifying that there aren't open issues using that version.Integration Team
3.✓Clone MDL-77940 and bump all versions, requires and dependencies along all plugins in codebase to planned release dates.Integration Team
4.✓✓Post a "Heads-up" message on the General Developer forum and MUA website.Platform Manager
5.✓✓Post a "Heads-up" message on the Partners forum.Security Officer
6.✓✓Post a "Heads-up" message on Twitter and other outlets.Marketing Officer
7.✓✓Identify security issues that need to be integrated using the security issues under integration filter (includes held, being integrated and ready issues).
  • Integrate from provided patches into supported branches (including branches supported only for security issues).
  • Ensure security issues are given priority in weekly integration and testing. Note that the --weekly releases to be performed this week are 100% normal and cover the standard (supported) branches. security branches will come as part of the release process later in the week.
Integration Team
8.✓✓Collect security issues into a clone of the last advisory in the list to prepare for release of Security Advisories.
  • Determine which security issues will be integrated.
  • Request CVE Identifiers by emailing issue descriptions to secalert@redhat.com with a message like: The following security issues have been discovered in Moodle. We request CVE Identifiers for these issues. We will be releasing the security announcements for these on moodle.org on XXX at 12noon AWST which is 04:00 UTCThe email subject should include the characters [vs]. The format needs to be plain text or encrypted. When granted CVE Identifiers, our issues should appear in the CVE list. More instructions are available (since 2015 we request identifiers from redhat and not from distros).
  • Post security issues as a reply to the "Heads-up" message on the Partners forum.
Security Officer
9.✓✓Release notes tasks:
  • Review and complete the release notes, making sure that issues have easily understandable summaries. Aim to have the release notes mostly complete on the Wednesday before release, and fully complete on the Friday, to give translators time to translate them.
  • Ensure all issues labelled with ui_change, api_change and release_notes are listed as UI changes, functional changed and fixes/improvements respectively in the release notes.
Platform Manager / Head of educational solutions
10.✓Last week managing / holding issues:
  • The "candidates queue" is handled automatically by the continuous queue manager (given it has correctly configured the release day). And it will automatically hold any issue arriving and not being "important".
  • The "current queue" has to be handled manually by the iTeam, so any issue there, awaiting, needs to be held, with the standard last week message and moved out from integration given:
    • It's not "important".
    • It has not been unhold previously.
  • If there is any issue already in progress, or there are doubts about any issue, always comment with the team. Depending on the remaining work and/or other criteria they may be acceptable and processed.
  • Ensure the next point effectively happens and the 2 lists are reviewed.
Integration Team
11.✓Organise a meeting to review the following issues: Product delivery lead
12.✓Upgrade moodle.org to beta release
13.✓Prepare pull requests for CI Repositories:Integration Team
14.✓Go through all points listed under 1 week prior in New docs version process.Community Manager
15.✓Update the BRANCHLIST and VERSIONLIST in the JSDoc and PHPDoc repositories with the list of supported versions. These are built on a Sunday so this should be done in the final week before release.Integration Team

Releasing​

Packaging​

info

This should happen immediately before the next integration cycle begins on Monday (that is, some days after last weekly, 2 days prior to official release).

#MajorMinorTaskResponsibility
1.✓✓Check list:
  • Make sure there are no real blockers introduced in the last weekly (install, upgrade...).
  • Confirm that the latest en_fix changes have been generated and integrated.
  • Verify that all security issues have been integrated and there are not leftovers in the security area (but for delayed majors, where some security issues may have been introduced along the delay and they will be postponed to next minors).
Integration Team
2.✓✓Verify all unit tests and integration tests have passed (check all current CI servers in use).Integration Team
3.✓Verify QA tests have passed.Integration Team
4.✓✓Timing prerequisites (don't continue with the process if not within the time window!):
  1. Latest weeklies before the minor/major releases must be done (moodle-package-extract-and-postprocess email should confirm this has happened around 09:00 Australia/Perth).
  2. Minor and major releases can be packaged at any time within the 96 hours before the automatic releasing time (Monday 09:00 Australia/Perth). And always, at very least, 4 hours before then (to allow the server to prepare all the stuff).
In practice, this means that the packaging of minor/major releases must be done, assuming that weeklies were rolled normally on Thursday, between Friday 09:00 and Monday 05:00 (Australia/Perth times). Breaking any of these rules will require manual operation in the server, forcing repackaging and releasing.

Follow the mdlrelease steps:
  • Run the prerelease.sh script to generate the target minor (--type minor) and major (--type major) releases and their corresponding tags. Pay attention to the complete output of the process: permissions, svg... changes (only for majors)... Triple verify all changes are correct! (can use the --show option to inspect them).
  • Follow the on-screen instructions.
  • Push changes to integration.git (but tags).

For major releases, only when a new STABLE branch has been created:
  • Run the prerelease.sh script (--type back-to-dev) to move master to next X.Y+1 development version/branch.
  • Push changes to integration.git - note this will lead to the "versions checker" failing for master, no problem, it will be fixed by the 1st cloned issue at #12 below).
  • Configure mdlrelease (config.sh) to know about the new MOODLE_XY stable branch.
  • Update the old CI server(s) by cloning all the "master" jobs to a new MXY view (trick: do it down-top, way easier). Don't forget to re-chain the jobs properly.
    • Adjust master compare DB jobs to check for upgrade from MOODLE_XY_STABLE.
    • Configure the "05. Check version.php files" jobs both for the involved branches (XY and master) to define the interval of versions allowed in each.
    • Run all them.
  • Update the new CI infrastructure:
    • MR1: Merge pull request for new version in CI Jobs Repository
    • MR2: Merge pull request for symlink in moodle-ci-runner repository.
    • MR3: Merge pull request for skipping the language upgrade in moodle-ci-runner repository.
    • Clone all the "B - master" jobs to a new "B - XY" view (there is a maintenance job to perform the operation).

Both with minors and majors, continue with the mdlrelease steps:
  • Once CI servers have ended and all jobs have passed, push tags and run the release.sh script to apply the changes to moodle.git.
  • If needed configure mdlrelease (config.sh) to get rid of any unsupported version and adjust stable and security branches. Make a pull request with the changes.
  • Don't forget the "After the release" tasks.
Integration Team
5.✓✓Post a "git repos updated & tagged" message on the Partner forumIntegration Team
6.✓In the performance comparison repository, set the new release commit hash as $basecommit in the master branch and create a new MOODLE_XY_STABLE branch from it. (example). Also, apply it to the current performance testing infrastructure.Integration Team
7.✓✓Wait for the automated (every 2 hours) moodle-package to finish building for all versions. Verify the process has ended successfully (email).Integration Team
8.✓
  • Create new windows packager script (cloning the current master one and configuring it).
  • Edit the stats.php script (in the moodle-local_downloadmoodleorg repository), adding the new branch to the versions. Push it. Next time the plugin is deployed (normally 1 week after release approx) the stats will start showing the new branch statistics.
Integration Team
9.✓✓In the Tracker:
  • Visit the versions page and make the release, bumping all remaining open bugs to the next point release. If there is not next (end of support), clean that version from all the issues having it as "Fix for". This must be done both for the Moodle Project and the Plugins project.
  • Archive all versions but the ones being released, verifying that there aren't open issues using that version.
  • Remove from all screens the custom fields (Pull X.Y Branch and Pull X.Y Diff URL) belonging to 100% unsupported branches and, finally, proceed to delete the fields themselves.

For major releases only:		Integration Team
10.✓
  • Create the new Pull X.Y Branch and Pull X.Y Diff URL custom fields and spread them to all the screens needing them (copy from previous ones). Order them properly on each screen. Re-index Tracker.
  • In the CI server edit the following jobs to make them meet the new Pull X.Y Branch field:
    At the same time, remove from those jobs any branch belonging to 100% unsupported branches.
  • Edit the Tracker - CI - Check marked as integrated job to configure the development branches (devbranches) (39, 310, 311, 400...) actually active. Normally only the branch corresponding to master, but also accepts a comma separated list when under parallel development ("310,400" or "311,400" for example). Ordered by planned release date.
  • Check/configure (again, this may be has been already executed few steps above) the 05. Check version.php files jobs both for the involved branches (XY and master) to define the interval of versions allowed in each.
Integration Team
11.✓✓In the Tracker:
For major releases only:
  • Create the new "X.Y regressions" version to be applied to every issue found to be a recent problem introduced by the major release. These issues should get priority after release.
  • Adjust the Latest regressions filter (requires login).

For all releases:		Integration Team
12.✓Clone MDL-78001 and MDL-78000, to be resolved ASAP.Integration Team
13.✓
  • Create the new MOODLE_XY_STABLE and lastbased-MOODLE_XY_STABLE branches in the security repo (branching from the just created upstream MOODLE_XY_STABLE branch).
  • Cherry pick any security commit present in the security repository's master branch to the new MOODLE_XY_STABLE branch (note that normally there should not be any, see point #1 check list above).
  • Verify the new security jobs exist for the new branch and check it's working in all servers, keeping only enabled the public one.
Integration Team
14.✓Protect the MOODLE_XY_STABLE branches in github interface to prevent non-FF accidents ( MDLSITE-4183 )Integration Team

Release day​

info

Usually on Monday

#MajorMinorTaskResponsibility
1.✓✓The execution of moodle-package-extract-and-postprocess X script may be needed if the releases are not going to be published on Monday but another weekday (X is the weekday, (1-7) starting in Monday).
By default it happens automatically around 09:00 AM Perth/Australia time (check emails then).		Integration Team
2.✓✓Verify release status, download pages and windows packages.Integration Team
3.✓✓If needed, modify the downloads cfg script (serverscripts) to decide about branches status, master visibility and windows availability. Push changes, will be auto-deployed in a few minutes.Integration Team
4.✓✓Add/update the release date, build number and link on the Releases page and date in new version pages.Integration Team
5.✓✓Notify all registered sys admins, including security notes with CVE identifiers, using the mailing list server.Security Officer
6.✓Post about the Major release on the moodle.org News forumHead of educational solutions
7.✓Post about minor releases on the moodle.org News forumPlatform Manager
8.✓Add the next branch code to the branchesall admin setting at lang.moodle.org. Review the list of supported versions and update the value of branchsupported there eventually, too.AMOS Maintainer
9.✓✓Verify, 24h after tagging, that https://moodle.org/dev/contributions.php has been updated with new versions. If not, file an urgent MDLSITE issue, crons must be running!Integration Team
10.✓For en and de Moodle Docs, update default redirects and enable email notifications.Moodle Docs Maintainer
11.✓Go through all points listed under Day of release in New docs version process.Community Manager
12.✓
  • Decide the Full demo, Code freeze and QA begins dates for the next X.(Y+1) major release and put them in the Moodle development calendar. They will be -7w, -6w and -5w before release date respectively.
  • Update the Full demo, Code freeze, QA begins and Release dates on the Roadmap page.
  • Notify the Community Manager of new dates to be added to the moodle.org calendar.
Product delivery lead
13.✓Add calendar events in the moodle.org calendar for coming Major and Minor releases up to the next Major release.Community Manager
14.✓✓Update release schedule image on Releases pagePlatform Manager
15.✓Important: This must be done once it's already release day @ UTC (aka, after Australia/Perth 08:00) or the queues manage with hold them again because it's still "last week before release".
The integration_held label will be removed only from bug issues awaiting integration and bug issues awaiting component leads review; they correspond to last-week bugs that were held because of them being unrelated with the release. Now they can be processed, under on-sync rules. Standard message will be used to explain the un-hold.		Integration Team

1 week after​

#MajorMinorTaskResponsibility
1.✓✓Publish the X.Y+ packages for download.moodle.org (should be automatic once weeklies are packaged).
For major releases, within the on-sync period, master packages will be generated using the prerelease.sh script (--type on-sync). That guarantees XY_STABLE and master versions are kept the same. The last week, when leaving the on-sync period, master package will be generated normally (implicit --type weekly), so versions will diverge and dev can continue separately from that point.		Integration Team
2.✓✓Update the version.php in git to be X.Y.Z+ during the next weekly integration processIntegration Team
3.✓✓Create a new release notes page for the next minor versions (using the release notes template.) Add note to top of relevant version page about security support.Platform Manager
4.✓✓Add all security advisories to Security news and release notes with links to security advisories
5.✓✓Notify about publications of CVE using form: https://cveform.mitre.org/Security Officer
6.✓Upgrade moodle.org and all other Moodle community sites (next sites first, then production)
7.✓Deprecations:
  • Create the next deprecation epic (for X.Y+1+2y) similar to MDL-78075 and add standard issues to it:
    • Final deprecation of lib/deprecatedlib.php (like MDL-78076)
    • Removal of deprecated Behat steps (like MDL-78077)
    • Removal of strings (like MDL-78078) and
    • Removal of renamed classes (MDL-78079).
The issue can be based/cloned on the previous one and linked ("has to be done after") with it.
Libraries:
  • Create the next third party libraries epic for X.Y+1, similar to MDL-78024. The issue should refer to the list of Moodle libraries credits. The issue can be based/cloned on the previous one and linked ("has to be done after") with it. Drag/link opened issues from the previous one, avoiding leaving unfinished stuff.
  • Review how previous library review epics are going and share about them.
Integration Team

2 weeks after​

#MajorMinorTaskResponsibility
1.✓Once the "On Sync" period ends:
  • Confirm that versions (XY_STABLE and master) diverged properly the last on-sync week. If not, proceed to it by manually bumping master the version to the current date @ integration.git.
  • Disable the "Manage queues on continuous" job. It has served us well.
  • The integration_held label will be removed from all issues awaiting integration and issues awaiting component leads review, so they will, automatically (see next point), be moved to the various integration queues. Send an unholding + rebase message to all those issues.
  • Enable the "Manage queues on normal" job. It will control the queues over the coming months, until the next freeze.
  • In the Jenkins CI server enable all the jobs corresponding to the new branch (B-XYZ view) that were created on packaging day.
Integration Team
2.✓The discussion about environmental requirements for next X.Y+1 major release (MDL-76724) will end and the issue will be resolved immediately. A new issue, about the requirements for the next X.Y+2 major release will be created at the same time by cloning the previous one and dragging any non-resolved detail (due date = 3w after release).Integration Team
3.✓Ensure Language pack for master (X.Y+1) is available and merge the pull request MR4 for stop skipping the language upgrade in moodle-ci-runner repository.Integration Team
4.✓Ensure release retrospectives are held for each of the LMS teams and results are actioned.Platform Manager / Head of educational solutions
5.✓

Prepare for the next version of PHP.

We need to ensure that the Moodle versions that will support the next version of PHP are compatible with it.

  • If an epic has not been created yet, create an epic issue for it (see MDL-76405 as an example)
  • Go through the list of changes for the upcoming PHP version, identify potential breaking changes in Moodle and add them to the epic.

Related links:

Integration team

1 month after​

#MajorMinorTaskResponsibility
1.✓Remove, in CI servers, all the jobs and views corresponding to branches which support has ended completely. (there is a maintenance job to perform the operation).Integration Team
2.✓Upgrade all the Moodle CI sites to recent major release by configuring the "Moodle CI Auto Upgrade" job in all them.Integration Team
3.✓Confirm that there isn't any remaining integration_held issue from latest release, proceeding to un-hold them immediately. Note that there may exist other "held" issues, unrelated with latest release. This process step does not affect them.Integration Team

See also​