ci: update changelog release automation

[endorama]

Motivation/summary

We recently changed how we manage the changelog. Our release automation needs to be updated.

For 9.0 release these changes are not necessary, as the release notes have been prepared manually.

See #16181

In draft as I'm still learning how things will play out. What I know so far:

• docs will be released from main
• older doc is not affected: this means our release workflow will need to be run from older branches (or we should create a new release workflow?)
• the content is broken down in multiple files but is not broken down by version, all release notes will be added to the same files (this may be revisited if automation becomes too complex, but does not seem to be the case)
• a "Next version" section will be available on top of each page
• backports are not expected, may still be done to ease backporting other changes

Checklist

• Update CHANGELOG.asciidoc
• Documentation has been updated

For functional changes, consider:

• Is it observable through the addition of either logging or metrics?
• Is its use being published in telemetry to enable product improvement?
• Have system tests been added to avoid regression?

How to test these changes

The best way is to run them locally in a Linux context (for those using MacOS).

Related issues

#16181

[mergify]

This pull request does not have a backport label. Could you fix it @endorama? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-7.17 is the label to automatically backport to the 7.17 branch.
• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
• backport-9./d is the label to automatically backport to the 9./d branch. /d is the digit.
• backport-8.x is the label to automatically backport to the 8.x branch.
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[v1v]

I've tested this PR in conjunction with #16645

For such, I created a test branch called test/my-new-docs using the above changeset and I ran:

$ export GITHUB_TOKEN=$(gh auth token)
$ make minor-release \
  RELEASE_VERSION=9.1.0 \
  PROJECT_OWNER=v1v \
  RELEASE_TYPE=minor \
  BASE_BRANCH=test/my-new-docs

And the logs/outcome :

INFO: Create GitHub label backport for the version 9.1.0
::group::create-github-label backport-9.1
✓ Label "backport-9.1" created in v1v/apm-server
::endgroup::
INFO: Create release branch and update new version 9.1.0
::group::create-branch 9.1
Already on 'test/my-new-docs'
Switched to a new branch '9.1'
::endgroup::
>> update-version
>> update-version-makefile
::group::git-diff
diff --git a/Makefile b/Makefile
index 0a2a77de8..9951bf26e 100644
--- a/Makefile
+++ b/Makefile
@@ -229,7 +229,7 @@ docs/spec: go.mod
 # Beats synchronisation.
 ##############################################################################
 
-BEATS_VERSION?=main
+BEATS_VERSION?=9.1
 BEATS_MODULE:=github.com/elastic/beats/v7
 
 .PHONY: update-beats
::endgroup::
::group::create-commit
 M Makefile
check that executables have shebangs.................(no files to check)Skipped
check that scripts with shebangs are executable..........................Passed
check for merge conflicts................................................Passed
[9.1 af672f69d] [Release] update version 9.1.0
 1 file changed, 1 insertion(+), 1 deletion(-)
::endgroup::
INFO: Create feature branch and update the versions. Target branch 9.1
::group::create-branch changelog-9.1
Already on '9.1'
Switched to a new branch 'changelog-9.1'
::endgroup::
>> update-changelog
>> updating changelog for 9.1
::group::git-diff
diff --git a/docs/release-notes/breaking-changes.md b/docs/release-notes/breaking-changes.md
index 62819a26f..ea971bfd9 100644
--- a/docs/release-notes/breaking-changes.md
+++ b/docs/release-notes/breaking-changes.md
@@ -20,6 +20,18 @@ Breaking changes can impact your Elastic applications, potentially disrupting no
 % **Action**<br> Steps for mitigating deprecation impact.
 % ::::
 
+## 9.1 [9-1]
+% **Release date:** Month day, year
+
+% ::::{dropdown} Title of breaking change
+% Description of the breaking change.
+% For more information, check [#PR-number]({{apm-pull}}PR-number).
+%
+% **Impact**<br> Impact of the breaking change.
+%
+% **Action**<br> Steps for mitigating deprecation impact.
+% ::::
+
 ## 9.0.0 [9-0-0]
 
 ::::{dropdown} Change server information endpoint "/" to only accept GET and HEAD requests
diff --git a/docs/release-notes/deprecations.md b/docs/release-notes/deprecations.md
index 6e269dd4a..8eb359a29 100644
--- a/docs/release-notes/deprecations.md
+++ b/docs/release-notes/deprecations.md
@@ -10,6 +10,18 @@ Review the deprecated functionality for Elastic APM. While deprecations have no
 ## Next version [next-version]
 % **Release date:** Month day, year
 
+% ::::{dropdown} Title of breaking change
+% Description of the breaking change.
+% For more information, check [#PR-number]({{apm-pull}}PR-number).
+%
+% **Impact**<br> Impact of the breaking change.
+%
+% **Action**<br> Steps for mitigating deprecation impact.
+% ::::
+
+## 9.1 [9-1]
+% **Release date:** Month day, year
+
 % ::::{dropdown} Deprecation title
 % Description of the deprecation.
 % For more information, check [PR #](PR link).
diff --git a/docs/release-notes/index.md b/docs/release-notes/index.md
index 44155eda9..e7a8ed94e 100644
--- a/docs/release-notes/index.md
+++ b/docs/release-notes/index.md
@@ -19,6 +19,17 @@ To check for security updates, go to [Security announcements for the Elastic sta
 
 ### Features and enhancements [elastic-apm-next-features-enhancements]
 
+%* 1 sentence describing the change. ([#PR number](https://github.com/elastic/apm-server/pull/PR number))
+
+### Fixes [elastic-apm-next-fixes]
+
+%* 1 sentence describing the change. ([#PR number](https://github.com/elastic/apm-server/pull/PR number))
+
+## 9.1 [9-1]
+% **Release date:** Month day, year
+
+### Features and enhancements [elastic-apm-next-features-enhancements]
+
 ### Fixes [elastic-apm-next-fixes]
 
 ## 9.0.0 [9-0-0]
::endgroup::
::group::create-commit
 M docs/release-notes/breaking-changes.md
 M docs/release-notes/deprecations.md
 M docs/release-notes/index.md
check that executables have shebangs.................(no files to check)Skipped
check that scripts with shebangs are executable..........................Passed
check for merge conflicts................................................Passed
[changelog-9.1 405b84c71] docs: Update changelogs for 9.1 release
 3 files changed, 35 insertions(+)
::endgroup::
INFO: Create feature branch and update the versions. Target branch test/my-new-docs
::group::create-branch update-9.1.0
Switched to branch 'test/my-new-docs'
Switched to a new branch 'update-9.1.0'
::endgroup::
>> update-mergify
Update mergify with backport-9.1
>> update-version
::group::git-diff
diff --git a/.mergify.yml b/.mergify.yml
index eecf2ae2d..1dc79a430 100644
--- a/.mergify.yml
+++ b/.mergify.yml
@@ -450,3 +450,17 @@ pull_request_rules:
         labels:
           - "backport"
         title: "[{{ destination_branch }}] {{ title }} (backport #{{ number }})"
+  - name: backport patches to 9.1 branch
+    conditions:
+      - merged
+      - base=main
+      - label=backport-9.1
+    actions:
+      backport:
+        assignees:
+          - "{{ author }}"
+        branches:
+          - "9.1"
+        labels:
+          - "backport"
+        title: "[{{ destination_branch }}] {{ title }} (backport #{{ number }})"
diff --git a/cmd/intake-receiver/version.go b/cmd/intake-receiver/version.go
index 588a42094..75521d1a0 100644
--- a/cmd/intake-receiver/version.go
+++ b/cmd/intake-receiver/version.go
@@ -18,4 +18,4 @@
 package main
 
 // version matches the APM Server's version
-const version = "9.1.0"
+const version = "9.2.0"
diff --git a/internal/version/version.go b/internal/version/version.go
index fb081dc0f..0e5623c25 100644
--- a/internal/version/version.go
+++ b/internal/version/version.go
@@ -18,4 +18,4 @@
 package version
 
 // Version holds the APM Server version.
-const Version = "9.1.0"
+const Version = "9.2.0"
::endgroup::
::group::create-commit
 M .mergify.yml
 M cmd/intake-receiver/version.go
 M internal/version/version.go
check that executables have shebangs.................(no files to check)Skipped
check that scripts with shebangs are executable..........................Passed
check for merge conflicts................................................Passed
[update-9.1.0 e478b7abe] [Release] update version 9.2.0
 3 files changed, 16 insertions(+), 2 deletions(-)
::endgroup::
>> update-changelog
>> updating changelog for 9.1.0
::group::git-diff
diff --git a/docs/release-notes/breaking-changes.md b/docs/release-notes/breaking-changes.md
index 62819a26f..4b724aa10 100644
--- a/docs/release-notes/breaking-changes.md
+++ b/docs/release-notes/breaking-changes.md
@@ -20,6 +20,18 @@ Breaking changes can impact your Elastic applications, potentially disrupting no
 % **Action**<br> Steps for mitigating deprecation impact.
 % ::::
 
+## 9.1.0 [9-1-0]
+% **Release date:** Month day, year
+
+% ::::{dropdown} Title of breaking change
+% Description of the breaking change.
+% For more information, check [#PR-number]({{apm-pull}}PR-number).
+%
+% **Impact**<br> Impact of the breaking change.
+%
+% **Action**<br> Steps for mitigating deprecation impact.
+% ::::
+
 ## 9.0.0 [9-0-0]
 
 ::::{dropdown} Change server information endpoint "/" to only accept GET and HEAD requests
diff --git a/docs/release-notes/deprecations.md b/docs/release-notes/deprecations.md
index 6e269dd4a..82a5fefa0 100644
--- a/docs/release-notes/deprecations.md
+++ b/docs/release-notes/deprecations.md
@@ -10,6 +10,18 @@ Review the deprecated functionality for Elastic APM. While deprecations have no
 ## Next version [next-version]
 % **Release date:** Month day, year
 
+% ::::{dropdown} Title of breaking change
+% Description of the breaking change.
+% For more information, check [#PR-number]({{apm-pull}}PR-number).
+%
+% **Impact**<br> Impact of the breaking change.
+%
+% **Action**<br> Steps for mitigating deprecation impact.
+% ::::
+
+## 9.1.0 [9-1-0]
+% **Release date:** Month day, year
+
 % ::::{dropdown} Deprecation title
 % Description of the deprecation.
 % For more information, check [PR #](PR link).
diff --git a/docs/release-notes/index.md b/docs/release-notes/index.md
index 44155eda9..9138fdb51 100644
--- a/docs/release-notes/index.md
+++ b/docs/release-notes/index.md
@@ -19,6 +19,17 @@ To check for security updates, go to [Security announcements for the Elastic sta
 
 ### Features and enhancements [elastic-apm-next-features-enhancements]
 
+%* 1 sentence describing the change. ([#PR number](https://github.com/elastic/apm-server/pull/PR number))
+
+### Fixes [elastic-apm-next-fixes]
+
+%* 1 sentence describing the change. ([#PR number](https://github.com/elastic/apm-server/pull/PR number))
+
+## 9.1.0 [9-1-0]
+% **Release date:** Month day, year
+
+### Features and enhancements [elastic-apm-next-features-enhancements]
+
 ### Fixes [elastic-apm-next-fixes]
 
 ## 9.0.0 [9-0-0]
::endgroup::
::group::create-commit
 M docs/release-notes/breaking-changes.md
 M docs/release-notes/deprecations.md
 M docs/release-notes/index.md
check that executables have shebangs.................(no files to check)Skipped
check that scripts with shebangs are executable..........................Passed
check for merge conflicts................................................Passed
[update-9.1.0 8212a6d17] [Release] update changelogs for 9.1 release
 3 files changed, 35 insertions(+)
::endgroup::
INFO: Push changes to v1v/apm-server and create the relevant Pull Requests
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 12 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 978 bytes | 978.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
remote: 
remote: Create a pull request for '9.1' on GitHub by visiting:
remote:      https://github.com/v1v/apm-server/pull/new/9.1
remote: 
To https://github.com/v1v/apm-server.git
 * [new branch]          9.1 -> 9.1
::group::create-pull-request
Enumerating objects: 29, done.
Counting objects: 100% (29/29), done.
Delta compression using up to 12 threads
Compressing objects: 100% (16/16), done.
Writing objects: 100% (16/16), 2.87 KiB | 2.87 MiB/s, done.
Total 16 (delta 11), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (11/11), completed with 10 local objects.
remote: 
remote: Create a pull request for 'update-9.1.0' on GitHub by visiting:
remote:      https://github.com/v1v/apm-server/pull/new/update-9.1.0
remote: 
To https://github.com/v1v/apm-server.git
 * [new branch]          update-9.1.0 -> update-9.1.0
--label 
https://github.com/v1v/apm-server/pull/75
::endgroup::
::group::create-pull-request
Enumerating objects: 13, done.
Counting objects: 100% (13/13), done.
Delta compression using up to 12 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 1.49 KiB | 1.49 MiB/s, done.
Total 7 (delta 5), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (5/5), completed with 5 local objects.
remote: 
remote: Create a pull request for 'changelog-9.1' on GitHub by visiting:
remote:      https://github.com/v1v/apm-server/pull/new/changelog-9.1
remote: 
To https://github.com/v1v/apm-server.git
 * [new branch]          changelog-9.1 -> changelog-9.1
--label 
https://github.com/v1v/apm-server/pull/76
::endgroup::

https://github.com/v1v/apm-server/tree/9.1 was created and the following PRs:

• 9.1: update docs, mergify, versions and changelogs v1v/apm-server#75
• 9.1: update docs v1v/apm-server#76

I had to do some tricky things in my end to support the gsed, I tried with alias sed=gsed but the bash... takes precedence.

[mergify]

This pull request is now in conflicts. Could you fix it @endorama? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b docs-automation upstream/docs-automation
git merge upstream/main
git push upstream docs-automation

[endorama]

I think PR v1v#76 is not needed as it targets a release branch and we don't need that with the new 9.x release notes.

I had to do some tricky things in my end to support the gsed, I tried with alias sed=gsed but the bash... takes precedence.

I addressed this by removing the check, in my tests sed did work as expected on MacOS (is being run on stdin and not on file, which seems to avoiding the need for -i bkp).

I also added and improved script that properly add the new version header in the right place.

[v1v]

I think PR v1v#76 is not needed as it targets a release branch and we don't need that with the new 9.x release notes.

Gotcha, I think that's something to be implemented in this PR, right? #16645 removed partially all the support for 8.x, and kept the former approach with the docs.

Let me know if you need help with