diff --git a/HOWTO.md b/HOWTO.md index 28707ef8bd..f205c6f4af 100644 --- a/HOWTO.md +++ b/HOWTO.md @@ -171,6 +171,13 @@ this process is performed using the above docker container. You'll likely want to run the git commands outside of the container and the `make ...` commands inside the container (so you don't have to setup git inside the docker container). +This process describes running `make ` in multiple places. These targets will +both regenerate the language bindings and then build and run any test suites. +Skipping tests should not be done by default, but most languages have a "gen" target +available - `make gen-` - which will only regenerate the bindings without +running tests. This can be used to split or speed up the process should any errors occur +and something needs to be repeated. + 1. It's easiest to do this on the master branch. Start by tagging the release version: ```shell @@ -213,13 +220,29 @@ inside the container (so you don't have to setup git inside the docker container ```shell make c haskell javascript rust - make javascript - git add c/include/libsbp/version.h haskell/sbp.cabal javascript/sbp/RELEASE-VERSION package.json package-lock.json rust/sbp/Cargo.toml + git add c/include/libsbp/version.h haskell/sbp.cabal rust/sbp/Cargo.toml git commit --amend -a -m 'Release ' git tag -f -a INCREMENTED_TAG -m "Version INCREMENTED_TAG of libsbp." ``` -4. Finally, build the docs: + + For JavaScript, needs to be run twice to update the package information + ```shell + make javascript + make javascript + git add javascript/sbp/RELEASE-VERSION package.json package-lock.json + git commit --amend -a -m 'Release ' + ``` + + For Kaitai + + ```shell + make kaitai + git add kaitai/ksy/sbp.ksy + git commit --amend -a -m 'Release ' + ``` + +4. Build the docs: ```shell make docs @@ -237,24 +260,24 @@ inside the container (so you don't have to setup git inside the docker container git tag -f -a INCREMENTED_TAG -m "Version INCREMENTED_TAG of libsbp." ``` -5. Verify that package dependencies, their version numbers, and the +5. Update the CHANGELOG details with `make release`. Submit a pull request and + get it merged. This requires a GitHub token to be loaded into your environment + at `CHANGELOG_GITHUB_TOKEN`. The Makefile will use docker to run the + tool that generates a `DRAFT_CHANGELOG.md`. + + It's generally a good idea to scrub any internal ticket numbers from + `DRAFT_CHANGELOG.md` as they add unnecessary noise for customers. + +6. Verify that package dependencies, their version numbers, and the libsbp version number in the C, Python, JavaScript, and LaTeX developer documentation are consistent. -6. Push the release to GitHub: +7. Push the release to GitHub: ```shell git push origin master ``` -7. Update the CHANGELOG details with `make release`. Submit a pull request and - get it merged. This requires a GitHub token to be loaded into your environment - at `CHANGELOG_GITHUB_TOKEN`. The Makefile will use docker to run the - tool that generates a `DRAFT_CHANGELOG.md`. - - It's generally a good idea to scrub any internal ticket numbers from - `DRAFT_CHANGELOG.md` as they add uncessary noise for customers. - -7. Create a release on +8. Create a release on [GitHub](https://github.com/swift-nav/libsbp/releases) and add the section for the new release from `DRAFT_CHANGELOG.md` to the release notes. @@ -266,15 +289,17 @@ inside the container (so you don't have to setup git inside the docker container [Protocol Documentation](https://github.com/swift-nav/libsbp/blob/v3.4.6/docs/sbp.pdf) ``` -8. Prep for the next development cycle. Add the new release section from - `DRAFT_CHANGELOG.md` to `CHANGELOG.md` and re-run `make release`. +9. Prep for the next development cycle. Create an empty commit so that version numbers + get regenerated with the `-alpha` tag of the next release, then rebuild all languages + + Again, javascript needs to be built twice to get the correct package versions ``` - vim/emacs/nano CHANGELOG.md # add new change log entries - git add CHANGELOG.md + git commit --allow-empty -m "prep for next release #no_auto_pr" make all - git add python/sbp/RELEASE-VERSION c/include/libsbp/version.h haskell/sbp.cabal javascript/sbp/RELEASE-VERSION package.json package-lock.json rust/sbp/Cargo.toml docs/sbp.pdf - git commit -m 'update CHANGELOG.md, prep for next release #no_auto_pr' + make javascript + git add python/sbp/RELEASE-VERSION c/include/libsbp/version.h haskell/sbp.cabal javascript/sbp/RELEASE-VERSION package.json package-lock.json rust/sbp/Cargo.toml docs/sbp.pdf kaitai/ksy/sbp.ksy + git commit -m 'prep for next release #no_auto_pr' git push origin master ``` @@ -318,37 +343,10 @@ To distribute Rust. Use the `cargo-release` tool: cargo install cargo-release ``` -**FIRST** just try running the `dist-rust` target: - -``` -make dist-rust -``` - -If that doesn't work (**status** it don't work, consider fixing the make target), -otherwise try releasing `sbp` and `sbp2json` crates separately, first `sbp`, -this will do a dry run first: - -``` -cargo release --exclude sbp2json -``` - -Then use `--execute` to actually run the release: - -``` -cargo release --exclude sbp2json --execute -``` +Once you have logged in to crates.io with `cargo`: -Next, release `sbp2son`, first do a dry-run: - -``` -cargo release --exclude sbp -``` - -Then, reset any modifications from the dry run, and then actually release `sbp2son`: - -``` -git checkout . -cargo release --exclude sbp --execute +```shell +cargo release --allow-branch HEAD --execute ``` Then rollback any commits that are created: @@ -362,6 +360,8 @@ git reset --hard v The build of the libsbp wheel can be done via the `libsbp-build` container described above. +You must have the correct token set in your environment to publish to PyPI. + ## Troubleshooting ### Error: `!!! No Python wheel (.whl) file found...` @@ -409,9 +409,15 @@ SonaType open source repo requires a GPG key for signatures. Generate GPG key v ```shell gpg --gen-key gpg --export-secret-keys >keys.gpg -gpg --keyserver keyserver.ubuntu.com --send-keys ``` +Export your public key +``` +gpg --export -a > pub.key +``` + +Go to [https://keyserver.ubuntu.com/#submitKey](https://keyserver.ubuntu.com/#submitKey) and upload your PUBLIC key + To locate the value for `signing.keyId` (needed below) run: ```shell @@ -464,7 +470,8 @@ staging repository and release to finish it off. Follow the instructions here for how to "close" and then "release" and staging repository on SonaType's repository manager: -- +- [Documentation](https://central.sonatype.org/publish/release) +- [Nexus Repository Manager](https://s01.oss.sonatype.org/#welcome) # Contributions diff --git a/scripts/tag.py b/scripts/tag.py new file mode 100755 index 0000000000..baef4fc118 --- /dev/null +++ b/scripts/tag.py @@ -0,0 +1,361 @@ +#!/usr/bin/env python + +import sys +from datetime import datetime +import os +import re +import argparse +import subprocess + +DOCKER_IMAGE = "swiftnav/libsbp-build:2023-12-19" +PWD = os.getcwd() + +PREP_FOR_NEXT_RELEASE_FINISHED_MSG = """ +A new commit has been generated and all language bindings rebuilt to take on a 'dirty' version number. + +After dismissing this message the commit will be shown .If everything appears ok you can push to master straight away. + +If there are any mistakes now is the time to correct them. Make any changes which are quired then update the tag by running: + +git add +git commit --amend -m "prep for next release #no_auto_pr" + +Once you have fixed everything you can push to master +""" + +TAG_FINISHED_MSG = """ +A new commit and tag have been created, all language bindings and documentation have been rebuilt, and the changelog updated. + +After dismissing this message the commit will be shown. If everything looks good you can push to master straight away and continue with distribution. + +If there are any mistakes now is the time to correct them. Make any changes which are required then update the tag by running: + +git add +git commit --amend -a -m "Release " +git tag -f -a -m "Version of libsbp." + +Once you have fixed everything you can push to master + +Once pushed prepare for the next release by running this script again with the "-p" flag. +""" + +COMMIT_MSG = "" +TAG_MSG = "" +TAG_NAME = "" + + +def run_command(cmd: list, expect_success=True, docker=False): + if docker: + cmd = [ + "docker", + "run", + "-it", + "--rm", + "-v", + f"{PWD}:/mnt/workspace", + "-t", + DOCKER_IMAGE, + ] + cmd + + result = subprocess.run(cmd, capture_output=True, text=True) + + if result.returncode != 0 and expect_success: + print(f"Command failed: {cmd}") + print(result.stdout) + print(result.stderr) + sys.exit(1) + + return result.stdout + + +def get_current_tag(): + return run_command( + ["git", "describe", "--match", "v*", "--always", "--tags"] + ).strip() + + +def current_commit_is_tag(): + return re.match(r"^v[0-9]+\.[0-9]+\.[0-9]+$", get_current_tag()) + + +def get_next_tag(): + current_tag = get_current_tag() + major, minor, patch = current_tag.split(".")[:3] + + if "-" in patch: + patch = patch.split("-")[0] + + return f"{major}.{minor}.{int(patch)+1}" + + +def fn_not_provided(): + assert False + + +class Step: + def __init__(self, name, fn=fn_not_provided, args={}): + self.name = name + self.fn = fn + self.args = args + + def run(self, index, total): + print(f"[{index}/{total}] {self.name}") + self.invoke() + + def invoke(self): + self.fn(**self.args) + + +class CreateInitialCommit(Step): + def __init__(self): + Step.__init__( + self, + "Create initial commit", + run_command, + {"cmd": ["git", "commit", "--allow-empty", "-m", COMMIT_MSG]}, + ) + + +class CreateInitialTag(Step): + def __init__(self): + Step.__init__( + self, + "Create initial tag", + run_command, + {"cmd": ["git", "tag", "-a", args.tag, "-m", TAG_MSG]}, + ) + + +class UpdateCommit(Step): + def __init__(self): + Step.__init__( + self, + "Amend commit", + run_command, + {"cmd": ["git", "commit", "--amend", "-a", "-m", COMMIT_MSG]}, + ) + + +class UpdateTag(Step): + def __init__(self): + Step.__init__( + self, + "Update tag", + run_command, + {"cmd": ["git", "tag", "-f", "-a", args.tag, "-m", TAG_MSG]}, + ) + + +class BuildLanguages(Step): + def __init__(self, languages): + if args.generate_only: + targets = ["gen-{}".format(lang) for lang in languages] + else: + targets = languages + Step.__init__( + self, + f"Build {"" if args.generate_only else "and test "}languages: {', '.join(languages)}", + run_command, + {"cmd": ["make", *targets], "docker": True}, + ) + + +class BuildDocumentation(Step): + def __init__(self): + Step.__init__( + self, + "Build documentation", + run_command, + {"cmd": ["make", "docs"], "docker": True}, + ) + + +class GenerateDraftChangelog(Step): + def __init__(self): + Step.__init__( + self, + "Generate draft changelog", + run_command, + {"cmd": ["make", "release"]}, + ) + + +class MergeChangelogs(Step): + def __init__(self): + Step.__init__(self, "Merge changelogs") + + def invoke(self): + with open("DRAFT_CHANGELOG.md", "r") as f: + draft = f.readlines() + + # The first 4 lines are just the title and "unreleased" headers which we will recreate later + draft = draft[4:] + + # The first line should now be the first real line of the "unreleased" section which is always a link to the full changelog. Find the next heading and discard everything afterwards + assert draft[0].startswith("[Full Changelog]") + + for i in range(1, len(draft)): + if draft[i].startswith("## [v"): + draft = draft[: i - 1] + break + + proposed = [ + f"## [{args.tag}](https://github.com/swift-nav/libsbp/tree/{args.tag}) ({datetime.today().strftime('%Y-%m-%d')})\n", + "\n", + ] + + # Strip out anything which looks like a Jira ticket number + for i in range(len(draft)): + proposed.append(re.sub(r"\\\[[A-Z]*-[0-9]*\\\](?=[^(])", r"", draft[i])) + proposed.append("\n") + print("Proposed new changelog section") + print("\n".join(proposed)) + + with open("CHANGELOG.md", "r") as f: + changelog = f.readlines() + + with open("CHANGELOG.md", "w") as f: + # Keep the first 2 lines from the origin alchangelog + f.writelines(changelog[0:2]) + + # Then the new section + f.writelines(proposed) + + # Then the rest of the original + f.writelines(changelog[2:]) + + os.remove("DRAFT_CHANGELOG.md") + + +class ShowFinishedBanner(Step): + def __init__(self, msg): + Step.__init__(self, "Finished") + self.__msg = msg + + def invoke(self): + print(self.__msg) + input("Press Enter to continue...") + + +class ShowHead(Step): + def __init__(self): + Step.__init__(self, "Show head", run_command, {"cmd": ["git", "show", "HEAD"]}) + + +if __name__ == "__main__": + if ( + not os.path.exists("spec") + or not os.path.exists("generator") + or not os.path.exists("scripts") + ): + print("This script must be run from the root of the libsbp repository") + sys.exit(1) + + if ( + subprocess.run(["git", "diff", "--exit-code"], capture_output=True).returncode + != 0 + ): + print( + "Working directory is not clean. Remove any and all changes before running this command" + ) + sys.exit(1) + + parser = argparse.ArgumentParser( + description=f"When run without arguments will tag the next version of libsbp which will be {get_next_tag()}" + ) + + parser.add_argument( + "-t", "--tag", type=str, required=False, default=get_next_tag(), help="New tag" + ) + parser.add_argument( + "-p", + "--prep_for_next_release", + action="store_true", + required=False, + default=False, + help="Prep for next release", + ) + parser.add_argument( + "-g", + "--generate_only", + action="store_true", + required=False, + default=False, + help="Don't run tests, just generate sources", + ) + + global args + args = parser.parse_args() + + steps = [] + + if args.prep_for_next_release: + if not current_commit_is_tag(): + print("Can only prep for next release from a properly tagged commit") + sys.exit(1) + + COMMIT_MSG = "prep for next release #no_auto_pr" + + steps.append(CreateInitialCommit()) + steps.append(BuildLanguages(["python"])) + steps.append(UpdateCommit()) + steps.append( + BuildLanguages( + ["java", "javascript", "protobuf", "c", "haskell", "javascript", "rust"] + ) + ) + steps.append(UpdateCommit()) + steps.append(BuildLanguages(["javascript"])) + steps.append(UpdateCommit()) + steps.append(BuildLanguages(["kaitai"])) + steps.append(UpdateCommit()) + steps.append(ShowFinishedBanner(PREP_FOR_NEXT_RELEASE_FINISHED_MSG)) + steps.append(ShowHead()) + + else: + COMMIT_MSG = f"Release {args.tag}" + TAG_MSG = f"Version {args.tag} of libsbp." + + if not re.match(r"^v[0-9]+\.[0-9]+\.[0-9]+$", args.tag): + print(f"Invalid tag: {args.tag}") + sys.exit(1) + + if ( + subprocess.run(["git", "show", args.tag], capture_output=True).returncode + == 0 + ): + print(f"Tag {args.tag} already exists") + sys.exit(1) + + input(f"About to release libsbp {args.tag}. Press Enter to continue...") + + steps.append(CreateInitialCommit()) + steps.append(CreateInitialTag()) + steps.append(BuildLanguages(["python"])) + steps.append(UpdateCommit()) + steps.append(UpdateTag()) + steps.append( + BuildLanguages( + ["java", "javascript", "protobuf", "c", "haskell", "javascript", "rust"] + ) + ) + steps.append(UpdateCommit()) + steps.append(UpdateTag()) + steps.append(BuildLanguages(["javascript"])) + steps.append(UpdateCommit()) + steps.append(UpdateTag()) + steps.append(BuildLanguages(["kaitai"])) + steps.append(UpdateCommit()) + steps.append(UpdateTag()) + steps.append(BuildDocumentation()) + steps.append(UpdateCommit()) + steps.append(UpdateTag()) + steps.append(GenerateDraftChangelog()) + steps.append(MergeChangelogs()) + steps.append(UpdateCommit()) + steps.append(UpdateTag()) + steps.append(ShowFinishedBanner(TAG_FINISHED_MSG)) + steps.append(ShowHead()) + + for i, step in enumerate(steps): + step.run(i + 1, len(steps))