feat: add script to update from main

[igobranco]

Add a script to make it more easy to update the current translations with the fixes and new translations from the upstream main.

This PR adds a Python script that updates the current translations from the upstream main translations.
So from a previous release we can make it more easy just receive the latest patched (fixed) and/or new translations.

Use cases:

1. Some translations is 'fixed' on transifex and with this script we can update the open release branch with the latest translations.
2. Instead of having to fix both release and head translations to operator can just receive the new translations.

---

Copied from the script comment:

Update release branch translations with fixes and new strings from the main branch.

This script synchronizes translation files between a release branch and an upstream branch
(typically 'main'). It supports both JSON and PO (gettext) translation files.

Key features:
- Fetches upstream translations from a git branch (no local clone needed)
- Merges translations preferring upstream fixes over local versions
- Adds new translation strings from upstream that don't exist locally
- Preserves PO file formatting using msgcat with --no-wrap
- Supports filtering by language code and file type

For JSON files:
- Parses both local and upstream JSON files
- Adds new translation keys from upstream
- Preserves existing local translations for unchanged keys
- Sorts keys alphabetically for consistency

For PO files:
- Uses msgcat with --use-first to prefer upstream msgstr values (fixes)
- Uses --no-wrap to preserve original line formatting
- Uses --sort-output for consistent entry ordering
- Maintains proper gettext format and metadata

Usage:
    python update_from_main.py [OPTIONS]

Examples:
    # Update all languages and file types from main branch
    python update_from_main.py --branch main
    
    # Update only Portuguese translations
    python update_from_main.py --language pt_PT
    
    # Update only PO files for multiple languages
    python update_from_main.py --language pt_PT --language es_419 --file-type po
    
    # Dry run to preview changes
    python update_from_main.py --dry-run
    
    # Update from a different upstream repository
    python update_from_main.py --branch nau/redwood.master --upstream-repo https://github.com/openedx/openedx-translations.git

[openedx-webhooks]

Thanks for the pull request, @igobranco!

This repository is currently maintained by @openedx/committers-translations.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details

---

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

[gabrieldamours]

Hi @openedx/committers-translations -- checking in on this PR. It's been stalled for some time, can we move it forward or perhaps close it?

[Agrendalath]

Hi @igobranco, I just noticed this while checking the hackathon board. Thanks for putting this together - the script is nicely documented, and it's great to see test coverage. However, I'd like to step back and ask whether we need it at all, because I think plain git serves this use case better. A couple of the current behaviors may not apply to common scenarios.

My main concern is the merge model. This script does a 2-way union with no merge base, whereas git merge/git rebase does a 3-way merge that knows the fork point. That distinction matters because:

• Without a merge base, the script can't tell "I intentionally changed this" from "this is just stale", so it can never surface a real conflict. It just silently resolves one way.
• For PO files, msgcat --use-first lists upstream first, so upstream wins every shared msgid. That would overwrite any custom commits one may have added to their fork. Say they added custom fixes to override the defaults, or pulled unreviewed translations because they needed higher translation coverage for a non-English instance. This script would not preserve these changes.
• For JSON, merge_translations only ever adds new keys and never overwrites, so it does the opposite of the PO path. The two file types resolve conflicts in opposite directions, which is confusing.

Separately, the sorting makes future diffs harder, which is the practical cost we'll feel most. dict(sorted(...)) for JSON and --sort-output for PO reorder entire files. Most files in the repo aren't alphabetically sorted now, so a run reorders them wholesale, making diffing a custom branch against main or a release branch much noisier in the future.

A couple of smaller things:

• In all-files mode, root_dir.rglob("*.json") will also pick up transifex_input.json source files, which I don't think we want to touch.
• A missing upstream file is an expected case, but run_command prints Error running command / stderr to stderr before fetch_upstream_file catches the exception, so normal runs look confusing.

Given all this, my suggestion is to drop the script and use git for the refresh, since forks share history with this repo and a 3-way merge handles this scenario natively. A downstream that tracks main can git merge main (or git rebase onto it) and let git preserve ordering, keep its custom commits, and raise real conflicts where both sides touched the same string. If the custom changes are small, cherry-picking those commits onto a fresh upstream is even simpler and keeps clean diffs against main and release branches over time.

If there's a recurring need I'm missing - for example, reconciling two full Transifex syncs of PO files, where line-level git merge gets too noisy - then a gettext-aware helper could be a better fit. In that case, we could consider:

1. Dropping the sorting.
2. Flipping the precedence to preserve local changes.
3. Excluding the transifex_input.json source files.
4. Aligning the JSON path with the PO path so they behave consistently.

Could you share a bit more about the scenario you had in mind?

[brian-smith-tcril]

While I understand maintaining translations across branches is painful, I do not believe this PR will address those problems.

The source of truth for translated strings is Transifex. This script does not change the translated strings in Transifex, meaning the changes this script makes will be overridden by the ones in Transifex when updates are made there (wiping out all changes that came from the script).

I know @OmarIthawi has done some work on a script that serves a similar purpose but ensures the strings are updated in Transifex, and we are currently investigating ways to utilize the branching functionality that was recently added to Transifex as well.

[igobranco]

Thanks for the thorough reviews, @Agrendalath and @brian-smith-tcril!

On using git merge instead (@Agrendalath): From experience, git-based merging of translation files is impractical in real deployments. Translation files tend to have many unrelated whitespace, ordering, or metadata diffs that make merge conflicts very noisy - even with a proper 3-way merge. That's why I've never relied on that approach in practice. The script gives more explicit control over what gets updated and what gets preserved.

On Transifex as the source of truth (@brian-smith-tcril): The script is intentionally scoped to local git branches only - it's not meant to push anything back to Transifex. The target use case is an operator or release manager who needs to quickly backport translation fixes to a release branch before (or instead of) waiting for a full Transifex sync cycle.

For context, I didn't actually use this script for our last major upgrade either - by then we always try to have all translations translated and reviewed on our own pt_PT locale. But my feeling is this script could still be handy for other operators who don't have that level of translation coverage.

That said, I'm very interested in the Transifex branching feature you mentioned. If Transifex now supports branching natively, that could be a cleaner long-term solution. Could you share more about where that investigation stands or point me to @OmarIthawi's work? I'd be happy to close this PR if there's a better path forward through Transifex itself.

[OmarIthawi]

Thanks @igobranco for the contribution and pardon the late review.

Please note that the recommended pattern for introducing overrides involves another strategy:

• Create a clean repo e.g. https://github.com/nelc/futurex-translations with similar directory structure but fetch smaller set of languages using atlas pull
• Have a list of replacements to make: https://github.com/nelc/futurex-translations/blob/open-release/teak.master/scripts/reword_list.csv
• Run a script to replace the words after the pull inline e.g. https://github.com/nelc/futurex-translations/blob/open-release/teak.master/scripts/reword_translations.py
• Optionally add custom translations that doesn't exists: https://github.com/nelc/futurex-translations/blob/open-release/teak.master/scripts/merge_custom_translations.py

Commit to github and pull from that repo.

This is the preferred approach compared to few alternatives we considered earlier.

I will close this PR.

[igobranco]

Thanks @OmarIthawi for your share! I think those are very good examples!
I would still prefer to have an additional step to merge the release and upstream (latest) translations.
I like your approach with custom translations. Recently we had to edit some files manually on our fork, but with your approach is much nicer!

• Release translations (for the base release) - git submodule
• Upstream translations (latest transifex changes) - git submodule
• Custom translations - exactly what you have on https://github.com/nelc/futurex-translations - on our we would include some additional translations.
• translations - "output"

[OmarIthawi]

I would still prefer to have an additional step to merge the release and upstream (latest) translations.

@igobranco I assume that you want to avoid duplicating effort and your Release better translated than Main, right?

This is a bigger problem than custom translations that we're working with Transifex to support. Merging release and main have a lot of edge cases that we'd rather offloading to Transifex because it has much more metadata than us.

[igobranco]

@OmarIthawi yes, I want to avoid duplicating efforts. I have come to cases where it was missing translations and/or revisions on a Open edX release. Other cases it was required a translation fixes (not good enough translations).
I want to avoid to having to manage translations on 2 different locations (Transifex latest + Transifex release or Transifex latest + forked openedx-translations). So I could say to the team not technical team just translate and/or review on the latest Transifex. The rest is for the technical team.
We couldn't use just the Transifex latest for an Open edX release because there are code that is removed and the translations and removed on Transifex in the mean time.
I think I have explain my motivation on wanting a mix of the translations.