Added i18n component and related scripts #1082

yihao03 · 2025-03-28T02:44:22Z

run yarn trans to start translation

…plits the text into manageable chunk

…output path

…e function

…xt handling and error reporting

…toring segment handling to use arrays for better management

…ling in Promise.all

…ity and consistency

… merging logic and remove unnecessary debug logs

…e TRANSLATE tags

…guages in xml

…ager version in package.json

…eadability

… file path usage

…lations inside SCHEME and SCHEMEINLINE tags

…ded try catch when parsing xml to json to report failed parsing possibly attributed to unsound xml structure.

openai api

- Created a new XML file for references (97references97.xml) containing a comprehensive list of references used in the SICP JS project. - Added a new XML file for the index preface (98indexpreface98.xml) to provide context and formatting for the index section. - Introduced a new XML file for the making section (99making99.xml) detailing the background, interactive features, and development history of the SICP JS project. - Updated subsection2.xml to close the previously open SUBSECTION tag and include a comment for clarity.

…LINE tags in XML parsing

yihao03 · 2025-04-12T07:10:47Z

breaking changes are made: xml repositories are divided into folders, currently consisting of en and cn folders to store translated content. The same applies to the json folder after running "yarn json". Frontend needs to be changed accordingly to fetch json files from the corresponding url

…logic

arnav-goel10

LGTM!

RichDom2185

Did not look at the contents of the zh_CN folder. I assume they are generated by AI? Or was there manual editing? If the former, please add a gitattributes file so that reviewers (and GitHub) knows it's machine generated

xml/en/chapter1/section1/subsection3.xml

xml/en/chapter3/section3/subsection4.xml

xml/en/chapter3/section3/subsection5.xml

xml/en/chapter3/section4/subsection2.xml

xml/en/chapter2/section3/subsection2.xml

RichDom2185 · 2025-07-05T04:15:59Z

.github/workflows/deploy-pages.yml

+      - name: Clone translated_xmls
+        run: |
+          git clone -b translated_xmls https://github.com/source-academy/sicp.git translated_xmls
+          mv translated_xmls/* xml/
+          rm -r translated_xmls


Why are we cloning from a specific branch instead of merging it to the default branch?

RichDom2185 · 2025-07-05T04:16:51Z

.github/workflows/translate-changed.yml

Could you explain what these new workflows are for?

This workflow is for triggering translations for all those English XMLs that are changed. The workflow is not set to be triggered on pushes yet because the AI translation is not very stable yet.

the translate-everything workflow is for translating all English XMLs

RichDom2185 · 2025-07-05T04:17:43Z

.github/workflows/translate-changed.yml

+          echo API_KEY=${{ secrets.OPENAI_KEY }} >> .env
+          # echo API_KEY=${{ secrets.OPENAI_KEY2 }} >> .env


If they are no longer used, the secret should be deleted.

In the start of the project, me and Yi Hao were both given an API key, so there are two secrets. Now one is used as a backup.

RichDom2185 · 2025-07-05T04:18:22Z

.github/workflows/translate-everything.yml

+          echo API_KEY=${{ secrets.OPENAI_KEY }} >> .env
+          # echo API_KEY=${{ secrets.OPENAI_KEY2 }} >> .env


coder114514 · 2025-07-05T11:25:47Z

Did not look at the contents of the zh_CN folder. I assume they are generated by AI? Or was there manual editing? If the former, please add a gitattributes file so that reviewers (and GitHub) knows it's machine generated

All the AI translated XMLs will go into the translated_xmls branch, so I will delete the zh_CN folder. Should I add gitattributes file to the translated_xmls branch to mark they are generated by AI?

RichDom2185 · 2025-07-05T11:30:35Z

Did not look at the contents of the zh_CN folder. I assume they are generated by AI? Or was there manual editing? If the former, please add a gitattributes file so that reviewers (and GitHub) knows it's machine generated

All the AI translated XMLs will go into the translated_xmls branch, so I will delete the zh_CN folder. Should I add gitattributes file to the translated_xmls branch to mark they are generated by AI?

How is the zh_CN folder different from the translated_xmls branch? Why not just merge the translations directly to master instead of using a separate branch?

coder114514 · 2025-07-05T11:35:12Z

Did not look at the contents of the zh_CN folder. I assume they are generated by AI? Or was there manual editing? If the former, please add a gitattributes file so that reviewers (and GitHub) knows it's machine generated

All the AI translated XMLs will go into the translated_xmls branch, so I will delete the zh_CN folder. Should I add gitattributes file to the translated_xmls branch to mark they are generated by AI?

How is the zh_CN folder different from the translated_xmls branch? Why not just merge the translations directly to master instead of using a separate branch?

They are the same. The translated_xmls branch is separately created because in the workflow for translation (translate-changed, translate-everything) an action called deploy which pushes files to a branch is used to push AI generated translations to GitHub, so the branch is created for this.

RichDom2185 · 2025-07-05T11:52:20Z

Did not look at the contents of the zh_CN folder. I assume they are generated by AI? Or was there manual editing? If the former, please add a gitattributes file so that reviewers (and GitHub) knows it's machine generated

All the AI translated XMLs will go into the translated_xmls branch, so I will delete the zh_CN folder. Should I add gitattributes file to the translated_xmls branch to mark they are generated by AI?

How is the zh_CN folder different from the translated_xmls branch? Why not just merge the translations directly to master instead of using a separate branch?

They are the same. The translated_xmls branch is separately created because in the workflow for translation (translate-changed, translate-everything) an action called deploy which pushes files to a branch is used to push AI generated translations to GitHub, so the branch is created for this.

I see, noted, and what is the rationale of running it in a workflow, as opposed to running it locally?

coder114514 · 2025-07-05T12:00:12Z

Did not look at the contents of the zh_CN folder. I assume they are generated by AI? Or was there manual editing? If the former, please add a gitattributes file so that reviewers (and GitHub) knows it's machine generated

All the AI translated XMLs will go into the translated_xmls branch, so I will delete the zh_CN folder. Should I add gitattributes file to the translated_xmls branch to mark they are generated by AI?

How is the zh_CN folder different from the translated_xmls branch? Why not just merge the translations directly to master instead of using a separate branch?

They are the same. The translated_xmls branch is separately created because in the workflow for translation (translate-changed, translate-everything) an action called deploy which pushes files to a branch is used to push AI generated translations to GitHub, so the branch is created for this.

I see, noted, and what is the rationale of running it in a workflow, as opposed to running it locally?

sicp.sourceacademy.org is a static site, so the translated XMLs need to be deployed too, and thus we are storing them on GitHub.

This is not really about the translation workflows. The translator can be run locally and then the generated content can be manually pushed to the translated_xmls branch.

RichDom2185 · 2025-08-19T16:30:35Z

@yihao03 I still don't understand why you removed the zh_CN directory. I know it's AI created. But we need it for deployment right? Why run it as a workflow and push to a separate branch? Why do we need the separate branch at all?

Just run the command locally and push it to master branch?

In my view, if it's going to be deployed, then it should me in the master branch not some other branch.

RichDom2185 · 2025-08-19T16:33:22Z

.github/workflows/translate-everything.yml

+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_dir: ./translation_output
+          force_orphan: false # leave the possiblity for direct modification on translated xmls


If you do this, it will overwrite the deployment no?

…pr/yihao03/1082

…o03/1082

RichDom2185 · 2025-08-19T16:40:56Z

i18n/ai_files/zh_CN/dictionary.txt

Are these files autogenerated? What is the ai_files folder used for?

Hi @RichDom2185, I have been out of the loop for a while. But I believe that ai_files are files for ai to refer to. While it hasn't been properly implemented/put to use, the plan is that it will have a folder structure that mirrors the xml/en directory where each file contains specific terms/instructions to translate the corresponding source file.
Based on our discussions with Prof @martin-henz , we agreed that users should not edit AI generated output directly to correct mistakes as the changes will be overwritten when we regenerate the output using AI, therefore it is better that should the translations need any amendment/improvement, it should be done by editing the prompt given to the AI model, which could be read from these files.

yihao03 · 2025-08-20T02:25:53Z

@yihao03 I still don't understand why you removed the zh_CN directory. I know it's AI created. But we need it for deployment right? Why run it as a workflow and push to a separate branch? Why do we need the separate branch at all?

Just run the command locally and push it to master branch?

In my view, if it's going to be deployed, then it should me in the master branch not some other branch.

Hi, this was a design decision made by @coder114514 as he was in charge of the deployment workflow, while I mainly worked on the translation logic. However, I do agree with you that zh_CN should live in the master branch as that was my original design too.

yihao03 and others added 27 commits March 12, 2025 22:12

integrated i18n into sicp repository

e82194a

Add .env to .gitignore

7e793de

attempt to create a translate function that attempts to recursively s…

098de27

…plits the text into manageable chunk

Refactor translate function to improve XML error handling and update …

10856a1

…output path

Update .gitignore and refactor translation logic to use recurTranslat…

4922e98

…e function

updated splitting logic to concat orphaned texts

9f21e76

Refactor XML parsing logic to use strict mode and enhance text escaping

29f1441

Fix error handling in recursivelyTranslate function and remove debug log

f0316dc

Refactor recursion logic in translate function to improve orphaned te…

e45ede2

…xt handling and error reporting

Enhance translation performance by measuring execution time and refac…

ba88b63

…toring segment handling to use arrays for better management

Refactor async execution in fancyName function and improve error hand…

d443ffb

…ling in Promise.all

Refactor XML escaping logic to use strongEscapeXML for improved secur…

da52eb5

…ity and consistency

Refactor segment handling in recursivelyTranslate function to improve…

3370e50

… merging logic and remove unnecessary debug logs

Add .gitignore for node_modules and update translation logic to handl…

d48e425

…e TRANSLATE tags

update lockfile

806a152

reorganised file structure and yarn prepare json to translate all lan…

3329c54

…guages in xml

Merge branch 'master' of https://github.com/yihao03/sicp

7a835c5

Fix file path resolution in translate function and update package man…

d110429

…ager version in package.json

Refactor translation logic to simplify segment handling and improve r…

a115312

…eadability

Enhance error handling and logging in translation process, streamline…

89841d6

… file path usage

added proper error logging and reporting. added ability to skip trans…

b37b503

…lations inside SCHEME and SCHEMEINLINE tags

modified index.js to properly handle updated repository structure. Ad…

48fe8df

…ded try catch when parsing xml to json to report failed parsing possibly attributed to unsound xml structure.

added troubleshoot toggle that skips calling

8e0e318

openai api

Refactor translation logic to improve handling of SCHEME and SCHEMEIN…

b9ce30a

…LINE tags in XML parsing

added filename to json error logging

5f23def

updated to remove file search annotation generated by api calls

429cff5

yihao03 added 2 commits April 12, 2025 15:13

ran prettier

a28c5f8

refined troubleshoot toggle, temporarily removed problemtic skipping …

c3be2c0

…logic

coder114514 added 5 commits June 5, 2025 21:44

update .env in workflows for translation

7ec10f1

add packageManager and license part in package.json of i18n

6f0c85c

remove unused module according to code review

a53b54d

replace MAXLEN with max_chunk_len according to code review

4125ea4

add definition of __dirname to processHandler.ts

626b0d2

coder114514 requested a review from arnav-goel10 June 5, 2025 14:03

coder114514 self-assigned this Jun 5, 2025

fix typo in the workflow translate-everything.yml

71395f4

arnav-goel10 approved these changes Jun 6, 2025

View reviewed changes

yihao03 and others added 2 commits June 10, 2025 22:36

Merge branch 'master' into master

1d4307e

Merge branch 'master' into master

a5c1a03

RichDom2185 requested changes Jul 5, 2025

View reviewed changes

fix file perms

92d1ce1

fix accidental change

af25668

remove zh_CN directory

ea0351a

RichDom2185 mentioned this pull request Aug 19, 2025

i18n support for sicp source-academy/frontend#3133

Draft

7 tasks

RichDom2185 requested changes Aug 19, 2025

View reviewed changes

RichDom2185 added 2 commits August 20, 2025 00:36

Merge branch 'master' of https://github.com/source-academy/sicp into …

b4b3c67

…pr/yihao03/1082

Merge branch 'master' of https://github.com/yihao03/sicp into pr/yiha…

c44cce6

…o03/1082

RichDom2185 reviewed Aug 19, 2025

View reviewed changes

RichDom2185 marked this pull request as draft August 19, 2025 17:06

RichDom2185 mentioned this pull request Aug 19, 2025

Add ZH translations #1105

Draft

yihao03 marked this pull request as ready for review August 20, 2025 02:24

		echo API_KEY=${{ secrets.OPENAI_KEY }} >> .env
		# echo API_KEY=${{ secrets.OPENAI_KEY2 }} >> .env

Added i18n component and related scripts #1082

Are you sure you want to change the base?

Added i18n component and related scripts #1082

Uh oh!

Conversation

yihao03 commented Mar 28, 2025

Uh oh!

yihao03 commented Apr 12, 2025

Uh oh!

arnav-goel10 left a comment

Choose a reason for hiding this comment

Uh oh!

RichDom2185 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coder114514 Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coder114514 commented Jul 5, 2025

Uh oh!

RichDom2185 commented Jul 5, 2025

Uh oh!

coder114514 commented Jul 5, 2025

Uh oh!

RichDom2185 commented Jul 5, 2025

Uh oh!

coder114514 commented Jul 5, 2025

Uh oh!

RichDom2185 commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yihao03 commented Aug 20, 2025

Uh oh!

Uh oh!

coder114514 Jul 5, 2025 •

edited

Loading

RichDom2185 commented Aug 19, 2025 •

edited

Loading