[Question] How come the Windows fix for test SlimSerializerTest.listWithSurrogatePairSerialize (change 61a0841) is any sane ?

[laurent-larizza]

** General Description **

It seems that the test mentioned in the subject has been amended subsequently to an effort of making the Slim protocol tests work on Windows (#1477)

The change was brought to the test, and states "Surrogate Length is different depending on OS". The change has been accepted into the codebase. Why so ?

• Java says "I encode Strings internally with 16-bit chars." (https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html)
• Unicode says "Whatever codepoint lies outside the BMP should be encoded with a surrogate pair when in UTF-16" (https://www.unicode.org/versions/Unicode16.0.0/UnicodeStandard-16.0.pdf §2.5.2)
• Java says "Well, OK. Therefore, surrogate pairs will be 2 consecutive chars in a String" (https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Character.html#unicode)
• Java also says "Oh, and therefore length will count the number on chars used (code units), not the number of codepoints." (https://docs.oracle.com/javase/8/docs/api/java/lang/String.html#length--)

In SLiM, the strings must be prefixed by their length as "The six+ digits are the number of UTF-16 code units in the string", that's what the doc says at https://fitnesse.org/FitNesse/UserGuide/WritingAcceptanceTests/SliM/SlimProtocol.html

The source file is encoded in UTF-8, and the supplementary character of the test is spliced directly into the string literal as F0 9F 80 9C twice, so correctly UTF-8 encoded. I suspect the real issue on Windows was because the compiler misinterpreted the string as being 8 windows-1252 characters instead of 5 UTF-8-encoded characters. I believe the correct fix for the test to be charset-independent would have been to replace the supplementary character by its correct escape sequence, namely \uD83C\uDC1C. (The actual surrogate pair)

Does anyone agree with my analysis ?

[six42]

Hi Laurnt-larizza,
thanks for your detailed analysis. You definetly understand more from UTF encoding then me.

My view is that the Java promise is to: build once and run everywhere. So Java should handle all OS dependencies.
In this case Java or the serialize function or the compiler is not keeping the promise.
The purpose of the test case is documenting this OS discrepancy.
It might be that using the escape sequence \uD83C\uDC1C doesn't have differnt behaviour and therefore would be OS agnostic. This would be then a good solution for such a problem in a real production use case.
The current test case has a vaild input and creates differnt outputs based on the OS you run it. That's important informtion and should be documented and therfore the current test case kept.

I think an actual fix should be applied to the compiler (as you mentioned it is a compiler issue). But this is out of scope from the Fitnesse project.

Why are you looking at this test case. Does it fails on your installation or do you have any other problem with surrogate chars in your test cases?

[laurent-larizza]

Hi, @six42, and thank you for answering on New Year's Eve.

Actually, I got interested in this case because I'm building a C++ Slim Server (working on stdin/stdout pipes) and as the base unit of communication for C++ is the byte, encodings as actually used by FitNesse become of prime interest. I looked at the documentation, the tests and the code to get all necessary information. That's where I dug into this and honed my Unicode knowledge, and the question arose.

I cannot easily reproduce the case, for reasons given hereafter. All I have is a theory: I believe that when the test failed and the different behaviour was first testified of on Windows, the cause of it was that the test was compiled without informing the compiler that the source file is encoded in UTF-8. Thing is, the byte sequence of the test String literal in the source file is 68 F0 9F 80 9C 6C 6C 6F is both valid UTF-8 (with 5 characters h \uD83C\uDC1C l l o, which counts for 6 UTF-16 code units since the code point of the funny character has 17 bits), and valid windows-1252 (with 8 1-byte characters, all valid). That would explain the discrepancy "between OSes".

If you read the Oracle Java 21 documentation here (https://docs.oracle.com/en/java/javase/21/intl/supported-encodings.html), you find :

The javac compiler assumes that .java source files are encoded with the default charset unless configured otherwise with the -encoding option.

Alternatively, if you prefer to save your source files with an encoding other than UTF-8, specify in the -encoding option the value of the native.encoding system property.

The same page states :

In JDK 17 and earlier releases, the default charset is determined when the Java runtime starts. On macOS, the default charset is UTF-8 except in the POSIX C locale. On other operating systems, it depends on the user's locale and the default encoding. For example, on Windows, it's a codepage-based charset such as windows-1252 or windows-31j.

What encoding was assumed by the compiler when compiling the test ? What was its default charset ? Does the behaviour actually happen when using the officially distributed fitnesse.jar on Windows?

I think the test as rewritten is invalid : when FitNesse runs, the source code encoding does not matter and any Unicode supplementary character that gets injected into a String comes from the Wiki pages (which may be wrongly encoded, but then the page must be fixed), and I believe that any such character would be correctly serialized, and in a way identical to the Mac & Linux case.

I don't have Windows handy, and I don't wish to fight with gradle or the installation of a Java development environment on my 2013 box yet to reproduce the test case failure. If I could validate this by myself, I would just have submitted a PR that would :

1. fix the test by using the escape sequence instead of a character
2. fix the Windows build by adding a systematic -encoding UTF-8 argument in the compilation command, since the project source files are all valid UTF-8.

But I don't want to submit a PR that I would not have thoroughly validated.

[jediwhale]

I believe your analysis is correct. Unfortunately I don't have a Windows environment available to verify it.

[laurent-larizza]

Hello @jediwhale, and thank you.

If anyone happens to have a Windows development environment, the experiment protocol I would therefore recommend would be the following, on a Windows installation :

1. Amend the gradle build by adding a systematic -encoding UTF-8 argument in the compilation command
2. Compile and run the tests.
3. Witness that the latest version SlimSerializerTest.listWithSurrogatePairSerialize then fails on Windows.
4. Revert commit a9246cc (since other fixes brought by Fix failling unit tests in windows OS #1477 look OK to me)
5. Compile and run the tests.
6. Witness that the test now succeeds on Windows, as it does on Linux and Mac.
7. Commit, PR, etc.

Optionally, replace the funny character (U+1F01C MAHJONG TILE FOUR OF CIRCLES character) by the correct surrogate pairs escape sequences \uD83C\uDC1C (double-checked here : https://www.fileformat.info/info/unicode/char/1f01c/index.htm)