Latin1 <=> UTF-X conversion + tests + benchmarks (simdutf#247)

* Validate latin

* Added Latin1 <=> UTF-X, tests TBD

* UTF -X <=> Latin1 conversions, test TBD

* Most Latin1 <=> UTF-X now passes compilation

* Implemented UTF32 => latin1 for archs on my CPU

* "Hello world" test UTF32 => Latin1

* Mostly works

* UTF32 => Latin1 test compiling but test failing

* Got a segfault here.

* Fixed segfault

* Less verbosity

* Same

* utf32 => Latin1 large input test

* test for Latin1 => UTF32 test. WIP

* fixed tests

* UTF32 => latin1 valid tests WIP

* Done with UTF 32 <=> Latin1 ?

* valid utf32 => latin1 tests

* UTF8 => Latin1 mostly done. Need tests.

* more tests  for UTF8 => Latin1

* Some more tests

* UTF8 => Latin1 with errors tests

* 4_byte too large test UTF8=>latin1 working

* utf8 => latin1 too_large_input test works

* Header bits test work  + small fixes

* UTF8 =>Latin1 /w errors test done

* utf8 => latin1 convert_valid tests done

* latin1 => UTF8 tests

* utf16le => latin1 tests

* utf16le => Latin1 with errors tests

* utf16le => latin1 tests

* utf16be => Latin1 tests

* latin1 => utf16be tests

* utf8 => Latin1 benchmarks

* fixing icu test attempt

* icu benchmark test

* Various minor fixes

* partial cleanup

* More cleaning up

* buggy cleanup

* better safe than sorry increnemental cleanup

* Very  incremental cleanup that copiles #2

* Slowly but surely!

* Small, sure,safe & incremental

* More incremental cleanup

* More increments

* More increments

* More increments

* More increments

* Preliminary Cleanup done

* Minor adjustment

* Latin1 => utf8 benchmarks (untested)

* latin1 => utf16le bench untested

* latin1 => utf32 benchmark + basic tests

* utf16 => latin1 benchmark

* utf32 => Latin1 benchmark

* fix wrong size causing segfaults...

* Latin1 => utf8 icu

* latin1 => utf16 iconv (prev was also iconv)

* Latin1 =>Utf32 iconv

* Latin1 => UTF16 Iconv bench

* UTF32 => Latin1 Iconv

* latin1 => utf8 icu bench

* Fix some buffer overflow errors

* Fixes to latin1 => UTF-16 bench

* Latin1 => UTF32 icu bench

* UTF16 => Latin1 ICU

* UTF32 => Latin1 ICU benches

* Working UTF16 =>Latin1 icu bench

* utf32 => Latin1 Icu working

* Some fixes

* Some fixes to benches + modest README changes

* README done + minor changes

* More ICU checks for safety

* Remove comments

* fixes

* Various fixes on the latin 1 branch

* Fixes.

* latin1_from_utf16 unused parameter deletion

* latin1_from_utf32 unused parameter deletion

* utf32 length from latin1 unused parameter deletion

* fixes

* utf16 length from latin1 unusued parameter deletion

* more parameter deletion

* more parameter deletions

* Windows fixes

* arm64 dummy/scalar  implementation

* More windows fixes

* Update tests/convert_utf32_to_latin1_with_errors_tests.cpp

Co-authored-by: Daniel Lemire <daniel@lemire.me>

* Update tests/convert_utf32_to_latin1_tests.cpp

Co-authored-by: Daniel Lemire <daniel@lemire.me>

* partial fixes

* Remove extraneous comments added by mistake

* remove extraneous comments

* Windows Fixes/fix utf32 => latin1 tests

* CMakeLists fixes

* More fixes

* Yet more fixes

* Made tests deterministic again + removed comments

* More Windows fixes

* s390/big endian fixes

* Windows fixes

* Cleaned up comments

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: Daniel Lemire <daniel@lemire.me>

Author: 3 people

README.md

@@ -56,6 +56,6 @@ if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU" AND NOT MINGW)
   target_link_libraries(simdutf_benchmarks_benchmark PUBLIC stdc++fs)
 endif()
 
-target_link_libraries(simdutf_benchmarks_benchmark PUBLIC simdutf simdutf::tests::helpers) 
+target_link_libraries(simdutf_benchmarks_benchmark PUBLIC simdutf simdutf::tests::helpers)
 
 add_library(simdutf::benchmarks::benchmark ALIAS simdutf_benchmarks_benchmark)

benchmarks/src/CMakeLists.txt

@@ -1112,7 +1112,7 @@ struct AppleEvents {
         u64 val = counters_1[idx] - counters_0[idx];
         printf("%14s: %llu\n", alias->alias, val);
     }*/
-    return performance_counters{
+    return performance_counters {
         counters_0[counter_map[0]], counters_0[counter_map[3]],
         counters_0[counter_map[2]], counters_0[counter_map[1]]};
   }

benchmarks/src/apple_arm_events.h

@@ -22,6 +22,7 @@
 #include <unicode/uconfig.h>
 #include <unicode/platform.h>
 #include <unicode/unistr.h>
+#include <unicode/ucnv.h>
 #endif
 
 #if ICONV_AVAILABLE
@@ -69,36 +70,60 @@ namespace simdutf::benchmarks {
         void run_validate_utf32_with_errors(const simdutf::implementation& implementation, size_t iterations);
         void run_count_utf8(const simdutf::implementation& implementation, size_t iterations);
         void run_count_utf16(const simdutf::implementation& implementation, size_t iterations);
+        void run_convert_latin1_to_utf8(const simdutf::implementation& implementation, size_t iterations);
+        void run_convert_latin1_to_utf16(const simdutf::implementation& implementation, size_t iterations);
+        void run_convert_latin1_to_utf32(const simdutf::implementation& implementation, size_t iterations);
+        void run_convert_utf8_to_latin1(const simdutf::implementation& implementation, size_t iterations);
+        void run_convert_utf8_to_latin1_with_errors(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_utf8_to_utf16(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_utf8_to_utf16_with_errors(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_utf8_to_utf32(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_utf8_to_utf32_with_errors(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_utf8_to_utf16_with_dynamic_allocation(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_utf8_to_utf32_with_dynamic_allocation(const simdutf::implementation& implementation, size_t iterations);
+        void run_convert_valid_utf8_to_latin1(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_valid_utf8_to_utf16(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_valid_utf8_to_utf32(const simdutf::implementation& implementation, size_t iterations);
+        void run_convert_utf16_to_latin1(const simdutf::implementation& implementation, size_t iterations);
+        void run_convert_utf16_to_latin1_with_errors(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_utf16_to_utf8(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_utf16_to_utf8_with_errors(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_utf16_to_utf32(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_utf16_to_utf32_with_errors(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_utf16_to_utf8_with_dynamic_allocation(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_utf16_to_utf32_with_dynamic_allocation(const simdutf::implementation& implementation, size_t iterations);
+        void run_convert_valid_utf16_to_latin1(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_valid_utf16_to_utf8(const simdutf::implementation& implementation, size_t iterations);
+        void run_convert_utf32_to_latin1(const simdutf::implementation& implementation, size_t iterations);
+        void run_convert_utf32_to_latin1_with_errors(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_utf32_to_utf8(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_utf32_to_utf8_with_errors(const simdutf::implementation& implementation, size_t iterations);
+        void run_convert_valid_utf32_to_latin1(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_valid_utf32_to_utf8(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_utf32_to_utf16(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_utf32_to_utf16_with_errors(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_valid_utf32_to_utf16(const simdutf::implementation& implementation, size_t iterations);
         void run_convert_valid_utf16_to_utf32(const simdutf::implementation& implementation, size_t iterations);
         void run_detect_encodings(const simdutf::implementation& implementation, size_t iterations);
 #if ICU_AVAILABLE
+        void run_convert_latin1_to_utf8_icu(size_t iterations);
+        void run_convert_latin1_to_utf16_icu(size_t iterations);
+        void run_convert_latin1_to_utf32_icu(size_t iterations);
+        void run_convert_utf8_to_latin1_icu(size_t iterations);
         void run_convert_utf8_to_utf16_icu(size_t iterations);
         void run_convert_utf16_to_utf8_icu(size_t iterations);
+        void run_convert_utf16_to_latin1_icu(size_t iterations);
+        void run_convert_utf32_to_latin1_icu(size_t iterations);
 #endif
 #if ICONV_AVAILABLE
+        void run_convert_latin1_to_utf8_iconv(size_t iterations);
+        void run_convert_latin1_to_utf16_iconv(size_t iterations);
+        void run_convert_latin1_to_utf32_iconv(size_t iterations);
+        void run_convert_utf8_to_latin1_iconv(size_t iterations);
         void run_convert_utf8_to_utf16_iconv(size_t iterations);
         void run_convert_utf16_to_utf8_iconv(size_t iterations);
+        void run_convert_utf16_to_latin1_iconv(size_t iterations);
+        void run_convert_utf32_to_latin1_iconv(size_t iterations);
 #endif
 #ifdef INOUE2008
         /**

benchmarks/src/benchmark.cpp

@@ -8,6 +8,7 @@ enum encoding_type {
         UTF16_BE = 4,   // BOM 0xfe 0xff
         UTF32_LE = 8,   // BOM 0xff 0xfe 0x00 0x00
         UTF32_BE = 16,   // BOM 0x00 0x00 0xfe 0xff
+        Latin1 = 32,
 
         unspecified = 0
 };

benchmarks/src/benchmark.h

@@ -10,9 +10,10 @@ enum error_code {
   TOO_LONG,     // We either have too many consecutive continuation bytes or the string starts with a continuation byte.
   OVERLONG,     // The decoded character must be above U+7F for two-byte characters, U+7FF for three-byte characters,
                 // and U+FFFF for four-byte characters.
-  TOO_LARGE,    // The decoded character must be less than or equal to U+10FFFF OR less than or equal than U+7F for ASCII.
+  TOO_LARGE,    // The decoded character must be less than or equal to U+10FFFF,less than or equal than U+7F for ASCII OR less than equal than U+FF for Latin1
   SURROGATE,    // The decoded character must be not be in U+D800...DFFF (UTF-8 or UTF-32) OR
-                // a high surrogate must be followed by a low surrogate and a low surrogate must be preceded by a high surrogate (UTF-16)
+                // a high surrogate must be followed by a low surrogate and a low surrogate must be preceded by a high surrogate (UTF-16) OR
+                // there must be no surrogate at all (Latin1)
   OTHER         // Not related to validation/transcoding.
 };