Merge branch 'next-major' into je/global-key

Author: jedelbo

.clang-format

@@ -1,4 +1,6 @@
 ---
+# Don't try to format any languages not described by this file
+DisableFormat: true
 BasedOnStyle: LLVM
 AccessModifierOffset: -4
 AlignAfterOpenBracket: Align
@@ -56,7 +58,6 @@ ConstructorInitializerIndentWidth: 4
 ContinuationIndentWidth: 4
 Cpp11BracedListStyle: true
 DerivePointerAlignment: false
-DisableFormat:   false
 ExperimentalAutoDetectBinPacking: false
 FixNamespaceComments: true
 ForEachMacros:
@@ -125,5 +126,10 @@ StatementMacros:
   - QT_REQUIRE_VERSION
 TabWidth:        4
 UseTab:          Never
+---
+Language: Cpp
+DisableFormat:   false
+---
+Language: ObjC
+DisableFormat:   false
 ...
-

.gitignore

@@ -25,6 +25,7 @@ CMakeScripts
 # Ignore build artifacts
 /build*
 /debug*
+/release*
 
 # Ignore all user files except one which includes execution path for test
 *.user
@@ -97,3 +98,8 @@ compile_commands.json
 # Swift package manager
 /.swiftpm
 /.build
+
+# Bindgen generated files and npm stuff
+/bindgen/generated/
+node_modules/
+tsconfig.tsbuildinfo

Android.bp

@@ -148,6 +148,7 @@ cc_defaults {
         "-fPIC",
         "-DREALM_NO_CONFIG",
         "-DREALM_HAVE_OPENSSL=1",
+        "-DREALM_INCLUDE_CERTS=1",
         "-DREALM_ENABLE_ENCRYPTION=1",
         "-DREALM_ENABLE_SYNC=1",
         "-DREALM_ENABLE_GEOSPATIAL=1",

CHANGELOG.md

@@ -12,12 +12,14 @@
 * Fixed equality queries on a Mixed property with an index possibly returning the wrong result if values of different types happened to have the same StringIndex hash. ([6407](https://github.com/realm/realm-core/issues/6407) since v11.0.0-beta.5).
 * If you have more than 8388606 links pointing to one specific object, the program will crash. ([#6577](https://github.com/realm/realm-core/issues/6577), since v6.0.0)
 * Query for NULL value in Dictionary<Mixed> would give wrong results ([6748])(https://github.com/realm/realm-core/issues/6748), since v10.0.0)
+* A Realm generated on a non-apple ARM 64 device and copied to another platform (and vice-versa) were non-portable due to a sorting order difference. This impacts strings or binaries that have their first difference at a non-ascii character. These items may not be found in a set, or in an indexed column if the strings had a long common prefix (> 200 characters). ([PR # 6670](https://github.com/realm/realm-core/pull/6670), since 2.0.0-rc7 for indexes, and since since the introduction of sets in v10.2.0)
 
 ### Breaking changes
 * Support for upgrading from Realm files produced by RealmCore v5.23.9 or earlier is no longer supported.
 * Remove `set_string_compare_method`, only one sort method is now supported which was previously called `STRING_COMPARE_CORE`.
 * BinaryData and StringData are now strongly typed for comparisons and queries. This change is especially relevant when querying for a string constant on a Mixed property, as now only strings will be returned. If searching for BinaryData is desired, then that type must be specified by the constant. In RQL the new way to specify a binary constant is to use `mixed = bin('xyz')` or `mixed = binary('xyz')`. ([6407](https://github.com/realm/realm-core/issues/6407)).
 * In the C API, `realm_collection_changes_get_num_changes` and `realm_dictionary_get_changes` have got an extra parameter to receive information on the deletion of the entire collection.
+* Sorting order of strings has changed to use standard unicode codepoint order instead of grouping similar english letters together. A noticeable change will be from "aAbBzZ" to "ABZabz". ([2573](https://github.com/realm/realm-core/issues/2573))
 
 ### Compatibility
 * Fileformat: Generates files with format v24. Reads and automatically upgrade from fileformat v10. If you want to upgrade from an earlier file format version you will have to use RealmCore v13.x.y or earlier.
@@ -29,6 +31,106 @@
 
 ----------------------------------------------
 
+# 13.23.2 Release notes
+
+### Enhancements
+* None.
+
+### Fixed
+* Deleting an object in an asymmetric table would cause a crash. Likely to solve [#1537](https://github.com/realm/realm-kotlin/issues/1537), since v12.1.0.
+* Implement BIO_CTRL_GET_KTLS_SEND and BIO_CTRL_GET_KTLS_RECV as OpenSSL versions < 3.0.4 depend on it.
+
+### Breaking changes
+* Added separate enum for callback handler result values in the platform networking C API. ([PR #7015](https://github.com/realm/realm-core/pull/7015))
+* Platform networking CAPI now uses different callback types depending on where callback is used. ([PR #7015](https://github.com/realm/realm-core/pull/7015))
+
+### Compatibility
+* Fileformat: Generates files with format v23. Reads and automatically upgrade from fileformat v5.
+
+-----------
+
+### Internals
+* Update JSON library from 3.7.3 to 3.11.2.
+* Updated async_write_binary in the Default Socket Provider to properly return a status to the callback handler. ([PR #7015](https://github.com/realm/realm-core/pull/7015))
+
+----------------------------------------------
+
+# 13.23.1 Release notes
+
+### Enhancements
+* Empty commits no longer trigger an extra invocation of the sync progress handler reporting the exact same information as the previous invocation ([PR #7031](https://github.com/realm/realm-core/pull/7031)).
+
+### Fixed
+* `SyncManager::path_for_realm()` would return `/<path>/filename.realm.realm` if `custom_file_name` was set to `filename.realm` and the file didn't exist. It would correctly return `/<path>/filename.realm` if the file already existed. After this fix `/<path>/filename.realm` is returned in all cases. ([#7038](https://github.com/realm/realm-core/issues/7038))
+* Fixed a bug preventing SSL handshake from completing successfuly due to failed hostname verification when linking against BoringSSL. (PR [#7034](https://github.com/realm/realm-core/pull/7034))
+* Updating subscriptions did not trigger Realm autorefreshes, sometimes resulting in async refresh hanging until another write was performed by something else ([PR #7031](https://github.com/realm/realm-core/pull/7031)).
+* Fix interprocess locking for concurrent realm file access resulting in a interprocess deadlock on FAT32/exFAT filesystems ([PR #6959](https://github.com/realm/realm-core/pull/6959)).
+
+### Breaking changes
+* None.
+
+### Compatibility
+* Fileformat: Generates files with format v23. Reads and automatically upgrade from fileformat v5.
+
+-----------
+
+### Internals
+* None.
+
+----------------------------------------------
+
+# 13.23.0 Release notes
+
+### Enhancements
+* Allow collections of non-embedded links in asymmetric objects. ([PR #7003](https://github.com/realm/realm-core/pull/7003))
+* Flexible sync API improvements:
+  - Erase Subscriptions by class type for C API.
+  - `MutableSubscriptionSet::erase(iterator)` now runs in constant time.
+  - Introduce `MutableSubscriptionSet::erase_by_id()`.
+  - Introduce `MutableSubscriptionSet::erase_by_class_name()`.
+  ([PR #7008](https://github.com/realm/realm-core/pull/7008))
+* Filtering by user-defined functions (PR [#7020](https://github.com/realm/realm-core/pull/7020))
+
+### Fixed
+* Fixed an issue where Android.bp builds would fail with SSL certificate validation errors because we didn't include the trusted CA roots bundle. (PR [#7022](https://github.com/realm/realm-core/pull/7022))
+
+### Breaking changes
+* None.
+
+### Compatibility
+* Fileformat: Generates files with format v23. Reads and automatically upgrade from fileformat v5.
+
+-----------
+
+### Internals
+* Update tests to use global logger. ([PR #6917](https://github.com/realm/realm-core/pull/6917))
+* TableView::get_num_results_excluded_by_limit() has been removed. Hopefully not used by any SDK.
+* bindgen is now part of the release
+
+----------------------------------------------
+
+# 13.22.0 Release notes
+
+### Enhancements
+* None.
+
+### Fixed
+* Fixed issue with double delete when using the CAPI for timers in platform networking ([#6993](https://github.com/realm/realm-core/issues/6993), since v13.3.0).
+* Receiving a write_not_allowed error from the server would have led to a crash. ([#6978](https://github.com/realm/realm-core/issues/6978), since v13.2.0)
+
+### Breaking changes
+* Platform Networking CAPI has been updated to provide separate functions (instead of 1) for executing callback handlers depending on purpose ([PR #6994](https://github.com/realm/realm-core/pull/6994)).
+
+### Compatibility
+* Fileformat: Generates files with format v23. Reads and automatically upgrade from fileformat v5.
+
+-----------
+
+### Internals
+* None.
+
+----------------------------------------------
+
 # 13.21.0 Release notes
 
 ### Enhancements

CMakeLists.txt

@@ -285,7 +285,7 @@ if(REALM_NEEDS_OPENSSL OR REALM_FORCE_OPENSSL)
     set(OPENSSL_USE_STATIC_LIBS ON)
     find_package(OpenSSL REQUIRED)
     set(REALM_HAVE_OPENSSL ON)
-    string(REGEX MATCH "^([0-9]+)\\.([0-9]+)" OPENSSL_VERSION_MAJOR_MINOR ${OPENSSL_VERSION})
+    string(REGEX MATCH "^([0-9]+)\\.([0-9]+)" OPENSSL_VERSION_MAJOR_MINOR "${OPENSSL_VERSION}")
 elseif(APPLE)
     set(REALM_HAVE_SECURE_TRANSPORT "1")
 endif()
@@ -331,6 +331,7 @@ include_directories(src)
 include_directories(${CMAKE_CURRENT_BINARY_DIR}/src) # For generated files (like config.h)
 
 add_subdirectory(src)
+add_subdirectory(bindgen)
 
 # Install the licence and changelog files
 install(FILES LICENSE CHANGELOG.md DESTINATION "doc/realm" COMPONENT devel)

Package.swift

@@ -3,7 +3,7 @@
 import PackageDescription
 import Foundation
 
-let versionStr = "13.21.0"
+let versionStr = "13.23.2"
 let versionPieces = versionStr.split(separator: "-")
 let versionCompontents = versionPieces[0].split(separator: ".")
 let versionExtra = versionPieces.count > 1 ? versionPieces[1] : ""
@@ -16,7 +16,7 @@ var cxxSettings: [CXXSetting] = [
     .define("REALM_ENABLE_ASSERTIONS", to: "1"),
     .define("REALM_ENABLE_ENCRYPTION", to: "1"),
     .define("REALM_ENABLE_SYNC", to: "1"),
-    .define("REALM_ENABLE_GEOSPATIAL", to: "0"),
+    .define("REALM_ENABLE_GEOSPATIAL", to: "1"),
 
     .define("REALM_VERSION_MAJOR", to: String(versionCompontents[0])),
     .define("REALM_VERSION_MINOR", to: String(versionCompontents[1])),
@@ -64,7 +64,6 @@ let notSyncServerSources: [String] = [
     "realm/cluster.cpp",
     "realm/cluster_tree.cpp",
     "realm/collection.cpp",
-    "realm/collection_list.cpp",
     "realm/collection_parent.cpp",
     "realm/column_binary.cpp",
     "realm/db.cpp",
@@ -73,6 +72,7 @@ let notSyncServerSources: [String] = [
     "realm/disable_sync_to_disk.cpp",
     "realm/error_codes.cpp",
     "realm/exceptions.cpp",
+    "realm/geospatial.cpp",
     "realm/global_key.cpp",
     "realm/group.cpp",
     "realm/group_writer.cpp",
@@ -397,14 +397,13 @@ let package = Package(
             ] + cxxSettings) as [CXXSetting]),
         .target(
             name: "RealmCore",
-            dependencies: ["Bid"],
+            dependencies: ["Bid", "s2geometry"],
             path: "src",
             exclude: ([
                 "CMakeLists.txt",
                 "external",
                 "realm/CMakeLists.txt",
                 "realm/exec",
-                "realm/geospatial.cpp",
                 "realm/object-store/CMakeLists.txt",
                 "realm/object-store/c_api",
                 "realm/object-store/impl/epoll",

README.md

@@ -1,3 +1,9 @@
+<p align="right">
+<a href='https://github.com/realm/realm-core/releases'><img src='https://img.shields.io/github/v/release/realm/realm-core' alt='Latest Release' /></a>
+<a href='https://coveralls.io/github/realm/realm-core'><img src='https://coveralls.io/repos/github/realm/realm-core/badge.svg' alt='Coverage Status' /></a>
+<a href='https://github.com/realm/realm-core'><img src='https://img.shields.io/github/license/realm/realm-core' alt='Source License' /></a>
+</p>
+
 ![Realm](doc/logo.png)
 
 Realm is a mobile database that runs directly inside phones, tablets or wearables - check out [realm.io](https://realm.io).

bindgen/.eslintignore

@@ -0,0 +1 @@
+realm-bindgen.ts

bindgen/.eslintrc.json

@@ -0,0 +1,16 @@
+{
+  "root": true,
+  "parser": "@typescript-eslint/parser",
+  "extends": [
+    "eslint:recommended",
+    "plugin:@typescript-eslint/recommended",
+    "plugin:prettier/recommended"
+  ],
+  "rules": {
+    "@typescript-eslint/explicit-function-return-type": "off",
+    "no-case-declarations": "off"
+  },
+  "env": {
+    "node": true
+  }
+}

bindgen/.mocharc.json

@@ -0,0 +1,6 @@
+{
+  "loader": "tsx",
+  "enable-source-maps": true,
+  "no-warnings": true,
+  "spec": "src/tests/*.test.ts"
+}

bindgen/.prettierrc.yaml

@@ -0,0 +1,2 @@
+trailingComma: all
+printWidth: 120

bindgen/CMakeLists.txt

@@ -0,0 +1,88 @@
+file(GLOB_RECURSE BINDGEN_LIB_TS_FILES
+    LIST_DIRECTORIES false
+    CONFIGURE_DEPENDS
+    ${CMAKE_CURRENT_SOURCE_DIR}/src/*.ts
+)
+list(FILTER BINDGEN_LIB_TS_FILES EXCLUDE REGEX "tests/[^/]*\.ts$")
+
+# Ninja buffers output so we need to tell tools to use colors even though stdout isn't a tty.
+if(CMAKE_GENERATOR MATCHES "^Ninja")
+    set(NODE_FORCE_COLOR ${CMAKE_COMMAND} -E env FORCE_COLOR=1)
+endif()
+set(NPX ${NODE_FORCE_COLOR} npx --prefix .) # without a prefix, npx changes the current directory and that is really confusing and hard to debug
+
+# This path is hard coded in the reader, and referred to in the specs to inform yaml-language-server of the schema
+set(SCHEMA_FILE ${CMAKE_CURRENT_SOURCE_DIR}/generated/spec.schema.json)
+add_custom_command(
+    OUTPUT ${SCHEMA_FILE}
+    WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
+    COMMAND ${NPX} typescript-json-schema ${CMAKE_CURRENT_SOURCE_DIR}/tsconfig.json RelaxedSpec --include ${CMAKE_CURRENT_SOURCE_DIR}/src/spec/relaxed-model.ts --out ${SCHEMA_FILE} --required --noExtraProps --aliasRefs
+    VERBATIM
+    MAIN_DEPENDENCY src/spec/relaxed-model.ts
+    DEPENDS
+        # Note: The json-schema generation task is extremely slow so we don't want to rerun it any time any TS file is modified.
+        # Instead we assume that it will only transitively import the listed files.
+        # (Right now there are no imports)
+)
+
+add_custom_target(BindgenSpecJsonSchema DEPENDS ${SCHEMA_FILE})
+
+# Stash these variables as properties so they can be read from the bindgen() function.
+# Using the BindgenSpecJsonSchema target for scoping, not because it actually has anything to do with that.
+set_property(TARGET BindgenSpecJsonSchema PROPERTY BINDGEN_LIB_TS_FILES "${BINDGEN_LIB_TS_FILES}")
+set_property(TARGET BindgenSpecJsonSchema PROPERTY NPX "${NPX}")
+
+function(bindgen)
+    set(options)
+    set(oneValueArgs TEMPLATE OUTDIR OPTIN)
+    set(multiValueArgs OUTPUTS SPECS SOURCES)
+    cmake_parse_arguments(PARSE_ARGV 0 BINDGEN "${options}" "${oneValueArgs}" "${multiValueArgs}")
+
+    if(BINDGEN_UNPARSED_ARGUMENTS)
+        message(FATAL_ERROR "Unexpected arguemnts to bindgen()" ${BINDGEN_UNPARSED_ARGUMENTS})
+    endif()
+
+    foreach(ARG TEMPLATE OUTDIR OUTPUTS) # SOURCES and SPECS are optional.
+        if (NOT BINDGEN_${ARG})
+            message(FATAL_ERROR "Argument ${ARG} is required to bindgen()")
+        endif()
+    endforeach()
+
+    # Don't want to set cmake_minimum_required() here because it has other effects and needs to be done early.
+    # Remove this once https://github.com/realm/realm-core/issues/6537 bumps realm-core's minimum.
+    if(NOT CMAKE_CURRENT_FUNCTION_LIST_DIR)
+        message(FATAL_ERROR "bindgen() requires cmake 3.17+")
+    endif()
+
+    get_property(BINDGEN_LIB_TS_FILES TARGET BindgenSpecJsonSchema PROPERTY BINDGEN_LIB_TS_FILES)
+    get_property(NPX TARGET BindgenSpecJsonSchema PROPERTY NPX)
+
+    set(CORE_SPEC_FILE ${CMAKE_CURRENT_FUNCTION_LIST_DIR}/spec.yml)
+
+    add_custom_command(
+        OUTPUT ${BINDGEN_OUTPUTS}
+        WORKING_DIRECTORY ${CMAKE_CURRENT_FUNCTION_LIST_DIR}
+        COMMAND ${NPX} tsx -- "${CMAKE_CURRENT_FUNCTION_LIST_DIR}/realm-bindgen.ts"
+            --spec "$<JOIN:${CORE_SPEC_FILE};${BINDGEN_SPECS},;--spec;>"
+            "$<$<BOOL:${BINDGEN_OPTIN}>:--opt-in;${BINDGEN_OPTIN}>"
+            --template "${BINDGEN_TEMPLATE}"
+            --output "${BINDGEN_OUTDIR}"
+        COMMAND_EXPAND_LISTS
+        VERBATIM
+        MAIN_DEPENDENCY ${BINDGEN_TEMPLATE}
+        DEPENDS
+            # from bindgen-lib
+            ${CORE_SPEC_FILE}
+            BindgenSpecJsonSchema
+            ${BINDGEN_LIB_TS_FILES}
+
+            # from sdk
+            ${BINDGEN_SPECS}
+            ${BINDGEN_SOURCES}
+            ${BINDGEN_OPTIN}
+    )
+
+    foreach(FILE ${BINDGEN_OUTPUTS})
+        set_property(SOURCE ${FILE} PROPERTY INCLUDE_DIRECTORIES "$<$<COMPILE_LANGUAGE:CXX>:${CMAKE_CURRENT_FUNCTION_LIST_DIR}/src>")
+    endforeach()
+endfunction()

bindgen/realm-bindgen.ts

@@ -0,0 +1,3 @@
+#!/usr/bin/env -S npx tsx --
+import { program } from "./src/program";
+program.parse(process.argv);