Merge branch 'next-major' into fsa/string-interning

Author: nicola-cab

.github/workflows/make-release.yml

@@ -12,6 +12,9 @@ jobs:
   finish_release:
     if: github.event.pull_request.merged == true && startsWith(github.head_ref, 'release/automated_v')
     runs-on: ubuntu-latest
+    environment:
+      name: Production
+      url: ${{ steps.publish-github-release.outputs.html_url }}
     steps:
     - name: Checkout
       uses: actions/checkout@v4
@@ -29,6 +32,7 @@ jobs:
       shell: bash
     - name: Make Tag and Publish Github Release
       uses: ncipollo/release-action@2c591bcc8ecdcd2db72b97d6147f871fcd833ba5 #! 1.14.0
+      id: publish-github-release
       with:
         bodyFile: extracted_changelog.md
         name: Realm Core v${{ steps.get-version.outputs.version }}

.github/workflows/prepare-release.yml

@@ -23,14 +23,16 @@ jobs:
       - name: Change Version
         run: tools/release-init.sh ${{ inputs.version }}
         shell: bash
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} # the gh command in the release script requires this token
       - name: Create Release PR
         id: prepare-pr
         uses: peter-evans/create-pull-request@6d6857d36972b65feb161a90e484f2984215f83e #! 6.0.5
         with:
           branch: release/automated_v${{ inputs.version }}
           title: Prepare for ${{ inputs.version }}
           draft: false
-          body-path: changes-since-last-tag.txt
+          body-path: pr-body.txt
           labels: no-jira-ticket
           commit-message: Prepare for release ${{ inputs.version }}
           token: ${{ secrets.GITHUB_TOKEN }}

.gitignore

@@ -110,4 +110,4 @@ ssh_agent_commands.sh
 
 # release script artifacts
 extracted_changelog.md
-changes-since-last-tag.txt
+pr-body.txt

CHANGELOG.md

@@ -1,10 +1,12 @@
-# NEXT RELEASE
+# NEXT MAJOR RELEASE
 
 ### Enhancements
 * <New feature description> (PR [#????](https://github.com/realm/realm-core/pull/????))
+* Storage of integers changed so that they take up less space in the file. This can cause commits and some queries to take a bit longer (PR [#7668](https://github.com/realm/realm-core/pull/7668))
 
 ### Fixed
 * <How do the end-user experience this issue? what was the impact?> ([#????](https://github.com/realm/realm-core/issues/????), since v?.?.?)
+* None.
 
 ### Breaking changes
 * None.
@@ -19,12 +21,43 @@
 
 ----------------------------------------------
 
+# 14.10.0 Release notes
+
+### Enhancements
+* Include the originating client reset error in AutoClientResetFailure errors. ([#7761](https://github.com/realm/realm-core/pull/7761))
+* Reduce the size of the local transaction log produced by creating objects, improving the performance of insertion-heavy transactions ([PR #7734](https://github.com/realm/realm-core/pull/7734)).
+
+### Fixed
+* Fix some client resets (such as migrating to flexible sync) potentially failing with AutoClientResetFailed if a new client reset condition (such as rolling back a flexible sync migration) occurred before the first one completed. ([PR #7542](https://github.com/realm/realm-core/pull/7542), since v13.11.0)
+* Encrypted files on Windows had a maximum size of 2GB even on x64 due to internal usage of `off_t`, which is a 32-bit type on 64-bit Windows ([PR #7698](https://github.com/realm/realm-core/pull/7698), since the introduction of encryption support on Windows in v3.0.0).
+* The encryption code no longer behaves differently depending on the system page size, which should entirely eliminate a recurring source of bugs related to copying encrypted Realm files between platforms with different page sizes. One known outstanding bug was ([RNET-1141](https://github.com/realm/realm-dotnet/issues/3592)), where opening files on a system with a larger page size than the writing system would attempt to read sections of the file which had never been written to ([PR #7698](https://github.com/realm/realm-core/pull/7698)).
+* There were several complicated scenarios which could result in stale reads from encrypted files in multiprocess scenarios. These were very difficult to hit and would typically lead to a crash, either due to an assertion failure or DecryptionFailure being thrown ([PR #7698](https://github.com/realm/realm-core/pull/7698), since v13.9.0).
+* Encrypted files have some benign data races where we can memcpy a block of memory while another thread is writing to a limited range of it. It is logically impossible to ever read from that range when this happens, but Thread Sanitizer quite reasonably complains about this. We now perform a slower operations when running with TSan which avoids this benign race ([PR #7698](https://github.com/realm/realm-core/pull/7698)).
+* Tokenizing strings for full-text search could pass values outside the range [-1, 255] to `isspace()`, which is undefined behavior ([PR #7698](https://github.com/realm/realm-core/pull/7698), since the introduction of FTS in v13.0.0).
+
+### Breaking changes
+* Any `stitch_` prefixed fields in the `BsonDocument` returned from `app::User::custom_data()` are being renamed on the server to have a `baas_` prefix instead ([PR #7769](https://github.com/realm/realm-core/pull/7769)).
+
+### Compatibility
+* Fileformat: Generates files with format v24. Reads and automatically upgrade from fileformat v10. If you want to upgrade from an earlier file format version you will have to use RealmCore v13.x.y or earlier.
+
+-----------
+
+### Internals
+* Removed references to `stitch_` fields in access tokens in sync unit tests ([PR #7769](https://github.com/realm/realm-core/pull/7769)).
+* Added back iOS simulator testing to evergreen after Jenkins went away ([PR #7758](https://github.com/realm/realm-core/pull/7758)).
+* `realm-trawler -c` did not work on Realm using SyncClient history ([PR #7734](https://github.com/realm/realm-core/pull/7734)).
+* `File::Map`'s move constructor and assignment operator left `m_fd` unchanged, which appears to have never actually resulted in problems with how it was used ([PR #7698](https://github.com/realm/realm-core/pull/7698)).
+
+----------------------------------------------
+
 # 14.9.0 Release notes
 
 ### Enhancements
 * Report the originating error that caused a client reset to occur. ([#6154](https://github.com/realm/realm-core/issues/6154))
 
 ### Fixed
+* Add a missing file from the bid library to the android blueprint. (PR [#7738](https://github.com/realm/realm-core/pull/7738))
 * After compacting, a file upgrade would be triggered. This could cause loss of data if schema mode is SoftResetFile ([#7747](https://github.com/realm/realm-core/issues/7747), since 14.0.0)
 * Add missing `REALM_APP_SERVICES` flag to the android blueprint. (PR [#7755](https://github.com/realm/realm-core/pull/7755))
 

Package.swift

@@ -3,7 +3,7 @@
 import PackageDescription
 import Foundation
 
-let versionStr = "14.9.0"
+let versionStr = "14.10.0"
 let versionPieces = versionStr.split(separator: "-")
 let versionCompontents = versionPieces[0].split(separator: ".")
 let versionExtra = versionPieces.count > 1 ? versionPieces[1] : ""

dependencies.yml

@@ -1,5 +1,5 @@
 PACKAGE_NAME: realm-core
-VERSION: 14.9.0
+VERSION: 14.10.0
 OPENSSL_VERSION: 3.2.0
 ZLIB_VERSION: 1.2.13
 # https://github.com/10gen/baas/commits

src/realm/alloc.cpp

@@ -120,9 +120,7 @@ char* Allocator::translate_less_critical(RefTranslation* ref_translation_ptr, re
     RefTranslation& txl = ref_translation_ptr[idx];
     size_t offset = ref - get_section_base(idx);
     char* addr = txl.mapping_addr + offset;
-#if REALM_ENABLE_ENCRYPTION
-    realm::util::encryption_read_barrier(addr, NodeHeader::header_size, txl.encrypted_mapping, nullptr);
-#endif
+    util::encryption_read_barrier(addr, NodeHeader::header_size, txl.encrypted_mapping);
     // if we know the translation is inside the slab area, we don't need to check
     // for anything beyond the header, and we don't need to check if decryption is needed
     auto size = known_in_slab ? 8 : NodeHeader::get_byte_size_from_header(addr);
@@ -138,27 +136,21 @@ char* Allocator::translate_less_critical(RefTranslation* ref_translation_ptr, re
     }
     if (REALM_LIKELY(!crosses_mapping)) {
         // Array fits inside primary mapping, no new mapping needed.
-#if REALM_ENABLE_ENCRYPTION
-        realm::util::encryption_read_barrier(addr, size, txl.encrypted_mapping, nullptr);
-#endif
+        util::encryption_read_barrier(addr, size, txl.encrypted_mapping);
         return addr;
     }
-    else {
-        // we need a cross-over mapping. If one is already established, use that.
-        auto xover_mapping_addr = txl.xover_mapping_addr.load(std::memory_order_acquire);
-        if (!xover_mapping_addr) {
-            // we need to establish a xover mapping - or wait for another thread to finish
-            // establishing one:
-            const_cast<Allocator*>(this)->get_or_add_xover_mapping(txl, idx, offset, size);
-            // reload (can be relaxed since the call above synchronizes on a mutex)
-            xover_mapping_addr = txl.xover_mapping_addr.load(std::memory_order_relaxed);
-        }
-        // array is now known to be inside the established xover mapping:
-        addr = xover_mapping_addr + (offset - txl.xover_mapping_base);
-#if REALM_ENABLE_ENCRYPTION
-        realm::util::encryption_read_barrier(addr, size, txl.xover_encrypted_mapping, nullptr);
-#endif
-        return addr;
+    // we need a cross-over mapping. If one is already established, use that.
+    auto xover_mapping_addr = txl.xover_mapping_addr.load(std::memory_order_acquire);
+    if (!xover_mapping_addr) {
+        // we need to establish a xover mapping - or wait for another thread to finish
+        // establishing one:
+        const_cast<Allocator*>(this)->get_or_add_xover_mapping(txl, idx, offset, size);
+        // reload (can be relaxed since the call above synchronizes on a mutex)
+        xover_mapping_addr = txl.xover_mapping_addr.load(std::memory_order_relaxed);
     }
+    // array is now known to be inside the established xover mapping:
+    addr = xover_mapping_addr + (offset - txl.xover_mapping_base);
+    util::encryption_read_barrier(addr, size, txl.xover_encrypted_mapping);
+    return addr;
 }
 } // namespace realm

src/realm/alloc.hpp

@@ -174,7 +174,7 @@ class Allocator {
     // into equal chunks.
     struct RefTranslation {
         char* mapping_addr;
-        uint64_t cookie;
+        uint64_t cookie = 0x1234567890;
         std::atomic<size_t> lowest_possible_xover_offset = 0;
 
         // member 'xover_mapping_addr' is used for memory synchronization of the fields
@@ -186,14 +186,12 @@ class Allocator {
 #if REALM_ENABLE_ENCRYPTION
         util::EncryptedFileMapping* encrypted_mapping = nullptr;
         util::EncryptedFileMapping* xover_encrypted_mapping = nullptr;
+#else
+        static inline util::EncryptedFileMapping* const encrypted_mapping = nullptr;
+        static inline util::EncryptedFileMapping* const xover_encrypted_mapping = nullptr;
 #endif
-        explicit RefTranslation(char* addr)
+        explicit RefTranslation(char* addr = nullptr)
             : mapping_addr(addr)
-            , cookie(0x1234567890)
-        {
-        }
-        RefTranslation()
-            : RefTranslation(nullptr)
         {
         }
         ~RefTranslation()
@@ -225,7 +223,7 @@ class Allocator {
     };
     // This pointer may be changed concurrently with access, so make sure it is
     // atomic!
-    std::atomic<RefTranslation*> m_ref_translation_ptr;
+    std::atomic<RefTranslation*> m_ref_translation_ptr{nullptr};
 
     /// The specified size must be divisible by 8, and must not be
     /// zero.
@@ -255,7 +253,7 @@ class Allocator {
     char* translate_critical(RefTranslation*, ref_type ref, bool known_in_slab = false) const noexcept;
     char* translate_less_critical(RefTranslation*, ref_type ref, bool known_in_slab = false) const noexcept;
     virtual void get_or_add_xover_mapping(RefTranslation&, size_t, size_t, size_t) = 0;
-    Allocator() noexcept;
+    Allocator() noexcept = default;
     size_t get_section_index(size_t pos) const noexcept;
     inline size_t get_section_base(size_t index) const noexcept;
 
@@ -274,11 +272,9 @@ class Allocator {
     //   used to detect if the allocator (and owning structure, e.g. Table)
     //   is recycled. Mismatch on this counter will cause accesors
     //   lower in the hierarchy to throw if access is attempted.
-    std::atomic<uint_fast64_t> m_content_versioning_counter;
-
-    std::atomic<uint_fast64_t> m_storage_versioning_counter;
-
-    std::atomic<uint_fast64_t> m_instance_versioning_counter;
+    std::atomic<uint_fast64_t> m_content_versioning_counter{0};
+    std::atomic<uint_fast64_t> m_storage_versioning_counter{0};
+    std::atomic<uint_fast64_t> m_instance_versioning_counter{0};
 
     inline uint_fast64_t get_storage_version(uint64_t instance_version)
     {
@@ -550,14 +546,6 @@ inline bool Allocator::is_read_only(ref_type ref) const noexcept
     return ref < m_baseline.load(std::memory_order_relaxed);
 }
 
-inline Allocator::Allocator() noexcept
-{
-    m_content_versioning_counter = 0;
-    m_storage_versioning_counter = 0;
-    m_instance_versioning_counter = 0;
-    m_ref_translation_ptr = nullptr;
-}
-
 // performance critical part of the translation process. Less critical code is in translate_less_critical.
 inline char* Allocator::translate_critical(RefTranslation* ref_translation_ptr, ref_type ref,
                                            bool known_in_slab) const noexcept
@@ -570,30 +558,23 @@ inline char* Allocator::translate_critical(RefTranslation* ref_translation_ptr,
         if (REALM_LIKELY(offset < lowest_possible_xover_offset)) {
             // the lowest possible xover offset may grow concurrently, but that will not affect this code path
             char* addr = txl.mapping_addr + offset;
-#if REALM_ENABLE_ENCRYPTION
-            realm::util::encryption_read_barrier(addr, NodeHeader::header_size, txl.encrypted_mapping,
-                                                 NodeHeader::get_byte_size_from_header);
-#endif
+            util::encryption_read_barrier(addr, NodeHeader::header_size, txl.encrypted_mapping);
+            size_t size = NodeHeader::get_byte_size_from_header(addr);
+            util::encryption_read_barrier(addr, size, txl.encrypted_mapping);
             return addr;
         }
-        else {
-            // the lowest possible xover offset may grow concurrently, but that will be handled inside the call
-            return translate_less_critical(ref_translation_ptr, ref, known_in_slab);
-        }
+        // the lowest possible xover offset may grow concurrently, but that will be handled inside the call
+        return translate_less_critical(ref_translation_ptr, ref, known_in_slab);
     }
     realm::util::terminate("Invalid ref translation entry", __FILE__, __LINE__, txl.cookie, 0x1234567890, ref, idx);
-    return nullptr;
 }
 
 inline char* Allocator::translate(ref_type ref) const noexcept
 {
-    auto ref_translation_ptr = m_ref_translation_ptr.load(std::memory_order_acquire);
-    if (REALM_LIKELY(ref_translation_ptr)) {
-        return translate_critical(ref_translation_ptr, ref);
-    }
-    else {
-        return do_translate(ref);
+    if (auto ptr = m_ref_translation_ptr.load(std::memory_order_acquire); REALM_LIKELY(ptr)) {
+        return translate_critical(ptr, ref);
     }
+    return do_translate(ref);
 }
 
 inline char* Allocator::translate_in_slab(ref_type ref) const noexcept