pg18 refactors #722
Draft
pg18 refactors #722
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It doesn't have any inherently unsafe functions that allow users to do things they otherwise wouldn't be able to do so; except more efficiently.
By using the WAL record as source of the block number, we don't have to rely on buffer pinning, which doesn't work in the WAL-redo process.
The vanilla records don't have cid, which is required for persistence during PS WAL redo.
A sequence's page can be evicted quite quickly with low shared_buffers configurations, which then causes values to be skipped. This prevents that, at the cost of higher WAL volume when sequences are involved.
This allows us to use lower (on average) LSNs in requests to PS, thus increasing the WAL lag tolerance of the pageserver. Note that callers to smgr_end_unlogged_build don't have to update LwLsnCache, as that call internally handles it.
This prevent excessive USE_PREFETCH redefinition warnings if the compiler gets USE_PREFETCH from elsewhere with a different definition value.
We also move certain phases of the CheckPoint process ahead of the final checkpoint record during shutdown, to allow those processes to write out their changes into WAL.
Specifically, hash vacuum & btree index (only) scans Co-authored-by: Konstantin Knizhnik <[email protected]>
In neon.com, the privileged role will be neon_superuser, but other platforms may need a different role name, hence the parameter.
Used for Logical Replication and various other system state changes. Required to keep state synchronized with Pageserver's Basebackup output, as we don't have systems in place to make direct writes at an LSN possible. TODO: Enforce using a subscription's FAILOVER option instead for LR? Co-authored-by: Konstantin Knizhnik <[email protected]>
Without this, it may ack higher than it, losing data on endpoint restart.
This improves the scalability of Neon's WAL systems through bypassing the archiving systems, which would require out-of-process WAL requests.
Rather than access(), which doesn't work with Neon's non-local relfilenodes.
TODO: the logic around wal_redo + local buffers in other places too.
Co-authored-by: Heikki Linnakangas <[email protected]>
The feature is disabled by default, but can be enabled by setting the neon_pgstat_file_size_limit integer limit to a value that's not 0 (disabled) Co-authored-by: Heikki Linnakangas <[email protected]>
when we're in the process of dropping subscriptions inherited by a neon branch. Because these slots are still needed by the parent branch subscriptions. For regular slots we handle this by setting the slot_name to NONE before calling DROP SUBSCRIPTION, but tablesync slots are not exposed to SQL. rely on GUC disable_logical_replication_subscribers=true to know that we're in the Neon-specific process of dropping subscriptions.
If an extension's control file is present, but not all the files that we expected, we'll notify a sidecar process to download and install these files, after which we'll try accessing again. This allows faster starts and dynamic installation of various files without impacting the cold start time as much, as most extensions are not accessed during or immediately after startup. Author: Anastasia Lubennikova <[email protected]> Author: Tristan Partin <[email protected]>
This allows the smgr to make storage choices based on the relpersistence of the relation, which is critical in future commits. Committed separately to make rebasing & backpatching easier.
This allows other systems to use the type. Committed separately for ease of rebasing & conflict resolution.
Committed separately to make rebasing efforts easier in the future.
It enables relation builds to be bulk-logged after all relation pages have been filled. Committed separately for ease of rebasing.
This allows us to remove SLRU segments from Neon's minimal-sized basebackups.
Neon doesn't currently handle tablespaces in the way PostgreSQL expects, so the original tablespace.out would not allow all tests to succeed.
Specifically, --sync-safekeepers and --wal-redo
This makes neon's WalProposer work.
Used to ensure extensions aren't compatible-by-default between Neon and vanilla PostgreSQL - there are some important differences.
Not really used in PostgreSQL, but used for wal-redo in Pageserver.
Instead we'll crash and restart when this happens, allowing a generally better experience for normal users which don't get so close to the limits that they would cause issues.
Used by Databricks' authentication system to authenticate their users.
Changes include: - Removing smgr_[start/finish/end]_unlogged_build from f_smgr, making it a set of hooks. - Renaming smgr_read_slru_segment, and making it a hook The smgrrelation was never used, so the API didn't make much sense. - Auto-adjust smgr relation persistence based on buffer pin levels. This allows the smgr->which method to work better than it did before. - Adjust some places' passing of relpersistence, from rel->rd_rel->relpersistence, to smgr->relpersistence. This enables Neon's hot-switching of relpersistence - Add relpersistence to PendingRelDelete Also allow this setting to be changed on-the-fly; again, enabling Neon to do local buffered index builds. - Change buffers that are getting logged to BM_PERMANENT Again, locally buffered index builds - "Fix" various HeapAM redo methods We can't really read buffers yet in PG18 in wal-redo postgres, so we try to avoid that here. TODO: Find a proper solution to the WAL-redo PostgreSQL stuff - we're currently really screwed with the AIO system; it adds a lot of complexity in a system we'd prefer to keep as simple as possible. Maybe now is the time to finally improve our walredo process to stop relying on main PG's buffer infra and link in our own bufmgr.c code?
* pg hooks for online table (#24) Adding hooks for PG online table. - PreOnlineTableOp_hook - PostOnlineTableOp_hook - OnlineTableSecurityLabel_hook The 3 hooks are all used to bypass the ownership check on an online table. The Pre/Post hooks are used to allow non-online table owner to perform a few DDLs on the table, e.g., CREATE INDEX. - Pre hook will check the DDL type and whether the referenced relation is an online table. If the conditions are met, it will set the session user as the table owner. The rest of the PG code will work since the session is acting as the table owner now. - Post hook will restore the current user to the session user. OnlineTableSecurityLabel_hook is used to allow customers to add who can perform DDL on an online table. By default, the databricks_superuser can. OnlineTableSecurityLabel_hook will check if the session user is one of the roles attached to the online table. If it is, it will allow updating the label to add / remove roles. Otherwise, it will throw the permission error. Why we add them? 1. These 3 hooks are very specific for online table. We don't intend to use them for other purposes. 2. This is also the reason we are not overloading the public hook also. I need to be very specific on the code point where we want to set as table owner. standard_ProcessUtility contains too many stuff. I don't want to make our change generalize to all PG utilities. The extension change is inside https://github.com/databricks-eng/hadron/pull/492 * First pass of review. --------- Co-authored-by: Haoyu Huang <[email protected]>
The CI builds Postgres from the source code in this branch installed in the neon vendor/postgres-vXX directory, and then runs the Postgres regression tests and a very simple Neon test that just does CREATE EXTENSION neon. This allows making sure we can still build Neon with the current code in the branch, and load the neon shared library at the server's start-up. Actual neon testing is still maintained in the neon repository, the goal of this CI action is to have a minimum amount of feedback when contributing to the Postgres repository.
To be squashed into the relevant commit
Also removes the debug logging and assertions that were added to find the issue The issue was that we failed to correctly initialize the page if the page was the first page of a segment, and should thus have a long header.
MMeent
force-pushed
the
REL_18_STABLE_neon
branch
2 times, most recently
from
f6d9f02 to
eab5446
Compare
MMeent
force-pushed
the
REL_18_STABLE_neon
branch
from
0d78936 to
cf7506a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.