Implement AWS key value store #2883

ogghead · 2024-10-10T14:30:29Z

Hi folks! I am creating this draft PR to solicit feedback on an initial AWS key value store implementation. I appreciate any and all discussions on this PR!

Some points for thought:

Implementation uses DynamoDB, though for large blob storage, S3 is preferable (DynamoDB can only store <=400KB size records), see Add an S3 key/value storage provider interface #2606. DynamoDB is cheaper and faster for performing many rapid reads/writes of small amounts of data though, and is roughly in the same niche as Azure CosmosDB
Auth currently requires generating AWS STS token credentials and passing them to the Spin app in a runtime config file. https://github.com/spinkube/skips/pull/9/files discusses better patterns to fetch credentials. Curious to hear thoughts on how this implementation can integrate better with that proposal!
The Azure key-value implementation supports reading credentials from environment variables, however the AWS Rust SDK does not offer a synchronous API to load config and would require the MakeKeyValueStore::make_store function to be async for all implementations -- leading to a chain of async function coloring. It is possible to manually fill the SdkConfig object and I did this to pass STS tokens from a runtime config file, but it would be ideal to rely on the SDK's defaults and many credential loading fallbacks if possible. Curious on thoughts for how to best handle env var credential loading

itowlson · 2024-10-28T21:35:24Z

Kia ora @ogghead and thanks for this. We've got work going on in #2895 to implement some additional key-value interfaces, and I think it works better to land that one first, then have this PR include all the AWS stuff. This is partly down to what has the biggest compatibility implications combined with the present release timeline, but also will hopefully provide enough infrastructure to make extending this PR to the new interfaces easy! I hope that's okay with you.

In the meantime, I'll try to have a look at your points for thought!

itowlson · 2024-10-28T23:52:40Z

Implementation uses DynamoDB, though for large blob storage, S3 is preferable

I'd say this is fine for now. We could call this a DynamoDB KV store, which would leave us flexibility to later to add a S3 KV backend if users had large object use cases - nothing here would preclude that as long as we think of it as an "AWS product X" store rather than an "AWS" store.

Auth currently requires generating AWS STS token credentials

I don't think we are bound to offer a "tokens in the runtime config" option if that doesn't make sense or is painful to implement. I'm not sure we can rely on the workload identity idea from that SKIP across all Spin runtime environments, but absolutely open to doing things differently as appropriate.

it would be ideal to rely on the SDK's defaults and many credential loading fallbacks if possible

This is what we do in the SQS trigger. There's no credential configuration, we just load the SDK and let figure out the credentials, whether from ambient EVs or whatever. I'm told that's idiomatic enough, so I'd have no problem with doing the same thing here. I'm sure someone will shout out if they do - but we could presumably retrofit additional configuration methods if need be - the Cosmos one certainly went through a few sets of extensions...

ogghead · 2024-10-29T00:10:53Z

Kia ora @ogghead and thanks for this. We've got work going on in #2895 to implement some additional key-value interfaces, and I think it works better to land that one first, then have this PR include all the AWS stuff. This is partly down to what has the biggest compatibility implications combined with the present release timeline, but also will hopefully provide enough infrastructure to make extending this PR to the new interfaces easy! I hope that's okay with you.

In the meantime, I'll try to have a look at your points for thought!

Sounds good to me! I'm excited to see that work land. It makes sense to hold off on this for now, then rework after that is merged and support all WASI KV interfaces for AWS.

Thanks for taking a look!

ogghead · 2024-10-29T00:56:42Z

Implementation uses DynamoDB, though for large blob storage, S3 is preferable

I'd say this is fine for now. We could call this a DynamoDB KV store, which would leave us flexibility to later to add a S3 KV backend if users had large object use cases - nothing here would preclude that as long as we think of it as an "AWS product X" store rather than an "AWS" store.

Great callout! The config specifies "Dynamo" as the KV store type, so should hopefully be flexible to add other backends. I will keep this in mind when implementing the full WASI KV interface

Auth currently requires generating AWS STS token credentials

I don't think we are bound to offer a "tokens in the runtime config" option if that doesn't make sense or is painful to implement. I'm not sure we can rely on the workload identity idea from that SKIP across all Spin runtime environments, but absolutely open to doing things differently as appropriate.

The runtime config token setup is (from local testing) working -- but the "use the default SDK config loading" is proving challenging with the current interface constraints. Mainly as the default AWS config loader is implemented as an async function.

it would be ideal to rely on the SDK's defaults and many credential loading fallbacks if possible

This is what we do in the SQS trigger. There's no credential configuration, we just load the SDK and let figure out the credentials, whether from ambient EVs or whatever. I'm told that's idiomatic enough, so I'd have no problem with doing the same thing here. I'm sure someone will shout out if they do - but we could presumably retrofit additional configuration methods if need be - the Cosmos one certainly went through a few sets of extensions...

Agreed, this is the pattern I followed in the Kinesis trigger as well. I would love to have this here too! The challenge is function coloring from AWS config default loader function -- using that async function here appeared to require a chain of refactoring across the general KV traits, but I must admit that my async Rust knowledge hit a wall when trying to reconcile the changes required for that.

itowlson · 2024-10-29T01:33:49Z

Ah, I misread your comment about using the default SDK config - sorry about that.

It seems like you could call Tokio's block_on to wrap the async call in a blocking wrapper. It's not going to win any prizes for elegance but should, hopefully, get the job done. The section "A synchronous interface to mini-redis" on the Tokio "Bridging to sync code" page seems like it might be close to what you want, although you may well have tried that already? We do have some Giant Async Brains floating around who might be able to help if you share what you tried and what you ran into.

ogghead · 2024-10-29T15:02:21Z

Ah, I misread your comment about using the default SDK config - sorry about that.

It seems like you could call Tokio's block_on to wrap the async call in a blocking wrapper. It's not going to win any prizes for elegance but should, hopefully, get the job done. The section "A synchronous interface to mini-redis" on the Tokio "Bridging to sync code" page seems like it might be close to what you want, although you may well have tried that already? We do have some Giant Async Brains floating around who might be able to help if you share what you tried and what you ran into.

Good callout -- I tried using block_on to get this working with

Handle::current().block_on(aws_config::load_defaults(BehaviorVersion::latest()))

as well as

let rt = tokio::runtime::Builder::new_current_thread()
                    .enable_all()
                    .build()?;
 rt.block_on(aws_config::load_defaults(BehaviorVersion::latest()))

While these do compile, I see crashes immediately on Spin app startup with

Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks.

Alternatively, when I went down the path of asyncifying everything required to await this function, I hit a wall at store_from_toml_fn where closures are returned. Async closure enhancements in Rust might be needed to make the closures returned there async, but this is where my knowledge of async Rust was lacking.

If any Giant Async Brains (or anyone) have ideas on the best path forward on this, much appreciated!

itowlson · 2024-10-29T20:37:48Z

All right. I think I have a way around this for you. "But," in the words of Deep Thought, "you're not going to like it."

So I asked the Giant Async Brains about blocking on load_defaults and they said "don't do that." Instead the idea is to create a future for the Client and capture that in a lazy or once-cell instead. Then await it each time you want to use it (which will be cheap after the first time, especially compared to the network activity that follows).

Here is what I did, which seems to work (but you may find a less awful way, this was just the first stab that didn't make the compiler mad at me):

Add the async-once-cell crate
Change KeyValueAwsDynamo::client to be a (deep breath) async_once_cell::Lazy<Client, std::pin::Pin<Box<dyn std::future::Future<Output = Client> + Send>>>
In KeyValueAwsDynamo::new, shunt the existing code into an async move. Then Box::pin the async move. And capture that as a future. Then put that into a Lazy::from_future. So it looks like:

        let client_fut: std::pin::Pin<Box<dyn std::future::Future<Output = Client> + Send>> = Box::pin(async move {
            let config = match auth_options {
                KeyValueAwsDynamoAuthOptions::RuntimeConfigValues(config) => /* as current */,
                KeyValueAwsDynamoAuthOptions::Environmental => {
                    aws_config::load_defaults(BehaviorVersion::latest()).await  // as before but uncommented
                }
            };
            Client::new(&config)
        });

        let client_cell = async_once_cell::Lazy::from_future(client_fut);

        Ok(Self { client: client_cell, table })

(Some of the naming here is poor, this was throwaway code.)

Change StoreManager::get to get_unpin().await the Lazy:

    async fn get(&self, name: &str) -> Result<Arc<dyn Store>, Error> {
        Ok(Arc::new(AwsDynamoStore {
            _name: name.to_owned(),
            client: self.client.get_unpin().await.clone(),  // <-- this bit
            table: self.table.clone(),
        }))
    }

NOTE: this breaks StoreManager::summary. You'll need to either make that async or capture the summary info as extra fields, but this should be routine. (I hope. I admit punting on this.)

Let me know if you need more info or want a proper diff.

ogghead · 2024-10-29T22:26:54Z

Excellent! This is exactly the Galaxy Async Brain thinking I was sorely lacking 😄

I will give this a go tonight, thanks for the tips!

ogghead · 2024-10-30T03:54:17Z

Can confirm this worked like a charm! Pushed the changes to reflect and I will keep an eye on the full WASI KV implementation PR. I am in your debt for your help on this @itowlson :)

One does not simply walk into async_once_cell::Lazy<Client, std::pin::Pin<Box<dyn std::future::Future<Output = Client> + Send>>> (or Mordor)

itowlson · 2024-10-30T04:33:06Z

I'm delighted to have helped! Thanks once again for your effort, your patience, and your good humour throughout this...

...

...because you will need them when I call that debt in. ominous music and cheesy lightning FX in which the viewer can vaguely make out the looming shape of wasi:blobstore

(also, and at the risk of bathos, please ignore MQTT CI failures - it's a known flake)

ogghead · 2024-11-08T14:23:22Z

I have implemented atomic and batch operations -- the logic is tentatively all in place. I will test the operations with a component using WASI-KV and then (barring any uncovered issues) mark this ready for review

itowlson

Thanks for this! I'm not very qualified to review the Dynamo stuff but @endocrimes has kindly volunteered to look at it. In the meantime just a few comments and questions.

itowlson · 2024-11-11T22:11:46Z

crates/key-value-aws/src/store.rs

+}
+
+struct AwsDynamoStore {
+    _name: String,


Is the underscore because this is dead code? (I appreciate this has been through a lot of iteration and maybe it got lost in the process!) If this is needed but never used, it would be good to comment why it's needed; if it's not needed, maybe remove it?

Indeed, this appears to be dead -- further investigation shows it may also be dead in the Azure implementation (likely where I utilized my brilliant ctrl+c, ctrl+v technique) -- I went ahead and removed this in both implementations

itowlson · 2024-11-11T22:14:28Z

crates/key-value-aws/src/store.rs

+use spin_factor_key_value::{log_error, Cas, Error, Store, StoreManager, SwapError};
+
+pub struct KeyValueAwsDynamo {
+    table: Arc<String>,


It wasn't clear to me why these were Arc given that client isn't. Maybe merits a comment?

That is fair -- the reasoning for client not being Arc is that client already wraps an Arc:

pub struct Client { handle: Arc<Handle>, }

It should be low cost to clone the client without another Arc was my initial thought, but definitely open to thoughts! I will add a comment to this effect

Yeah, it would have made sense to me if KeyValueAwsDynamo was Clone, but I didn't see a Clone implementation. And although Client is clone, I don't think async_once_cell::Lazy<Client, std:something::Terrifying<...>> is, which now makes me wonder if this might be a hangover from a previous iteration?

You are quite correct -- cloning these fields is only done in StoreManager::get and StoreManager::summary and the overall object is never cloned. But, I now realize that the get function itself returns an Arc -- so this may be unnecessary overhead.

In the Store implementation for AwsDynamoStore though, creating CAS handles through the new_compare_and_swap function made it seem ideal to use Arcs/low cost cloned objects for the client and table to allow creating many parallel CAS handles at low cost. That doesn't preclude containing owned data in the KeyValueAwsDynamo and creating an Arc for table in StoreManager::get though, would that be preferable?

I agree that the considerations for AwsDynamoStore are different - it was specifically this KeyValueAwsDynamo struct that was puzzling me. But yeah if the expectation is that these fields will be repeatedly cloned then it makes sense to Arc them once here. And with client, as you say, it's already cheap to await (except for the first time) and clone the result. Thanks for the patient explanation - I'm happy now.

It is certainly possible that this is premature optimization and caching of the store is done in a table higher in the orchestration of key-value factors -- open to tweaking this! I appreciate the discussion :) It does look like we could get away without making region an Arc as it is never cloned (outside of formatting a string of course), so going to change that to a String in KeyValueAwsDynamo

itowlson · 2024-11-11T22:36:17Z

crates/key-value-aws/src/store.rs

+    }
+
+    async fn exists(&self, key: &str) -> Result<bool, Error> {
+        Ok(self.get_item(key).await?.is_some())


This looks like it fully downloads the value if present. Is that necessary just to check if the key exists? (It's fine if the answer is "yes" - this is me being ignorant about Dynamo. But if existence checks have time and egress cost implications that we might want to capture those in the docs. And now you have me wondering if the same applies to Cosmos...!)

This is a good callout -- while I do not know of a specific operation meant to check whether an item with a specific key exists in DynamoDB, we can definitely avoid downloading the entire value by returning only a specific key (in this case, just the PK). I will update with this optimization, but I cannot speak for whether CosmosDB supports something similar

Sorry, the Cosmos mention was more a note to self to see what it does and put a note in the docs if it downloaded potentially large data. Definitely not trying to put that on your plate.

ogghead · 2024-11-12T01:09:02Z

crates/key-value-aws/src/store.rs

+impl Cas for CompareAndSwap {
+    async fn current(&self) -> Result<Option<Vec<u8>>, Error> {
+        // TransactGetItems fails if concurrent writes are in progress on an item
+        let output = self


Thinking aloud -- this can be brought closer to ensuring a unique lock: TransactWrite can return the VAL key while setting a lock key -- only under the condition that the lock key doesn't already exist. Combined with deleting the lock key on swap, this could guarantee that only one process can acquire "the lock" assuming all processes call current before swap. However, this does not prevent another process calling swap without first calling current and ignoring the lock. But, it may still be worth making this change to get as close as possible to a transaction -- curious to hear thoughts on the best approach for this!

Scratch that, that is only possible for the UpdateItem operation and not the Update inside a TransactWrite. I suspect using UpdateItem will actually work fine here but checking to confirm behavior is idempotent

Indeed it appears to be, so I will make this change which hopefully drastically simplify the calls in CAS

endocrimes

Dynamo uses eventually consistent reads by default - which is fine for applications written against Dynamo, but I believe our Spin KV contract is that you at least read-your-writes. Dynamo charging 2x the cost for a Consistent Read makes this a compelling tradeoff regardless, but we may need to document that fact (and potentially make ConsistentReads an opt-in configuration?)

Otherwise this seems reasonably sound to me (modulo questions about CAS) - thanks for the PR!

endocrimes · 2024-11-12T11:01:12Z

crates/key-value-aws/src/store.rs

+                .table_name(self.table.as_str())
+                .projection_expression(PK);
+
+            if let Some(keys) = last_evaluated_key {


This would potentially be slightly easier to read with the SDK's paginator (https://docs.rs/aws-sdk-dynamodb/latest/aws_sdk_dynamodb/operation/scan/builders/struct.ScanFluentBuilder.html#method.into_paginator) - but I'm extremely not a Rustacean 😅 (and this otherwise is doing ~the same thing afaict)

Good point -- I was unaware of this utility! I will look into replacing custom logic with the paginator.

endocrimes · 2024-11-12T11:50:04Z

crates/key-value-aws/src/store.rs

+}
+
+#[async_trait]
+impl Cas for CompareAndSwap {


I'm a little curious about the CAS impl here - With the lock attribute not expiring (afaik?), it seems like there are a few crash/process-killing/bad-usage cases here that could result in an item being forever locked?

Could it not potentially be better served with a Consistent Read for current, and then a Conditional write for the swap? something like the cli:

aws dynamodb update-item \ --table-name MyTable \ --key '...' \ --update-expression "SET VAL = :newval" \ --condition-expression "VAL = :currval"

That would also make it easier to understand what happens in race cases if the same key is both cas'd and set/set-many concurrently.

But if there's a common dynamo pattern I've missed please definitely let me know 😅

Agreed -- there is definitely a situation where crashes could leave the lock sitting there -- my hope was that this is minimal as:

All non-atomic writes use Put (erasing the lock)

Swapping the value deletes the lock regardless of whether it was acquired

But, if someone calls current, then experienced hardware failure or a business logic crash, and then tries to rerun the whole component logic, they could find themselves in this bad state where the lock has not been released. I may have been overly hasty to refactor the CAS to hold a unique lock on data rather than using an optimistic lock with a version key. Your solution is elegant -- my only concern with using the VAL itself for conditional update was increased cost of sending it over the wire. I was previously using a version key for this and I think it's possible to use that in an update operation to do this comparison at lower cost: cache the incremented version key during current and then assert it is the same during swap.

I'll play around with a few options for optimistic locking on this today

ogghead · 2024-11-12T15:23:52Z

Dynamo uses eventually consistent reads by default - which is fine for applications written against Dynamo, but I believe our Spin KV contract is that you at least read-your-writes. Dynamo charging 2x the cost for a Consistent Read makes this a compelling tradeoff regardless, but we may need to document that fact (and potentially make ConsistentReads an opt-in configuration?)

Otherwise this seems reasonably sound to me (modulo questions about CAS) - thanks for the PR!

Good point! I was mulling over whether to add a configuration to specify strongly consistent reads, and this comment makes it clear that would be good to have (and potentially default to strongly consistent to maintain consistency of behavior with other implementations?) It should be quick to add, will push that up either this morning or tonight

ogghead · 2024-11-13T05:03:14Z

Ok! I may have gotten a bit pedantic with the states for Cas but this is hopefully in line with what you were imagining @endocrimes and cleans up. That said, I uncovered some odd behavior while testing and published an "int test Spin app" repo here for reference

For AWS, I observed panics here

Caused by:
    0: error while executing at wasm backtrace:
           0: 0x4812a - wit-component:shim!indirect-wasi:keyvalue/[email protected]
           1: 0xf034 - test_dynamo.wasm!spin_sdk::wit::wasi::keyvalue::atomics::swap::hfb1256b17b155b22
           2: 0x64fc - test_dynamo.wasm!test_dynamo::handle_test_dynamo::hb0dc3a6709598c91
           3: 0x77ba - test_dynamo.wasm!spin_executor::run::h6238cdfeaf27259c
           4: 0x954c - test_dynamo.wasm!wasi:http/[email protected]#handle
    1: CasError::CasFailed(Resource { rep: 4, state: "own (not in table)" })

For sanity, I tested with sqlite to see the "reference behavior" but there I hit panics here

Caused by:
    0: error while executing at wasm backtrace:
           0: 0x1bd0e - test_dynamo.wasm!__rust_start_panic
           1: 0x1bba7 - test_dynamo.wasm!rust_panic
           2: 0x1bad5 - test_dynamo.wasm!std::panicking::rust_panic_with_hook::h6e665b71c8f50b27
           3: 0x1ac0f - test_dynamo.wasm!std::panicking::begin_panic_handler::{{closure}}::hb63ceb92de73cef0
           4: 0x1ab75 - test_dynamo.wasm!std::sys::backtrace::__rust_end_short_backtrace::hf18948010daec5d9
           5: 0x1b3b1 - test_dynamo.wasm!rust_begin_unwind
           6: 0x299bf - test_dynamo.wasm!core::panicking::panic_fmt::h7916dc01f99baff2
           7: 0x2b1cd - test_dynamo.wasm!core::panicking::assert_failed_inner::h636831bd2590e950
           8: 0x3538 - test_dynamo.wasm!core::panicking::assert_failed::h4b4824a9d77e94de
           9: 0x51e8 - test_dynamo.wasm!test_dynamo::handle_test_dynamo::hb0dc3a6709598c91
          10: 0x77ba - test_dynamo.wasm!spin_executor::run::h6238cdfeaf27259c
          11: 0x954c - test_dynamo.wasm!wasi:http/[email protected]#handle
    1: wasm trap: wasm `unreachable` instruction executed

It is certainly possible that these issues are caused by my own environment but the AWS one appears to potentially be rooted in higher level orchestration logic around atomic retries here

rylev · 2024-11-13T09:11:15Z

crates/key-value-aws/src/store.rs

+    }
+
+    async fn delete(&self, key: &str) -> Result<(), Error> {
+        if self.exists(key).await? {


Why do we need to check for key existence? Does client.delete_item fail if the item doesn't exist? If so, we still have a race condition between the call to exists and delete_item where the item might be deleted and the error could occur.

Good point! It appears that the delete operation only fails if a condition is set. I will remove this check as the API does not throw errors when running delete on nonexistent items

rylev · 2024-11-13T09:13:08Z

crates/key-value-aws/src/store.rs

+        let mut results = Vec::with_capacity(keys.len());
+
+        if keys.is_empty() {
+            return Ok(results);
+        }


Nit: with_capacity does allocation which we can avoid in the empty key case by moving the initialization down.

Suggested change

let mut results = Vec::with_capacity(keys.len());

if keys.is_empty() {

return Ok(results);

}

if keys.is_empty() {

return Ok(Vec::new());

}

let mut results = Vec::with_capacity(keys.len());

Fair, I may be missing some finer details of Vec memory allocation but the docs seem to imply that Vec::new and Vec::with_capacity(0) should behave the same:

However, the pointer might not actually point to allocated memory. In particular, if you construct a Vec with capacity 0 via Vec::new, vec![], Vec::with_capacity(0), or by calling shrink_to_fit on an empty Vec, it will not allocate memory.

Regardless, I will make this change to ensure no allocations occur in this case!

Actually, this got me thinking -- would it be better to include these empty list checks here in the KV host implementation? That would ensure consistency across all implementations when handling empty lists for batch operations. I am imagining adding this check after fetching the store on 285 (so that permissions can be checked still) -- as well as on other batch operations there. Let me know if that is something that would be desirable, otherwise I can make this change here

As a side note, I see potentially unnecessary vec allocations at that level

Signed-off-by: Darwin Boersma <[email protected]>

…ons easier Signed-off-by: Darwin Boersma <[email protected]>

Signed-off-by: Darwin Boersma <[email protected]>

…returned values for getItem calls Signed-off-by: Darwin Boersma <[email protected]>

…istency Signed-off-by: Darwin Boersma <[email protected]>

itowlson · 2024-11-13T19:24:55Z

cc @devigned for the possible issue in the SQLite (or host?) implementation (#2883 (comment))

devigned · 2024-11-13T21:46:17Z

cc @devigned for the possible issue in the SQLite (or host?) implementation (#2883 (comment))

I'm at KubeCon right now, but I will try to give it a look when I have a break. At first glance, it seems like the CAS resource is not registered in the CAS resource table. The test code looked correct to me, so this is likely a bug in the CAS implementation.

…needed exists check, higher level filtering of empty get_all queries, sqlite handle null value before swap Signed-off-by: Darwin Boersma <[email protected]>

ogghead · 2024-11-13T22:17:16Z

cc @devigned for the possible issue in the SQLite (or host?) implementation (#2883 (comment))

I'm at KubeCon right now, but I will try to give it a look when I have a break. At first glance, it seems like the CAS resource is not registered in the CAS resource table. The test code looked correct to me, so this is likely a bug in the CAS implementation.

I appreciate the quick responses! On lunch so pushed up a few fixes for latest comments. I did some additional testing and the issue on sqlite get_many does appear to be inconsistent and I can reproduce with AWS too -- I have a hunch that caching of deleted values could be involved as adding sleeps in between delete_many and get_many operations allows consistent passes of that test for sqlite.

I did end up adding one fallback in the sqlite implementation in swap for null old_value as I was seeing an edge case crash here, please let me know if the modified behavior isn't desirable! With this change in place, I can confirm that both sqlite and AWS crash at the same place -- though I have yet to trace why the error including the new CAS isn't being returned user-side and instead logs/crashes somewhere

…d to client Signed-off-by: Darwin Boersma <[email protected]>

Signed-off-by: Darwin Boersma <[email protected]>

ogghead · 2024-11-14T14:35:58Z

Ok, I have solved all the observed issues:

Added CasError to this mapping here and tweaked the definition of swap here to return a Result<(), CasError> directly rather than wrapped in anyhow::Error -- this allowed the CasError to be passed back to the client component without hitting a trap
Flushed state at the start of get_many following this pattern -- this ensures there are no cached deleted objects during get_many

Pushed up fixes as well as validated that my test app passes all situations with AWS -- the sqlite implementation passes all situations barring the last one "Two handles, read nonexistent object, only one writes successfully" -- it might need a custom enum to capture the difference between an unknown CAS object and one that was fetched/is expected to be None -- but I could be misinterpreting the expected CAS behavior on an unknown object, let me know if we should actually treat unknown previous object as expected to not exist in database at all.

I appreciate all the great feedback on this PR!

ogghead force-pushed the dynamo-key-value-store branch from 74e7705 to 0ecf501 Compare October 30, 2024 03:43

ogghead force-pushed the dynamo-key-value-store branch from 891cdc7 to 917f811 Compare November 5, 2024 14:43

ogghead force-pushed the dynamo-key-value-store branch from d7b5ec1 to 2e71a72 Compare November 10, 2024 20:34

ogghead marked this pull request as ready for review November 11, 2024 15:27

ogghead mentioned this pull request Nov 11, 2024

compare and swap design WebAssembly/wasi-keyvalue#53

Open

itowlson requested a review from endocrimes November 11, 2024 22:23

itowlson reviewed Nov 11, 2024

View reviewed changes

ogghead commented Nov 12, 2024

View reviewed changes

endocrimes reviewed Nov 12, 2024

View reviewed changes

rylev reviewed Nov 13, 2024

View reviewed changes

ogghead added 7 commits November 13, 2024 07:27

Add AWS key value store

12f3dad

Signed-off-by: Darwin Boersma <[email protected]>

Update to DynamoKeyValueStore to make other AWS KV store implementati…

3b6e6aa

…ons easier Signed-off-by: Darwin Boersma <[email protected]>

Compiling with todos for new functionality

74cfa2b

Signed-off-by: Darwin Boersma <[email protected]>

Implemented first draft batch operations

ef2e26e

Signed-off-by: Darwin Boersma <[email protected]>

More updates, partial CAS implementation

2460378

Signed-off-by: Darwin Boersma <[email protected]>

CAS implementation

c3d587b

Signed-off-by: Darwin Boersma <[email protected]>

Updates from testing

2400dc0

Signed-off-by: Darwin Boersma <[email protected]>

ogghead added 3 commits November 13, 2024 07:27

Final updates to ensure better atomicity

e805726

Signed-off-by: Darwin Boersma <[email protected]>

Removed arcs, adjusted atomic functions to use updateItem, minimized …

de4f334

…returned values for getItem calls Signed-off-by: Darwin Boersma <[email protected]>

enum for CAS states, use paginator, add configuration for strong cons…

ed7299f

…istency Signed-off-by: Darwin Boersma <[email protected]>

Use transaction in increment and swap for better atomicity, remove un…

1cc913c

…needed exists check, higher level filtering of empty get_all queries, sqlite handle null value before swap Signed-off-by: Darwin Boersma <[email protected]>

ogghead force-pushed the dynamo-key-value-store branch from 3e9c8e5 to 1cc913c Compare November 13, 2024 21:57

Updated to handle CasError in wasm mapping, ensure new cas is returne…

8db1174

…d to client Signed-off-by: Darwin Boersma <[email protected]>

ogghead force-pushed the dynamo-key-value-store branch from ca9edbb to 8db1174 Compare November 14, 2024 14:06

Flush in get_many to ensure updated cache

6d8570b

Signed-off-by: Darwin Boersma <[email protected]>

Implement AWS key value store #2883

Are you sure you want to change the base?

Implement AWS key value store #2883

Conversation

ogghead commented Oct 10, 2024

itowlson commented Oct 28, 2024

itowlson commented Oct 28, 2024

ogghead commented Oct 29, 2024

ogghead commented Oct 29, 2024

itowlson commented Oct 29, 2024

ogghead commented Oct 29, 2024

itowlson commented Oct 29, 2024

ogghead commented Oct 29, 2024

ogghead commented Oct 30, 2024

itowlson commented Oct 30, 2024

ogghead commented Nov 8, 2024

itowlson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ogghead Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ogghead Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ogghead Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ogghead Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

endocrimes left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ogghead commented Nov 12, 2024

ogghead commented Nov 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itowlson commented Nov 13, 2024 • edited Loading

devigned commented Nov 13, 2024 • edited Loading

ogghead commented Nov 13, 2024

ogghead commented Nov 14, 2024

ogghead Nov 12, 2024 •

edited

Loading

ogghead Nov 12, 2024 •

edited

Loading

ogghead Nov 12, 2024 •

edited

Loading

ogghead Nov 12, 2024 •

edited

Loading

endocrimes left a comment •

edited

Loading

itowlson commented Nov 13, 2024 •

edited

Loading

devigned commented Nov 13, 2024 •

edited

Loading