Session & Transaction Support

[Turnerj]

Support two levels of transactions - transaction that wrap the SaveChanges call from the MongoDbContext and the internal batch write calls in the DbEntityWriter.

https://mongodb.github.io/mongo-csharp-driver/2.7/what_is_new/
https://mongodb.github.io/mongo-csharp-driver/2.7/reference/driver/crud/sessions_and_transactions/

Will need to maintain backwards compatibility when the MongoDB server isn't running v4

More investigation will be required on how best to implement this functionality.

[Turnerj]

Having a closer look at how transactions are supported in MongoDB, it currently only works on replica sets. Because of this, I'm delaying this functionality till there is support for standalone servers.

[Turnerj]

While transactions are still not supported on standalone servers, apparently it can work on a server with the inMemory storage provider as long as you use the right command line flag.

If this is possible and the CI can run tests with and without that flag enabled, it still might be doable.

That said, MongoFramework would need to detect whether transactions are supported and the API for it would need to be worked out. Perhaps transactions should be "hidden" from the user's view but creating a context creates a session which is used for all subsequent calls. This would give the desired result as long as the context is made aware when to "commit" the changes.

Alternatively, it could be a wrapping class around the context, put in a using statement, with its own SaveChanges call etc. It sets up the session and transaction, it calls the context to save, it then commits the transaction.

Getting the session to writer might be difficult - maybe another level of abstraction or some how having the IMongoDbConnection interface expose the session.

[dthk-cogmatix]

Here is how I've implemented quick and dirty session/transaction support:

1. MongoDBConnection holds a reference to a Current Session, as well as basic Session/Transaction handling methods.

private IClientSessionHandle _currentSession;
public IClientSessionHandle CurrentSession;
public void EndSession();
public async Task<IClientSessionHandle> BeginTransactionAsync();
public async Task CommitTransactionAsync();
public async Task RollbackTransactionAsync();

2. MongoDBContext wraps up the Connection (Just to make consumption friendly)

public bool HasActiveTransaction => this.Connection.CurrentSession != null;
public IClientSessionHandle GetCurrentTransaction();
public async Task<IClientSessionHandle> BeginTransactionAsync();
public async Task CommitTransactionAsync(IClientSessionHandle transaction);
public void RollbackTransaction();

3. Implement session detection in CommandWriter, EntityIndexWriter, MongoFrameworkQueryProvider. (Basically find any MongoClient method that can be functionally overloaded for a session).

Example

try
{
  if(Connection.CurrentSession == null)
    underlyingResult = GetCollection().Aggregate(pipeline).ToEnumerable();
  else
    underlyingResult = GetCollection().Aggregate(Connection.CurrentSession, pipeline).ToEnumerable();
}

I'm not advocating this be how MongoFramework implements it, but it definitely is a stop-gap measure until a more mature/robust solution is available.

This met my requirements of:

1. Handle transactional boundaries in a CQRS DDD project. I can begin a transaction on my first command, handle the command, dispatch events, handle the events, and perform any additional processing until everything has fired. Then I can commit the transaction.

2. Allows me to still call Save periodically to write commands to the server without committing. This way I can retrieve database generated IDs and "chunk" work up before final commitment.

Things to note:

1. This is implemented very similarly to how MongoDB added on support for sessions. All interaction is in an implicit session/transaction. If we don't pass any sessionID, it gets wrapped in the current implicit session and it's business as usual for the framework. If we explicitly create a new session (Like the above implementation), then we just pass the session alongside any of our operations that are in scope of the explicit transaction. This approach means the Framework isn't implement sessions and transactions internally, but gives the user the ability to determine when they need to explicitly create a transaction and expand the scope of the session.

2. I'm not trying to put logic in to determine whether the server supports transactions and do something differently. This may be necessary if internally the framework wanted to perform activities atomically (database prep/maintenance work), but for a consumer of the framework, I want to know when I explicitly request a transaction/session and have it fail when it's not supported (instead of internally hiding / swallowing this).

3. I'm not handling some of the complex scenarios and prioritizing operations. So if I have a UnitOfWork with collection creation mixed in with session/transaction behavior, I don't bubble the collection activities to execute first, outside of a session. This may not be that important, but there are caveats here I haven't explored deeply.

Observations/Opportunities:
I don't think MongoDBConnection is the place to handle the session. It works based on the current internals of MongoFramework, but I've also seen the connection passed around to a lot of "dead ends." I'm sure this is plumbing for future or remnants of legacy code. I think the MongoDBConnection concept should be revisited because MongoDBConnection houses a MongoClient. MongoClient really should be scoped as a single instance. By placing a reference to a CurrentSession with the Connection and the Client, it goes against guidance for usage of MongoClient. This all works, because the MongoClient internally handles connection pooling (assuming exact same connection strings used - which it is), but seems like we could be more intentful and explicit with usage.

[Turnerj]

Lots of good thoughts you've provided @dthk-cogmatix !

I'm thinking perhaps something like:

Some methods that actually create the session. The implementation behind IMongoDbSession basically wraps the MongoClient.

public interface IMongoDbConnection
{
  IMongoDbSession CreateSession();
  Task<IMongoDbSession> CreateSessionAsync();
}

Like you mentioned, a consumption friendly pattern.

public interface IMongoDbContext
{
  IMongoDbSession CreateSession();
  Task<IMongoDbSession> CreateSessionAsync();
}

I'm personally a fan of using the disposable to complete certain actions. It can be useful like if something throws an exception, the dispose method can then auto-rollback without needing your own try/catch setup. Maybe I'll make the dispose action configurable via enum or something.

public interface IMongoDbSession : IDisposable, IAsyncDisposable
{
  IMongoDbSessionReference Reference { get; }
  void Rollback(); //Sync disposable does a sync rollback
  Task RollbackAsync(); //Async disposable does an async rollback
  void Commit();
  Task CommitAsync();
}

Usage:

using (var session = myContext.CreateSessionAsync())
{
  myContext.MyDbSet.Where(a => true).WithSession(session.Reference);
  myContext.SaveChangesAsync(session.Reference);
}

Methods like SaveChanges would have an additional parameter taking in the IMongoDbSessionReference (I'm not personally attached to that name - it is a bit wordy). I've opted to pass this rather than IMongoDbSession as these methods shouldn't have access to the Commit or Rollback methods.

For the MongoDbQueryProvider, I'd have an extension method like WithSession(IMongoDbSessionReference session) which would set the session on the query.

What are your thoughts on something like that? I want to try and keep away from exposing the MongoDB driver underneath as I may step away from using it in the future.

[dthk-cogmatix]

I like what you've proposed and it's a much more thoughtful implementation of what I've mentioned above. I completely agree with wrapping the Session and usage of IDisposable. I think the only other add I would consider is putting an IsDisposed property on the Session? That way you can minimize having to catch and swallow ObjectDisposedExceptions?

[Turnerj]

Yeah, that's a fair call having a IsDisposed property - I could see that being useful if, for some reason, one couldn't use using directly for how they structured their code. Probably safer than not throwing ObjectDisposedException when the object is disposed.

[Turnerj]

With #156 landing which (primarily) flips the whole "heavy DbSet/light context" around, adding session/transaction support becomes easier.

That said, reviewing the latest documentation on MongoDB's site, there are some potential issues depending on the DB version:

• In MongoDB 4.2 and earlier, you cannot create collections in transactions. Write operations that result in document inserts (e.g. insert or update operations with upsert: true) must be on existing collections if run inside transactions.
• Starting in MongoDB 4.4, you can create collections in transactions implicitly or explicitly. You must use MongoDB drivers updated for 4.4, however. See Create Collections and Indexes In a Transaction for details.

Additionally...

Starting in MongoDB 4.4 with feature compatibility version (fcv) "4.4", you can create collections and indexes inside a multi-document transaction unless the transaction is a cross-shard write transaction.
[ ... ]
When creating an index inside a transaction [1], the index to create must be on either:

• a non-existing collection. The collection is created as part of the operation.
• a new empty collection created earlier in the same transaction.

Effectively, in certain MongoDB versions I can't create collections or indexes dynamically while in a session - kinda annoying if all your code is in sessions as nothing would ever work! When I can create collections or indexes in a session, I am restricted to where the indexes can apply.

Hoping that something post MongoDB v4.4 will address some of these things can make it a little less of a hassle. Unfortunately though, it would still mean potentially applying a minimum supported MongoDB version.

[dthk-cogmatix]

Thoughts from "the field."

I've been running a my own patched version of MongoFramework for the past 9 months where I've added session/transaction support. I've been using a Mongo Client pre 4.4 and have approached indexes and collection creation as a manual activity prior to code deployment. This is similar to (older) Entity Framework behavior, unless you specify in initialization to have Entity Framework handle the schema creation. I don't see a "huge" issue here, but it definitely doesn't feel as clean, polished, or robust.

Thoughts on how to handle this:
#1) Imitate Entity Framework behavior. Create initialization handlers/code that allow the developer to specify initialization behavior. Delete(), Create(), CreateIfNotExists(), EnsureCreated(), etc. It seems like the executing context of this code wouldn't really require a session. This seems like the appropriate time to check if Collections exist for all entities and if any necessary indexes are in place.

#2) Further simplify the process and have an option flag that is like: AutoCreateCollections. Document behavior and call it good enough for now. If someone is using client 4.4+ and sets this to true, no issue. If it's set to true and the client is 4.2, they will receive an exception during creation. Not terrible....

#3) Prioritize operations and executions: Pre-Transaction and during Transaction. If you're tracking all changes locally before submitting to the server, you could then prioritize execution of collections/indexes before beginning a transaction and sending to the server.

I'm not sure how the code in the latest version is looking, but that may dictate which approach is most appropriate.