Support for Turso

[FleetAdmiralJakob]

The connection to the turso database seems to consist out of the db URL and the the db auth token. diesel only allows the connection via one db URL with includes both. how can I connect my turso db with diesel?

[weiznich]

I've never used turso, so I cannot answer that.

The official answer here is: Diesel officially supports SQLite, PostgreSQL and MySQL as backends. These backends are tested and expected to work.

Everything else is not supported. It might work or it might not, depending on what the other database does. If it doesn't work the correct way to make it work is to implement a custom connection type (maybe including a custom backend) that handles the specific requirements of that database as third party crate.

[weiznich]

I don't have a detailed design yet, but the grand idea is something like the following:

diesel.toml contains a field extensions = ["extension_name", "/path/to/extension"] which contains references to CLI applications that can be used as extension.

diesel-cli then calls on startup each of these extensions via std::process::Command and passes a json request to them to query capabilities.

I have the following capabilities in mind:

• Custom backend (basically registers that a certain URL prefix should be forwarded to the extension) for diesel migration run/revert/…, diesel print-schema
• Custom migration harness (basically registers that this extensions manages "migrations" for the cli)
• Schema generation hook (the extension gets the schema as input and can generate code based on this, possibly also for tweaking the type mapping?)

We would likely want to have a diesel-cli-extension crate which defines the data structures used for the json IPC and ideally provides traits that extensions can implement for each capability and that hide the boilerplate code of registering the actual extension and managing the communication. In diesel-cli we likely can use the same crate to pass in the data and call out to the extension. We also should take care of making the used communication datastructures somewhat forward/backward compatible so that the extension version doesn't necessarily need to be on par with the diesel core version.

Now that's the end goal to reach, which means a starting version should likely concentrate only on one capability as additional ones can then be added step by step as required.

[najtin]

Thanks for sharing your thoughts. Regarding the api you proposed three interfaces. I list the corresponding traits below each item.

• Custom backend (basically registers that a certain URL prefix should be forwarded to the extension) for diesel migration run/revert/…, diesel print-schema

• Backend, Connection
• Backend depends on many other Traits

• Custom migration harness (basically registers that this extensions manages "migrations" for the cli)

• MigrationHarness<DB: Backend>

• Schema generation hook (the extension gets the schema as input and can generate code based on this, possibly also for tweaking the type mapping?)

• I couldn't find any trait for this. It seems like this is currently handled with pattern matching e.g. here diesel_cli/src/infer_schema_internals/inference.rs#193
• I think it would be simplify things in the future if a corresponding trait would be created and used instead.

We would likely want to have a diesel-cli-extension crate which defines the data structures used for the json IPC and ideally provides traits that extensions can implement for each capability and that hide the boilerplate code of registering the actual extension and managing the communication. In diesel-cli we likely can use the same crate to pass in the data and call out to the extension.

I agree with that vision.

We also should take care of making the used communication datastructures somewhat forward/backward compatible so that the extension version doesn't necessarily need to be on par with the diesel core version.

I agree with this, but I don't think there should be separate interface for extensions. I think the existing traits should mark the boundary. The main reason for this is that the person that implements the custom backend already has to implement a number of traits from diesel. It would only create additional work if these had to be implemented twice for different interfaces in diesel_cli.

Now that's the end goal to reach, which means a starting version should likely concentrate only on one capability as additional ones can then be added step by step as required.

I want to mention that to implement a custom backend, we have to touch the schema generation code anyways. So I suggest we focus on the following traits: Backend, Connection and the to be created schema_generation trait. Ideally, every function in a trait could then be mapped to a corresponding json rpc/ipc.

There is a big problem with this approach. Traits can not be implemented at runtime. This is a problem because of traits like like SqlDialect or TypeMetadata, which Backend depends on.

pub trait SqlDialect: TrustedBackend {
    type ReturningClause;
    type OnConflictClause;
    //...
}

pub trait TypeMetadata {
    type TypeMetadata;
    type MetadataLookup: ?Sized;
}

There is just no way to implement traits at runtime. So the only chance to retain full compatibility would be to add a dynamic way to call the methods that are currently generated at compile time depending on these types. Here is an example for a function whose generation depends on the type of SqlDialect:

impl<T, DB, const N: usize> CanInsertInSingleQuery<DB> for [T; N]
where
DB: Backend+ SqlDialect<InsertWithDefaultKeyword = sql_dialect::default_keyword_for_insert::IsoSqlDefaultKeyword>
{
    fn rows_to_insert(&self) -> Option<usize> {
        Some(N)
    }
}

As of right know I have no idea how to address this issue, but I will try to come up with something.

MetadataLookup here is also problematic for another reason. It serves a double purpose as an indicator and as a way to access the LookUp function. I think it should be split into the indicator and the implementation of a lookup function.

pub trait TypeMetadata {
    type TypeMetadata;
    type MetadataLookupIndicator: ?Sized;
    
    /// Determine the type metadata for the given `type_name` (from postgres)
    fn lookup_type(&mut self, type_name: &str, schema: Option<&str>) -> TypeMetadata {
        unimplemented!()
    }
}

[weiznich]

The traits in diesel are relatively decoupled from what we would need in diesel-cli. All the traits you listed are relevant for a diesel integration, but not for a diesel-cli integration.

Also to be clear here: There are currently no publicly useful traits in diesel-cli as it is currently not designed to handle this kind of extension. Designing and implementing that would be the main work here.

That what I have in mind is that a potential diesel-cli-extension crate would for example provide a interface like this (that's just a placeholder for now, someone needs to do the real design):

enum ExtensionCapablities {
     CustomBackend{url_prefix: String} // e.g. `turso://`
     CustomMigrationHarness, 
     /* Whatever else we come up with */
}

trait DieselCliExtension {
     fn capabilties(&self) -> ExtensionCapabilities;
}

struct PrintSchemaData {
    tables: TableData,
}

trait DieselCliExtensionCustomBackend {
    // exact methods need to be designed
    fn print_schema(&self, url: String) -> Result<PrintSchemaData>;
    fn run_migration(&self, url: String, content: String) -> Result<()>;
}
// other traits for other capabilities

Then diesel-cli-extension would provide a function/whatever to register an extension in your custom extension binary main function. That would expose the corresponding command line interface + JSON RPC for communicating with diesel. By that I mean that we just have a few subcommands that accept a json input (for the function arguments) and provide a json output (for the results).

diesel-cli itself would then implement these traits for it's build-in backends + have another impl that internally calls the extensions via std::process::Command. That would internally again "just" serialize the arguments and deserialize the results.

Overall I hope that this would decouple the main CLI binary from the extensions by allowing them to step forward as they want. Also with that you don't need to implement any traits on runtime.

[najtin]

Thanks for your reply. What I was advocating for was that there is a single way to extend diesel and diesel_cli, so that extension maintainers don't have to implement and care about two completely different ways of interacting with diesel. However, diesel can currently not be extended in a way that enables RPC to an (at compile time) unknown backend because there is no way to implement traits at runtime. Workarounds, like enumerating all possible types before hand, are infeasible due to the number of types.

Realizing that there is no easy solution to get a unified interface for diesel and diesel_cli without major rewrites, I agree that the next best approach is maintaining two interfaces: one via the traits needed for diesel, and a second for diesel_cli that can be implemented through rpc. You were probably already aware of that, but I needed to think it through. Sorry for that.

I will start working on a prototype and see where that leads me.

[weiznich]

What I was advocating for was that there is a single way to extend diesel and diesel_cli, so that extension maintainers don't have to implement and care about two completely different ways of interacting with diesel.

While I understand what you are trying to say here, I still argue that we are not talking about the same thing here. For diesel this set of traits already exists. For diesel-cli there is no such set of traits and most of the things that are currently implemented for the supported backend differ per backend. That means each custom backend will likely use a completely different implementation for this functionality anyway. It's nothing that is shared with diesel in any sort (beside that it might use the connection impl to issue queries to inspect the database). Even the TypeMetadata trait you've mentioned before is not helpful for diesel-cli. It translates a type name back to a type identifier (oid in postgres) which is exactly the opposite operation diesel-cli. Diesel-cli needs to translate type identifiers (like oid's) to type names to put them in the schema.rs file.

As written before: In the end we hopefully should end up with another set of traits wrapping the necessary functionality for diesel-cli. If a custom backend can then use any of the existing implementations we can still look at moving that code into yet another crate that implements the necessary traits based on a generic connection.

[alshdavid]

Would also like to use Turso. I have a a serverless application that uses Turso as a database (generous free tier) or SQLite locally (during development).

[weiznich]

Adding the same question again doesn't change anything an the answers given above. Such behaviour is not welcome here.

Otherwise any relevant information was already given in the thread above, so please read that. I won't repeat them here.

[jlucaso1]

Hi guys, I was experimenting using diesel with both libsql and turso (pure rust).

oxidezap/whatsapp-rust#330 this is a example of usage

In this history is using only libsql instead original sqlite3: https://github.com/jlucaso1/libsqlite3-libsql-sys/tree/1089a98bfc0b5e25463c05d998efece02f05c250

and this is using pure turso (is with my turso fork to implement somethings in FFI): https://github.com/jlucaso1/libsqlite3-libsql-sys