diff --git a/docs/Readme.md b/docs/Readme.md new file mode 100644 index 000000000..357df1b2f --- /dev/null +++ b/docs/Readme.md @@ -0,0 +1,33 @@ +# Overview + +LiteDB v5 is a single-file, serverless NoSQL database for .NET. It stores JSON-like documents, offers LINQ-friendly queries, and is designed to be fast, lightweight, and easy to embed in desktop, web, or mobile applications. + +## Documentation map + +Use the following guides to explore LiteDB's feature set: + +| Topic | Summary | +| --- | --- | +| [Getting Started](../getting-started/) | Install LiteDB, connect to a database, and perform basic CRUD operations with strongly typed collections. | +| [Data Structure](../data-structure/) | Learn how LiteDB organizes data internally, including pages, collections, and how documents are persisted on disk. | +| [Object Mapping](../object-mapping/) | Configure the mapper that turns your POCO classes into BSON documents, including custom type conversions and attributes. | +| [Collections](../collections/) | Manage collections, control identity keys, and work with typed and dynamic documents. | +| [BsonDocument](../bsondocument/) | Work directly with `BsonDocument` when you need dynamic schemas or advanced document manipulation. | +| [Expressions](../expressions/) | Compose expressions to filter, project, and update documents with LiteDB's JSON-path-inspired language. | +| [DbRef](../dbref/) | Model relationships across collections with references while keeping documents denormalized when appropriate. | +| [Connection String](../connection-string/) | Configure the LiteDB engine with connection string options such as file paths, timeouts, and journaling. | +| [FileStorage](../filestorage/) | Store and stream files alongside your documents with the GridFS-inspired FileStorage API. | +| [Indexes](../indexes/) | Accelerate queries with single-field, compound, and expression-based indexes, including multikey support. | +| [Encryption](../encryption/) | Protect data files using password-based AES encryption and understand how keys are derived. | +| [Pragmas](../pragmas/) | Tune engine behavior at runtime with pragmas such as size limits, timeouts, and default collations. | +| [Collation](../collation/) | Control string comparison rules by customizing collations per database. | +| [Queries](../queries/) | Explore the query API, including typed queries, aggregation, and filtering across indexes. | +| [Concurrency](../concurrency/) | Understand how LiteDB handles locking, transactions, and multi-threaded access. | +| [How LiteDB Works](../how-litedb-works/) | Dive into the engine design, including the page allocator, WAL file, and recovery process. | +| [Repository Pattern](../repository-pattern/) | Wrap LiteDB with repositories for domain-driven designs and testability. | +| [Versioning](../versioning/) | See how LiteDB evolves, how semantic versioning applies, and where to find upgrade notes. | +| [Changelog](../changelog/) | Review notable changes across releases. | + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/bsondocument/Readme.md b/docs/bsondocument/Readme.md new file mode 100644 index 000000000..d128b02ac --- /dev/null +++ b/docs/bsondocument/Readme.md @@ -0,0 +1,61 @@ +# BsonDocument + +The `BsonDocument` class is LiteDB’s implementation of documents. Internally, a `BsonDocument` stores key-value pairs in a `Dictionary`. + +```csharp +var customer = new BsonDocument(); +customer["_id"] = ObjectId.NewObjectId(); +customer["Name"] = "John Doe"; +customer["CreateDate"] = DateTime.Now; +customer["Phones"] = new BsonArray { "8000-0000", "9000-000" }; +customer["IsActive"] = true; +customer["IsAdmin"] = new BsonValue(true); +customer["Address"] = new BsonDocument +{ + ["Street"] = "Av. Protasio Alves" +}; +customer["Address"]["Number"] = "1331"; +``` + +LiteDB supports documents up to 16MB after BSON serialization. + +## Document keys + +* Keys are case-insensitive +* Duplicate keys are not allowed +* LiteDB keeps the original key order, including mapped classes. The only exception is for `_id` field, which will always be the first field. + +## Document values + +* Values can be any BSON value data type: Null, Int32, Int64, Decimal, Double, String, Embedded Document, Array, Binary, ObjectId, Guid, Boolean, DateTime, MinValue, MaxValue +* When a field is indexed, the value must occupy less than 1024 bytes after BSON serialization. +* `_id` field cannot be: `Null`, `MinValue` or `MaxValue` +* `_id` is unique indexed field, so value must occupy less than 1024 bytes + +## Related .NET types + +* `BsonValue` + - Holds any BSON data type, including null, arrays, or documents. + - Provides implicit constructors for supported .NET data types. + - Is immutable. + - Exposes the underlying .NET value via the `RawValue` property. +* `BsonArray` + - Supports `IEnumerable`. + - Allows array items with different BSON types. +* `BsonDocument` + - Missing fields always return `BsonValue.Null`. + +```csharp +// Testing BSON value data type +if(customer["Name"].IsString) { ... } + +// Helper to get .NET type +string str = customer["Name"].AsString; +``` + +To use other .NET data types you need a custom `BsonMapper` class. + + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/changelog/Readme.md b/docs/changelog/Readme.md new file mode 100644 index 000000000..39cb16198 --- /dev/null +++ b/docs/changelog/Readme.md @@ -0,0 +1,48 @@ +# ChangeLog + +## New Features + +* Add support to NETStandard 2.0 (with support to `Shared` mode) +* New document `Expression` parser/executor - see [Expression Wiki](https://github.com/mbdavid/LiteDB/wiki/Expressions) +* Support index creation with expressions + +```csharp + col.EnsureIndex(x => x.Name, "LOWER($.Name)"); + col.EnsureIndex("GrandTotal", "SUM($.Items[*].Qtd * $.Items[*].Price)"); +``` + + + Query with `Include` it´s supported in Engine level with ANY nested includes + `C# + col.Include(x => x.Users) + .Include(x => x.Users[0].Address) + .Include(x => x.Users[0].Address.City) + .Find(...)` +* Support complex Linq queries using `LinqQuery` compiler (works as linq to object) + + + `col.Find(x => x.Name == "John" && x.Items.Length.ToString().EndsWith == "0")` +* Better execution plan (with debug info) in multi query statements +* No more external journal file - use same datafile to store temporary data +* Fixed concurrency problems (keeps thread/process safe) +* Convert `Query.And` to `Query.Between` when possible +* Add support to `Query.Between` open/close interval +* **Same datafile from LiteDB `v3` (no upgrade needed)** + +## Shell + +* New UPDATE/SELECT statements in shell +* Shell commands parser/executor are back into LiteDB.dll +* Better shell error messages in parser with position in error +* Print query execution plan in debug + `(Seek([Age] > 10) and Filter([Name] startsWith "John"))` + (preparing to new visual LiteDB database management tool) + +## Breaking changes + +* Remove transactions +* Remove auto-id register function for custom type +* Remove index definitions on mapper (fluent/attribute) +* Remove auto create index on query execution. If the index is not found do full scan search (use `EnsureIndex` on initialize database) + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/collation/Readme.md b/docs/collation/Readme.md new file mode 100644 index 000000000..a1784f96a --- /dev/null +++ b/docs/collation/Readme.md @@ -0,0 +1,22 @@ +# Collation + +A collation is a special pragma (for more info, see [Pragmas](../pragmas)) that allows users to specify a culture and string compare options for a datafile. + +Collation is a read-only pragma and can only be changed with a rebuild. + +A collation is specified with the format `CultureName/CompareOption1[,CompareOptionN]`. For more info about compare options, check the [.NET documentation](https://docs.microsoft.com/en-us/dotnet/api/system.globalization.compareoptions). + +Datafiles are always created with `CultureInfo.CurrentCulture` as their culture and with `IgnoreCase` as the compare option. The collation can be change by rebuilding the datafile. + +Internally, the culture info is stored in the header of the datafile using its LCID value. Cultures with LCID value 4096 are not supported. When an unsupported culture is detected, the rebuild defaults to `CultureInfo.InvariantCulture`. + +## Examples + +* `rebuild {"collation": "en-US/None"};` rebuilds the datafile with the `en-US` culture and regular string comparison +* `rebuild {"collation": "en-GB/IgnoreCase"};` rebuilds the datafile with the `en-GB` culture and case-insensitive string comparison +* `rebuild {"collation": "pt-BR/IgnoreCase,IgnoreSymbols"};` rebuilds the datafile with the `pt-BR` culture and case-insensitive string comparison that also ignores symbols (white spaces, punctuation, math symbols etc.) + + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/collections/Readme.md b/docs/collections/Readme.md new file mode 100644 index 000000000..a8838fa7d --- /dev/null +++ b/docs/collections/Readme.md @@ -0,0 +1,88 @@ +# Collections + +Documents are stored and organized in collections. `LiteCollection` is the generic class that manages them in LiteDB. A collection name must: + +* Contain only letters, numbers, or underscores (`_`). +* Be case-insensitive. +* Avoid the `_` prefix (reserved for internal storage). +* Avoid the `$` prefix (reserved for system and virtual collections). + +The total size of all collection names in a database is limited to 8,000 bytes. If you expect to maintain hundreds of collections, keep the names short (about 10 characters per collection yields roughly 800 collections). + +Collections are created automatically when you first call `Insert` or `EnsureIndex`. Read, update, or delete operations against a non-existent collection will fail rather than create it. + +`LiteCollection` works for both typed and untyped scenarios. When `T` is `BsonDocument`, LiteDB keeps the document schema-less; otherwise, it maps between `T` and `BsonDocument` under the hood. The two snippets below are equivalent: + +```csharp +// Typed collection +using (var db = new LiteDatabase("mydb.db")) +{ + var customers = db.GetCollection("customers"); + + customers.Insert(new Customer { Id = 1, Name = "John Doe" }); + customers.EnsureIndex(x => x.Name); + + var match = customers.FindOne(x => x.Name == "john doe"); +} + +// Untyped collection (T is BsonDocument) +using (var db = new LiteDatabase("mydb.db")) +{ + var customers = db.GetCollection("customers"); + + customers.Insert(new BsonDocument { ["_id"] = 1, ["Name"] = "John Doe" }); + customers.EnsureIndex("Name"); + + var match = customers.FindOne("$.Name = 'john doe'"); +} +``` + +## System Collections + +System collections are special collections that provide information about the datafile. All system collections start with `$`. All system collections, with the exception of `$file`, are read-only. + +| Collection | Description | +| --- | --- | +| $cols | Lists all collections in the datafile, including the system collections. | +| $database | Shows general info about the datafile. | +| $indexes | Lists all indexes in the datafile. | +| $sequences | Lists all the sequences in the datafile. | +| $transactions | Lists all the open transactions in the datafile. | +| $snapshots | Lists all existing snapshots. | +| $open\_cursors | Lists all the open cursors in the datafile. | +| $dump(pageID) | Lists advanced info about the desired page. If no pageID is provided, lists all the pages. | +| $page\_list(pageID) | Lists basic info about the desired page. If no pageID is provided, lists all the pages. | +| $query(subquery) | Takes a query as string and returns the result of the query. Can be used for simulating subqueries. **Experimental**. | +| $file(path) | See below. | + +## The `$file` system collection + +`$file` reads from and writes to external files. + +* `SELECT $ INTO $FILE('customers.json') FROM Customers` exports every document from the `Customers` collection to a JSON file. +* `SELECT $ FROM $FILE('customers.json')` reads a JSON file back into the result set. + +LiteDB also offers limited CSV support. Only primitive types are supported, and the schema of the first document controls the output columns. Additional fields are ignored when writing. + +* `SELECT $ INTO $FILE('customers.csv') FROM Customers` exports the collection to CSV. +* `SELECT $ FROM $FILE('customers.csv')` imports rows from a CSV file. + +The `$file` parameter can be one of the following: + +* `$file("filename.json|csv")` — shorthand that infers the format from the file extension. +* `$file({ ... })` — detailed configuration using an object literal. + +| Option | Applies to | Description | +| --- | --- | --- | +| `filename` | All | Target file path. | +| `format` | All | `"json"` or `"csv"`. | +| `encoding` | All | Text encoding (defaults to `"utf-8"`). | +| `overwrite` | All | Overwrite existing files when `true`. | +| `indent` | JSON | Indentation size used when `pretty` is `true`. | +| `pretty` | JSON | Emit indented JSON when `true`. | +| `delimiter` | CSV | Column separator (defaults to `","`). | +| `header` | CSV | When `true`, writes a header row; pass an array to control the header order when reading. | + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/concurrency/Readme.md b/docs/concurrency/Readme.md new file mode 100644 index 000000000..80b5db03f --- /dev/null +++ b/docs/concurrency/Readme.md @@ -0,0 +1,22 @@ +# Concurrency + +LiteDB v4 supports both thread-safe and process-safe patterns: + +* You can create a new instance of `LiteRepository`, `LiteDatabase` or `LiteEngine` in each use (process-safe) +* You can share a single `LiteRepository`, `LiteDatabase` or `LiteEngine` instance across your threads (thread-safe) + +With the first option (process-safe), each operation opens the datafile, acquires a read or write lock, performs the work, and then closes the file. Locks are implemented using `FileStream.Lock` for both read and write modes. Always wrap the database in a `using` statement to ensure the file is closed promptly. + +With the second option (thread-safe), LiteDB controls concurrency using the .NET `ReaderWriterLockSlim` class. This allows many concurrent readers and a single exclusive writer. All threads share the same database instance, and each method coordinates access. + +## Recommendation + +Running a single shared instance is significantly faster than spinning up multiple instances. Each separate instance must perform expensive operations—open, lock, unlock, read, and close. Each instance also manages its own cache; if it handles only one operation, it will discard cached pages immediately. A single instance keeps pages cached and shares them across all reader threads. + +If your application runs in a single process (mobile apps, ASP.NET sites), prefer a single database instance shared across all threads. + +You can also enable `Exclusive` mode in the connection string. In exclusive mode, LiteDB does not check the header page for external changes and relies on in-memory locking only (via `ReaderWriterLockSlim`). + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/connection-string/Readme.md b/docs/connection-string/Readme.md new file mode 100644 index 000000000..c9425f6bb --- /dev/null +++ b/docs/connection-string/Readme.md @@ -0,0 +1,44 @@ +# Connection String + +LiteDatabase can be initialized using a connection string with the `key1=value1; key2=value2; ...` syntax. If there is no `=`, LiteDB assumes the string is the `Filename`. Quote values (`"` or `'`) when they contain reserved characters such as `;` or `=`. **Keys and values are case-insensitive.** + +## Options + +| Key | Type | Description | Default value | +| --- | --- | --- | --- | +| Filename | string | Full or relative path to the datafile. Supports `:memory:` for an in-memory database or `:temp:` for an on-disk temporary database (the file is deleted when the database closes). **Required.** | — | +| Connection | string | Connection type (“direct” or “shared”) | “direct” | +| Password | string | Encrypt the datafile with AES using a password. | null (no encryption) | +| InitialSize | string or long | Initial size for the datafile (strings support “KB”, “MB”, and “GB”). | 0 | +| ReadOnly | bool | Open the datafile in read-only mode. | false | +| Upgrade | bool | Upgrade the datafile if it is from an older version before opening it. | false | + +### Connection Type + +LiteDB offers 2 types of connections: `Direct` and `Shared`. This affects how the engine opens the data file. + +* `Direct`: Opens the datafile in exclusive mode and keeps it open until `Dispose()`. No other process can open the file. This mode is recommended because it’s faster and benefits from caching. +* `Shared`: Closes the datafile after each operation. Locks use `Mutex`. This mode is slower but allows multiple processes to open the same file. + +> The Shared mode only works in .NET implementations that provide named mutexes. Its multi-process capabilities will only work in platforms that implement named mutexes as system-wide mutexes. + +## Example + +### App.config + +```xml + + + +``` + +### C# + +```csharp +System.Configuration.ConfigurationManager.ConnectionStrings["LiteDB"].ConnectionString +``` + + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/data-structure/Readme.md b/docs/data-structure/Readme.md new file mode 100644 index 000000000..a5017d0bc --- /dev/null +++ b/docs/data-structure/Readme.md @@ -0,0 +1,111 @@ +# Data Structure + +LiteDB stores data as documents, which are JSON-like objects containing key-value pairs. Documents are a schema-less data structure. Each document stores both its data and its structure. + +```json +{ + _id: 1, + name: { first: "John", last: "Doe" }, + age: 37, + salary: 3456.0, + createdDate: { $date: "2014-10-30T00:00:00.00Z" }, + phones: ["8000-0000", "9000-0000"] +} +``` + +* `_id` contains document primary key - a unique value in collection +* `name` contains an embedded document with `first` and `last` fields +* `age` contains a `Int32` value +* `salary` contains a `Double` value +* `createDate` contains a `DateTime` value +* `phones` contains an array of `String` + +LiteDB stores documents in collections. A collection is a group of related documents that have a set of shared indices. Collections are analogous to tables in relational databases. + +## BSON + +LiteDB stores documents using BSON (Binary JSON). BSON is a binary representation of JSON with additional type information. In the documents, the value of a field can be any of the BSON data types, including other documents, arrays, and arrays of documents. BSON is a fast and simple way to serialize documents in binary format. + +LiteDB uses only a subset of [BSON data types](http://bsonspec.org/spec.html). See all supported LiteDB BSON data types and .NET equivalents. + +| BSON Type | .NET type | +| --- | --- | +| MinValue | - | +| Null | Any .NET object with `null` value | +| Int32 | `System.Int32` | +| Int64 | `System.Int64` | +| Double | `System.Double` | +| Decimal | `System.Decimal` | +| String | `System.String` | +| Document | `System.Collection.Generic.Dictionary` | +| Array | `System.Collection.Generic.List` | +| Binary | `System.Byte[]` | +| ObjectId | `LiteDB.ObjectId` | +| Guid | `System.Guid` | +| Boolean | `System.Boolean` | +| DateTime | `System.DateTime` | +| MaxValue | - | + +> Following the BSON specification, `DateTime` values are stored only up to the miliseconds. +> All `DateTime` values are converted to UTC on storage and converted back to local time on retrieval. + +## Extended JSON + +To serialize a BSON document to JSON, LiteDB uses an extended version of JSON so as not to lose any BSON type information. Extended data types are represented as embedded documents, using a key starting with `$` and string value. + +| BSON data type | JSON representation | Description | +| --- | --- | --- | +| ObjectId | `{ "$oid": "507f1f55bcf96cd799438110" }` | 12 bytes in hex format | +| Date | `{ "$date": "2015-01-01T00:00:00Z" }` | UTC and ISO-8601 format | +| Guid | `{ "$guid": "ebe8f677-9f27-4303-8699-5081651beb11" }` | | +| Binary | `{ "$binary": "VHlwZSgaFc3sdcGFzUpcmUuLi4=" }` | Byte array in base64 string format | +| Int64 | `{ "$numberLong": "12200000" }` | | +| Decimal | `{ "$numberDecimal": "122.9991" }` | | +| MinValue | `{ "$minValue": "1" }` | | +| MaxValue | `{ "$maxValue": "1" }` | | + +LiteDB implements JSON in its `JsonSerializer` static class. + +If you want to convert your object type to a BsonValue, you must use a `BsonMapper`. + +```csharp +var customer = new Customer { Id = 1, Name = "John Doe" }; + +var doc = BsonMapper.Global.ToDocument(customer); + +var jsonString = JsonSerialize.Serialize(doc); +``` + +`JsonSerialize` also supports `TextReader` and `TextWriter` to read/write directly from a file or `Stream`. + +## ObjectId + +`ObjectId` is a 12 bytes BSON type: + +* `Timestamp`: Value representing the seconds since the Unix epoch (4 bytes) +* `Machine`: Machine identifier (3 bytes) +* `Pid`: Process id (2 bytes) +* `Increment`: A counter, starting with a random value (3 bytes) + +In LiteDB, documents are stored in a collection that requires a unique `_id` field that acts as a primary key. Because `ObjectIds` are small, most likely unique, and fast to generate, LiteDB uses `ObjectIds` as the default value for the `_id` field if the `_id` field is not specified. + +Unlike the Guid data type, ObjectIds are sequential, so it’s a better solution for indexing. ObjectIds use hexadecimal numbers represented as strings. + +```csharp +var id = ObjectId.NewObjectId(); + +// You can get creation datetime from an ObjectId +var date = id.CreationTime; + +// ObjectId is represented in hex value +Debug.WriteLine(id); +"507h096e210a18719ea877a2" + +// Create an instance based on hex representation +var nid = new ObjectId("507h096e210a18719ea877a2"); +``` + + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/dbref/Readme.md b/docs/dbref/Readme.md new file mode 100644 index 000000000..7ad092ec6 --- /dev/null +++ b/docs/dbref/Readme.md @@ -0,0 +1,110 @@ +# DbRef + +LiteDB is a document database, so there is no JOIN between collections. You can use embedded documents (sub-documents) or create a reference between collections. To create a reference you can use `[BsonRef]` attribute or use the`DbRef` method from the fluent API mapper. + +## Mapping a reference on database initialization + +```csharp +public class Customer +{ + public int CustomerId { get; set; } + public string Name { get; set; } +} + +public class Order +{ + public int OrderId { get; set; } + public Customer Customer { get; set; } +} +``` + +If no custom mapping is created, when you save an `Order`, `Customer` is saved as an embedded document with no link to any other collection. Any changes made to documents in the `customers` collection will not be reflected in the `orders` collection. + +``` +Order => { _id: 123, Customer: { CustomerId: 99, Name: "John Doe" } } +``` + +If you want to store only a reference to a customer in `Order`, you can decorate your class: + +```csharp +public class Order +{ + public int OrderId { get; set; } + + [BsonRef("customers")] // where "customers" is the collection to be referenced + public Customer Customer { get; set; } +} +``` + +Note that `BsonRef` decorates the full object being referenced, not an int `customerid` field that references an object in the other collection. + +Or use fluent API: + +```csharp +BsonMapper.Global.Entity() + .DbRef(x => x.Customer, "customers"); // where "customers" are Customer collection name +``` + +**Note:** `Customer` needs to have a `[BsonId]` defined. + +Now, when you store `Order` you are storing only the reference. + +``` +Order => { _id: 123, Customer: { $id: 4, $ref: "customers"} } +``` + +## Querying results + +When you query a document with a cross-collection reference, you can auto load references using the `Include` method before query. + +```csharp +var orders = db.GetCollection("orders"); + +var order1 = orders + .Include(x => x.Customer) + .FindById(1); +``` + +DbRef also support `List` or `Array`, like: + +```csharp +public class Product +{ + public int ProductId { get; set; } + public string Name { get; set; } + public decimal Price { get; set; } +} + +public class Order +{ + public int OrderId { get; set; } + public DateTime OrderDate { get; set; } + public List Products { get; set; } +} + +BsonMapper.Global.Entity() + .DbRef(x => x.Products, "products"); +``` + +If the `Products` field is null or an empty list, the value will be preserved when being mapped from a `BsonDocument` to an `Order`. If you do not use `Include` in query, every `Product` in `Products` will be loaded with the id field set and all other fields null or default. + +In v4, this include process occurs on BsonDocument engine level. It also support any level of include, just using `Path` syntax: + +```csharp +orders.Include(new string[] { "$.Customer", "$.Products[*]" }); +``` + +If you are using `LiteCollection` or `Repository` you can also use Linq syntax: + +``` +// repository fluent syntax +db.Query() + .Include(x => x.Customer) + .Include(x => x.Products) + .ToList(); +``` + + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/encryption/Readme.md b/docs/encryption/Readme.md new file mode 100644 index 000000000..545194a56 --- /dev/null +++ b/docs/encryption/Readme.md @@ -0,0 +1,12 @@ +# Encryption + +LiteDB uses salted AES (as defined by [RFC 2898](https://tools.ietf.org/html/rfc2898)) as its encryption. This is implemented by the `Rfc2898DeriveBytes` class. + +The `Aes` object used for cryptography is initialized with `PaddingMode.None` and `CipherMode.ECB`. + +The password for an encrypted datafile is defined in the connection string (for more info, check [Connection String](../connection-string)). The password can only be changed or removed by rebuilding the datafile (for more info, check Rebuild Options in [Pragmas](../pragmas#rebuildOptions)). + + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/expressions/Readme.md b/docs/expressions/Readme.md new file mode 100644 index 000000000..165b1ebbf --- /dev/null +++ b/docs/expressions/Readme.md @@ -0,0 +1,135 @@ +# Expressions + +Expressions are path or formulas to access and modify the data inside a document. Based on the concept of JSON path ( LiteDB supports a similar syntax to navigate inside a document. + +In previous versons, LiteDB used lambda expressions directly on objects. This was very flexible, but also had poor perfomance. LiteDB v5 uses `BsonExpression`s, which are expressions that can be directly applied to a `BsonDocument`. + +`BsonExpression`s can either be used natively (there is an implicit conversion between `string` and `BsonExpression`) or by mapping a lambda expression (methods that take a lambda expression do this automatically). + +* Path starts with `$`: `$.Address.Street`, where `$` represents the root document. The `$` symbol are optional and default in document navigation (`Address.Street` works too) +* Int values are defined by `[0-9]*`: `123` +* Double values are defined by `[0-9].[0-9]`: `123.45` +* Strings are represented with a single/double quote: `'Hello World'` +* Null is represented by `null` +* Bool is represented using `true` or `false` keywords. +* Document starts with `{ key1: , key2: ... }` +* Arrays are represented with `[, , ...]` +* Functions are represented with `FUNCTION_NAME(par1, par2, ...)`: `LOWER($.Name)` + +Examples: + +* `$.Price` +* `$.Price + 100` +* `SUM($.Items[*].Price)` + +Expressions can be used in many ways: + +* Creating an index based on an expression: + + `collection.EnsureIndex("idx_name", "LOWER($.Name)", false)` + + `collection.EnsureIndex(x => x.Name.ToLower())` +* Querying documents inside a collection based on expression (full scan search) + + `collection.Find("SUBSTRING($.Name, 0, 1) = 'T'")` +* Update using SQL syntax + + `UPDATE customers SET Name = LOWER($.Name) WHERE _id = 1` +* Creating new document result in SELECT shell command + + `SELECT { upper_titles: ARRAY(UPPER($.Books[*].Title)) } WHERE $.Name LIKE "John%"` +* Querying documents using the SQL syntax + + `SELECT $.Name, $.Phones[@.Type = "Mobile"] FROM customers` + +## Path + +* `$` - Root document +* `$.Name` - Field `Name` +* `$.Name.First` - Field `First` from `Name` subdocument +* `$.Books` - Returns the array of books +* `$.Books[0]` - Returns the first book inside Books array +* `$.Books[*]` - Returns every book inside Books +* `$.Books[*].Title` Returns the title from every book in Books +* `$.Books[-1]` - Returns the last book inside Books array + +Path also supports expressions to filter child nodes + +* `$.Books[@.Title = 'John Doe']` - Returns all books where `Title` is `'John Doe'` +* `$.Books[@.Price > 100].Title` - Returns all titles where `Price` is greater than `100` + +Inside an array, `@` acts as a sub-iterator, pointing to the current sub-document. It’s possible use functions inside expressions too: + +* `$.Books[SUBSTRING(LOWER(@.Title), 0, 1) = 't']` - Returns all books whose `Title` starts with `'T'` or `'t'`. + +### Difference between `$` and `*` + +In SQL query, it is possible use both `$` and `*`. They have different functionalities: + +* `$` represents current root document. When `$` is used, you are referencing the root document. If neither `$` nor `*` are present, `$` is assumed. +* `*` represent a group of documents. Used when `GROUP BY` is present or when you want to return a single value in a query (`SELECT COUNT(*) FROM customers`). + +`SELECT $ FROM customers` returns `IEnumerable` result (`N` documents). +`SELECT * FROM customers` returns a single value, a `BsonArray` with all documents result inside. + +## Functions + +Functions are used to manipulate data in expressions. A few examples will be provided for each category of functions. For a complete list of functions, check the API documentation. + +### Aggregate Functions + +Aggregate functions take an array as input and return a single value. + +* `COUNT(arr)` - Returns the number of elements in the array `arr` +* `AVG(arr)` - Returns the average value in the array `arr` +* `LAST(arr)` - Returns the last element in the array `arr` + +### DataType Functions + +DataType functions provide explicit data type conversion. + +* `STRING(expr)` - Returns the result of `expr` converted to string +* `INT32(expr)` - Tries to convert the result of `expr` to an `Int32`, returning `null` if not possible +* `DATETIME(expr)` - Tries to convert the result of `expr` to a `DateTime`, returning `null` if not possible + +### Date Functions + +* `YEAR(date)` - Returns the year value from `date` +* `DATEADD('year', 3, date)` - Returns a new date with 3 years added to date +* `DATEDIFF('day', dateStart, dateEnd)` - Returns the difference in days between `dateEnd` and `dateStart` + +### Math Functions + +* `ABS(num)` - Returns the absolute value of `num` +* `ROUND(num, digits)` - Returns `num` rounded to `digits` digits +* `POW(base, exp)` - Returns `base` to the power of `exp` + +### String Functions + +* `UPPER(str)` - Returns `str` in uppercase +* `TRIM(str)` - Returns a new string without leading and trailing white spaces +* `REPLACE(str, old, new)` - Returns a new string with every ocurrence of `old` in `str` replaced by `new` + +### High-Order Functions + +High-Order functions take an array and a lambda expression that is applied to every document in the array. Use the `@` symbol to represent inner looped value. + +* `MAP(arr => expr)` returns a new array with the map expression applied to each element + + + `MAP([1,2,3] => @*2)` returns `[2,4,6]` + + `MAP([{a:1, b:2}, {a:3, b:4}] => @.a)` returns `[1,3]` +* `FILTER(arr => expr)` returns a new array containing only the elements for which the filter expression returns `true` + + + `FILTER([1,2,3,4,5] => @ > 3)` returns `[4,5]` + + `FILTER([{a:1, b:2}, {a:2}] => @.b != null)` returns `[{a:1, b:2}]` +* `SORT(arr => expr)` returns a new array sorted by the result of `expr` in ascending order + -`SORT([3,2,5,1,4] => @)` returns `[1,2,3,4,5]` + -`SORT([{a:2}, {a:1, b:2}] => @.a)` returns `[{a:1, b:2}, {a:2}]` +* `SORT(arr => expr, order)` returns a new array sorted by the result of `expr` with the order defined by `order` (ascending if `order` is `1` or `'asc'`, descending if `order` is `-1` or `'desc'`) + -`SORT([3,2,5,1,4] => @, 'desc')` returns `[5,4,3,2,1]` + -`SORT([{a:1, b:2}, {a:2}] => @.a, -1)` returns `[{a:2}, {a:1, b:2}]` + +### Misc Functions + +* `JSON(str)` - Takes a string representation of a JSON and returns a `BsonValue` containing the parsed document +* `CONCAT(arr1, arr2)` - Returns a new array containg the concatenation between arrays `arr1` and `arr2` +* `RANDOM(min, max)` - Returns a random `Int32` between `min` and `max` + + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/filestorage/Readme.md b/docs/filestorage/Readme.md new file mode 100644 index 000000000..5f5fb372c --- /dev/null +++ b/docs/filestorage/Readme.md @@ -0,0 +1,86 @@ +# FileStorage + +To keep its memory profile slim, LiteDB limits the size of a documents to 1MB. For most documents, this is plenty. However, 1MB is too small for a useful file storage. For this reason, LiteDB implements `FileStorage`, a custom collection to store files and streams. + +`FileStorage` uses two special collections: + +* The first collection stores file references and metadata only (by default it is called `_files`) + +```json +{ + _id: "my-photo", + filename: "my-photo.jpg", + mimeType: "image/jpg", + length: { $numberLong: "2340000" }, + chunks: 9, + uploadDate: { $date: "2020-01-01T00:00:00Z" }, + metadata: { "key1": "value1", "key2": "value2" } +} +``` + +* The second collection stores binary data in 255kB chunks (by default it is called `_chunks`) + +```json +{ + _id: { "f": "my-photo", "n": 0 }, + data: { $binary: "VHlwZSAob3Igc ... GUpIGhlcmUuLi4" } +} +{ + _id: { "f": "my-photo", "n": 1 }, + data: { $binary: "pGaGhlcmUuLi4 ... VHlwZSAob3Igc" } +} +{ + ... +} +``` + +Files are identified by an `_id` string value, with following rules: + +* Starts with a letter, number, `_`, `-`, `$`, `@`, `!`, `+`, `%`, `;` or `.` +* If contains a `/`, must be sequence with chars above + +To better organize many files, you can use `_id` as a `directory/file_id`. This will be a great solution to quickly find all files in a directory using the `Find` method. + +Example: `$/photos/2014/picture-01.jpg` + +The `FileStorage` collection contains simple methods like: + +* **`Upload`**: Send file or stream to database. Can be used with file or `Stream`. If file already exists, file content is overwritten. +* **`Download`**: Get your file from database and copy to `Stream` parameter +* **`Delete`**: Delete a file reference and all data chunks +* **`Find`**: Find one or many files in `_files` collection. Returns `LiteFileInfo` class, that can be download data after. +* **`SetMetadata`**: Update stored file metadata. This method doesn’t change the value of the stored file. It updates the value of `_files.metadata`. +* **`OpenRead`**: Find file by `_id` and returns a `LiteFileStream` to read file content as stream + +```csharp +// Gets a FileStorage with the default collections +var fs = db.FileStorage; + +// Gets a FileStorage with custom collection name +var fs = db.GetStorage("myFiles", "myChunks"); + +// Upload a file from file system +fs.Upload("$/photos/2014/picture-01.jpg", @"C:\Temp\picture-01.jpg"); + +// Upload a file from a Stream +fs.Upload("$/photos/2014/picture-01.jpg", "picture-01.jpg", stream); + +// Find file reference only - returns null if not found +LiteFileInfo file = fs.FindById("$/photos/2014/picture-01.jpg"); + +// Now, load binary data and save to file system +file.SaveAs(@"C:\Temp\new-picture.jpg"); + +// Or get binary data as Stream and copy to another Stream +file.CopyTo(Response.OutputStream); + +// Find all files references in a "directory" +var files = fs.Find("$/photos/2014/"); +``` + +`FileStorage` does not support transactions to avoid putting all of the file in memory before storing it on disk. Transactions *are* used per chunk. Each uploaded chunk is committed in a single transaction. + + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/getting-started/Readme.md b/docs/getting-started/Readme.md new file mode 100644 index 000000000..65b0c1467 --- /dev/null +++ b/docs/getting-started/Readme.md @@ -0,0 +1,103 @@ +# Getting Started + +LiteDB is a simple, fast, and lightweight embedded .NET document database inspired by MongoDB. The API intentionally mirrors the official MongoDB .NET API so that common persistence patterns feel familiar. + +## Installation + +LiteDB is a serverless database—there is no service to install or configure. You can: + +* Copy [LiteDB.dll](https://github.com/mbdavid/LiteDB/releases) next to your application binaries and reference it directly. +* Install the NuGet package with `Install-Package LiteDB`. + +When hosting in IIS, be sure the application pool identity has write permissions to the folder that will contain your `.db` file. + +## First steps + +The snippet below creates a database, inserts a document, and runs a simple query: + +```csharp +// Create your POCO class entity +public class Customer +{ + public int Id { get; set; } + public string Name { get; set; } + public string[] Phones { get; set; } + public bool IsActive { get; set; } +} + +// Open database (or create if it doesn't exist) +using (var db = new LiteDatabase(@"C:\Temp\MyData.db")) +{ + // Get a collection (or create, if it doesn't exist) + var customers = db.GetCollection("customers"); + + // Create your new customer instance + var customer = new Customer + { + Name = "John Doe", + Phones = new[] { "8000-0000", "9000-0000" }, + IsActive = true + }; + + // Insert new customer document (Id will be auto-incremented) + customers.Insert(customer); + + // Update a document inside the collection + customer.Name = "Jane Doe"; + customers.Update(customer); + + // Index documents using the Name property + customers.EnsureIndex(x => x.Name); + + // Use LINQ to query documents (filter, sort, transform) + var results = customers.Query() + .Where(x => x.Name.StartsWith("J")) + .OrderBy(x => x.Name) + .Select(x => new { x.Name, NameUpper = x.Name.ToUpper() }) + .Limit(10) + .ToList(); + + // Create a multikey index on phone numbers + customers.EnsureIndex(x => x.Phones); + + // Query by phone number + var match = customers.FindOne(x => x.Phones.Contains("8888-5555")); +} +``` + +## Custom mapping + +If the automatic mapper does not serialize a type the way you expect, register a custom serializer: + +```csharp +BsonMapper.Global.RegisterType +( + serialize: obj => + { + var doc = new BsonDocument(); + doc["DateTime"] = obj.DateTime.Ticks; + doc["Offset"] = obj.Offset.Ticks; + return doc; + }, + deserialize: doc => new DateTimeOffset(doc["DateTime"].AsInt64, new TimeSpan(doc["Offset"].AsInt64)) +); +``` + +## Working with files + +LiteDB ships with FileStorage for handling binary data: + +```csharp +// Get file storage with an integer identifier +var storage = db.GetStorage(); + +// Upload a file from the file system into the database +storage.Upload(123, @"C:\Temp\picture-01.jpg"); + +// And download it later +storage.Download(123, @"C:\Temp\copy-of-picture-01.jpg"); +``` + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/how-litedb-works/Readme.md b/docs/how-litedb-works/Readme.md new file mode 100644 index 000000000..b472397a0 --- /dev/null +++ b/docs/how-litedb-works/Readme.md @@ -0,0 +1,53 @@ +# How LiteDB Works + +## File Format + +LiteDB is a single file database. But databases have many different types of information, like indexes, collections, documents. To manage this, LiteDB implements database pages concepts. Page is a block of same information type and has 4096 bytes. Page is the minimum read/write operation on disk file. There are 6 page types: + +* **Header Page**: Contains database information, like file version, data file size and pointer to free list pages. Is the first page on database (`PageID` = 0). +* **Collection Page**: Each collection use one page and hold all collection information, like name, indexes, pointers and options. All collections are connected each others by a double linked list. +* **Index Page**: Used to hold index nodes. LiteDB implement skip list indexes, so each node has one key and levels link pointers to others index nodes. +* **Data Page**: Data page contains data blocks. Each data block represent an document serialized in BSON format. If a document is bigger than one page, data block use a link pointer to an extended page. +* **Extend Page**: Big documents that need more than one page, are serialized in multiples extended pages. Extended pages are double linked to create a single data segment that to store documents. Each extend page contains only one document or a chunk of a document. +* **Empty Page**: When a page is excluded becomes a empty page. Empty pages will be use on next page request (for any kind of page). + +Each page has a own header and content. Header is used to manage common data structure like PageID, PageType, Free Space. Content are implement different on each page type. + +### Page free space + +Index pages and data pages contains a collection of elements (index nodes and data blocks). This pages can store data and keep with available space to more. To hold this free space on each page type, LiteDB implements free list pages. + +Free list are a double linked list, sorted by available space. When database need a page to store data use this list to search first available page. After this, `PagerService` fix page order or remove from free list if there is no more space on page. + +To create near data related, each collection contains an data free list. So, in a data page, all data blocks are of same collection. The same occurs in indexes. Each index (on each collection) contains your own free list. This solution consume more disk space but are much faster to read/write operations because data are related and near one with other. If you get all documents in a collection, database needs read less pages on disk. + +## Limits + +* Collection Name: + + Pattern: `A-Z`, `_-`, `0-9` + + Maxlength of 60 chars + + Case insensitive +* Index Name: + + Pattern: `A-Z`, `_-`, `0-9` (`.` for nested document) + + Maxlength of 60 chars + + Case sensitive +* BsonDocument Key Name: + + `A-Z`, `_-`, `0-9` + + Case sensitive +* FileStorage FileId: + + Pattern: same used in file system path/file. + + Case sensitive +* Collections: 3000 bytes to all collections names (each collection has 8 bytes of overhead) +* Documents per collections: `UInt.MaxValue` +* FileStorage Max File Length: 2Gb per file +* Index key size: 512 bytes after BSON serialization +* BSON document size: no limit (but highly recommended to keep as little as possible, under 200kb) +* Nested Depth for BSON Documents: 20 +* Page Size: 4096 bytes +* DataPage Reserved Bytes: 2039 bytes (like PCT FREE) +* IndexPage Reserved Bytes: 100 bytes (like PCT FREE) +* Database Max Length: In theory, `UInt.MaxValue` \* PageSize (4096) = 16TB ~ Too big! + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/index.xml.md b/docs/index.xml.md new file mode 100644 index 000000000..8e5ed3442 --- /dev/null +++ b/docs/index.xml.md @@ -0,0 +1,158 @@ +# Documentation RSS feed + +```xml + + + + Overview on LiteDB :: A .NET embedded NoSQL database + /docs/ + Recent content in Overview on LiteDB :: A .NET embedded NoSQL database + Hugo -- gohugo.io + en-us + Wed, 28 Nov 2018 15:14:39 +1000 + + + + + + Getting Started + /docs/getting-started/ + Mon, 01 Jan 0001 00:00:00 +0000 + + /docs/getting-started/ + LiteDB is a simple, fast and lightweight embedded .NET document database. LiteDB was inspired by the MongoDB database and its API is very similar to the official MongoDB .NET API. +How to install LiteDB is a serverless database, so there is no installation. Just copy LiteDB.dll into your Bin folder and add it as Reference. Or, if you prefer, you can install via NuGet: Install-Package LiteDB. If you are running in a web environment, make sure that your IIS user has write permission to the data folder. + + + + Data Structure + /docs/data-structure/ + Mon, 01 Jan 0001 00:00:00 +0000 + + /docs/data-structure/ + LiteDB stores data as documents, which are JSON-like objects containing key-value pairs. Documents are a schema-less data structure. Each document stores both its data and its structure. +{ _id: 1, name: { first: &#34;John&#34;, last: &#34;Doe&#34; }, age: 37, salary: 3456.0, createdDate: { $date: &#34;2014-10-30T00:00:00.00Z&#34; }, phones: [&#34;8000-0000&#34;, &#34;9000-0000&#34;] } _id contains document primary key - a unique value in collection name contains an embedded document with first and last fields age contains a Int32 value salary contains a Double value createDate contains a DateTime value phones contains an array of String LiteDB stores documents in collections. + + + + Object Mapping + /docs/object-mapping/ + Mon, 01 Jan 0001 00:00:00 +0000 + + /docs/object-mapping/ + The LiteDB mapper converts POCO classes documents. When you get a ILiteCollection&lt;T&gt; instance from LiteDatabase.GetCollection&lt;T&gt;, T will be your document type. If T is not a BsonDocument, LiteDB internally maps your class to BsonDocument. To do this, LiteDB uses the BsonMapper class: +// Simple strongly-typed document public class Customer { public ObjectId CustomerId { get; set; } public string Name { get; set; } public DateTime CreateDate { get; set; } public List&lt;Phone&gt; Phones { get; set; } public bool IsActive { get; set; } } var typedCustomerCollection = db. + + + + Collections + /docs/collections/ + Mon, 01 Jan 0001 00:00:00 +0000 + + /docs/collections/ + Documents are stored and organized in collections. LiteCollection is a generic class that is used to manage collections in LiteDB. Each collection must have a unique name: + Contains only letters, numbers and _ Collection names are case insensitive Collection names starting with _ are reserved for internal storage use Collection names starting with $ are reserved for internal system/virtual collections The total size of all the collections names in a database is limited to 8000 bytes. + + + + BsonDocument + /docs/bsondocument/ + Mon, 01 Jan 0001 00:00:00 +0000 + + /docs/bsondocument/ + The BsonDocument class is LiteDB&rsquo;s implementation of documents. Internally, a BsonDocument stores key-value pairs in a Dictionary&lt;string, BsonValue&gt;. +var customer = new BsonDocument(); customer[&#34;_id&#34;] = ObjectId.NewObjectId(); customer[&#34;Name&#34;] = &#34;John Doe&#34;; customer[&#34;CreateDate&#34;] = DateTime.Now; customer[&#34;Phones&#34;] = new BsonArray { &#34;8000-0000&#34;, &#34;9000-000&#34; }; customer[&#34;IsActive&#34;] = true; customer[&#34;IsAdmin&#34;] = new BsonValue(true); customer[&#34;Address&#34;] = new BsonDocument { [&#34;Street&#34;] = &#34;Av. Protasio Alves&#34; }; customer[&#34;Address&#34;][&#34;Number&#34;] = &#34;1331&#34;; LiteDB supports documents up to 16MB after BSON serialization. + + + + Expressions + /docs/expressions/ + Mon, 01 Jan 0001 00:00:00 +0000 + + /docs/expressions/ + Expressions are path or formulas to access and modify the data inside a document. Based on the concept of JSON path (http://goessner.net/articles/JsonPath/), LiteDB supports a similar syntax to navigate inside a document. +In previous versons, LiteDB used lambda expressions directly on objects. This was very flexible, but also had poor perfomance. LiteDB v5 uses BsonExpressions, which are expressions that can be directly applied to a BsonDocument. +BsonExpressions can either be used natively (there is an implicit conversion between string and BsonExpression) or by mapping a lambda expression (methods that take a lambda expression do this automatically). + + + + DbRef + /docs/dbref/ + Mon, 01 Jan 0001 00:00:00 +0000 + + /docs/dbref/ + LiteDB is a document database, so there is no JOIN between collections. You can use embedded documents (sub-documents) or create a reference between collections. To create a reference you can use [BsonRef] attribute or use theDbRef method from the fluent API mapper. +Mapping a reference on database initialization public class Customer { public int CustomerId { get; set; } public string Name { get; set; } } public class Order { public int OrderId { get; set; } public Customer Customer { get; set; } } If no custom mapping is created, when you save an Order, Customer is saved as an embedded document with no link to any other collection. + + + + Connection String + /docs/connection-string/ + Mon, 01 Jan 0001 00:00:00 +0000 + + /docs/connection-string/ + LiteDatabase can be initialized using a string connection, with key1=value1; key2=value2; ... syntax. If there is no = in your connection string, LiteDB assume that your connection string contains only the Filename. Values can be quoted (&quot; or ') if they contain special characters (like ; or =). Keys and values are case-insensitive. +Options Key Type Description Default value Filename string Full or relative path to the datafile. + + + + FileStorage + /docs/filestorage/ + Mon, 01 Jan 0001 00:00:00 +0000 + + /docs/filestorage/ + To keep its memory profile slim, LiteDB limits the size of a documents to 1MB. For most documents, this is plenty. However, 1MB is too small for a useful file storage. For this reason, LiteDB implements FileStorage, a custom collection to store files and streams. +FileStorage uses two special collections: + The first collection stores file references and metadata only (by default it is called _files) { _id: &#34;my-photo&#34;, filename: &#34;my-photo. + + + + Indexes + /docs/indexes/ + Mon, 01 Jan 0001 00:00:00 +0000 + + /docs/indexes/ + LiteDB improves search performance by using indexes on document fields or expressions. Each index storess the value of a specific expression ordered by the value (and type). Without an index, LiteDB must execute a query using a full document scan. Full document scans are inefficient because LiteDB must deserialize every document in the collection. +Index Implementation Indexes in LiteDB are implemented using Skip lists. Skip lists are double linked sorted list with up to 32 levels. + + + + Encryption + /docs/encryption/ + Mon, 01 Jan 0001 00:00:00 +0000 + + /docs/encryption/ + LiteDB uses salted AES (as defined by RFC 2898) as its encryption. This is implemented by the Rfc2898DeriveBytes class. +The Aes object used for cryptography is initialized with PaddingMode.None and CipherMode.ECB. +The password for an encrypted datafile is defined in the connection string (for more info, check Connection String). The password can only be changed or removed by rebuilding the datafile (for more info, check Rebuild Options in Pragmas). + + + + Pragmas + /docs/pragmas/ + Mon, 01 Jan 0001 00:00:00 +0000 + + /docs/pragmas/ + In LiteDB v5, pragmas are variables that can alter the behavior of a datafile. They are stored in the header of the datafile. + Name Read-only Data type Description Default value USER_VERSION no int Reserved for version control by the user. Does not affect the behavior of the datafile. 0 COLLATION yes (can be changed with a rebuild) string (internally stored as int) Check Collation. + + + + Collation + /docs/collation/ + Mon, 01 Jan 0001 00:00:00 +0000 + + /docs/collation/ + A collation is a special pragma (for more info, see Pragmas) that allows users to specify a culture and string compare options for a datafile. +Collation is a read-only pragma and can only be changed with a rebuild. +A collation is specified with the format CultureName/CompareOption1[,CompareOptionN]. For more info about compare options, check the .NET documentation. +Datafiles are always created with CultureInfo.CurrentCulture as their culture and with IgnoreCase as the compare option. + + + + +``` + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/indexes/Readme.md b/docs/indexes/Readme.md new file mode 100644 index 000000000..a67d029a2 --- /dev/null +++ b/docs/indexes/Readme.md @@ -0,0 +1,100 @@ +# Indexes + +LiteDB improves search performance by using indexes on document fields or expressions. Each index stores the value of a specific expression ordered by both value and type. Without an index, LiteDB must execute a query using a full document scan, deserializing every document in the collection. + +## Index Implementation + +Indexes in LiteDB are implemented using **skip lists**. Skip lists are doubly linked sorted lists with up to 32 levels. They are easy to implement and statistically balanced. + +Insert and search operations have an average complexity of *O*(log n). This means that in a collection with 1 million documents, a search operation over an indexed expression will take about 13 steps to find the desired document. If you want to know more about skip lists, [check this great video](https://www.youtube.com/watch?v=kBwUoWpeH_Q). + +Given that collections are schema-less, it is possible for an expression to return different data types when applied over different documents. In such cases, the documents will be ordered by the type returned by the indexing expression, according to the table below: + +| BSON Type | Order | +| --- | --- | +| MinValue | 1 | +| Null | 2 | +| Int32, Int64, Double, Decimal | 3 | +| String | 4 | +| Document | 5 | +| Array | 6 | +| Binary | 7 | +| ObjectId | 8 | +| Guid | 9 | +| Boolean | 10 | +| DateTime | 11 | +| MaxValue | 12 | + +* When comparing documents, the values in the key-value pairs are compared in the order that they appear until a tie is broken. +* When comparing arrays, every position is compared until a tie is broken. +* All numeric values are converted to `Decimal` before comparison. + +## Primary key (= auto id) + +By default, an index over `_id` is created upon the first insertion. + +## EnsureIndex() + +Indexes are created via `EnsureIndex`. This method creates the index if it does not exist and does nothing if already exists. + +An index can be created over any valid `BsonExpression`. + +```json +{ + "_id": 1, + "Address": { + "Street": "Av. Protasio Alves, 1331", + "City": "Porto Alegre", + "Country": "Brazil" + } +} +``` + +* You can use `EnsureIndex("Address")` to create an index on the embedded `Address` document. +* `EnsureIndex("Address.Street")` creates an index on the nested `Street` field using dotted notation. +* Indexes operate on `BsonDocument` field names. If you are using a custom `ResolvePropertyName` or `[BsonField]` attribute, reference the resolved document field name rather than the property name. See [Object Mapping](../object-mapping/). +* You can use a lambda expression to define an index field in a strongly typed collection: `EnsureIndex(x => x.Name)`. +* Indexes are identified by a unique name. + +## Multikey indexes + +When you create an index on an array field, LiteDB indexes each element so you can search for any value. + +```csharp +public class Customer +{ + public int Id { get; set; } + public string Name { get; set; } + public string[] Phones { get; set; } +} + +var customers = db.GetCollection("customers"); + +customers.Insert(new Customer { Name = "John", Phones = new string[] { "1", "2", "5" } }); +customers.Insert(new Customer { Name = "Doe", Phones = new string[] { "1", "8" } }); + +customers.EnsureIndex(x => x.Phones); + +var result = customers.Find(x => x.Phones.Contains("1")); // returns both documents +``` + +## Expressions + +You can create an index based on the result of an expression, including expressions that return multikey values. This allows you to index information that is not stored directly in a single field. + +* `collection.EnsureIndex("Name", "LOWER($.Name)")` +* `collection.EnsureIndex("Total", "SUM($.Items[*].Price)")` +* `collection.EnsureIndex("CheapBooks", "LOWER($.Books[@.Price < 20].Title)")` + +See [Expressions](/docs/expressions/) for more details about expressions. + +## Limitations + +* Even if multiple indexed expressions are used on a query, only one of the indexes is used, with the remaining expressions being filtered using a full scan. +* Index keys must occupy less than 1024 bytes +* Up to 255 indexes per collections, including the `_id` primary key, but limited to 8096 bytes for index definition. Each index uses: `41` bytes + `LEN(name)` + `LEN(expression)` + + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/object-mapping/Readme.md b/docs/object-mapping/Readme.md new file mode 100644 index 000000000..095bc0eca --- /dev/null +++ b/docs/object-mapping/Readme.md @@ -0,0 +1,176 @@ +# Object Mapping + +The LiteDB mapper converts POCO classes documents. When you get a `ILiteCollection` instance from `LiteDatabase.GetCollection`, `T` will be your document type. If `T` is not a `BsonDocument`, LiteDB internally maps your class to `BsonDocument`. To do this, LiteDB uses the `BsonMapper` class: + +```csharp +// Simple strongly-typed document +public class Customer +{ + public ObjectId CustomerId { get; set; } + public string Name { get; set; } + public DateTime CreateDate { get; set; } + public List Phones { get; set; } + public bool IsActive { get; set; } +} + +var typedCustomerCollection = db.GetCollection("customer"); + +var schemelessCollection = db.GetCollection("customer"); // is BsonDocument +``` + +## Mapper conventions + +`BsonMapper.ToDocument()` auto converts each property of a class to a document field following these conventions: + +* Properties can be read-only or read/write +* The class should have an `Id` property, `Id` property, a property with `[BsonId]` attribute or mapped by the fluent API. +* A property can be decorated with `[BsonIgnore]` in order not to be mapped to a document field +* A property can be decorated with `[BsonField("fieldName")]` to customize the name of the document field +* No circular references are allowed +* By default, max depth of 20 inner classes (this can be changed in the `BsonMapper`) +* You can use `BsonMapper` global instance (`BsonMapper.Global`) or a custom instance and pass to `LiteDatabase` in its constructor. Keep this instance in a single place to avoid re-creating the mappings each time you use a database. + +In addition to basic BSON types, `BsonMapper` maps others .NET types to BSON data type: + +| .NET type | BSON type | +| --- | --- | +| `Int16`, `UInt16`, `Byte`, `SByte` | Int32 | +| `UInt32` , `UInt64` | Int64 | +| `Single` | Double | +| `Char`, `Enum` | String | +| `IList` | Array | +| `T[]` | Array | +| `IDictionary` | Document | +| Any other .NET type | Document | + +* `Nullable` are accepted. If value is `null` the BSON type is Null, otherwise the mapper will use `T?`. +* For `IDictionary`, `K` key must be `String` or a simple type (convertible using `Convert.ToString(..)`). + +### Constructors + +Starting with version 5 of LiteDB you can use `BsonCtorAttribute` to indicate which constructor the mapper must use. Fields no longer need to have a public setter and can be initialized by the constructor. + +```csharp +public class Customer +{ + public ObjectId CustomerId { get; } + public string Name { get; } + public DateTime CreationDate { get; } + public bool IsActive { get; } + + public Customer(string name, bool isActive) + { + CustomerId = ObjectId.NewObjectId(); + Name = name; + CreationDate = DateTime.Now; + IsActive = true; + } + + [BsonCtor] + public Customer(ObjectId _id, string name, DateTime creationDate, bool isActive) + { + CustomerId = _id; + Name = name; + CreationDate = creationDate; + IsActive = isActive; + } +} + +var typedCustomerCollection = db.GetCollection("customer"); +``` + +When `GetCollection` is called, it tries to create instances of `T` by searching for a constructor in the following order: + +* First, it searches for a constructor with `BsonCtorAttribute` +* Then, it searches for a parameterless constructor (and assumes all serialized fields are public and all serialized properties have public setters) +* Finally, it searches for a constructor whose parameters names match with the names of the fields in the document + +Please note that all the parameters in the constructor annotated with `BsonCtorAttribute` must be of a simple type, `BsonDocument` or `BsonArray`. + +### Register a custom type + +You can register your own map function, using the `RegisterType` instance method. To register, you need to provide both serialize and deserialize functions. + +```csharp +BsonMapper.Global.RegisterType +( + serialize: (uri) => uri.AbsoluteUri, + deserialize: (bson) => new Uri(bson.AsString) +); +``` + +* `serialize` function receives an instance of `T` and returns an instance of `BsonValue` +* `deserialize` function receives an instance of `BsonValue` and returns an instance of `T` +* `RegisterType` supports complex objects via `BsonDocument` or `BsonArray` + +### Mapping options + +`BsonMapper` class settings: + +| Name | Default | Description | +| --- | --- | --- | +| `SerializeNullValues` | false | Serialize field if value is `null` | +| `TrimWhitespace` | true | Trim strings properties before mapping to document | +| `EmptyStringToNull` | true | Empty strings convert to `null` | +| `ResolvePropertyName` | (s) => s | A function to map property name to document field name | +| `EnumAsInteger` | false | Map enum to `string` (default) or to `int` | +| `IncludeFields` | false | If mapper should include all class fields | +| `IncludeNonPublic` | false | If mapper should include all private/protected fields/properties | +| `ResolveCollectionName` | typeof(T).Name | When collection name are omitted, use this collection name resolver function | + +Please note that Linq expressions in typed collections will only work over Enum fields if `EnumAsInteger = true`. + +`BsonMapper` offers 2 predefined functions to resolve property names: `UseCamelCase()` and `UseLowerCaseDelimiter('_')`. + +```csharp +BsonMapper.Global.UseLowerCaseDelimiter('_'); + +public class Customer +{ + public int CustomerId { get; set; } + + public string FirstName { get; set; } + + [BsonField("customerLastName")] + public string LastName { get; set; } +} + +var doc = BsonMapper.Global.ToDocument(new Customer { FirstName = "John", LastName = "Doe" }); + +var id = doc["_id"].AsInt; +var john = doc["first_name"].AsString; +var doe = doc["customerLastName"].AsString; +``` + +## AutoId + +There are 4 built-in auto-id functions implemented: + +* `ObjectId`: `ObjectId.NewObjectId()` +* `Guid`: `Guid.NewGuid()` method +* `Int32/Int64`: New collection sequence + +AutoId is only used when there is no `_id` field in the document upon insertion. In strongly-typed documents, `BsonMapper` removes the `_id` field for empty values (like `0` for `Int` or `Guid.Empty` for `Guid`). +Please note that AutoId requires the id field to have a public setter. + +While the AutoId can be used similarly to a sequence in a relational database, it behaves slightly differently. AutoIds are not persisted and have to be recreated in memory when the file is reopened, so it is possible that an id from a previously deleted document ends up being reused. + +## Fluent Mapping + +LiteDB offers a complete fluent API to create custom mappings without using attributes, keeping you domain classes without external references. + +Fluent API uses `EntityBuilder` to add custom mappings to your classes. + +```csharp +var mapper = BsonMapper.Global; + +mapper.Entity() + .Id(x => x.MyCustomKey) // set your document ID + .Ignore(x => x.DoNotSerializeThis) // ignore this property (do not store) + .Field(x => x.CustomerName, "cust_name"); // rename document field +``` + + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/pragmas/Readme.md b/docs/pragmas/Readme.md new file mode 100644 index 000000000..79e6fab01 --- /dev/null +++ b/docs/pragmas/Readme.md @@ -0,0 +1,40 @@ +# Pragmas + +In LiteDB v5, pragmas are variables that can alter the behavior of a datafile. They are stored in the header of the datafile. + +| Name | Read-only | Data type | Description | Default value | +| --- | --- | --- | --- | --- | +| USER\_VERSION | no | int | Reserved for version control by the user. Does not affect the behavior of the datafile. | 0 | +| COLLATION | yes (can be changed with a rebuild) | string (internally stored as int) | Check [Collation](../collation). | `CurrentCulture` and `IgnoreCase` | +| TIMEOUT | no | int | Maximum amount of time (in seconds) that the engine waits for a shared resouce to be unlocked. | 60 | +| LIMIT\_SIZE | no | long | Maximum size (in bytes) that the datafile can grow to. Cannot be smaller than the current datafile size. Cannot be smaller than 4 pages (32768 bytes). | long.MaxValue | +| UTC\_DATE | no | bool | If `false`, dates are converted to local time on retrieval. Storage format is not affected (always in UTC). | false | +| CHECKPOINT | no | int | Maximum number of pages to be stored in the log before a soft checkpoint. If set to `0`, auto-checkpoint and shutdown checkpoint are disabled. | 1000 | + +### Examples + +* `select pragmas from $database;` returns the pragmas in the current datafile +* `pragma USER_VERSION = 1;` sets USER\_VERSION to 1 +* `pragma UTC_DATE = true;` sets UTC\_DATE to true + +## Rebuild Options + +Rebuild options are used to configure a rebuild. + +| Name | Data type | Description | Default value | +| --- | --- | --- | --- | +| collation | string | Check [Collation](../collation). | null (will use `CurrentCulture` and `IgnoreCase` if null) | +| password | string | Defines the password for an encrypted datafile. | null (datafile will not be encrypted) | + +If the `rebuild` command is issued without options, both are assumed to be null. + +Rebuilds are also useful to defragment a datafile, making it smaller and faster to access. + +* `rebuild;` rebuilds the database with the default collation and no password +* `rebuild {"collation": "en-GB/IgnoreCase"};` rebuilds the datafile with the `en-GB` culture and case-insensitive string comparison +* `rebuild {"collation": "pt-BR/None", "password" : "1234"};` rebuilds the datafile with the `pt-BR` culture, case-sensitive string comparison and sets the password to “1234” + + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/queries/Readme.md b/docs/queries/Readme.md new file mode 100644 index 000000000..47603aeb5 --- /dev/null +++ b/docs/queries/Readme.md @@ -0,0 +1,146 @@ +# Queries + +Query filter document inside a collection in three ways: + +* Indexed based search (best option). See +* Full scan on BsonDocument (slower but more powerful) +* LINQ to object (slower but convenient) + +## Query implementations + +`Query` is a static class that creates a query criteria. Each method represents a different criteria operation that can be used to query documents. + +* **`Query.All`** - Returns all documents. Can be specified an index field to read in ascending or descending index order. +* **`Query.EQ`** - Find document are equals (==) to value. +* **`Query.LT/LTE`** - Find documents less then (<) or less then equals (<=) to value. +* **`Query.GT/GTE`** - Find documents greater then (>) or greater then equals (>=) to value. +* **`Query.Between`** - Find documents between start/end value. +* **`Query.In`** - Find documents that are equals of listed values. +* **`Query.Not`** - Find documents that are NOT equals (!=) to value. +* **`Query.StartsWith`** - Find documents that strings starts with value. Valid only for string data type. +* **`Query.Contains`** - Find documents that strings contains value. Valid only for string data type. This query do index search, only index scan (slow for many documents). +* **`Query.Where`** - Find documents based in a `Func` predicate, where `BsonValue` are each key in index. It’s a full index scan based query. +* **`Query.And`** - Apply intersection between two queries results. +* **`Query.Or`** - Apply union between two queries results. + +```csharp +var results = col.Find(Query.EQ("Name", "John Doe")); + +var results = col.Find(Query.GTE("Age", 25)); + +var results = col.Find(Query.And( + Query.EQ("FirstName", "John"), Query.EQ("LastName", "Doe") +)); + +var results = col.Find(Query.StartsWith("Name", "Jo")); + +// Query using multikey index (where products are an array of embedded documents) +var results = col.Find(Query.GT("Products[*].Price", 100)) + +// Execute Func in each key in Name index +var results = col.Find(Query.Where("Name", name => name.AsString.Length > 20)); + +// get last added 100 objects of the collection +var results = collection.Find(Query.All(Query.Descending), limit: 100); + +// find top 100 oldest persons aged between 20 and 30 +var results = col.Find(Query.And(Query.All("Age", Query.Descending), Query.Between("Age", 20, 30)), limit: 100); +``` + +In all queries: + +* In index search, **Field** must be an index name or field in document. +* When no index using, **Field** can be `Path` or an `Expression` +* **Field** name on left side, **Value** (or values) on right side +* Queries are executed in `BsonDocument` class before mapping to your object. You need to use the `BsonDocument` field name and BSON types values. If you are using a custom `ResolvePropertyName` or `[BsonField]` attribute, you must use your document field name and not the property name on your type. See [Object Mapping](Object-Mapping). + +## Find(), FindById(), FindOne() and FindAll() + +Collections are 4 ways to return documents: + +* **`FindAll`**: Returns all documents on collection +* **`FindOne`**: Returns `FirstOrDefault` result of `Find()` +* **`FindById`**: Returns `SingleOrDefault` result of `Find()` by using primary key `_id` index. +* **`Find`**: Return documents using `Query` builder or LINQ expression on collection. + +`Find()` supports `Skip` and `Limit` parameters. These operations are used at the index level, so it’s more efficient than in LINQ to Objects. + +`Find()` method returns an `IEnumerable` of documents. If you want do more complex filters, value as expressions, sorting or transforms results you can use LINQ to Objects. + +Returning an `IEnumerable` your code still connected to datafile. Only when you finish consume all data, datafile will be disconected. + +```csharp +col.EnsureIndex(x => x.Name); + +var result = col + .Find(Query.EQ("Name", "John Doe")) // This filter is executed in LiteDB using index + .Where(x => x.CreationDate >= x.DueDate.AddDays(-5)) // This filter is executed by LINQ to Object + .OrderBy(x => x.Age) + .Select(x => new + { + FullName = x.FirstName + " " + x.LastName, + DueDays = x.DueDate - x.CreationDate + }); // Transform +``` + +## Count() and Exists() + +These two methods are useful because you can count documents (or check if a document exists) without deserializing the document. + +```csharp +// This way is more efficient +var count = collection.Count(Query.EQ("Name", "John Doe")); + +// Than use Find + Count +var count = collection.Find(Query.EQ("Name", "John Doe")).Count(); +``` + +* In the first count, LiteDB uses the index to search and count the number of index occurrences of “Name = John” without deserializing and mapping the document. +* If the `Name` field does not have an index, LiteDB will deserialize the document but will not run the mapper. Still faster than `Find().Count()` +* The same idea applies when using `Exists()`, which is again better than using `Count() >= 1`. Count needs to visit all matched results and `Exists()` stops on first match (similar to LINQ’s `Any` extension method). + +## Min() and Max() + +LiteDB uses a skip list implementation for indexes (See ). Collections offer `Min` and `Max` index values. The implementation is: + +* **`Min`** - Read head index node (MinValue BSON data type) and move to next node. This node is the lowest value in index. If index are empty, returns MinValue. Lowest value is not the first value! +* **`Max`** - Read tail index node (MaxValue BSON data type) and move to previous node. This node is the highest value on index. If index are empty, returns MaxValue. Highest value is not the last value! + +Min/Max required a created index in field. + +## LINQ expressions + +Some LiteDB methods support predicates to allow you to easily query strongly typed documents. If you are working with `BsonDocument`, you need to use classic `Query` class methods. + +```csharp +col.Find(x => x.Name == "John Doe") +// Query.EQ("Name", "John Doe") + +col.Find(x => x.Age > 30) +// Query.GT("Age", 30) + +col.Find(x => x.Name.StartsWith("John") && x.Age > 30) +// Query.And(Query.StartsWith("Name", "John"), Query.GT("Age", 30)) + +// where PhoneNumbers is string[] +col.Find(x => x.PhoneNumbers.Contains("555-1234")) +// Query.EQ("PhoneNumbers", "555-1234") + +// create index on Number inside phone array +col.EnsureIndex(x => x.Phones[0].Number); // ignore 0 index: it's just a syntax to access child +// db.EnsureIndex("col", "Phones[*].Number", false, "$.Phones[*].Number) + +col.Find(x => x.Phones.Select(z => z.Number == "555-1234")) // another way to access child +// Query.EQ("Phones[*].Numbers", "555-1234") + +col.Find(x => !(x.Age > 30)) +// Query.Not(Query.GT("Age", 30)) +``` + +* LINQ implementations are: `==, !=, >, >=, <, <=, StartsWith, Contains (string and IEnumerable), Equals, &&, ||, ! (not)` +* Property name support inner document field: `x => x.Name.Last == "Doe"` +* Behind the scenes, LINQ expressions are converted to `Query` implementations using the `QueryVisitor` class. + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/repository-pattern/Readme.md b/docs/repository-pattern/Readme.md new file mode 100644 index 000000000..f4d96cdfb --- /dev/null +++ b/docs/repository-pattern/Readme.md @@ -0,0 +1,40 @@ +# Repository Pattern + +`LiteRepository` is a new class to access your database. LiteRepository is implemented over `LiteDatabase` and is just a layer to quick access your data without `LiteCollection` class and fluent query + +```csharp +using(var db = new LiteRepository(connectionString)) +{ + // simple access to Insert/Update/Upsert/Delete + db.Insert(new Product { ProductName = "Table", Price = 100 }); + + db.Delete(x => x.Price == 100); + + // query using fluent query + var result = db.Query() + .Include(x => x.Customer) // add dbref 1x1 + .Include(x => x.Products) // add dbref 1xN + .Where(x => x.Date == DateTime.Today) // use indexed query + .Where(x => x.Active) // used indexes query + .ToList(); + + var p = db.Query() + .Where(x => x.ProductName.StartsWith("Table")) + .Where(x => x.Price < 200) + .Limit(10) + .ToEnumerable(); + + var c = db.Query() + .Where(txtName.Text != null, x => x.Name == txtName.Text) // conditional filter + .ToList(); + +} +``` + +Collection names could be omited and will be resolved by new `BsonMapper.ResolveCollectionName` function (default: `typeof(T).Name`). + +This API was inspired by the excellent [NPoco Micro-ORM](https://github.com/schotime/NPoco). + +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.* diff --git a/docs/versioning.md b/docs/versioning/versioning.md similarity index 95% rename from docs/versioning.md rename to docs/versioning/versioning.md index 661c1b600..817a25456 100644 --- a/docs/versioning.md +++ b/docs/versioning/versioning.md @@ -24,14 +24,14 @@ The first prerelease that precedes the 6.0.0 release (commit `a0298891ddcaf7ba48 GitVersion is registered as a local dotnet tool. Restore the tool once (`dotnet tool restore`) and use one of the helpers: ```powershell -# PowerShell (Windows, macOS, Linux) +## PowerShell (Windows, macOS, Linux) ./scripts/gitver/gitversion.ps1 # show version for HEAD ./scripts/gitver/gitversion.ps1 dev~3 # inspect an arbitrary commit ./scripts/gitver/gitversion.ps1 -Json # emit raw JSON ``` ```bash -# Bash (macOS, Linux, Git Bash on Windows) +## Bash (macOS, Linux, Git Bash on Windows) ./scripts/gitver/gitversion.sh # show version for HEAD ./scripts/gitver/gitversion.sh dev~3 # inspect an arbitrary commit ./scripts/gitver/gitversion.sh --json # emit raw JSON @@ -55,3 +55,6 @@ Both scripts resolve the git ref to a SHA, execute GitVersion with the repositor For historical reference, the `v6.0.0-prerelease.0001` tag remains anchored to commit `a0298891ddcaf7ba48c679f1052a6f442f6c094f`, ensuring version ordering continues correctly from the original timeline. +--- + +*Made with ♥ by the LiteDB team – [@mbdavid](https://twitter.com/mbdavid) – MIT License.*