diff --git a/semantic-kernel/concepts/ai-services/TOC.yml b/semantic-kernel/concepts/ai-services/TOC.yml index 8d24a54b..fd35b379 100644 --- a/semantic-kernel/concepts/ai-services/TOC.yml +++ b/semantic-kernel/concepts/ai-services/TOC.yml @@ -6,4 +6,6 @@ - name: Embedding generation href: embedding-generation/TOC.yml - name: AI Integrations - href: integrations.md \ No newline at end of file + href: integrations.md +- name: Realtime + href: realtime.md \ No newline at end of file diff --git a/semantic-kernel/concepts/ai-services/index.md b/semantic-kernel/concepts/ai-services/index.md index 3e473aec..9ce5d2fd 100644 --- a/semantic-kernel/concepts/ai-services/index.md +++ b/semantic-kernel/concepts/ai-services/index.md @@ -14,21 +14,23 @@ One of the main features of Semantic Kernel is its ability to add different AI s Within Semantic Kernel, there are interfaces for the most popular AI tasks. In the table below, you can see the services that are supported by each of the SDKs. -| Services | C# | Python | Java | Notes | -|-----------------------------------|:----:|:------:|:----:|-------| -| [Chat completion](./chat-completion/index.md) | ✅ | ✅ | ✅ | -| Text generation | ✅ | ✅ | ✅ | -| Embedding generation (Experimental) | ✅ | ✅ | ✅ | -| Text-to-image (Experimental) | ✅ | ✅ | ❌ | -| Image-to-text (Experimental) | ✅ | ❌ | ❌ | -| Text-to-audio (Experimental) | ✅ | ✅ | ❌ | -| Audio-to-text (Experimental) | ✅ | ✅ | ❌ | +| Services | C# | Python | Java | Notes | +| --------------------------------------------- | :---: | :----: | :---: | ----- | +| [Chat completion](./chat-completion/index.md) | ✅ | ✅ | ✅ | +| Text generation | ✅ | ✅ | ✅ | +| Embedding generation (Experimental) | ✅ | ✅ | ✅ | +| Text-to-image (Experimental) | ✅ | ✅ | ❌ | +| Image-to-text (Experimental) | ✅ | ❌ | ❌ | +| Text-to-audio (Experimental) | ✅ | ✅ | ❌ | +| Audio-to-text (Experimental) | ✅ | ✅ | ❌ | +| [Realtime](./realtime.md) (Experimental) | ❌ | ✅ | ❌ | > [!TIP] > In most scenarios, you will only need to add chat completion to your kernel, but to support multi-modal AI, you can add any of the above services to your kernel. ## Next steps + To learn more about each of the services, please refer to the specific articles for each service type. In each of the articles we provide sample code for adding the service to the kernel across multiple AI service providers. > [!div class="nextstepaction"] -> [Learn about chat completion](./chat-completion/index.md) \ No newline at end of file +> [Learn about chat completion](./chat-completion/index.md) diff --git a/semantic-kernel/concepts/ai-services/integrations.md b/semantic-kernel/concepts/ai-services/integrations.md index a8d34ec5..d30a6984 100644 --- a/semantic-kernel/concepts/ai-services/integrations.md +++ b/semantic-kernel/concepts/ai-services/integrations.md @@ -18,21 +18,22 @@ With the available AI connectors, developers can easily build AI agents with swa ### AI Services -| Services | C# | Python | Java | Notes | -|-----------------------------------|:----:|:------:|:----:|-------| -| Text Generation | ✅ | ✅ | ✅ | Example: Text-Davinci-003 | -| Chat Completion | ✅ | ✅ | ✅ | Example: GPT4, Chat-GPT | -| Text Embeddings (Experimental) | ✅ | ✅ | ✅ | Example: Text-Embeddings-Ada-002 | -| Text to Image (Experimental) | ✅ | ✅ | ❌ | Example: Dall-E | -| Image to Text (Experimental) | ✅ | ❌ | ❌ | Example: Pix2Struct | -| Text to Audio (Experimental) | ✅ | ✅ | ❌ | Example: Text-to-speech | -| Audio to Text (Experimental) | ✅ | ✅ | ❌ | Example: Whisper | +| Services | C# | Python | Java | Notes | +| ------------------------------ | :---: | :----: | :---: | -------------------------------- | +| Text Generation | ✅ | ✅ | ✅ | Example: Text-Davinci-003 | +| Chat Completion | ✅ | ✅ | ✅ | Example: GPT4, Chat-GPT | +| Text Embeddings (Experimental) | ✅ | ✅ | ✅ | Example: Text-Embeddings-Ada-002 | +| Text to Image (Experimental) | ✅ | ✅ | ❌ | Example: Dall-E | +| Image to Text (Experimental) | ✅ | ❌ | ❌ | Example: Pix2Struct | +| Text to Audio (Experimental) | ✅ | ✅ | ❌ | Example: Text-to-speech | +| Audio to Text (Experimental) | ✅ | ✅ | ❌ | Example: Whisper | +| Realtime (Experimental) | ❌ | ✅ | ❌ | Example: gpt-4o-realtime-preview | ## Additional plugins If you want to extend the functionality of your AI agent, you can use plugins to integrate with other Microsoft services. Here are some of the plugins that are available for Semantic Kernel: -| Plugin | C# | Python | Java | Description | -| ---------- | :-: | :----: | :--: | ----------- | -| Logic Apps | ✅ | ✅ | ✅ | Build workflows within Logic Apps using its available connectors and import them as plugins in Semantic Kernel. [Learn more](../plugins/adding-logic-apps-as-plugins.md). | -| Azure Container Apps Dynamic Sessions | ✅ | ✅ | ❌ | With dynamic sessions, you can recreate the Code Interpreter experience from the Assistants API by effortlessly spinning up Python containers where AI agents can execute Python code. [Learn more](/azure/container-apps/sessions). | +| Plugin | C# | Python | Java | Description | +| ------------------------------------- | :---: | :----: | :---: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Logic Apps | ✅ | ✅ | ✅ | Build workflows within Logic Apps using its available connectors and import them as plugins in Semantic Kernel. [Learn more](../plugins/adding-logic-apps-as-plugins.md). | +| Azure Container Apps Dynamic Sessions | ✅ | ✅ | ❌ | With dynamic sessions, you can recreate the Code Interpreter experience from the Assistants API by effortlessly spinning up Python containers where AI agents can execute Python code. [Learn more](/azure/container-apps/sessions). | diff --git a/semantic-kernel/concepts/ai-services/realtime.md b/semantic-kernel/concepts/ai-services/realtime.md new file mode 100644 index 00000000..80eec9ad --- /dev/null +++ b/semantic-kernel/concepts/ai-services/realtime.md @@ -0,0 +1,189 @@ +--- +title: Realtime AI Integrations for Semantic Kernel +description: Learn about realtime multi-modal AI integrations available in Semantic Kernel. +author: eavanvalkenburg +ms.topic: conceptual +ms.author: edvan +ms.date: 02/26/2025 +ms.service: semantic-kernel +--- + +# Realtime Multi-modal APIs + +The first realtime API integration for Semantic Kernel has been added, it is currently only available in Python and considered experimental. This is because the underlying services are still being developed and are subject to changes and we might need to make breaking changes to the API in Semantic Kernel as we learn from customers how to use this and as we add other providers of these kinds of models and APIs. +## Realtime Client abstraction + +To support different realtime APIs from different vendors, using different protocols, a new client abstraction has been added to the kernel. This client is used to connect to the realtime service and send and receive messages. +The client is responsible for handling the connection to the service, sending messages, and receiving messages. The client is also responsible for handling any errors that occur during the connection or message sending/receiving process. + +### Realtime API + +Any realtime client implements the following methods: + +| Method | Description | +| ---------------- | ------------------------------------------------------------------------------------------------------------------ | +| `create_session` | Creates a new session | +| `update_session` | Updates an existing session | +| `delete_session` | Deletes an existing session | +| `receive` | This is a asynchronous generator method that listens for messages from the service and yields them as they arrive. | +| `send` | Sends a message to the service | + +### Python implementations + +The python version of Semantic Kernel currently supports the following realtime clients: + +| Client | Protocol | Modalities | Function calling enabled | Description | +| ------ | --------- | ------------ | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| OpenAI | Websocket | Text & Audio | Yes | The OpenAI Realtime API is a websocket based api that allows you to send and receive messages in realtime, this connector uses the OpenAI Python package to connect and receive and send messages. | +| OpenAI | WebRTC | Text & Audio | Yes | The OpenAI Realtime API is a WebRTC based api that allows you to send and receive messages in realtime, it needs a webRTC compatible audio track at session creation time. | +| Azure | Websocket | Text & Audio | Yes | The Azure Realtime API is a websocket based api that allows you to send and receive messages in realtime, this uses the same package as the OpenAI websocket connector. | + +## Getting started + +To get started with the Realtime API, you need to install the `semantic-kernel` package with the `realtime` extra. + +```bash +pip install semantic-kernel[realtime] +``` + +Depending on how you want to handle audio, you might need additional packages to interface with speakers and microphones, like `pyaudio` or `sounddevice`. + +### Websocket clients + +Then you can create a kernel and add the realtime client to it, this shows how to do that with a AzureRealtimeWebsocket connection, you can replace AzureRealtimeWebsocket with OpenAIRealtimeWebsocket without any further changes. + +```python +from semantic_kernel.connectors.ai.open_ai import ( + AzureRealtimeWebsocket, + AzureRealtimeExecutionSettings, + ListenEvents, +) +from semantic_kernel.contents import RealtimeAudioEvent, RealtimeTextEvent + +# this will use environment variables to get the api key, endpoint, api version and deployment name. +realtime_client = AzureRealtimeWebsocket() +settings = AzureRealtimeExecutionSettings(voice='alloy') +async with realtime_client(settings=settings, create_response=True): + async for event in realtime_client.receive(): + match event: + # receiving a piece of audio (and send it to a undefined audio player) + case RealtimeAudioEvent(): + await audio_player.add_audio(event.audio) + # receiving a piece of audio transcript + case RealtimeTextEvent(): + # Semantic Kernel parses the transcript to a TextContent object captured in a RealtimeTextEvent + print(event.text.text, end="") + case _: + # OpenAI Specific events + if event.service_type == ListenEvents.SESSION_UPDATED: + print("Session updated") + if event.service_type == ListenEvents.RESPONSE_CREATED: + print("\nMosscap (transcript): ", end="") +``` + +There are two important things to note, the first is that the `realtime_client` is an async context manager, this means that you can use it in an async function and use `async with` to create the session. +The second is that the `receive` method is an async generator, this means that you can use it in a for loop to receive messages as they arrive. + +### WebRTC client + +The setup of a WebRTC connection is a bit more complex and so we need a extra parameter when creating the client. This parameter, `audio_track` needs to be a object that implements the `MediaStreamTrack` protocol of the `aiortc` package, this is also demonstrated in the samples that are linked below. + +To create a client that uses WebRTC, you would do the following: + +```python +from semantic_kernel.connectors.ai.open_ai import ( + ListenEvents, + OpenAIRealtimeExecutionSettings, + OpenAIRealtimeWebRTC, +) +from aiortc.mediastreams import MediaStreamTrack + +class AudioRecorderWebRTC(MediaStreamTrack): + # implement the MediaStreamTrack methods. + +realtime_client = OpenAIRealtimeWebRTC(audio_track=AudioRecorderWebRTC()) +# Create the settings for the session +settings = OpenAIRealtimeExecutionSettings( + instructions=""" +You are a chat bot. Your name is Mosscap and +you have one goal: figure out what people need. +Your full name, should you need to know it, is +Splendid Speckled Mosscap. You communicate +effectively, but you tend to answer with long +flowery prose. +""", + voice="shimmer", +) +audio_player = AudioPlayer +async with realtime_client(settings=settings, create_response=True): + async for event in realtime_client.receive(): + match event.event_type: + # receiving a piece of audio (and send it to a undefined audio player) + case "audio": + await audio_player.add_audio(event.audio) + case "text": + # the model returns both audio and transcript of the audio, which we will print + print(event.text.text, end="") + case "service": + # OpenAI Specific events + if event.service_type == ListenEvents.SESSION_UPDATED: + print("Session updated") + if event.service_type == ListenEvents.RESPONSE_CREATED: + print("\nMosscap (transcript): ", end="") +``` + +Both of these samples receive the audio as RealtimeAudioEvent and then they pass that to a unspecified audio_player object. + +### Audio output callback + +Next to this we have a parameter called `audio_output_callback` on the `receive` method and on the class creation. This callback will be called first before any further handling of the audio and gets a `numpy` array of the audio data, instead of it being parsed into AudioContent and returned as a RealtimeAudioEvent that you can then handle, which is what happens above. This has shown to give smoother audio output because there is less overhead between the audio data coming in and it being given to the player. + +This example shows how to define and use the `audio_output_callback`: + +```python +from semantic_kernel.connectors.ai.open_ai import ( + ListenEvents, + OpenAIRealtimeExecutionSettings, + OpenAIRealtimeWebRTC, +) +from aiortc.mediastreams import MediaStreamTrack + +class AudioRecorderWebRTC(MediaStreamTrack): + # implement the MediaStreamTrack methods. + +class AudioPlayer: + async def play_audio(self, content: np.ndarray): + # implement the audio player + +realtime_client = OpenAIRealtimeWebRTC(audio_track=AudioRecorderWebRTC()) +# Create the settings for the session +settings = OpenAIRealtimeExecutionSettings( + instructions=""" +You are a chat bot. Your name is Mosscap and +you have one goal: figure out what people need. +Your full name, should you need to know it, is +Splendid Speckled Mosscap. You communicate +effectively, but you tend to answer with long +flowery prose. +""", + voice="shimmer", +) +audio_player = AudioPlayer +async with realtime_client(settings=settings, create_response=True): + async for event in realtime_client.receive(audio_output_callback=audio_player.play_audio): + match event.event_type: + # no need to handle case: "audio" + case "text": + # the model returns both audio and transcript of the audio, which we will print + print(event.text.text, end="") + case "service": + # OpenAI Specific events + if event.service_type == ListenEvents.SESSION_UPDATED: + print("Session updated") + if event.service_type == ListenEvents.RESPONSE_CREATED: + print("\nMosscap (transcript): ", end="") +``` + +### Samples + +There are four samples in [our repo](https://github.com/microsoft/semantic-kernel/tree/main/python/samples/concepts/realtime), they cover both the basics using both websockets and WebRTC, as well as a more complex setup including function calling. Finally there is a more [complex demo](https://github.com/microsoft/semantic-kernel/tree/main/python/samples/demos/call_automation) that uses [Azure Communication Services](/azure/communication-services/) to allow you to call your Semantic Kernel enhanced realtime API. diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/TOC.yml b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/TOC.yml index b4c46306..de9c5f57 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/TOC.yml +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/TOC.yml @@ -6,6 +6,8 @@ href: azure-cosmosdb-mongodb-connector.md - name: Azure CosmosDB NoSQL connector href: azure-cosmosdb-nosql-connector.md +- name: Chroma connector + href: chroma-connector.md - name: Couchbase connector href: couchbase-connector.md - name: Elasticsearch connector diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-ai-search-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-ai-search-connector.md index 305ce851..a85ee0d4 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-ai-search-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-ai-search-connector.md @@ -17,27 +17,59 @@ ms.service: semantic-kernel The Azure AI Search Vector Store connector can be used to access and manage data in Azure AI Search. The connector has the following characteristics. -| Feature Area | Support | -|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| -| Collection maps to | Azure AI Search Index | -| Supported key property types | string | -| Supported data property types | <ul><li>string</li><li>int</li><li>long</li><li>double</li><li>float</li><li>bool</li><li>DateTimeOffset</li><li>*and enumerables of each of these types*</li></ul> | -| Supported vector property types | ReadOnlyMemory\<float\> | -| Supported index types | <ul><li>Hnsw</li><li>Flat</li></ul> | -| Supported distance functions | <ul><li>CosineSimilarity</li><li>DotProductSimilarity</li><li>EuclideanDistance</li></ul> | -| Supported filter clauses | <ul><li>AnyTagEqualTo</li><li>EqualTo</li></ul> | -| Supports multiple vectors in a record | Yes | -| IsFilterable supported? | Yes | -| IsFullTextSearchable supported? | Yes | -| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +::: zone pivot="programming-language-csharp" + +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure AI Search Index | +| Supported key property types | string | +| Supported data property types | <ul><li>string</li><li>int</li><li>long</li><li>double</li><li>float</li><li>bool</li><li>DateTimeOffset</li><li>*and enumerables of each of these types*</li></ul> | +| Supported vector property types | ReadOnlyMemory\<float\> | +| Supported index types | <ul><li>Hnsw</li><li>Flat</li></ul> | +| Supported distance functions | <ul><li>CosineSimilarity</li><li>DotProductSimilarity</li><li>EuclideanDistance</li></ul> | +| Supported filter clauses | <ul><li>AnyTagEqualTo</li><li>EqualTo</li></ul> | +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | +| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +::: zone-end +::: zone pivot="programming-language-python" +| Feature Area | Support | +| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure AI Search Index | +| Supported key property types | string | +| Supported data property types | <ul><li>string</li><li>int</li><li>long</li><li>double</li><li>float</li><li>bool</li><li>DateTimeOffset</li><li>*and iterables of each of these types*</li></ul> | +| Supported vector property types | list[float], list[int], ndarray | +| Supported index types | <ul><li>Hnsw</li><li>Flat</li></ul> | +| Supported distance functions | <ul><li>CosineSimilarity</li><li>DotProductSimilarity</li><li>EuclideanDistance</li><li>Hamming</li></ul> | +| Supported filter clauses | <ul><li>AnyTagEqualTo</li><li>EqualTo</li></ul> | +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | +::: zone-end +::: zone pivot="programming-language-java" +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure AI Search Index | +| Supported key property types | string | +| Supported data property types | <ul><li>string</li><li>int</li><li>long</li><li>double</li><li>float</li><li>bool</li><li>DateTimeOffset</li><li>*and enumerables of each of these types*</li></ul> | +| Supported vector property types | ReadOnlyMemory\<float\> | +| Supported index types | <ul><li>Hnsw</li><li>Flat</li></ul> | +| Supported distance functions | <ul><li>CosineSimilarity</li><li>DotProductSimilarity</li><li>EuclideanDistance</li></ul> | +| Supported filter clauses | <ul><li>AnyTagEqualTo</li><li>EqualTo</li></ul> | +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | +| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +::: zone-end ## Limitations Notable Azure AI Search connector functionality limitations. -| Feature Area | Workaround | -|--------------------------------------------------------------------------------------| -----------------------------------------------------------------------------------------------| -| Configuring full text search analyzers during collection creation is not supported. | Use the Azure AI Search Client SDK directly for collection creation | +| Feature Area | Workaround | +| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------- | +| Configuring full text search analyzers during collection creation is not supported. | Use the Azure AI Search Client SDK directly for collection creation | ::: zone pivot="programming-language-csharp" diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md index 46fbbf96..c50ed06f 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md @@ -13,30 +13,50 @@ ms.service: semantic-kernel > [!WARNING] > The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. -::: zone pivot="programming-language-csharp" - ## Overview The Azure CosmosDB MongoDB Vector Store connector can be used to access and manage data in Azure CosmosDB MongoDB (vCore). The connector has the following characteristics. -| Feature Area | Support | -|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| -| Collection maps to | Azure Cosmos DB MongoDB (vCore) Collection + Index | -| Supported key property types | string | -| Supported data property types | <ul><li>string</li><li>int</li><li>long</li><li>double</li><li>float</li><li>decimal</li><li>bool</li><li>DateTime</li><li>*and enumerables of each of these types*</li></ul> | -| Supported vector property types | <ul><li>ReadOnlyMemory\<float\></li><li>ReadOnlyMemory\<double\></li></ul> | -| Supported index types | <ul><li>Hnsw</li><li>IvfFlat</li></ul> | -| Supported distance functions | <ul><li>CosineDistance</li><li>DotProductSimilarity</li><li>EuclideanDistance</li></ul> | -| Supported filter clauses | <ul><li>EqualTo</li></ul> | -| Supports multiple vectors in a record | Yes | -| IsFilterable supported? | Yes | -| IsFullTextSearchable supported? | No | -| StoragePropertyName supported? | No, use BsonElementAttribute instead. [See here for more info.](#data-mapping) | +::: zone pivot="programming-language-csharp" + +| Feature Area | Support | +| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure Cosmos DB MongoDB (vCore) Collection + Index | +| Supported key property types | string | +| Supported data property types | <ul><li>string</li><li>int</li><li>long</li><li>double</li><li>float</li><li>decimal</li><li>bool</li><li>DateTime</li><li>*and enumerables of each of these types*</li></ul> | +| Supported vector property types | <ul><li>ReadOnlyMemory\<float\></li><li>ReadOnlyMemory\<double\></li></ul> | +| Supported index types | <ul><li>Hnsw</li><li>IvfFlat</li></ul> | +| Supported distance functions | <ul><li>CosineDistance</li><li>DotProductSimilarity</li><li>EuclideanDistance</li></ul> | +| Supported filter clauses | <ul><li>EqualTo</li></ul> | +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | No | +| StoragePropertyName supported? | No, use BsonElementAttribute instead. [See here for more info.](#data-mapping) | +::: zone-end +::: zone pivot="programming-language-python" +| Feature Area | Support | +| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure Cosmos DB MongoDB (vCore) Collection + Index | +| Supported key property types | string | +| Supported data property types | <ul><li>string</li><li>int</li><li>long</li><li>double</li><li>float</li><li>decimal</li><li>bool</li><li>DateTime</li><li>*and iterables of each of these types*</li></ul> | +| Supported vector property types | <ul><li>list[float]</li><li>list[int]</li></ul><li>ndarray</li></ul> | +| Supported index types | <ul><li>Hnsw</li><li>IvfFlat</li></ul> | +| Supported distance functions | <ul><li>CosineDistance</li><li>DotProductSimilarity</li><li>EuclideanDistance</li></ul> | +| Supported filter clauses | <ul><li>EqualTo</li></ul><ul><li>AnyTagsEqualTo</li></ul> | +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | No | +::: zone-end +::: zone pivot="programming-language-java" +More info coming soon. +::: zone-end ## Limitations This connector is compatible with Azure Cosmos DB MongoDB (vCore) and is *not* designed to be compatible with Azure Cosmos DB MongoDB (RU). +::: zone pivot="programming-language-csharp" + ## Getting started Add the Azure CosmosDB MongoDB Vector Store connector NuGet package to your project. @@ -165,9 +185,52 @@ public class Hotel ::: zone-end ::: zone pivot="programming-language-python" -## Coming soon +## Getting started -More info coming soon. +Add the Azure CosmosDB MongoDB Vector Store dependencies to your environment. Because the Azure CosmosDB MongoDB connector is built on the MongoDB Atlas connector and uses the same client as that one, you need to install with these extras: + +```bash +pip install semantic-kernel[azure, mongo] +``` + +You can then create the vector store. + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBforMongoDBStore + +# If the right environment settings are set, namely AZURE_COSMOS_DB_MONGODB_CONNECTION_STRING and optionally AZURE_COSMOS_DB_MONGODB_DATABASE_NAME, this is enough to create the Store: +store = AzureCosmosDBforMongoDBStore() +``` + +Alternatively, you can also pass in your own mongodb client if you want to have more control over the client construction: + +```python +from pymongo import AsyncMongoClient +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBforMongoDBStore + +client = AsyncMongoClient(...) +store = AzureCosmosDBforMongoDBStore(mongo_client=client) +``` + +When a client is passed in, Semantic Kernel will not close the connection for you, so you need to ensure to close it, for instance with a `async with` statement. + +You can also create a collection directly, without the store. + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBforMongoDBCollection + +# `hotel` is a class created with the @vectorstoremodel decorator +collection = AzureCosmosDBforMongoDBCollection( + collection_name="my_collection", + data_model_type=hotel +) +``` + +## Serialization + +Since the Azure CosmosDB for MongoDB connector needs a simple dict with the fields corresponding to the index as the input, the serialization is quite easy, it only uses a predetermined key `_id`, so we replace the key of the data model with that if it is not already `_id`. + +For more details on this concept see the [serialization documentation](./../serialization.md). ::: zone-end ::: zone pivot="programming-language-java" diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/chroma-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/chroma-connector.md new file mode 100644 index 00000000..17dc15cc --- /dev/null +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/chroma-connector.md @@ -0,0 +1,104 @@ +--- +title: Using the Semantic Kernel Chroma Vector Store connector (Preview) +description: Contains information on how to use a Semantic Kernel Vector store connector to access and manipulate data in ChromaDB. +zone_pivot_groups: programming-languages +author: eavanvalkenburg +ms.topic: conceptual +ms.author: edvan +ms.date: 02/26/2025 +ms.service: semantic-kernel +--- + +# Using the Chroma connector (Preview) + +> [!WARNING] +> The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. + +::: zone pivot="programming-language-csharp" + +## Not supported + +Not supported. + +::: zone-end +::: zone pivot="programming-language-python" + +## Overview + +The Chroma Vector Store connector can be used to access and manage data in Chroma. The connector has the +following characteristics. + +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------------- | +| Collection maps to | Chroma collection | +| Supported key property types | string | +| Supported data property types | All types that are supported by System.Text.Json (either built-in or by using a custom converter) | +| Supported vector property types | <ul><li>list[float]</li><li>list[int]</li><li>ndarray</li></ul> | +| Supported index types | <ul><li>HNSW</li></ul> | +| Supported distance functions | <ul><li>CosineSimilarity</li><li>DotProductSimilarity</li><li>EuclideanSquaredDistance</li></ul> | +| Supported filter clauses | <ul><li>AnyTagEqualTo</li><li>EqualTo</li></ul> | +| Supports multiple vectors in a record | No | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | + +## Limitations + +Notable Chroma connector functionality limitations. + +| Feature Area | Workaround | +| ------------------ | ------------------------------------------------------------------------------------------------------------------------- | +| Client-server mode | Use the client.HttpClient and pass the result to the `client` parameter, we do not support a AsyncHttpClient at this time | +| Chroma Cloud | Unclear at this time, as Chroma Cloud is still in private preview | + +## Getting Started + +Add the Chroma Vector Store connector dependencies to your project. + +```bash +pip install semantic-kernel[chroma] +``` + +You can then create the vector store. + +```python +from semantic_kernel.connectors.memory.chroma import ChromaStore + +store = ChromaStore() +``` + +Alternatively, you can also pass in your own mongodb client if you want to have more control over the client construction: + +```python +from chromadb import Client +from semantic_kernel.connectors.memory.chroma import ChromaStore + +client = Client(...) +store = ChromaStore(client=client) +``` + +You can also create a collection directly, without the store. + +```python +from semantic_kernel.connectors.memory.chroma import ChromaCollection + +# `hotel` is a class created with the @vectorstoremodel decorator +collection = ChromaCollection( + collection_name="my_collection", + data_model_type=hotel +) +``` + +## Serialization + +The Chroma client returns both `get` and `search` results in tabular form, this means that there are between 3 and 5 lists being returned in a dict, the lists are 'keys', 'documents', 'embeddings', and optionally 'metadatas' and 'distances'. The Semantic Kernel Chroma connector will automatically convert this into a list of `dict` objects, which are then parsed back to your data model. + +It could be very interesting performance wise to do straight serialization from this format into a dataframe-like structure as that saves a lot of rebuilding of the data structure. This is not done for you, even when using container mode, you would have to specify this yourself, for more details on this concept see the [serialization documentation](./../serialization.md). + +::: zone-end +::: zone pivot="programming-language-java" + +## Not supported + +Not supported. + +::: zone-end diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/index.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/index.md index 15b30d1d..ef0c44cc 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/index.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/index.md @@ -22,60 +22,59 @@ Semantic Kernel provides a number of out-of-the-box Vector Store integrations ma ::: zone pivot="programming-language-csharp" -| Vector Store Connectors | C# | Uses officially supported SDK | Maintainer / Vendor | -|------------------------------------------------------------|:---------------:|:---------------------------------:|:----------------------------------:| -| [Azure AI Search](./azure-ai-search-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Cosmos DB MongoDB (vCore)](./azure-cosmosdb-mongodb-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Cosmos DB No SQL](./azure-cosmosdb-nosql-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Couchbase](./couchbase-connector.md) | ✅ | ✅ | Couchbase | -| [Elasticsearch](./elasticsearch-connector.md) | ✅ | ✅ | Elastic | -| Chroma | Planned | | | -| [In-Memory](./inmemory-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | -| Milvus | Planned | | | -| [MongoDB](./mongodb-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Pinecone](./pinecone-connector.md) | ✅ | ❌ | Microsoft Semantic Kernel Project | -| [Postgres](./postgres-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Qdrant](./qdrant-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Redis](./redis-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| Sql Server | Planned | | | -| [SQLite](./sqlite-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Volatile (In-Memory)](./volatile-connector.md) | Deprecated (use In-Memory) | N/A | Microsoft Semantic Kernel Project | -| [Weaviate](./weaviate-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| Vector Store Connectors | C# | Uses officially supported SDK | Maintainer / Vendor | +| ------------------------------------------------------------------ | :------------------------: | :---------------------------: | :-------------------------------: | +| [Azure AI Search](./azure-ai-search-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Cosmos DB MongoDB (vCore)](./azure-cosmosdb-mongodb-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Cosmos DB No SQL](./azure-cosmosdb-nosql-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Couchbase](./couchbase-connector.md) | ✅ | ✅ | Couchbase | +| [Elasticsearch](./elasticsearch-connector.md) | ✅ | ✅ | Elastic | +| Chroma | Planned | | | +| [In-Memory](./inmemory-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | +| Milvus | Planned | | | +| [MongoDB](./mongodb-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Pinecone](./pinecone-connector.md) | ✅ | ❌ | Microsoft Semantic Kernel Project | +| [Postgres](./postgres-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Qdrant](./qdrant-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Redis](./redis-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| Sql Server | Planned | | | +| [SQLite](./sqlite-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Volatile (In-Memory)](./volatile-connector.md) | Deprecated (use In-Memory) | N/A | Microsoft Semantic Kernel Project | +| [Weaviate](./weaviate-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | ::: zone-end ::: zone pivot="programming-language-python" -| Vector Store Connectors | Python | Uses officially supported SDK | Maintainer / Vendor | -|------------------------------------------------------------|:---------------:|:----------------------------------:|:----------------------------------:| -| [Azure AI Search](./azure-ai-search-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Cosmos DB MongoDB (vCore)](./azure-cosmosdb-mongodb-connector.md) | In Development | ✅ | Microsoft Semantic Kernel Project | -| [Cosmos DB No SQL](./azure-cosmosdb-nosql-connector.md) | In Development | ✅ | Microsoft Semantic Kernel Project | -| [Elasticsearch](./elasticsearch-connector.md) | Planned | | | -| Chroma | Planned | | | -| [In-Memory](./inmemory-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | -| Milvus | Planned | | | -| [MongoDB](./mongodb-connector.md) | In Development | ✅ | Microsoft Semantic Kernel Project | -| [Pinecone](./pinecone-connector.md) | In Development | ✅ | Microsoft Semantic Kernel Project | -| [Postgres](./postgres-connector.md) | ✅ | | Microsoft Semantic Kernel Project | -| [Qdrant](./qdrant-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Redis](./redis-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| Sql Server | Planned | | | -| [SQLite](./sqlite-connector.md) | In Development | ✅ | Microsoft Semantic Kernel Project | -| [Volatile (In-Memory)](./volatile-connector.md) | Deprecated (use In-Memory) | N/A | Microsoft Semantic Kernel Project | -| [Weaviate](./weaviate-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | +| Vector Store Connectors | Python | Uses officially supported SDK | Maintainer / Vendor | +| ------------------------------------------------------------------ | :-----: | :---------------------------: | :-------------------------------: | +| [Azure AI Search](./azure-ai-search-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Cosmos DB MongoDB (vCore)](./azure-cosmosdb-mongodb-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Cosmos DB No SQL](./azure-cosmosdb-nosql-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Chroma](./chroma-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Elasticsearch](./elasticsearch-connector.md) | Planned | | | +| Faiss | Planned | | | +| [In-Memory](./inmemory-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | +| [MongoDB](./mongodb-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Pinecone](./pinecone-connector.md) | Planned | ✅ | Microsoft Semantic Kernel Project | +| [Postgres](./postgres-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Qdrant](./qdrant-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Redis](./redis-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| SQL Server | Planned | ✅ | Microsoft Semantic Kernel Project | +| SQLite | Planned | ✅ | Microsoft Semantic Kernel Project | +| [Weaviate](./weaviate-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | ::: zone-end ::: zone pivot="programming-language-java" -| Vector Store Connectors | Java | Uses officially supported SDK | Maintainer / Vendor | -|------------------------------------------------------------|:--------------:|:----------------------------------:|:----------------------------------:| -| [Azure AI Search](./azure-ai-search-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| HSQLDB | Use [JDBC](./jdbc-connector.md) | ✅ | Microsoft Semantic Kernel Project | -| [JDBC](./jdbc-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| MySQL | Use [JDBC](./jdbc-connector.md) | ✅ | Microsoft Semantic Kernel Project | -| Postgres | Use [JDBC](./jdbc-connector.md) | | Microsoft Semantic Kernel Project | -| [Redis](./redis-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| SQLite | Use [JDBC](./jdbc-connector.md) | ✅ | Microsoft Semantic Kernel Project | -| [Volatile (In-Memory)](./volatile-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | +| Vector Store Connectors | Java | Uses officially supported SDK | Maintainer / Vendor | +| ------------------------------------------------- | :-----------------------------: | :---------------------------: | :-------------------------------: | +| [Azure AI Search](./azure-ai-search-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| HSQLDB | Use [JDBC](./jdbc-connector.md) | ✅ | Microsoft Semantic Kernel Project | +| [JDBC](./jdbc-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| MySQL | Use [JDBC](./jdbc-connector.md) | ✅ | Microsoft Semantic Kernel Project | +| Postgres | Use [JDBC](./jdbc-connector.md) | | Microsoft Semantic Kernel Project | +| [Redis](./redis-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| SQLite | Use [JDBC](./jdbc-connector.md) | ✅ | Microsoft Semantic Kernel Project | +| [Volatile (In-Memory)](./volatile-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | ::: zone-end diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/mongodb-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/mongodb-connector.md index 6c406610..8671e1a3 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/mongodb-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/mongodb-connector.md @@ -13,25 +13,45 @@ ms.service: semantic-kernel > [!WARNING] > The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. -::: zone pivot="programming-language-csharp" - ## Overview The MongoDB Vector Store connector can be used to access and manage data in MongoDB. The connector has the following characteristics. -| Feature Area | Support | -|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| -| Collection maps to | MongoDB Collection + Index | -| Supported key property types | string | -| Supported data property types | <ul><li>string</li><li>int</li><li>long</li><li>double</li><li>float</li><li>decimal</li><li>bool</li><li>DateTime</li><li>*and enumerables of each of these types*</li></ul> | -| Supported vector property types | <ul><li>ReadOnlyMemory\<float\></li><li>ReadOnlyMemory\<double\></li></ul> | -| Supported index types | N/A | -| Supported distance functions | <ul><li>CosineSimilarity</li><li>DotProductSimilarity</li><li>EuclideanDistance</li></ul> | -| Supported filter clauses | <ul><li>EqualTo</li></ul> | -| Supports multiple vectors in a record | Yes | -| IsFilterable supported? | Yes | -| IsFullTextSearchable supported? | No | -| StoragePropertyName supported? | No, use BsonElementAttribute instead. [See here for more info.](#data-mapping) | +::: zone pivot="programming-language-csharp" + +| Feature Area | Support | +| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | MongoDB Collection + Index | +| Supported key property types | string | +| Supported data property types | <ul><li>string</li><li>int</li><li>long</li><li>double</li><li>float</li><li>decimal</li><li>bool</li><li>DateTime</li><li>*and enumerables of each of these types*</li></ul> | +| Supported vector property types | <ul><li>ReadOnlyMemory\<float\></li><li>ReadOnlyMemory\<double\></li></ul> | +| Supported index types | N/A | +| Supported distance functions | <ul><li>CosineSimilarity</li><li>DotProductSimilarity</li><li>EuclideanDistance</li></ul> | +| Supported filter clauses | <ul><li>EqualTo</li></ul> | +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | No | +| StoragePropertyName supported? | No, use BsonElementAttribute instead. [See here for more info.](#data-mapping) | +::: zone-end +::: zone pivot="programming-language-python" + +| Feature Area | Support | +| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | MongoDB Collection + Index | +| Supported key property types | string | +| Supported data property types | <ul><li>string</li><li>int</li><li>long</li><li>double</li><li>float</li><li>decimal</li><li>bool</li><li>DateTime</li><li>*and iterables of each of these types*</li></ul> | +| Supported vector property types | <ul><li>list[float]</li><li>list[int]</li><li>ndarray</li></ul> | +| Supported index types | <ul><li>Hnsw</li><li>IvfFlat</li></ul> | +| Supported distance functions | <ul><li>CosineDistance</li><li>DotProductSimilarity</li><li>EuclideanDistance</li></ul> | +| Supported filter clauses | <ul><li>EqualTo</li></ul><ul><li>AnyTagsEqualTo</li></ul> | +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | No | +::: zone-end +::: zone pivot="programming-language-java" +More info coming soon. +::: zone-end +::: zone pivot="programming-language-csharp" ## Getting started @@ -136,9 +156,52 @@ public class Hotel ::: zone-end ::: zone pivot="programming-language-python" -## Coming soon +## Getting started -More info coming soon. +Add the MongoDB Atlas Vector Store dependencies to your environment. It needs the `pymongo` package which is included in the mongo extra: , you need to install with these extras: + +```bash +pip install semantic-kernel[mongo] +``` + +You can then create the vector store. + +```python +from semantic_kernel.connectors.memory.mongodb_atlas import MongoDBAtlasStore + +# If the right environment settings are set, namely MONGODB_ATLAS_CONNECTION_STRING and optionally MONGODB_ATLAS_DATABASE_NAME and MONGODB_ATLAS_INDEX_NAME, this is enough to create the Store: +store = MongoDBAtlasStore() +``` + +Alternatively, you can also pass in your own mongodb client if you want to have more control over the client construction: + +```python +from pymongo import AsyncMongoClient +from semantic_kernel.connectors.memory.mongodb_atlas import MongoDBAtlasStore + +client = AsyncMongoClient(...) +store = MongoDBAtlasStore(mongo_client=client) +``` + +When a client is passed in, Semantic Kernel will not close the connection for you, so you need to ensure to close it, for instance with a `async with` statement. + +You can also create a collection directly, without the store. + +```python +from semantic_kernel.connectors.memory.mongodb_atlas import MongoDBAtlasCollection + +# `hotel` is a class created with the @vectorstoremodel decorator +collection = MongoDBAtlasCollection( + collection_name="my_collection", + data_model_type=hotel +) +``` + +## Serialization + +Since the MongoDB Atlas connector needs a simple dict with the fields corresponding to the index as the input, the serialization is quite easy, it only uses a predetermined key `_id`, so we replace the key of the data model with that if it is not already `_id`. + +For more details on this concept see the [serialization documentation](./../serialization.md). ::: zone-end ::: zone pivot="programming-language-java" diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/weaviate-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/weaviate-connector.md index 52a0178a..b3efc390 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/weaviate-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/weaviate-connector.md @@ -13,33 +13,51 @@ ms.service: semantic-kernel > [!WARNING] > The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. -::: zone pivot="programming-language-csharp" - ## Overview The Weaviate Vector Store connector can be used to access and manage data in Weaviate. The connector has the following characteristics. -| Feature Area | Support | -|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| -| Collection maps to | Weaviate Collection | -| Supported key property types | Guid | -| Supported data property types | <ul><li>string</li><li>byte</li><li>short</li><li>int</li><li>long</li><li>double</li><li>float</li><li>decimal</li><li>bool</li><li>DateTime</li><li>DateTimeOffset</li><li>Guid</li><li>*and enumerables of each of these types*</li></ul> | -| Supported vector property types | <ul><li>ReadOnlyMemory\<float\></li><li>ReadOnlyMemory\<double\></li></ul> | -| Supported index types | <ul><li>Hnsw</li><li>Flat</li><li>Dynamic</li></ul> | -| Supported distance functions | <ul><li>CosineDistance</li><li>NegativeDotProductSimilarity</li><li>EuclideanSquaredDistance</li><li>Hamming</li><li>ManhattanDistance</li></ul>| -| Supported filter clauses | <ul><li>AnyTagEqualTo</li><li>EqualTo</li></ul> | -| Supports multiple vectors in a record | Yes | -| IsFilterable supported? | Yes | -| IsFullTextSearchable supported? | Yes | -| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +::: zone pivot="programming-language-csharp" + +| Feature Area | Support | +| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Weaviate Collection | +| Supported key property types | Guid | +| Supported data property types | <ul><li>string</li><li>byte</li><li>short</li><li>int</li><li>long</li><li>double</li><li>float</li><li>decimal</li><li>bool</li><li>DateTime</li><li>DateTimeOffset</li><li>Guid</li><li>*and enumerables of each of these types*</li></ul> | +| Supported vector property types | <ul><li>ReadOnlyMemory\<float\></li><li>ReadOnlyMemory\<double\></li></ul> | +| Supported index types | <ul><li>Hnsw</li><li>Flat</li><li>Dynamic</li></ul> | +| Supported distance functions | <ul><li>CosineDistance</li><li>NegativeDotProductSimilarity</li><li>EuclideanSquaredDistance</li><li>Hamming</li><li>ManhattanDistance</li></ul> | +| Supported filter clauses | <ul><li>AnyTagEqualTo</li><li>EqualTo</li></ul> | +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | +| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +::: zone-end +::: zone pivot="programming-language-python" +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Weaviate Collection | +| Supported key property types | Guid | +| Supported data property types | <ul><li>string</li><li>byte</li><li>short</li><li>int</li><li>long</li><li>double</li><li>float</li><li>decimal</li><li>bool</li><li>*and iterables of each of these types*</li></ul> | +| Supported vector property types | <ul><li>list[float]</li><li>list[int]</li><li>ndarray</li></ul> | +| Supported index types | <ul><li>Hnsw</li><li>Flat</li><li>Dynamic</li></ul> | +| Supported distance functions | <ul><li>CosineDistance</li><li>NegativeDotProductSimilarity</li><li>EuclideanSquaredDistance</li><li>Hamming</li><li>ManhattanDistance</li></ul> | +| Supported filter clauses | <ul><li>AnyTagEqualTo</li><li>EqualTo</li></ul> | +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | +::: zone-end +::: zone pivot="programming-language-java" +Coming soon. +::: zone-end ## Limitations Notable Weaviate connector functionality limitations. -| Feature Area | Workaround | -|------------------------------------------------------------------------| -----------------------------------------------------------------------------------------------| -| Using the 'vector' property for single vector objects is not supported | Use of the 'vectors' property is supported instead. | +| Feature Area | Workaround | +| ---------------------------------------------------------------------- | --------------------------------------------------- | +| Using the 'vector' property for single vector objects is not supported | Use of the 'vectors' property is supported instead. | > [!WARNING] > Weaviate requires collection names to start with an upper case letter. If you do not provide a collection name with an upper case letter, Weaviate will return an error when you try and create your collection. The error that you will see is `Cannot query field "mycollection" on type "GetObjectsObj". Did you mean "Mycollection"?` where `mycollection` is your collection name. In this example, if you change your collection name to `Mycollection` instead, this will fix the error. @@ -181,9 +199,64 @@ public class Hotel ::: zone-end ::: zone pivot="programming-language-python" -## Coming soon +## Getting Started -More info coming soon. +Add the Weaviate Vector Store connector dependencies to your project. + +```bash +pip install semantic-kernel[weaviate] +``` + +You can then create the vector store, it uses environment settings to connect: + +For using Weaviate Cloud: + +- url: WEAVIATE_URL +- api_key: WEAVIATE_API_KEY + +For using Weaviate Local (i.e. Weaviate in a Docker container): + +- local_host: WEAVIATE_LOCAL_HOST +- local_port: WEAVIATE_LOCAL_PORT +- local_grpc_port: WEAVIATE_LOCAL_GRPC_PORT + +If you want to use embedded: + +- use_embed: WEAVIATE_USE_EMBED + +These should be set exclusively, so only one set of the above is present, otherwise it will raise an exception. + +```python +from semantic_kernel.connectors.memory.weaviate import WeaviateStore + +store = WeaviateStore() +``` + +Alternatively, you can also pass in your own mongodb client if you want to have more control over the client construction: + +```python +import weaviate +from semantic_kernel.connectors.memory.weaviate import WeaviateStore + +client = weaviate.WeaviateAsyncClient(...) +store = WeaviateStore(async_client=client) +``` + +You can also create a collection directly, without the store. + +```python +from semantic_kernel.connectors.memory.weaviate import WeaviateCollection + +# `hotel` is a class created with the @vectorstoremodel decorator +collection = WeaviateCollection( + collection_name="my_collection", + data_model_type=hotel +) +``` + +## Serialization + +The Weaviate client returns it's own objects which are parsed and turned into dicts in the regular flow, for more details on this concept see the [serialization documentation](./../serialization.md). ::: zone-end ::: zone pivot="programming-language-java"