diff --git a/docs/ai-integration/generating-embeddings/assets/add-ai-task-6.png b/docs/ai-integration/generating-embeddings/assets/add-ai-task-6.png index 42d20a25c5..2e165b07f5 100644 Binary files a/docs/ai-integration/generating-embeddings/assets/add-ai-task-6.png and b/docs/ai-integration/generating-embeddings/assets/add-ai-task-6.png differ diff --git a/docs/ai-integration/generating-embeddings/content/_embeddings-generation-task-csharp.mdx b/docs/ai-integration/generating-embeddings/content/_embeddings-generation-task-csharp.mdx index 68198c5af1..e4b4c7eace 100644 --- a/docs/ai-integration/generating-embeddings/content/_embeddings-generation-task-csharp.mdx +++ b/docs/ai-integration/generating-embeddings/content/_embeddings-generation-task-csharp.mdx @@ -2,6 +2,8 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; @@ -14,8 +16,8 @@ import CodeBlock from '@theme/CodeBlock'; to understand the embeddings generation process flow. * In this article: - * [Configuring an embeddings generation task - from the Studio](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx#configuring-an-embeddings-generation-task---from-the-studio) - * [Configuring an embeddings generation task - from the Client API](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx#configuring-an-embeddings-generation-task---from-the-client-api) + * [Configuring an embeddings generation task - from the Studio](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx#configuring-an-embeddings-generation-task-from-the-studio) + * [Configuring an embeddings generation task - from the Client API](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx#configuring-an-embeddings-generation-task-from-the-client-api) * [Define source using PATHS](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx#configure-an-embeddings-generation-task---define-source-using-paths) * [Define source using SCRIPT](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx#configure-an-embeddings-generation-task---define-source-using-script) * [Chunking methods and tokens](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx#chunking-methods-and-tokens) @@ -23,7 +25,7 @@ import CodeBlock from '@theme/CodeBlock'; -## Configuring an embeddings generation task - from the Studio + * **Define the general task settings**: @@ -93,7 +95,7 @@ import CodeBlock from '@theme/CodeBlock'; 4. **Add path configuration** Click to add the specified to the list. 5. **List of paths** - Displays the document properties you added for embedding generation. + This table displays the document properties you added for embedding generation. * **Define the embeddings source - using SCRIPT**: @@ -141,18 +143,24 @@ import CodeBlock from '@theme/CodeBlock'; 1. **Querying** This label indicates that this section configures parameters only for embeddings - generated by the task for **search terms** in vector search queries. + generated by the task for **search terms in vector search queries**. 2. **Chunking method** Select the method for splitting the search term into chunks. Learn more in [Chunking methods and tokens](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx#chunking-methods-and-tokens). 3. **Max tokens per chunk** Enter the maximum number of tokens allowed per chunk (this depends on the service provider). - 4. **Embeddings cache expiration** + 4. **Overlap tokens** + Enter the number of tokens to repeat at the start of each chunk from the end of the previous one. + This helps preserve context between chunks by carrying over some tokens from one to the next. + Applies only to the _"Plain Text: Split Paragraphs"_ and _"Markdown: Split Paragraphs"_ chunking methods. + 5. **Embeddings cache expiration** Set the expiration period for documents stored in the `@embeddings-cache` collection. These documents contain embeddings generated from the search terms, serving as a cache for these embeddings. The default period is `14` days. Learn more in [The embeddings cache collection](../../../ai-integration/generating-embeddings/embedding-collections.mdx#the-embeddings-cache-collection). -## Configuring an embeddings generation task - from the Client API + + + @@ -314,7 +322,9 @@ EmbeddingsTransformation = new EmbeddingsTransformation() -## Chunking methods and tokens + + + **Tokens and processing limits**: @@ -432,7 +442,9 @@ html.strip(htmlText | [htmlText], maxTokensPerChunk); | **maxTokensPerChunk / maxTokensPerLine** | `number` | The maximum number of tokens allowed per chunk.
Default is `512`. | | **overlapTokens** | `number` (optional) | The number of tokens to overlap between consecutive chunks. Helps preserve context continuity across chunks (e.g., between paragraphs).
Default is `0`. | -## Syntax + + + #### The embeddings generation task configuration: @@ -463,7 +475,7 @@ public class EmbeddingsGenerationConfiguration : EtlConfiguration` | A list of properties inside documents that contain text to be embedded, along with their chunking settings. | | **EmbeddingsTransformation** | `EmbeddingsTransformation ` | An object that contains a script defining the transformations and processing applied to the source text before generating embeddings. | | **Quantization** | `VectorEmbeddingType ` | The quantization type for the generated embeddings. | -| **ChunkingOptionsForQuerying** | `ChunkingOptions ` | The chunking method and maximum token limit used when processing search terms in vector search queries. | +| **ChunkingOptionsForQuerying** | `ChunkingOptions ` | The chunking method, maximum token limit, and overlap tokens used when processing search terms in vector search queries. | | **EmbeddingsCacheExpiration** | `TimeSpan ` | The expiration period for documents in the [Embedding cache collection](../../../ai-integration/generating-embeddings/embedding-collections.mdx#the-embeddings-cache-collection) that contain embeddings generated from source documents. | | **EmbeddingsCacheForQueryingExpiration** | `TimeSpan ` | The expiration period for documents in the embedding cache collection that contain embeddings generated from search terms in vector search queries. | @@ -518,3 +530,5 @@ public enum VectorEmbeddingType public AddEmbeddingsGenerationOperation(EmbeddingsGenerationConfiguration configuration); ``` + + \ No newline at end of file diff --git a/docs/ai-integration/generating-embeddings/embeddings-generation_start.mdx b/docs/ai-integration/generating-embeddings/embeddings-generation_start.mdx index 22c140be71..8cb3666fd8 100644 --- a/docs/ai-integration/generating-embeddings/embeddings-generation_start.mdx +++ b/docs/ai-integration/generating-embeddings/embeddings-generation_start.mdx @@ -1,14 +1,14 @@ --- -title: "Generating embeddings: Start" +title: "Generating embeddings" sidebar_label: Start sidebar_position: 0 --- import Admonition from '@theme/Admonition'; - import CardWithImage from "@site/src/components/Common/CardWithImage"; import CardWithImageHorizontal from "@site/src/components/Common/CardWithImageHorizontal"; import ColGrid from "@site/src/components/ColGrid"; + import embedGenStartApiImage from "./assets/embeddings-generation_start_api-image.png"; import embedGenStartStudioImage from "./assets/embeddings-generation_start_studio-image.png"; import aiSearchArticleImage from "./assets/ai-search-article-cover.webp"; @@ -17,43 +17,102 @@ import ayendeBlogImage from "@site/static/img/from-ayende-com.webp"; import webinarThumbnailPlaceholder from "@site/static/img/webinar.webp"; import discordLargeThumbnailPlaceholder from "@site/static/img/discord-lg.webp"; -# Generating embeddings - + + ### Create embeddings to enable AI-powered similarity search. -[Embeddings](https://en.wikipedia.org/wiki/Embedding_(machine_learning)) are numeric vectors that you can create for data (like a text or an image) to capture meanings, contexts, or relationships related to the data. You can then search the data by running intelligent queries over its embeddings using [vector search](../../ai-integration/vector-search/vector-search_start) to find content by similarity rather than exact match. -- RavenDB allows you to create embeddings using native [ongoing embeddings-generation tasks](../../ai-integration/generating-embeddings/embeddings-generation-task) that systematically process document collections and convert document fields (like texts or arrays) into embeddings. To create the embeddings, the tasks can use either an external AI model (such as OpenAI) or RavenDB's default embedding model. -- You can also create embeddings using external embeddings providers and store them in your database (e.g., to handle other content types such as images). -- You can avoid pre-generating embeddings, and let vector search operations generate embeddings on-the-fly, while searching. -- Embeddings can be used by other RavenDB AI features. E.g., [AI agents](../../ai-integration/ai-agents/ai-agents_start) can use vector search to retrieve relevant data requested by the LLM. + +* [Embeddings](https://en.wikipedia.org/wiki/Embedding_(machine_learning)) are numeric vectors that represent the meaning, context, or relationships within the data (e.g., text or images). + They allow you to search for similar content using [Vector search](../../ai-integration/vector-search/vector-search_start), rather than exact matches. + +* RavenDB can generate embeddings automatically using [Embeddings-generation tasks](../../ai-integration/generating-embeddings/embeddings-generation-task) that you define. + These tasks process text fields from documents and convert them into vectors using either an AI model from external providers + (e.g., [OpenAI](../../ai-integration/connection-strings/open-ai)) or [RavenDB’s built-in model](../../ai-integration/connection-strings/embedded). + The resulting vectors are stored in a dedicated collection in your database. + +* Alternatively, you can skip pre-generating embeddings via _Embedding-generation tasks_ and let RavenDB generate them on-the-fly during dynamic vector search queries. + Learn more in [Vector search using a dynamic query](../../ai-integration/vector-search/vector-search-using-dynamic-query). + +* RavenDB does not generate embeddings for non-text content such as images. + However, you can import embeddings for such content from external providers and store them in your database for similarity search. + +* Embeddings also power other AI features in RavenDB, such as [GenAI](../../ai-integration/gen-ai-integration/gen-ai_start) and [AI agents](../../ai-integration/ai-agents/ai-agents_start), + which may use vector search queries to retrieve relevant documents in response to LLM requests. + +* On this start page, you'll find: + * [Use cases](../../ai-integration/generating-embeddings/embeddings-generation_start#use-cases) + * [Technical documentation links](../../ai-integration/generating-embeddings/embeddings-generation_start#technical-documentation-links-) + * [Related in-depth articles](../../ai-integration/generating-embeddings/embeddings-generation_start#related-in-depth-articles) + * [Related lives & videos](../../ai-integration/generating-embeddings/embeddings-generation_start#related-lives--videos) + + + +--- ### Use cases -Embeddings generation tasks can be used to prepare your data for AI-powered search, analysis, and usage, e.g., for - + +Embeddings-generation tasks can prepare your data for AI-powered search, analysis, and retrieval. +Common use cases include: + * **Enterprise knowledge bases** - Generate embeddings for thousands of documents, policies, and procedures to enable instant semantic search + Generate embeddings for thousands of documents, policies, and procedures to enable fast, semantic search. * **Legal document libraries** - Process case law, contracts, and regulations to build searchable legal repositories + Process case law, contracts, and regulations to build a searchable legal knowledge base. * **Product catalogs** - Convert product descriptions, specifications, and reviews into embeddings for enhanced e-commerce search + Convert product descriptions, specifications, and reviews into embeddings for intelligent e-commerce search and recommendations. * **Content management systems** - Transform blog posts, articles, and marketing materials into searchable vector representations + Transform blog posts, articles, and marketing materials into searchable vector embeddings. -### Technical documentation -Learn about generating, storing, and using embeddings in RavenDB. +--- - - - - +### Technical documentation links ✨ + +* [Generating embeddings - overview](../../ai-integration/generating-embeddings/overview) +* [Configure an embeddings generation task - from the Studio](../../ai-integration/generating-embeddings/embeddings-generation-task) +* [Configure an embeddings generation task - from the Client API](../../ai-integration/generating-embeddings/embeddings-generation-task#configuring-an-embeddings-generation-task---from-the-client-api) +* [The embedding collections](../../ai-integration/generating-embeddings/embedding-collections) + +--- + +### Related in-depth articles -#### Learn more: In-depth embeddings generation articles - - + + -### Related lives & Videos +--- + +### Related lives & videos + Learn more about enhancing your applications using vector search operations. + - - + + \ No newline at end of file diff --git a/versioned_docs/version-7.1/ai-integration/generating-embeddings/assets/add-ai-task-6.png b/versioned_docs/version-7.1/ai-integration/generating-embeddings/assets/add-ai-task-6.png index 42d20a25c5..2e165b07f5 100644 Binary files a/versioned_docs/version-7.1/ai-integration/generating-embeddings/assets/add-ai-task-6.png and b/versioned_docs/version-7.1/ai-integration/generating-embeddings/assets/add-ai-task-6.png differ diff --git a/versioned_docs/version-7.1/ai-integration/generating-embeddings/content/_embeddings-generation-task-csharp.mdx b/versioned_docs/version-7.1/ai-integration/generating-embeddings/content/_embeddings-generation-task-csharp.mdx index 68198c5af1..e4b4c7eace 100644 --- a/versioned_docs/version-7.1/ai-integration/generating-embeddings/content/_embeddings-generation-task-csharp.mdx +++ b/versioned_docs/version-7.1/ai-integration/generating-embeddings/content/_embeddings-generation-task-csharp.mdx @@ -2,6 +2,8 @@ import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from '@site/src/components/Panel'; @@ -14,8 +16,8 @@ import CodeBlock from '@theme/CodeBlock'; to understand the embeddings generation process flow. * In this article: - * [Configuring an embeddings generation task - from the Studio](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx#configuring-an-embeddings-generation-task---from-the-studio) - * [Configuring an embeddings generation task - from the Client API](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx#configuring-an-embeddings-generation-task---from-the-client-api) + * [Configuring an embeddings generation task - from the Studio](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx#configuring-an-embeddings-generation-task-from-the-studio) + * [Configuring an embeddings generation task - from the Client API](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx#configuring-an-embeddings-generation-task-from-the-client-api) * [Define source using PATHS](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx#configure-an-embeddings-generation-task---define-source-using-paths) * [Define source using SCRIPT](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx#configure-an-embeddings-generation-task---define-source-using-script) * [Chunking methods and tokens](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx#chunking-methods-and-tokens) @@ -23,7 +25,7 @@ import CodeBlock from '@theme/CodeBlock'; -## Configuring an embeddings generation task - from the Studio + * **Define the general task settings**: @@ -93,7 +95,7 @@ import CodeBlock from '@theme/CodeBlock'; 4. **Add path configuration** Click to add the specified to the list. 5. **List of paths** - Displays the document properties you added for embedding generation. + This table displays the document properties you added for embedding generation. * **Define the embeddings source - using SCRIPT**: @@ -141,18 +143,24 @@ import CodeBlock from '@theme/CodeBlock'; 1. **Querying** This label indicates that this section configures parameters only for embeddings - generated by the task for **search terms** in vector search queries. + generated by the task for **search terms in vector search queries**. 2. **Chunking method** Select the method for splitting the search term into chunks. Learn more in [Chunking methods and tokens](../../../ai-integration/generating-embeddings/embeddings-generation-task.mdx#chunking-methods-and-tokens). 3. **Max tokens per chunk** Enter the maximum number of tokens allowed per chunk (this depends on the service provider). - 4. **Embeddings cache expiration** + 4. **Overlap tokens** + Enter the number of tokens to repeat at the start of each chunk from the end of the previous one. + This helps preserve context between chunks by carrying over some tokens from one to the next. + Applies only to the _"Plain Text: Split Paragraphs"_ and _"Markdown: Split Paragraphs"_ chunking methods. + 5. **Embeddings cache expiration** Set the expiration period for documents stored in the `@embeddings-cache` collection. These documents contain embeddings generated from the search terms, serving as a cache for these embeddings. The default period is `14` days. Learn more in [The embeddings cache collection](../../../ai-integration/generating-embeddings/embedding-collections.mdx#the-embeddings-cache-collection). -## Configuring an embeddings generation task - from the Client API + + + @@ -314,7 +322,9 @@ EmbeddingsTransformation = new EmbeddingsTransformation() -## Chunking methods and tokens + + + **Tokens and processing limits**: @@ -432,7 +442,9 @@ html.strip(htmlText | [htmlText], maxTokensPerChunk); | **maxTokensPerChunk / maxTokensPerLine** | `number` | The maximum number of tokens allowed per chunk.
Default is `512`. | | **overlapTokens** | `number` (optional) | The number of tokens to overlap between consecutive chunks. Helps preserve context continuity across chunks (e.g., between paragraphs).
Default is `0`. | -## Syntax + + + #### The embeddings generation task configuration: @@ -463,7 +475,7 @@ public class EmbeddingsGenerationConfiguration : EtlConfiguration` | A list of properties inside documents that contain text to be embedded, along with their chunking settings. | | **EmbeddingsTransformation** | `EmbeddingsTransformation ` | An object that contains a script defining the transformations and processing applied to the source text before generating embeddings. | | **Quantization** | `VectorEmbeddingType ` | The quantization type for the generated embeddings. | -| **ChunkingOptionsForQuerying** | `ChunkingOptions ` | The chunking method and maximum token limit used when processing search terms in vector search queries. | +| **ChunkingOptionsForQuerying** | `ChunkingOptions ` | The chunking method, maximum token limit, and overlap tokens used when processing search terms in vector search queries. | | **EmbeddingsCacheExpiration** | `TimeSpan ` | The expiration period for documents in the [Embedding cache collection](../../../ai-integration/generating-embeddings/embedding-collections.mdx#the-embeddings-cache-collection) that contain embeddings generated from source documents. | | **EmbeddingsCacheForQueryingExpiration** | `TimeSpan ` | The expiration period for documents in the embedding cache collection that contain embeddings generated from search terms in vector search queries. | @@ -518,3 +530,5 @@ public enum VectorEmbeddingType public AddEmbeddingsGenerationOperation(EmbeddingsGenerationConfiguration configuration); ``` + + \ No newline at end of file diff --git a/versioned_docs/version-7.1/ai-integration/generating-embeddings/embeddings-generation_start.mdx b/versioned_docs/version-7.1/ai-integration/generating-embeddings/embeddings-generation_start.mdx index 22c140be71..8cb3666fd8 100644 --- a/versioned_docs/version-7.1/ai-integration/generating-embeddings/embeddings-generation_start.mdx +++ b/versioned_docs/version-7.1/ai-integration/generating-embeddings/embeddings-generation_start.mdx @@ -1,14 +1,14 @@ --- -title: "Generating embeddings: Start" +title: "Generating embeddings" sidebar_label: Start sidebar_position: 0 --- import Admonition from '@theme/Admonition'; - import CardWithImage from "@site/src/components/Common/CardWithImage"; import CardWithImageHorizontal from "@site/src/components/Common/CardWithImageHorizontal"; import ColGrid from "@site/src/components/ColGrid"; + import embedGenStartApiImage from "./assets/embeddings-generation_start_api-image.png"; import embedGenStartStudioImage from "./assets/embeddings-generation_start_studio-image.png"; import aiSearchArticleImage from "./assets/ai-search-article-cover.webp"; @@ -17,43 +17,102 @@ import ayendeBlogImage from "@site/static/img/from-ayende-com.webp"; import webinarThumbnailPlaceholder from "@site/static/img/webinar.webp"; import discordLargeThumbnailPlaceholder from "@site/static/img/discord-lg.webp"; -# Generating embeddings - + + ### Create embeddings to enable AI-powered similarity search. -[Embeddings](https://en.wikipedia.org/wiki/Embedding_(machine_learning)) are numeric vectors that you can create for data (like a text or an image) to capture meanings, contexts, or relationships related to the data. You can then search the data by running intelligent queries over its embeddings using [vector search](../../ai-integration/vector-search/vector-search_start) to find content by similarity rather than exact match. -- RavenDB allows you to create embeddings using native [ongoing embeddings-generation tasks](../../ai-integration/generating-embeddings/embeddings-generation-task) that systematically process document collections and convert document fields (like texts or arrays) into embeddings. To create the embeddings, the tasks can use either an external AI model (such as OpenAI) or RavenDB's default embedding model. -- You can also create embeddings using external embeddings providers and store them in your database (e.g., to handle other content types such as images). -- You can avoid pre-generating embeddings, and let vector search operations generate embeddings on-the-fly, while searching. -- Embeddings can be used by other RavenDB AI features. E.g., [AI agents](../../ai-integration/ai-agents/ai-agents_start) can use vector search to retrieve relevant data requested by the LLM. + +* [Embeddings](https://en.wikipedia.org/wiki/Embedding_(machine_learning)) are numeric vectors that represent the meaning, context, or relationships within the data (e.g., text or images). + They allow you to search for similar content using [Vector search](../../ai-integration/vector-search/vector-search_start), rather than exact matches. + +* RavenDB can generate embeddings automatically using [Embeddings-generation tasks](../../ai-integration/generating-embeddings/embeddings-generation-task) that you define. + These tasks process text fields from documents and convert them into vectors using either an AI model from external providers + (e.g., [OpenAI](../../ai-integration/connection-strings/open-ai)) or [RavenDB’s built-in model](../../ai-integration/connection-strings/embedded). + The resulting vectors are stored in a dedicated collection in your database. + +* Alternatively, you can skip pre-generating embeddings via _Embedding-generation tasks_ and let RavenDB generate them on-the-fly during dynamic vector search queries. + Learn more in [Vector search using a dynamic query](../../ai-integration/vector-search/vector-search-using-dynamic-query). + +* RavenDB does not generate embeddings for non-text content such as images. + However, you can import embeddings for such content from external providers and store them in your database for similarity search. + +* Embeddings also power other AI features in RavenDB, such as [GenAI](../../ai-integration/gen-ai-integration/gen-ai_start) and [AI agents](../../ai-integration/ai-agents/ai-agents_start), + which may use vector search queries to retrieve relevant documents in response to LLM requests. + +* On this start page, you'll find: + * [Use cases](../../ai-integration/generating-embeddings/embeddings-generation_start#use-cases) + * [Technical documentation links](../../ai-integration/generating-embeddings/embeddings-generation_start#technical-documentation-links-) + * [Related in-depth articles](../../ai-integration/generating-embeddings/embeddings-generation_start#related-in-depth-articles) + * [Related lives & videos](../../ai-integration/generating-embeddings/embeddings-generation_start#related-lives--videos) + + + +--- ### Use cases -Embeddings generation tasks can be used to prepare your data for AI-powered search, analysis, and usage, e.g., for - + +Embeddings-generation tasks can prepare your data for AI-powered search, analysis, and retrieval. +Common use cases include: + * **Enterprise knowledge bases** - Generate embeddings for thousands of documents, policies, and procedures to enable instant semantic search + Generate embeddings for thousands of documents, policies, and procedures to enable fast, semantic search. * **Legal document libraries** - Process case law, contracts, and regulations to build searchable legal repositories + Process case law, contracts, and regulations to build a searchable legal knowledge base. * **Product catalogs** - Convert product descriptions, specifications, and reviews into embeddings for enhanced e-commerce search + Convert product descriptions, specifications, and reviews into embeddings for intelligent e-commerce search and recommendations. * **Content management systems** - Transform blog posts, articles, and marketing materials into searchable vector representations + Transform blog posts, articles, and marketing materials into searchable vector embeddings. -### Technical documentation -Learn about generating, storing, and using embeddings in RavenDB. +--- - - - - +### Technical documentation links ✨ + +* [Generating embeddings - overview](../../ai-integration/generating-embeddings/overview) +* [Configure an embeddings generation task - from the Studio](../../ai-integration/generating-embeddings/embeddings-generation-task) +* [Configure an embeddings generation task - from the Client API](../../ai-integration/generating-embeddings/embeddings-generation-task#configuring-an-embeddings-generation-task---from-the-client-api) +* [The embedding collections](../../ai-integration/generating-embeddings/embedding-collections) + +--- + +### Related in-depth articles -#### Learn more: In-depth embeddings generation articles - - + + -### Related lives & Videos +--- + +### Related lives & videos + Learn more about enhancing your applications using vector search operations. + - - + + \ No newline at end of file