You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/develop/ai/search-and-query/vectors.md
+70-7Lines changed: 70 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -25,7 +25,7 @@ To quickly get started, check out the [Redis vector quickstart guide]({{< relref
25
25
26
26
## Overview
27
27
28
-
1.[**Create a vector index**]({{< relref "develop/ai/search-and-query/vectors#create-a-vector-index" >}}): Redis maintains a secondary index over your data with a defined schema (including vector fields and metadata). Redis supports [`FLAT`]({{< relref "develop/ai/search-and-query/vectors#flat-index" >}}) and [`HNSW`]({{< relref "develop/ai/search-and-query/vectors#hnsw-index" >}}) vector index types.
28
+
1.[**Create a vector index**]({{< relref "develop/ai/search-and-query/vectors#create-a-vector-index" >}}): Redis maintains a secondary index over your data with a defined schema (including vector fields and metadata). Redis supports [`FLAT`]({{< relref "develop/ai/search-and-query/vectors#flat-index" >}}), [`HNSW`]({{< relref "develop/ai/search-and-query/vectors#hnsw-index" >}}) and [`SVS-VAMANA`]({{< relref "develop/ai/search-and-query/vectors#svs-vamana-index" >}}) vector index types.
29
29
1.[**Store and update vectors**]({{< relref "develop/ai/search-and-query/vectors#store-and-update-vectors" >}}): Redis stores vectors and metadata in hashes or JSON objects.
30
30
1.[**Search with vectors**]({{< relref "develop/ai/search-and-query/vectors#search-with-vectors" >}}): Redis supports several advanced querying strategies with vector fields including k-nearest neighbor ([KNN]({{< relref "develop/ai/search-and-query/vectors#knn-vector-search" >}})), [vector range queries]({{< relref "develop/ai/search-and-query/vectors#vector-range-queries" >}}), and [metadata filters]({{< relref "develop/ai/search-and-query/vectors#filters" >}}).
31
31
1.[**Configure vector queries at runtime**]({{< relref "develop/ai/search-and-query/vectors#runtime-query-params" >}}).
@@ -106,12 +106,12 @@ Choose the `HNSW` index type when you have larger datasets (> 1M documents) or w
106
106
[`HNSW`](https://arxiv.org/ftp/arxiv/papers/1603/1603.09320.pdf) supports a number of additional parameters to tune
107
107
the accuracy of the queries, while trading off performance.
|`M`| Max number of outgoing edges (connections) for each node in a graph layer. On layer zero, the max number of connections will be `2 * M`. Higher values increase accuracy, but also increase memory usage and index build time. The default is 16.|
112
-
|`EF_CONSTRUCTION`| Max number of connected neighbors to consider during graph building. Higher values increase accuracy, but also increase index build time. The default is 200.|
113
-
|`EF_RUNTIME`| Max top candidates during KNN search. Higher values increase accuracy, but also increase search latency. The default is 10.|
114
-
|`EPSILON`| Relative factor that sets the boundaries in which a range query may search for candidates. That is, vector candidates whose distance from the query vector is `radius * (1 + EPSILON)` are potentially scanned, allowing more extensive search and more accurate results, at the expense of run time. The default is 0.01.|
|`M`| Max number of outgoing edges (connections) for each node in a graph layer. On layer zero, the max number of connections will be `2 * M`. Higher values increase accuracy, but also increase memory usage and index build time. | 16|
112
+
|`EF_CONSTRUCTION`| Max number of connected neighbors to consider during graph building. Higher values increase accuracy, but also increase index build time. |200 |
113
+
|`EF_RUNTIME`| Max top candidates during KNN search. Higher values increase accuracy, but also increase search latency. | 10|
114
+
|`EPSILON`| Relative factor that sets the boundaries in which a range query may search for candidates. That is, vector candidates whose distance from the query vector is `radius * (1 + EPSILON)` are potentially scanned, allowing more extensive search and more accurate results, at the expense of run time. |0.01 |
115
115
116
116
**Example**
117
117
@@ -129,6 +129,58 @@ FT.CREATE documents
129
129
130
130
In the example above, an index named `documents` is created over hashes with the key prefix `docs:` and an `HNSW` vector field named `doc_embedding` with five index attributes: `TYPE`, `DIM`, `DISTANCE_METRIC`, `M`, and `EF_CONSTRUCTION`.
131
131
132
+
### SVS-VAMANA index
133
+
134
+
Scalable Vector Search (SVS) is an Intel project featuring a graph-based vector search algorithm that is optimized to work with compression methods to reduce memory usage. You can read more about the project [here](https://intel.github.io/ScalableVectorSearch/intro.html). Support for SVS-VAMANA indexing was added in Redis 8.2.
135
+
136
+
Choose the `SVS-VAMANA` index type when all of the following requirements apply:
137
+
- High search performance and scalability are more important than exact search accuracy (similar to HNSW)
|`TYPE`| Vector type (`FLOAT16` or `FLOAT32`). Note: `COMPRESSION` is supported with both types. |
146
+
|`DIM`| The width, or number of dimensions, of the vector embeddings stored in this field. In other words, the number of floating point elements comprising the vector. `DIM` must be a positive integer. The vector used to query this field must have the exact same dimensions as the field itself. |
|`COMPRESSION`| Compression algorithm (`LVQ8`, `LVQ4`, `LVQ4x4`, `LVQ4x8`, `LeanVec4x8`, or `LeanVec8x8`). Vectors will be compressed during indexing. See these Intel pages for best practices on using these algorithms: [`COMPRESSION` settings](https://intel.github.io/ScalableVectorSearch/howtos.html#compression-setting) and [`LeanVec`](https://intel.github.io/ScalableVectorSearch/python/experimental/leanvec.html). | None |
156
+
|`CONSTRUCTION_WINDOW_SIZE`| The search window size to use during graph construction. A higher search window size will yield a higher quality graph since more overall vertexes are considered, but will increase construction time. | 200 |
157
+
|`GRAPH_MAX_DEGREE`| The maximum node degree in the graph. A higher max degree may yield a higher quality graph in terms of recall for performance, but the memory footprint of the graph is directly proportional to the maximum degree. | 32 |
158
+
|`SEARCH_WINDOW_SIZE`| The size of the search window. Increasing the search window size and capacity generally yields more accurate but slower search results. | 10 |
159
+
|`EPSILON`| The range search approximation factor. | 0.01 |
160
+
|`TRAINING_THRESHOLD`| The number of vectors after which training is triggered. Applicable only when used with `COMPRESSION`. If a value is provided, it be less than `100 * DEFAULT_BLOCK_SIZE`, where `DEFAULT_BLOCK_SIZE` is 1024. |`10 * DEFAULT_BLOCK_SIZE`|
161
+
|`LEANVEC_DIM`| The dimension used when using `LeanVec4x8` or `LeanVec8x8` compression for dimensionality reduction. If a value is provided, it should be less than `DIM`. |`DIM / 2`|
162
+
163
+
{{< warning >}}
164
+
On non-Intel platforms, `SVS-VAMANA` with `COMPRESSION` will fall back to Intel’s basic scalar quantization implementation.
165
+
{{< /warning >}}
166
+
167
+
**Example**
168
+
169
+
```
170
+
FT.CREATE documents
171
+
ON HASH
172
+
PREFIX 1 docs:
173
+
SCHEMA doc_embedding VECTOR SVS-VAMANA 12
174
+
TYPE FLOAT32
175
+
DIM 1536
176
+
DISTANCE_METRIC COSINE
177
+
GRAPH_MAX_DEGREE 40
178
+
CONSTRUCTION_WINDOW_SIZE 250
179
+
COMPRESSION LVQ8
180
+
```
181
+
182
+
In the example above, an index named `documents` is created over hashes with the key prefix `docs:` and an `SVS-VAMANA` vector field named `doc_embedding` with six index attributes: `TYPE`, `DIM`, `DISTANCE_METRIC`, `GRAPH_MAX_DEGREE`, `CONSTRUCTION_WINDOW_SIZE` and `COMPRESSION`.
183
+
132
184
### Distance metrics
133
185
134
186
Redis supports three popular distance metrics to measure the degree of similarity between two vectors $u$, $v$ $\in \mathbb{R}^n$, where $n$ is the length of the vectors:
|`EF_RUNTIME`| The maximum number of top candidates to hold during the KNN search. Higher values lead to more accurate results at the expense of a longer query runtime. | The value passed during index creation. The default is 10. |
379
431
|`EPSILON`| The relative factor that sets the boundaries for a vector range query. Vector candidates whose distance from the query vector is `radius * (1 + EPSILON)` are potentially scanned, allowing a more extensive search and more accurate results at the expense of runtime. | The value passed during index creation. The default is 0.01. |
380
432
433
+
**SVS-VAMANA**
434
+
435
+
Optional runtime parameters for SVS-VAMANA indexes are:
436
+
437
+
| Parameter | Description | Default value |
438
+
|:----------|:------------|:--------------|
439
+
|`SEARCH_WINDOW_SIZE`| The size of the search window (applies only to KNN searches). | 10 or the value that was passed upon index creation. |
440
+
|`EPSILON`| The range search approximation factor. | 0.01 or the value that was passed upon index creation. |
441
+
|`USE_SEARCH_HISTORY`| When building an index, either the contents of the search buffer is used or the entire search history is used. The latter case may yield a slightly better graph at the cost of more search time. Boolean options are `OFF`, `ON`, and `AUTO`. `AUTO` is always evaluated internally as `ON`. |`AUTO`|
442
+
|`SEARCH_BUFFER_CAPACITY`| A tuning parameter used in compressed `SVS-VAMANA` indexes, which are using a two-level compression type (`LVQ<X>x<Y>` or one of the `LeanVec` types), that determines the number of vector candidates to collect in the first level of the search, before the re-ranking level (which is the second level). |`SEARCH_WINDOW_SIZE`|
0 commit comments