diff --git a/ydb/docs/en/core/_assets/resource pool.drawio b/ydb/docs/en/core/_assets/resource pool.drawio new file mode 100644 index 000000000000..02ff8ce47de5 --- /dev/null +++ b/ydb/docs/en/core/_assets/resource pool.drawio @@ -0,0 +1,133 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/ydb/docs/en/core/_assets/resource_pool.png b/ydb/docs/en/core/_assets/resource_pool.png new file mode 100644 index 000000000000..66411c3bac8c Binary files /dev/null and b/ydb/docs/en/core/_assets/resource_pool.png differ diff --git a/ydb/docs/en/core/_assets/resources_weight.drawio b/ydb/docs/en/core/_assets/resources_weight.drawio new file mode 100644 index 000000000000..5e37ed89cbfa --- /dev/null +++ b/ydb/docs/en/core/_assets/resources_weight.drawio @@ -0,0 +1,158 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/ydb/docs/en/core/_assets/resources_weight.png b/ydb/docs/en/core/_assets/resources_weight.png new file mode 100644 index 000000000000..cb9f2d191452 Binary files /dev/null and b/ydb/docs/en/core/_assets/resources_weight.png differ diff --git a/ydb/docs/en/core/concepts/_assets/recources_weight.png b/ydb/docs/en/core/concepts/_assets/recources_weight.png new file mode 100644 index 000000000000..2ca20e6f2eec Binary files /dev/null and b/ydb/docs/en/core/concepts/_assets/recources_weight.png differ diff --git a/ydb/docs/en/core/concepts/_assets/resource_pool.png b/ydb/docs/en/core/concepts/_assets/resource_pool.png new file mode 100644 index 000000000000..aa6a48efb2a5 Binary files /dev/null and b/ydb/docs/en/core/concepts/_assets/resource_pool.png differ diff --git a/ydb/docs/en/core/concepts/glossary.md b/ydb/docs/en/core/concepts/glossary.md index eedaa3b9983b..941d1c2a1da2 100644 --- a/ydb/docs/en/core/concepts/glossary.md +++ b/ydb/docs/en/core/concepts/glossary.md @@ -246,6 +246,14 @@ A **coordination node** is a schema object that allows client applications to cr A **semaphore** is an object within a [coordination node](#coordination-node) that provides a synchronization mechanism for distributed applications. Semaphores can be persistent or ephemeral and support operations like creation, acquisition, release, and monitoring. Learn more about [semaphores in {{ ydb-short-name }}](./datamodel/coordination-node.md#semaphore). +{% if feature_resource_pool == true and feature_resource_pool_classifier == true %} + +**Resource pool** is a schema object that describes the restrictions placed on the resources (CPU, RAM, etc.) available for executing queries in this resource pool. A query is always executed in some resource pool. By default, all queries are executed in a resource pool named `default`, which does not impose any restrictions. More information about using resource pools can be found in the article (link) + +**Resource pool classifier** is an object designed to manage the distribution of queries between resource pools(link). It describes the rules by which a pool of resources is selected for each query. These classifiers are global for the entire database(link) and apply to all queries entering it. More information about their use can be found in the article (link) + +{% endif %} + ### YQL {#yql} **YQL ({{ ydb-short-name }} Query Language)** is a high-level language for working with the system. It is a dialect of [ANSI SQL](https://en.wikipedia.org/wiki/SQL). There's a lot of content covering YQL, including a [tutorial](../dev/yql-tutorial/index.md), [reference](../yql/reference/syntax/index.md), and [recipes](../yql/reference/recipes/index.md). diff --git a/ydb/docs/en/core/dev/resource-consumption-management.md b/ydb/docs/en/core/dev/resource-consumption-management.md new file mode 100644 index 000000000000..0f60de36dfcc --- /dev/null +++ b/ydb/docs/en/core/dev/resource-consumption-management.md @@ -0,0 +1,246 @@ +# Workload Manager — resource consumption management + +[Resource pools](../concepts/glossary.md#) allow you to isolate [database](../concepts/glossary.md#database ) resources between running queries or configure resource allocation strategies in case of oversubscription (allocating more resources than are available in system). All resource pools are equal, without any hierarchy, and influence each other only when there is a general shortage of resources. + +For example, one typical resource isolation scenario is to separate two classes of consumers (customer/client/user): + +1. A regular robotic process that generates a report once a day. + +2. Analysts who perform ad hoc queries. + +{% note warning %} + +The presented functionality for managing resource consumption is in the Preview stage + +{% endnote %} + +## Creating a resource-pool + +The example below shows the syntax for creating a separate resource-pool named "olap" on which to run analytics queries: + +```yql +CREATE RESOURCE POOL olap WITH ( + CONCURRENT_QUERY_LIMIT=10, + QUEUE_SIZE=1000, + DATABASE_LOAD_CPU_THRESHOLD=80, + RESOURCES_WEIGHT=100, + QUERY_CPU_LIMIT_PERCENT_PER_NODE=50, + TOTAL_CPU_LIMIT_PERCENT_PER_NODE=70 +) +``` + +For a complete list of resource pool parameters, see the help for (link). Some parameters are global for the entire database (for example, `CONCURRENT_QUERY_LIMIT` , `QUEUE_SIZE` , `DATABASE_LOAD_CPU_THRESHOLD` ), while others apply only to one compute node (for example, `QUERY_CPU_LIMIT_PERCENT_PER_NODE` , `TOTAL_CPU_LIMIT_PERCENT_PER_NODE` , `QUERY_MEMORY_LIMIT_PERCENT_PER_NODE` ). CPU can be shared between all pools in case of oversubscription on one compute node using `RESOURCES_WEIGHT` . + +![resource_pools](../_assets/resource_pool.png) + +Let's look at the example above what these parameters actually mean and how they will affect resource allocation. Let's say the database {{ ydb-short-name }} has $10$ nodes allocated for $10 vCPU$. In total, such a database contains $100 vCPU$. Then on each node for a resource pool named `olap` the following will be allocated: + +$\frac{10 vCPU \cdot TOTAL\_CPU\_LIMIT\_PERCENT\_PER\_NODE}{100} = 10 vCPU \cdot 0.7 = 7 vCPU$ + +In total, with an even distribution of resources across the entire database, the resource pool will be allocated: + +$7 vCPU \cdot 10 \text{ (nodes)} = 70 vCPU$ + +For one query in this resource pool the following will be allocated: + +$\frac{10 vCPU \cdot TOTAL\_CPU\_LIMIT\_PERCENT\_PER\_NODE}{100} \cdot \frac{QUERY\_CPU\_LIMIT\_PERCENT\_PER\_NODE}{100} = 10 vCPU \cdot 0.7 \cdot 0.5 = 3.5 vCPU$ + +### How CONCURRENT_QUERY_LIMIT и QUEUE_SIZE {#concurrent_query_limit} works + +Let's say there are already 9 querys running in the `olap` resource pool . When a new query arrives, its execution will immediately start in parallel with the other 9 queries. Now there will be 10 querys running in the pool. If the 11th query arrives in the pool, it will not start executing, but will be placed in a waiting queue. When at least one of the 10 running querys completes, the 11th query will be removed from the queue and begin executing. + +If there are already $QUEUE\_SIZE = 1000$ querys in the queue , then when sending the 1001st query, the client will immediately receive an error in response, and this query will not be executed. Error example: + +```text +Issues: +
: Error: Request was rejected, number of local pending querys is 20, number of global delayed/running querys is 0, sum of them is larger than allowed limit 1 (including concurrent query limit 1) for pool olap +
: Error: Query failed during adding/waiting in workload pool olap +``` + +The number of concurrently executed queries is affected not only by `CONCURRENT_QUERY_LIMIT` , but also by `DATABASE_LOAD_CPU_THRESHOLD` . + +### How works DATABASE_LOAD_CPU_THRESHOLD {#database_load_cpu_threshold} + +When a query enters a resource pool that has `DATABASE_LOAD_CPU_THRESHOLD` set, 10% of the available CPU on the node is immediately reserved, based on the assumption that the query will at least require that amount of resources. Then, every 10 seconds, resource consumption across the entire database is recalculated, allowing the initial 10% estimate to be refined. This means that if more than 10 querys simultaneously arrive at a cluster node, then no more than 10 querys will be launched for execution, and the rest will wait for clarification of the actual CPU consumption. + +As with `CONCURRENT_QUERY_LIMIT` , when the specified load threshold is exceeded, querys are sent to a waiting queue. + +### Resource allocation according to RESOURCES_WEIGHT {#resources_weight} + +![resource_pools](../_assets/resources_weight.png) + +The `RESOURCES_WEIGHT` parameter starts working only in case of oversubscription and if there is more than one resource pool in the system. In the current implementation, `RESOURCES_WEIGHT` only affects the allocation of `vCPU` resources. When querys appear in the resource pool, it begins to participate in resource allocation. To do this, the limits in the pools are recalculated according to the [Max-min fairness](https://en.wikipedia.org/wiki/Max-min_fairness) algorithm. The redistribution of resources itself is performed on each computing node individually, as shown in the figure above. + +Let's say we have a node on the system with $10 vCPU$ available. Limitations are set to: + +- $TOTAL\_CPU\_LIMIT\_PERCENT\_PER\_NODE = 30$, +- $QUERY\_CPU\_LIMIT\_PERCENT\_PER\_NODE = 50$. + +In this case, the resource pool will have a limit of $3 vCPU$ per node and $1.5 vCPU$ per query in that pool (Figure *a*). If there are 4 such pools on the system and they are all trying to use the maximum resources, that would be $12 vCPU$, which is more than the limit of available resources on the node ($10 vCPU$). In this case, `RESOURCES_WEIGHT` comes into effect, and each pool will be allocated $2.5 vCPU$ (Figure *b*). + +If you need to increase the allocated resources for a particular pool, you can change its weight, for example, to 200. Then this pool will receive $3 vCPU$, and the other pools will equally share the remaining $7 vCPU$, which will be $\frac{7}{3} vCPU$ per pool (Figure *c*). + +{% note warning %} + +The current resource allocation algorithm may change in the future without maintaining backward compatibility. + +{% endnote %} + +## Default resource pool + +Even if no resource pool has been created, there is always a `default` resource pool in the system that cannot be deleted. Any query running in the system always belongs to some pool - there is no situation where a query is not tied to any resource pool. By default, the `default` resource pool settings look like this: + +```yql +CREATE RESOURCE POOL default WITH ( + CONCURRENT_QUERY_LIMIT=-1, + QUEUE_SIZE=-1, + DATABASE_LOAD_CPU_THRESHOLD=-1, + RESOURCES_WEIGHT=-1, + QUERY_MEMORY_LIMIT_PERCENT_PER_NODE=-1, + QUERY_CPU_LIMIT_PERCENT_PER_NODE=-1, + TOTAL_CPU_LIMIT_PERCENT_PER_NODE=-1 +) +``` + +This means that the `default` resource pool does not have any restrictions applied: it operates independently of other pools and has no restrictions on the resources it can consume. In the `default` resource pool , you can change parameters using the query (link), with the exception of the parameters `CONCURRENT_QUERY_LIMIT`, `DATABASE_LOAD_CPU_THRESHOLD` and `QUEUE_SIZE` . This limitation is intentional to minimize the risks associated with incorrectly configuring the default resource pool. + +## Resource pool ACL management + +Access rights must be granted according to the permissions described in the reference for (link) to create, modify, or delete a resource pool. For example, to create resource pools, you need to have `CREATE TABLE` permission to the `.metadata/workload_manager/pools` directory , which can be issued with a query like this: + +```yql +GRANT CREATE TABLE ON `.metadata/workload_manager/pools` TO user1; +``` + +## Creating a resource pool classifier + +[Resorce pool classifier](../concepts/glossary.md#resource-pool-classifier) allows you to set rules by which querys will be distributed between resource pools. The example below is a resource pool qualifier that sends querys from all users to a resource pool named `olap`: + +```yql +CREATE RESOURCE POOL CLASSIFIER olap_classifier +WITH ( + RESOURCE_POOL = 'olap', + MEMBER_NAME = 'all-users@well-known' +); +``` + +- `RESOURCE_POOL` - the name of the resource pool to which a query that satisfies the requirements specified in the resource pool classifier will be sent. +- `MEMBER_NAME` — a group of users or user whose querys will be sent to the specified resource pool. + +## Resource pool classifier ACL management + +There are no restrictions on the use of the resource pool classifier - they are global for the entire database and available to all users. To create, delete, or modify a resource pool classifier, you must have `ALL` permission on the entire database, which can be issued with a query like: + +```yql +GRANT ALL ON `/my_db` TO user1; +``` + +To use a resource pool classifier, the user must have access to the resource pool that the classifier refers to. + +## How to select a resource pool classifier in case of conflicts + +```yql +CREATE RESOURCE POOL CLASSIFIER olap1_classifier +WITH ( + RESOURCE_POOL = 'olap1', + MEMBER_NAME = 'user1@domain' +); + +CREATE RESOURCE POOL CLASSIFIER olap2_classifier +WITH ( + RESOURCE_POOL = 'olap2', + MEMBER_NAME = 'user1@domain' +); +``` + +Let's say there are two resource pool classifiers with conflicting conditions, and the user `user1@domain` matches both resource pools: `olap1` and `olap2` . If no classifier existed in the system before, then `RANK=1000` is set for `olap1` , and `RANK =2000` for `olap2` . Resource pool classifiers with lower `RANK` values ​​have higher priority. In this example, since `olap1` has a higher priority `RANK` than `olap2` , it will be selected. + +You can also independently set `RANK` for resource pool classifiers when creating using the syntactic construction (link), or change `RANK` for existing resource pool classifiers using (link). + +There cannot be two classifiers with the same `RANK` value in the system , which makes it possible to unambiguously determine which resource pool will be selected in the event of conflicting conditions. + +## Example of a priority resource pool + +Let's consider an example of a resource allocation problem between a team of analysts and a fictitious director (CEO). It is important for the CEO to have priority over computing resources that are used for analytical tasks, but it is useful to ensure that the analytics team can utilize more cluster resources during periods of time when the CEO is not using resources. The configuration for this scenario might look like this: + +```yql +CREATE RESOURCE POOL olap WITH ( + CONCURRENT_QUERY_LIMIT=20, + QUEUE_SIZE=100, + DATABASE_LOAD_CPU_THRESHOLD=80, + RESOURCES_WEIGHT=20, + QUERY_CPU_LIMIT_PERCENT_PER_NODE=80, + TOTAL_CPU_LIMIT_PERCENT_PER_NODE=100 +); + +CREATE RESOURCE POOL the_ceo WITH ( + CONCURRENT_QUERY_LIMIT=20, + QUEUE_SIZE=100, + RESOURCES_WEIGHT=100, + QUERY_CPU_LIMIT_PERCENT_PER_NODE=100, + TOTAL_CPU_LIMIT_PERCENT_PER_NODE=100 +); +``` + +In the example above, two resource pools are created: `olap` for the analyst team and `the_ceo` for the CEO. + +- **Resource pool 'olap'**: + + - Has a weight of 20. + - The limit on queries that can be run when the database is overloaded is 80% of available resources. + +- **Resource pool 'the_ceo'**: + + - Has more weight - 80. + - Has no restrictions on queries that can be launched when overloaded. + +## Diagnostics + +### Query plan + +Detailed information about query plans can be found on the page [ structure of query plans ]( ../yql/query_plans.md ). To get information about the resource pool used, you need to run a command to obtain statistics in the `json-unicode` format . Example command: + +```bash +ydb -p sql -s 'select 1' --stats full --format json-unicode +``` + +In the body of the query plan obtained using the above command, you can find useful attributes for diagnosing work with the resource pool. An example of such information: + +```json +"Node Type" : "Query", +"Stats" : { + "TotalDurationUs": 28795, + "ProcessCpuTimeUs": 45, + "Compilation": { + "FromCache": false, + "CpuTimeUs": 7280, + "DurationUs": 21700 + }, + "ResourcePoolId": "default", + "QueuedTimeUs": 0 +}, +"PlanNodeType" : "Query" +``` + +Useful attributes: + +- `TotalDurationUs` — total query execution time, including queue time; +- `ResourcePoolId` — the name of the resource pool to which the query was bound; +- `QueuedTimeUs` — total time the query waited in the queue. + +### Metrics + +Information about resource pool metrics can be found in [metrics reference](../reference/observability/metrics/index.md#resource_pools). + +### System Views + +Information about system views related to resource pools and resource pool qualifiers can be found at (link). + +### See also + +* (create-resource-pool.md) +* (alter-resource-pool.md) +* (drop-resource-pool.md) +* (create-resource-pool-classifier.md) +* (alter-resource-pool-classifier.md) +* (drop-resource-pool-classifier.md) \ No newline at end of file diff --git a/ydb/docs/en/core/dev/toc_p.yaml b/ydb/docs/en/core/dev/toc_p.yaml index 891e5c82eac8..998ec6ea1bf0 100644 --- a/ydb/docs/en/core/dev/toc_p.yaml +++ b/ydb/docs/en/core/dev/toc_p.yaml @@ -22,6 +22,8 @@ items: href: vector-indexes.md - name: Query plans optimization href: query-plans-optimization.md +- name: Resource consumption management + href: resource-consumption-management.md - name: Query hints href: query-hints.md - name: Batch upload diff --git a/ydb/docs/en/core/devops/observability/monitoring.md b/ydb/docs/en/core/devops/observability/monitoring.md index 101bd334a5a4..8492bcefc600 100644 --- a/ydb/docs/en/core/devops/observability/monitoring.md +++ b/ydb/docs/en/core/devops/observability/monitoring.md @@ -50,7 +50,20 @@ To set up {{ ydb-short-name }} cluster monitoring using [Prometheus](https://pro 1. Edit the Prometheus [configuration file](https://github.com/ydb-platform/ydb/tree/main/ydb/deploy/grafana_dashboards/local_ydb_prometheus.yml): +<<<<<<< HEAD 1. In the `targets` section, specify the addresses of all {{ ydb-short-name }} cluster servers and the ports of each storage node and database node running on the server. +======= + 1. In the `targets` section of [`ydbd-storage.yml`](https://github.com/ydb-platform/ydb/tree/main/ydb/deploy/prometheus/ydbd-storage.yml), specify the addresses of all {{ ydb-short-name }} cluster servers and the ports of storage nodes running on the servers. + + ```json + - labels: + container: ydb-static + targets: + - "ydb-s1.example.com:8765" + - "ydb-s2.example.com:8765" + - "ydb-s3.example.com:8765" + ``` +>>>>>>> 2da234665 (YDBDOCS-1394-TRANSLATION (#27515)) For example, for a cluster of three servers, where each server runs one storage node on port 8765 and two database nodes on ports 8766 and 8767, you need to specify nine addresses for all metric subgroups except disk metrics (for disk metric subgroups, you only need to specify storage node addresses): @@ -68,7 +81,28 @@ To set up {{ ydb-short-name }} cluster monitoring using [Prometheus](https://pro - ydb-s3.example.com:8767 ``` +<<<<<<< HEAD For a local single-node YDB cluster, specify one address in the `targets` section: +======= + 1. In the `targets` section of [`ydbd-database.yml`](https://github.com/ydb-platform/ydb/tree/main/ydb/deploy/prometheus/ydbd-database.yml), specify the addresses of all {{ ydb-short-name }} cluster servers and the ports of all database nodes running on the servers. + + ```json + - labels: + container: ydb-dynamic + targets: + - "ydb-s1.example.com:31002" + - "ydb-s1.example.com:31012" + - "ydb-s1.example.com:31022" + - "ydb-s2.example.com:31002" + - "ydb-s2.example.com:31012" + - "ydb-s2.example.com:31022" + - "ydb-s3.example.com:31002" + - "ydb-s3.example.com:31012" + - "ydb-s3.example.com:31022" + ``` + + For a local single-node YDB cluster, specify one address in the targets section: +>>>>>>> 2da234665 (YDBDOCS-1394-TRANSLATION (#27515)) ```json - targets: ["localhost:8765"] diff --git a/ydb/docs/en/core/reference/observability/metrics/index.md b/ydb/docs/en/core/reference/observability/metrics/index.md index d8dd0c763afd..7c3af676643b 100644 --- a/ydb/docs/en/core/reference/observability/metrics/index.md +++ b/ydb/docs/en/core/reference/observability/metrics/index.md @@ -120,3 +120,12 @@ You can analyze a transaction's execution time using a histogram counter. The in | `topic.write.message_size_bytes`
`HIST_RATE`, pieces | A histogram counter. The intervals are specified in bytes. It shows the number of messages which size falls within the boundaries of the interval.
Labels:
- _topic_ – the name of the topic. | | `topic.write.lag_milliseconds`
`HIST_RATE`, pieces | A histogram counter. The intervals are specified in milliseconds. It shows the number of messages where the difference between the write time and the message creation time falls within the specified interval.
Labels:
- _topic_ – the name of the topic. | +## Resource pool metrics {#resource_pools} + +| Metric name
Type, units of measurement | Description
Tags | +| ----- | ----- | +| `kqp.workload_manager.CpuQuotaManager.AverageLoadPercentage`
`RATE`, pieces | Average database load, the `DATABASE_LOAD_CPU_THRESHOLD` works based on this metric. | +| `kqp.workload_manager.InFlightLimit`
`GAUGE`, pieces | Limit on the number of simultaneously running requests. | +| `kqp.workload_manager.GlobalInFly`
`GAUGE`, pieces | The current number of simultaneously running requests. Displayed only for pools with `CONCURRENT_QUERY_LIMIT` or `DATABASE_LOAD_CPU_THRESHOLD` enabled | +| `kqp.workload_manager.QueueSizeLimit`
`GAUGE`, pieces | Queue size of pending requests. | +| `kqp.workload_manager.GlobalDelayedRequests`
`GAUGE`, pieces | The number of requests waiting in the execution queue. Only visible for pools with `CONCURRENT_QUERY_LIMIT` or `DATABASE_LOAD_CPU_THRESHOLD` enabled . | \ No newline at end of file diff --git a/ydb/docs/ru/core/devops/observability/monitoring.md b/ydb/docs/ru/core/devops/observability/monitoring.md index c691186445d4..cbc7732a5759 100644 --- a/ydb/docs/ru/core/devops/observability/monitoring.md +++ b/ydb/docs/ru/core/devops/observability/monitoring.md @@ -71,7 +71,40 @@ http://:/counters/counters=/prometheu Для локального однонодового кластера YDB, в секции `targets` прописывается один адрес: ```json +<<<<<<< HEAD - targets: ["localhost:8765"] +======= + - labels: + container: ydb-static + targets: + - "localhost:8765" + ``` + + 1. В файле [`ydbd-database.yml`](https://github.com/ydb-platform/ydb/tree/main/ydb/deploy/prometheus/ydbd-database.yml) в секции `targets` укажите адреса всех серверов кластера {{ ydb-short-name }} и порты узлов баз данных, работающих на серверах. + + ```json + - labels: + container: ydb-dynamic + targets: + - "ydb-s1.example.com:31002" + - "ydb-s1.example.com:31012" + - "ydb-s1.example.com:31022" + - "ydb-s2.example.com:31002" + - "ydb-s2.example.com:31012" + - "ydb-s2.example.com:31022" + - "ydb-s3.example.com:31002" + - "ydb-s3.example.com:31012" + - "ydb-s3.example.com:31022" + ``` + + Для локального однонодового кластера YDB, в секции `targets` укажите один адрес: + + ```json + - labels: + container: ydb-dynamic + targets: + - "localhost:8765" +>>>>>>> 2da234665 (YDBDOCS-1394-TRANSLATION (#27515)) ``` 1. При необходимости, в секции `tls_config` укажите [сертификат центра регистрации](../deployment-options/manual/initial-deployment.md#tls-certificates) (Certification Authority, CA), которым подписаны остальные сертификаты TLS кластера {{ ydb-short-name }}: diff --git a/ydb/docs/ru/core/public-materials/videos/2025.md b/ydb/docs/ru/core/public-materials/videos/2025.md index 610cef084f0b..736d33dd3c9d 100644 --- a/ydb/docs/ru/core/public-materials/videos/2025.md +++ b/ydb/docs/ru/core/public-materials/videos/2025.md @@ -1,5 +1,80 @@ # Видеозаписи 2025 +<<<<<<< HEAD +======= +## CI/CD как драйвер разработки + +{% include notitle [use_cases_tag](../_includes/tags.md#use_cases) %} + +Доклад Кирилла Сюзева посвящен эволюции CI/CD в сервисе SourceCraft, его архитектуре, выбору и анализу различных решений в области CI/CD (GitLab, GitHub, Arcadia CD), а также решению технических проблем. Одной из таких проблем является Real Time Logging, ключом к решению которой стало использование YDB как места для хранения логов и передаче их пользователю. + +Доклад будет интересен DevOps-инженерам, разработчикам программного обеспечения. + +@[youtube](https://www.youtube.com/watch?v=_2aeqTQALj0&) + +[Слайды](https://presentations.ydb.tech/2025/ru/infra_source_craft/presentation.pdf) + +## Эксплуатация распределенной системы на примере YDB + +{% include notitle [database_internals_tag](../_includes/tags.md#database_internals) %} + +Лекция, рассказанная Олегом Бондарем, в рамках Школы Анализа Данных, посвящена эксплуатации распределённой системы на примере кластеров YDB. В её рамках рассматриваются требования к инфраструктуре, инструменты, обеспечивающие удобство управления кластерами, а также подходы к резервному копированию и восстановлению данных. Особое внимание уделяется методам обновления системы без прерывания обслуживания, управлению хостами и дисковыми ресурсами, мерам обеспечения безопасности — включая шифрование, управление правами доступа, аутентификацию и аудит — а также диагностике и мониторингу работоспособности и производительности кластеров. + +Лекция будет полезна администраторам баз данных, инженерам баз данных и DevOps-инженерам. + +{% list tabs %} + +- YouTube + + @[youtube](https://youtu.be/Dy0VtzQatag?list=PL6Wui14DvQPwuUE1tijVmiBSl2LCLH3ru) + +- VK + + @[vk](https://vk.com/video_ext.php?oid=-84793390&id=456239889&hd=2&autoplay=0) + +{% endlist %} + +## Как YDB выполняет запросы + +{% include notitle [database_internals_tag](../_includes/tags.md#database_internals) %} + +Лекция, расказанная Виталием Гридневым в рамках Школы Анализа Данных, даёт вводное представление об архитектуре YDB: рассматриваются детерминистические транзакции и MVCC, объясняется жизненный цикл выполнения запроса и роль ключевых компонентов системы. На практических примерах показано, как обрабатываются запросы в кластере. Отдельно разбирается реализация вторичных индексов и особенности их добавления к существующим таблицам. В завершение освещаются подходы к тестированию YDB, включая методы верификации корректности и обеспечения надёжности. + +Материал будет полезен backend-разработчикам, инженерам распределённых систем, SRE/DevOps-специалистам и QA-инженерам, работающим с высоконагруженными СУБД. + +{% list tabs %} + +- YouTube + + @[youtube](https://youtu.be/s8KB7dKvBEc?list=PL6Wui14DvQPwuUE1tijVmiBSl2LCLH3ru) + +- VK + + @[vk](https://vk.com/video_ext.php?oid=-84793390&id=456239888&hd=2&autoplay=0) + +{% endlist %} + +## Петабайт в {{ ydb-short-name }} over HDD в процессингах Яндекс.Метрики + +{% include notitle [database_internals_tag](../_includes/tags.md#database_internals) %} + +Антон Барабанов в своем докладе рассказывает о работе с данными в Яндекс.Метрике, включая процессинг данных с использованием {{ ydb-short-name }}. Обсуждаются различные вопросы, возникающие при работе с базами данных — от сложностей с записью и чтением данных до неравномерной нагрузки на узлы кластера, а также вопросы оптимизации запросов, структуры данных и экономической выгоды использования SSD и HDD. + +Эти темы будут интересны инженерам баз данных, разработчикам баз данных и администраторам баз данных. + +@[youtube](https://youtu.be/wsRV7jyM9Gk) + +## Основы векторного и полнотекстового поиска + +{% include notitle [database_internals_tag](../_includes/tags.md#database_internals) %} + +В видео рассмотрены основы векторного и полнотекстового поиска, включая принципы работы векторной базы данных и алгоритмы поиска (точные и приближённые). Обсуждены методы оптимизации поиска — кластеризация, квантование, использование графовых алгоритмов и других техник. Также затронуты вопросы ранжирования, борьбы со спамом и реализации векторного поиска в базах данных. + +Доклад будет полезен специалистам по анализу данных, ML-инженерам, разработчикам баз данных и backend-разработчикам. + +@[vk](https://vk.com/video_ext.php?oid=-84793390&id=456239927&hd=2&autoplay=0) + +>>>>>>> 2da234665 (YDBDOCS-1394-TRANSLATION (#27515)) ## Эволюция векторного поиска в {{ ydb-short-name }}: от базовых методов к масштабируемому глобальному индексу {#2025-saint-highload-vector-search} {% include notitle [database_internals_tag](../_includes/tags.md#database_internals) %} diff --git a/ydb/docs/ru/core/reference/configuration/query_service_config.md b/ydb/docs/ru/core/reference/configuration/query_service_config.md index 49a16840a9bb..3453a3e6891e 100644 --- a/ydb/docs/ru/core/reference/configuration/query_service_config.md +++ b/ydb/docs/ru/core/reference/configuration/query_service_config.md @@ -30,7 +30,7 @@ || || `available_external_data_sources` | пустой список -| Список с разрешенными типами внешних источников. Применяется при `all_external_data_sources_are_available: false`. +| Список с разрешенными типами внешних источников. Применяется при `all_external_data_sources_are_available: false`. Возможные значения: diff --git a/ydb/docs/ru/core/yql/reference/syntax/insert_into.md b/ydb/docs/ru/core/yql/reference/syntax/insert_into.md index d385992cd8a8..ab6b1213cf60 100644 --- a/ydb/docs/ru/core/yql/reference/syntax/insert_into.md +++ b/ydb/docs/ru/core/yql/reference/syntax/insert_into.md @@ -70,7 +70,6 @@ SELECT key FROM my_table_source; {% endif %} - {% if feature_federated_queries %} При работе с [внешними файловыми источниками данных](../../../concepts/datamodel/external_data_source.md) можно дополнительно указывать ряд параметров: @@ -102,3 +101,30 @@ SELECT {% endif %} +<<<<<<< HEAD +======= +## INSERT INTO ... RETURNING {insert-into-returning} + +Используется для вставки строк и одновременного возврата значений из них. Это позволяет получить информацию о вставленных записях за один запрос, избавляя от необходимости выполнять последующий SELECT. + +### Примеры + +* Возврат всех значений вставленной строки + +```yql +INSERT INTO some_table (id, year, color, price) +VALUES (1103, 2023, 'blue', 400) +RETURNING *; +``` + +* Возврат конкретных столбцов + +```yql +INSERT INTO some_table (id, color, price) +VALUES + (1101, 'red', 200), + (1102, 'green', 300) +RETURNING id, price; +``` + +>>>>>>> 2da234665 (YDBDOCS-1394-TRANSLATION (#27515)) diff --git a/ydb/docs/ru/core/yql/reference/syntax/update.md b/ydb/docs/ru/core/yql/reference/syntax/update.md index 8f8c5a8e2569..d2d4671fa0d0 100644 --- a/ydb/docs/ru/core/yql/reference/syntax/update.md +++ b/ydb/docs/ru/core/yql/reference/syntax/update.md @@ -33,6 +33,33 @@ UPDATE my_table ON SELECT * FROM $to_update; ``` +<<<<<<< HEAD +======= +## UPDATE ... RETURNING {update-returning} + +Используется для обновления строк и одновременного возврата их новых значений. Это позволяет получить информацию об обновленных записях за один запрос, избавляя от необходимости выполнять дополнительный SELECT. + +### Примеры + +* Возврат всех значений обновленных строк + +```yql +UPDATE orders +SET status = 'shipped' +WHERE order_date < '2023-01-01' +RETURNING *; +``` + +* Возврат конкретных столбцов + +```yql +UPDATE products +SET price = price * 0.9 +WHERE category = 'Electronics' +RETURNING product_id, name, price AS new_price; +``` + +>>>>>>> 2da234665 (YDBDOCS-1394-TRANSLATION (#27515)) {% if feature_batch_operations %} ## См. также