Merge branch 'main' of https://github.com/ClickHouse/clickhouse-docs into Blargian-patch-965414

Author: Blargian

contribute/autogenerated-documentation-from-source.md

@@ -61,18 +61,16 @@ TO DO
 ## Functions
 
 Documentation for functions is autogenerated from the documentation in system tables (`system.functions`) which is 
-registered along with the function registration in the C++ code.
+registered along with the function registration in the C++ code. ([Example](https://github.com/ClickHouse/ClickHouse/blob/a0216c8120f00479f50237e41c897c2bf244e1ef/src/Functions/FunctionChar.cpp#L116-L158))
 
-To add a new page:
-- add SQL to generate the markdown from system tables
-- update `autogenerate-settings.sh` in the section which specifies which markdown file contents to
-  copy to which file in between the `<!--AUTOGENERATED_START-->` and `<!--AUTOGENERATED_END-->`
+This [script](https://github.com/ClickHouse/clickhouse-docs/blob/main/scripts/settings/autogenerate-settings.sh) is used to generate
+the markdown docs for functions.
 
-The following pages are autogenerated from the source code for functions:
-- [Arithmetic](/sql-reference/functions/arithmetic-functions) ([SQL](../scripts/settings/arithmetic-functions.sql))
-- [Arrays](/sql-reference/functions/array-functions) ([SQL](../scripts/settings/array-functions.sql))
-- [Bit](/sql-reference/functions/bit-functions) ([SQL](../scripts/settings/bit-functions.sql))
-- [Bitmap](/sql-reference/functions/bitmap-functions) ([SQL](../scripts/settings/bitmap-functions.sql))
-- [Comparison](/sql-reference/functions/comparison-functions) ([SQL](../scripts/settings/comparison-functions.sql))
-- [Conditional](/sql-reference/functions/conditional-functions) ([SQL](../scripts/settings/conditional-functions.sql))
+Adding a new page can be done by:
 
+1. Adding the function [category](https://github.com/ClickHouse/ClickHouse/blob/85700c135ccad89d3651a7a92ca63bb989743ba6/src/Common/FunctionDocumentation.cpp#L222-L273) to the [list of function categories to generate](https://github.com/ClickHouse/clickhouse-docs/blob/c31d64bc5d71c6c3afc5b520dbb73386e303447b/scripts/settings/autogenerate-settings.sh#L253-L278).
+2. The docs are automatically generated for the categories defined above by [this SQL code](https://github.com/ClickHouse/clickhouse-docs/blob/main/scripts/settings/generate-functions.sql). It produces a temporary markdown file with convention: `category-functions.md`. If the category name contains spaces, e.g. "String Splitting" then the temporary file will be `string_splitting-functions.md`.
+3. Register the temporary file name [here](https://github.com/ClickHouse/clickhouse-docs/blob/c31d64bc5d71c6c3afc5b520dbb73386e303447b/scripts/settings/autogenerate-settings.sh#L372)
+4. Register the destination file you want to insert the markdown into [here](https://github.com/ClickHouse/clickhouse-docs/blob/c31d64bc5d71c6c3afc5b520dbb73386e303447b/scripts/settings/autogenerate-settings.sh#L399)
+
+The script takes the content of the temporary markdown file and inserts it into the destination file between tags `<!--AUTOGENERATED_START-->` and `<!--AUTOGENERATED_END-->`. If those tags do not exist you will get a warning.

docs/_snippets/_community_monitoring.md

@@ -0,0 +1,7 @@
+## Community monitoring solutions {#community-monitoring}
+
+The ClickHouse community has developed comprehensive monitoring solutions that integrate with popular observability stacks. [ClickHouse Monitoring](https://github.com/duyet/clickhouse-monitoring) provides a complete monitoring setup with pre-built dashboards. This open source project offers a quick-start approach for teams looking to implement ClickHouse monitoring with established best practices and proven dashboard configurations.
+
+:::note
+Like other direct database monitoring approaches, this solution queries ClickHouse system tables directly, which prevents instances from idling and impacts cost optimization. 
+:::

docs/_snippets/_direct_observability_integration_options.md

@@ -0,0 +1,40 @@
+import Image from '@theme/IdealImage';
+import AdvancedDashboard from '@site/static/images/cloud/manage/monitoring/advanced_dashboard.png';
+import NativeAdvancedDashboard from '@site/static/images/cloud/manage/monitoring/native_advanced_dashboard.png';
+
+### Direct Grafana plugin integration {#direct-grafana}
+
+The ClickHouse data source plugin for Grafana enables visualization and exploration of data directly from ClickHouse using system tables. This approach works well for monitoring performance and creating custom dashboards for detailed system analysis.
+For plugin installation and configuration details, see the ClickHouse [data source plugin](/integrations/grafana). For a complete monitoring setup using the Prometheus-Grafana mix-in with pre-built dashboards and alerting rules, see [Monitor ClickHouse with the new Prometheus-Grafana mix-in](https://clickhouse.com/blog/monitor-with-new-prometheus-grafana-mix-in).
+
+### Direct Datadog Integration {#direct-datadog}
+
+Datadog offers a Clickhouse Monitoring plugin for its agent which queries system tables directly. This integration provides comprehensive database monitoring with cluster awareness through clusterAllReplicas functionality. 
+:::note
+This integration is not recommended for ClickHouse Cloud deployments due to incompatibility with cost-optimizing idle behavior and operational limitations of the cloud proxy layer.
+:::
+
+### Using system tables directly {#system-tables}
+
+Users can perform deep query performance analysis by connecting to ClickHouse system tables, particularly `system.query_log` and querying directly. Using either the SQL console or clickhouse client, teams can identify slow queries, analyze resource usage, and track usage patterns across the organization.
+
+**Query Performance Analysis**
+
+Users can use the system table query logs to perform Query Performance Analysis.
+
+**Example query**: Find the top 5 long-running queries across all cluster replicas:
+
+```sql
+SELECT
+    type,
+    event_time, 
+    query_duration_ms,
+    query,
+    read_rows,
+    tables
+FROM clusterAllReplicas(default, system.query_log)
+WHERE event_time >= (now() - toIntervalMinute(60)) AND type='QueryFinish'
+ORDER BY query_duration_ms DESC
+LIMIT 5
+FORMAT VERTICAL
+```

docs/_snippets/_observability_integration_options.md

@@ -0,0 +1,25 @@
+import Image from '@theme/IdealImage';
+import AdvancedDashboard from '@site/static/images/cloud/manage/monitoring/advanced_dashboard.png';
+import NativeAdvancedDashboard from '@site/static/images/cloud/manage/monitoring/native_advanced_dashboard.png';
+
+## Integration examples {#examples}
+
+External integration allows organizations to maintain established monitoring workflows, leverage existing team expertise with familiar tools, and integrate ClickHouse monitoring with broader infrastructure observability without disrupting current processes or requiring significant retraining investments.
+Teams can apply existing alerting rules and escalation procedures to ClickHouse metrics, while correlating database performance with application and infrastructure health within a unified observability platform. This approach maximizes ROI on current monitoring setups and enables faster troubleshooting through consolidated dashboards and familiar tooling interfaces.
+
+### Grafana Cloud monitoring {#grafana}
+
+Grafana provides ClickHouse monitoring through both direct plugin integration and Prometheus-based approaches. The Prometheus endpoint integration maintains operational separation between monitoring and production workloads while enabling visualization within existing Grafana Cloud infrastructure. See [Grafana's ClickHouse documentation](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/integrations/integration-reference/integration-clickhouse/) for configuration guidance.
+
+### Datadog monitoring {#datadog}
+Datadog is developing a dedicated API integration that will provide proper cloud service monitoring while respecting service idling behavior. In the interim, teams can use the OpenMetrics integration approach via ClickHouse Prometheus endpoints for operational separation and cost-efficient monitoring. For configuration guidance, see [Datadog's Prometheus and OpenMetrics integration documentation](https://docs.datadoghq.com/integrations/openmetrics/).
+
+### ClickStack {#clickstack}
+
+ClickStack is ClickHouse's recommended observability solution for deep system analysis and debugging, providing a unified platform for logs, metrics, and traces using ClickHouse as the storage engine. This approach relies on HyperDX, the ClickStack UI, connecting directly to the system tables inside your ClickHouse instance.
+HyperDX ships with a ClickHouse focused dashboard with tabs for Selects, Inserts, and Infrastructure. Teams can also use Lucene or SQL syntax to search system tables and logs, as well as create custom visualizations via Chart Explorer for detailed system analysis. 
+This approach is ideal for debugging complex issues, performance analysis, and deep system introspection rather than real-time production alerting.
+
+:::note
+Note that this approach will wake idle services as HyperDX queries the system tables directly.
+:::

docs/_snippets/_users-and-roles-common.md

@@ -120,49 +120,51 @@ With this set of examples:
 
 Roles are used to define groups of users for certain privileges instead of managing each user separately.
 
-1.  Create a role to restrict users of this role to only see `column1` in database `db1` and `table1`:
+<VerticalStepper headerLevel="h5">
+
+##### Create a role to restrict users of this role to only see `column1` in database `db1` and `table1`: {#create-column-role}
 
     ```sql
     CREATE ROLE column1_users;
     ```
 
-2.  Set privileges to allow view on `column1`
+##### Set privileges to allow view on `column1` {#set-column-privileges}
 
     ```sql
     GRANT SELECT(id, column1) ON db1.table1 TO column1_users;
     ```
 
-3.  Add the `column_user` user to the `column1_users` role
+##### Add the `column_user` user to the `column1_users` role {#add-column-user-to-role}
 
     ```sql
     GRANT column1_users TO column_user;
     ```
 
-4.  Create a role to restrict users of this role to only see selected rows, in this case, only rows containing `A` in `column1`
+##### Create a role to restrict users of this role to only see selected rows, in this case, only rows containing `A` in `column1` {#create-row-role}
 
     ```sql
     CREATE ROLE A_rows_users;
     ```
 
-5.  Add the `row_user` to the `A_rows_users` role
+##### Add the `row_user` to the `A_rows_users` role {#add-row-user-to-role}
 
     ```sql
     GRANT A_rows_users TO row_user;
     ```
 
-6.  Create a policy to allow view on only where `column1` has the values of `A`
+##### Create a policy to allow view on only where `column1` has the values of `A` {#create-row-policy}
 
     ```sql
     CREATE ROW POLICY A_row_filter ON db1.table1 FOR SELECT USING column1 = 'A' TO A_rows_users;
     ```
 
-7.  Set privileges to the database and table
+##### Set privileges to the database and table {#set-db-table-privileges}
 
     ```sql
     GRANT SELECT(id, column1, column2) ON db1.table1 TO A_rows_users;
     ```
 
-8.  grant explicit permissions for other roles to still have access to all rows
+##### Grant explicit permissions for other roles to still have access to all rows {#grant-other-roles-access}
 
     ```sql
     CREATE ROW POLICY allow_other_users_filter 
@@ -173,17 +175,21 @@ Roles are used to define groups of users for certain privileges instead of manag
     When attaching a policy to a table, the system will apply that policy, and only those users and roles defined will be able to do operations on the table, all others will be denied any operations. In order to not have the restrictive row policy applied to other users, another policy must be defined to allow other users and roles to have regular or other types of access.
     :::
 
+</VerticalStepper>
+
 ## Verification {#verification}
 
 ### Testing role privileges with column restricted user {#testing-role-privileges-with-column-restricted-user}
 
-1. Log into the clickhouse client using the `clickhouse_admin` user
+<VerticalStepper headerLevel="h5">
+
+##### Log into the clickhouse client using the `clickhouse_admin` user {#login-admin-user}
 
    ```bash
    clickhouse-client --user clickhouse_admin --password password
    ```
 
-2. Verify access to database, table and all rows with the admin user.
+##### Verify access to database, table and all rows with the admin user. {#verify-admin-access}
 
    ```sql
    SELECT *
@@ -201,13 +207,13 @@ Roles are used to define groups of users for certain privileges instead of manag
    └────┴─────────┴─────────┘
    ```
 
-3. Log into the ClickHouse client using the `column_user` user
+##### Log into the ClickHouse client using the `column_user` user {#login-column-user}
 
    ```bash
    clickhouse-client --user column_user --password password
    ```
 
-4. Test `SELECT` using all columns
+##### Test `SELECT` using all columns {#test-select-all-columns}
 
    ```sql
    SELECT *
@@ -230,7 +236,7 @@ Roles are used to define groups of users for certain privileges instead of manag
    Access is denied since all columns were specified and the user only has access to `id` and `column1`
    :::
 
-5. Verify `SELECT` query with only columns specified and allowed:
+##### Verify `SELECT` query with only columns specified and allowed: {#verify-allowed-columns}
 
    ```sql
    SELECT
@@ -250,15 +256,19 @@ Roles are used to define groups of users for certain privileges instead of manag
    └────┴─────────┘
    ```
 
+</VerticalStepper>
+
 ### Testing role privileges with row restricted user {#testing-role-privileges-with-row-restricted-user}
 
-1. Log into the ClickHouse client using `row_user`
+<VerticalStepper headerLevel="h5">
+
+##### Log into the ClickHouse client using `row_user` {#login-row-user}
 
    ```bash
    clickhouse-client --user row_user --password password
    ```
 
-2. View rows available
+##### View rows available {#view-available-rows}
 
    ```sql
    SELECT *
@@ -278,37 +288,41 @@ Roles are used to define groups of users for certain privileges instead of manag
    Verify that only the above two rows are returned, rows with the value `B` in `column1` should be excluded.
    :::
 
+</VerticalStepper>
+
 ## Modifying users and roles {#modifying-users-and-roles}
 
 Users can be assigned multiple roles for a combination of privileges needed. When using multiple roles, the system will combine the roles to determine privileges, the net effect will be that the role permissions will be cumulative.
 
 For example, if one `role1` allows for only select on `column1` and `role2` allows for select on `column1` and `column2` then the user will have access to both columns.
 
-1. Using the admin account, create new user to restrict by both row and column with default roles
+<VerticalStepper headerLevel="h5">
+
+##### Using the admin account, create new user to restrict by both row and column with default roles {#create-restricted-user}
 
    ```sql
    CREATE USER row_and_column_user IDENTIFIED BY 'password' DEFAULT ROLE A_rows_users;
    ```
 
-2. Remove prior privileges for `A_rows_users` role
+##### Remove prior privileges for `A_rows_users` role {#remove-prior-privileges}
 
    ```sql
    REVOKE SELECT(id, column1, column2) ON db1.table1 FROM A_rows_users;
    ```
 
-3. Allow `A_row_users` role to only select from `column1`
+##### Allow `A_row_users` role to only select from `column1` {#allow-column1-select}
 
    ```sql
    GRANT SELECT(id, column1) ON db1.table1 TO A_rows_users;
    ```
 
-4. Log into the ClickHouse client using `row_and_column_user`
+##### Log into the ClickHouse client using `row_and_column_user` {#login-restricted-user}
 
    ```bash
    clickhouse-client --user row_and_column_user --password password;
    ```
 
-5. Test with all columns:
+##### Test with all columns: {#test-all-columns-restricted}
 
    ```sql
    SELECT *
@@ -327,7 +341,7 @@ For example, if one `role1` allows for only select on `column1` and `role2` allo
    SELECT(id, column1, column2) ON db1.table1. (ACCESS_DENIED)
    ```
 
-6. Test with limited allowed columns:
+##### Test with limited allowed columns: {#test-limited-columns}
 
    ```sql
    SELECT
@@ -344,6 +358,7 @@ For example, if one `role1` allows for only select on `column1` and `role2` allo
    │  2 │ A       │
    └────┴─────────┘
    ```
+</VerticalStepper>
 
 ## Troubleshooting {#troubleshooting}