diff --git a/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit-playground.png b/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit-playground.png new file mode 100644 index 0000000000..cce8aa868b Binary files /dev/null and b/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit-playground.png differ diff --git a/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit-test.png b/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit-test.png new file mode 100644 index 0000000000..db5519e8fd Binary files /dev/null and b/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit-test.png differ diff --git a/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_create-index.png b/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_create-index.png new file mode 100644 index 0000000000..3b75e9ace8 Binary files /dev/null and b/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_create-index.png differ diff --git a/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_create-schema.png b/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_create-schema.png new file mode 100644 index 0000000000..f82b75d1e5 Binary files /dev/null and b/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_create-schema.png differ diff --git a/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_schema-definition.png b/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_schema-definition.png new file mode 100644 index 0000000000..85af5c9778 Binary files /dev/null and b/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_schema-definition.png differ diff --git a/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_schema-tab.png b/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_schema-tab.png new file mode 100644 index 0000000000..c71d436fc7 Binary files /dev/null and b/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_schema-tab.png differ diff --git a/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_test-index.png b/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_test-index.png new file mode 100644 index 0000000000..2ace8bc419 Binary files /dev/null and b/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_test-index.png differ diff --git a/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_test-index_results.png b/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_test-index_results.png new file mode 100644 index 0000000000..2be59d2f70 Binary files /dev/null and b/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_test-index_results.png differ diff --git a/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_test-index_results_preview.png b/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_test-index_results_preview.png new file mode 100644 index 0000000000..e29f15a643 Binary files /dev/null and b/docs/documents/schema-validation/auditing-document-compliance/assets/schema-validation_studio_audit_index_test-index_results_preview.png differ diff --git a/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit-playground.snagx b/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit-playground.snagx new file mode 100644 index 0000000000..8f43bfb16c Binary files /dev/null and b/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit-playground.snagx differ diff --git a/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit-test.snagx b/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit-test.snagx new file mode 100644 index 0000000000..02a0628b1e Binary files /dev/null and b/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit-test.snagx differ diff --git a/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_create-index.snagx b/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_create-index.snagx new file mode 100644 index 0000000000..79617094d1 Binary files /dev/null and b/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_create-index.snagx differ diff --git a/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_create-schema.snagx b/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_create-schema.snagx new file mode 100644 index 0000000000..3883e7597b Binary files /dev/null and b/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_create-schema.snagx differ diff --git a/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_schema-definition.snagx b/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_schema-definition.snagx new file mode 100644 index 0000000000..657ef0d83f Binary files /dev/null and b/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_schema-definition.snagx differ diff --git a/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_schema-tab.snagx b/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_schema-tab.snagx new file mode 100644 index 0000000000..7545329be7 Binary files /dev/null and b/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_schema-tab.snagx differ diff --git a/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_test-index.snagx b/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_test-index.snagx new file mode 100644 index 0000000000..ec0d3568f8 Binary files /dev/null and b/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_test-index.snagx differ diff --git a/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_test-index_results.snagx b/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_test-index_results.snagx new file mode 100644 index 0000000000..0da2f4e35c Binary files /dev/null and b/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_test-index_results.snagx differ diff --git a/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_test-index_results_preview.snagx b/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_test-index_results_preview.snagx new file mode 100644 index 0000000000..383742f910 Binary files /dev/null and b/docs/documents/schema-validation/auditing-document-compliance/assets/snagit/schema-validation_studio_audit_index_test-index_results_preview.snagx differ diff --git a/docs/documents/schema-validation/auditing-document-compliance/auditing-document-compliance_api.mdx b/docs/documents/schema-validation/auditing-document-compliance/auditing-document-compliance_api.mdx index 456295bc38..90f6fc5181 100644 --- a/docs/documents/schema-validation/auditing-document-compliance/auditing-document-compliance_api.mdx +++ b/docs/documents/schema-validation/auditing-document-compliance/auditing-document-compliance_api.mdx @@ -18,8 +18,8 @@ import Panel from "@site/src/components/Panel"; * Audit stored documents' compliance with a schema to identify and address any validation violations. - * You can run an [audit operation](../../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_api#running-an-audit-operation) that scans a document collection and generates validation violation reports. - * You can also audit document compliance [by index](../../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_api#audit-document-compliance-by-index), validating documents during indexing and embedding any validation error messages into the indexes. The documents can then be queried and managed based on their schema compliance. + * You can run an [audit operation](../../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_api#running-an-audit-operation) that scans a document collection or a part of it and generates a validation violations report. + * You can also audit document compliance [by index](../../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_api#audit-document-compliance-by-index), validating documents during indexing and embedding any validation error messages into the indexes. The indexed documents can then be queried and managed based on their schema compliance. * In this article: * [Running an audit operation](../../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_api#running-an-audit-operation) @@ -33,17 +33,18 @@ import Panel from "@site/src/components/Panel"; -## Running an audit operation + + Use this audit approach to examine the [compliance of documents in a collection](../../../documents/schema-validation/schema-validation_overview#running-an-audit-operation) with a validation schema. You can limit the number of documents to be checked and the number of error messages returned. This method is useful for one-time audits or periodic checks of data integrity. To execute this audit, run `StartSchemaValidationOperation`. The operation will scan the documents in the specified collection, validate each document against the provided JSON schema, and produce a report of any validation errors. - +### `StartSchemaValidationOperation` -#### `StartSchemaValidationOperation` constructor +#### `StartSchemaValidationOperation` constructor: ```csharp public StartSchemaValidationOperation(Parameters parameters) ``` @@ -51,7 +52,7 @@ public StartSchemaValidationOperation(Parameters parameters) -#### `StartSchemaValidationOperation` arguments +#### `StartSchemaValidationOperation` arguments: Pass the operation a `Parameters` object that specifies - ```csharp public class Parameters @@ -73,7 +74,7 @@ public class Parameters -#### `StartSchemaValidationOperation` results +#### `StartSchemaValidationOperation` results: The operation will run in the background and return a result object containing - - `ValidatedCount` The number of documents checked @@ -87,13 +88,13 @@ The operation will run in the background and return a result object containing - -### Example: -This example audits the `Orders` collection, ensuring each order has a `Customer` string and a non-negative `Total` property. The validation results are then retrieved and summarized. +### Example +In this example we audit the `Orders` collection, ensuring each order has a `Customer` string and a non-negative `Total` property. The validation results are then retrieved and summarized. ```csharp // Define a validation Schema as a string -// Customer must be a string -// Total must be a number >= 0 -// These fields are required +// `Customer` must be a string +// `Total` must be a number >= 0 +// Both fields must exist string schemaDefinition = @"{ ""properties"": { ""Customer"": { ""type"": ""string"" }, @@ -132,10 +133,10 @@ var result = await operation.WaitForCompletionAsync(TimeSp // Handle the results, e.g., print a validation summary // The number of inspected documents, and the number of documents with errors Console.WriteLine($"Validated: {result.ValidatedCount}, Errors: {result.ErrorCount}"); -// For each document with errors, print its ID and the associated error messages +// For each document with errors, print its Id and the associated error messages foreach (var error in result.Errors) { - Console.WriteLine($"Document ID: {error.Key}"); + Console.WriteLine($"Document Id: {error.Key}"); foreach (var line in error.Value.Split(new[] { '\n' }, StringSplitOptions.RemoveEmptyEntries)) { Console.WriteLine($"Error: {line}"); @@ -151,24 +152,35 @@ public class Order } ``` + -## Audit document compliance by index + -Use this audit approach to continuously validate document compliance during indexing and embed validation errors in the index. The index can then be queried to filter documents by their compliance status. +Use this audit approach to continuously validate document compliance during indexing and store validation error messages in the index. The index can then be queried to filter documents by their compliance status. This method is useful for checking data integrity in large or frequently changing datasets, where [a single audit operation](../../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_api#running-an-audit-operation) may be less efficient. -To validate documents by index: - +To validate documents by index: +#### Define a validation schema + +Define a validation schema that the index will use to validate documents during indexing. +The index will use the schema defined in its index configuration (see [below](../../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_api#defining-an-index-level-validation-schema) how to define it). +* If no schema is defined in the index configuration, the index will use the [database-level schema](../../../documents/schema-validation/write-validation/write-validation_api#creating-a-validation-schema). +* If no schema is defined either in the index configuration or at the database level, the index will not perform any validation. + + +**Note**: When the index uses a database-level schema, [disabling validation at the database level](../../../documents/schema-validation/write-validation/write-validation_api#2-create-the-schema-configuration) will also disable auditing by the index. +However, when the index uses an index-level schema, disabling database-level validation will **not** disable auditing by the index. + -Define a [validation schema](../../../documents/schema-validation/write-validation/write-validation_api#creating-a-validation-schema) and associate it with the collection whose documents you want to audit. +#### Defining an index-level validation schema: ```csharp // Define a validation Schema as a string -// Customer must be a string -// Total must be a number >= 0 -// These fields are required +// `Customer` must be a string +// `Total` must be a number >= 0 +// Both fields are required string schemaDefinition = @"{ ""properties"": { ""Customer"": { ""type"": ""string"" }, @@ -186,9 +198,9 @@ var schemaDefinitions = new IndexSchemaDefinitions -### Define and execute an index +#### Define and execute an index -* Add the index an **Errors** field to hold validation error messages. +* Add to the index an **Errors** field to hold validation error messages. ```csharp // Create the index definition var indexDefinition = new IndexDefinition @@ -208,10 +220,13 @@ var schemaDefinitions = new IndexSchemaDefinitions ```csharp // Create a map for the index, including a call to Schema.GetErrorsFor() const string OrdersValidationMap = @" - from doc in docs.Orders - select new + from doc in docs + let errors = Schema.GetErrorsFor(doc) + where errors != null && errors.Length > 0 + select new { - Errors = Schema.GetErrorsFor(doc) + Id = doc.Id, + Errors = errors } "; ``` @@ -226,13 +241,13 @@ var schemaDefinitions = new IndexSchemaDefinitions -### Query the index: +#### Query the index You can now query the index to find documents by their validation status and specific errors. ```csharp // Query the index and print validation errors using (var session = store.OpenAsyncSession()) { - // Retrieve results, containing validation errors, from the index + // Retrieve results that contain validation errors from the index var results = await session.Query("Orders_WithValidation") .Select(x => new { @@ -259,7 +274,8 @@ using (var session = store.OpenAsyncSession()) -### Full example: +### Full example + @@ -285,10 +301,13 @@ using (var session = store.OpenAsyncSession()) // Create a map for the index, including a call to Schema.GetErrorsFor() const string OrdersValidationMap = @" - from doc in docs.Orders - select new + from doc in docs + let errors = Schema.GetErrorsFor(doc) + where errors != null && errors.Length > 0 + select new { - Errors = Schema.GetErrorsFor(doc) + Id = doc.Id, + Errors = errors } "; @@ -328,8 +347,8 @@ using (var session = store.OpenAsyncSession()) // Query the index and print validation errors using (var session = store.OpenAsyncSession()) { - // Retrieve results, containing validation errors, from the index - var results = await session.Query("Orders_WithValidation") + // Retrieve results that contain validation errors from the index + var results = await session.Query("Orders_WithValidation") .Select(x => new { Id = RavenQuery.Metadata(x)["Id"] as string, // Also project the document Id @@ -387,7 +406,6 @@ public class IndexResult ""required"": [""Customer"", ""Total""] }"; - // Associate the schema with the Orders collection var schemaDefinitions = new IndexSchemaDefinitions { @@ -396,9 +414,12 @@ public class IndexResult // Create a map for the index, including a call to schema.getErrorsFor() const string OrdersValidationMap = @" - map('Orders', function (doc) { + map("@all_docs", (doc) => { + let errors = schema.getErrorsFor(doc); + if( errors != null && errors.length > 0) return { - Errors: schema.getErrorsFor(doc) + Id: id(doc), + Errors: errors }; })"; @@ -438,7 +459,7 @@ public class IndexResult // Query the index and print validation errors using (var session = store.OpenAsyncSession()) { - // Retrieve results, containing validation errors, from the index + // Retrieve results that contain validation errors from the index var results = await session.Query("Orders_WithValidation_JS") .Select(x => new { diff --git a/docs/documents/schema-validation/auditing-document-compliance/auditing-document-compliance_studio.mdx b/docs/documents/schema-validation/auditing-document-compliance/auditing-document-compliance_studio.mdx new file mode 100644 index 0000000000..5ba88d3d5c --- /dev/null +++ b/docs/documents/schema-validation/auditing-document-compliance/auditing-document-compliance_studio.mdx @@ -0,0 +1,176 @@ +--- +title: "Auditing document compliance: Studio" +hide_table_of_contents: true +sidebar_label: "Studio" +sidebar_position: 2 +--- + +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import CodeBlock from '@theme/CodeBlock'; +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; +import ContentFrame from "@site/src/components/ContentFrame"; +import Panel from "@site/src/components/Panel"; + +# Auditing document compliance: Studio + + +To check document compliance with a validation schema via Studio, you can: + - Run a manual [validation test operation](../../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_studio#running-a-validation-test), that goes through a collection or a part of it and reports invalid documents and their validation errors. + - Create an [audit index](../../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_studio#audit-documents-compliance-by-index), that validates documents during indexing and stores any validation error messages in the index, so the documents can be queried by compliance status and specific validation error messages. + +* In this article: + * [Running a validation test](../../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_studio#running-a-validation-test) + * [Test a schema using the Document Schema view](../../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_studio#test-a-schema-using-the-document-schema-view) + * [Test a schema using the Schema Playground](../../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_studio#test-a-schema-using-the-schema-playground) + * [Audit documents compliance by index](../../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_studio#audit-documents-compliance-by-index) + * [Define a validation schema](../../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_studio#define-a-validation-schema) + * [Create the audit index](../../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_studio#create-the-audit-index) + * [Query the index](../../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_studio#query-the-index) + + + + + +You can initiate a compliance test operation that validates documents against a schema, using either a defined schema listed in the [Document Schema view](../../../documents/schema-validation/write-validation/write-validation_studio#managing-and-testing-existing-schemas) or the [Schema playground](../../../documents/schema-validation/write-validation/write-validation_studio#the-schema-playground) views. + + + +### Test a schema using the Document Schema view + +The validation schemas that you already defined are displayed in the [Document Schema view](../../../documents/schema-validation/write-validation/write-validation_studio#creating-a-collection-schema), where you can test each schema against documents in the associated collection. + +![Test collection schema](./assets/schema-validation_studio_audit-test.png) + +* Use this method to verify schema effectiveness and documents compliance with schema constraints. +* Test results include the number and names of invalid documents, and the validation error messages generated for each document. +* You can validate an entire collection or a specified number of documents. This is helpful when working with large collections. +* Find [here](../../../documents/schema-validation/write-validation/write-validation_studio#managing-and-testing-existing-schemas) a detailed explanation of this view. + + + + + +### Test a schema using the Schema Playground + +The [Document Schema](../../../documents/schema-validation/write-validation/write-validation_studio#creating-a-collection-schema) view also provides an isolated playground environment for experimentation, allowing you to define schemas and try them out on your data without affecting your existing collections or schemas. + +![Document Schema view](./assets/ schema-validation_studio_audit-playground.png) + +* Use the playground to experiment with schema definitions and test them against documents in your collections. +* You can validate an entire collection or a specified number of documents. +* You can test multiple schemas in a single run, including multiple schemas for the same collection. This is helpful when you want to compare the effectiveness of different schema definitions, validate a specific field across collections, or test a large dataset. +* Test results are presented per schema, detailing the number and names of invalid documents and the validation error messages generated for each document. +* Find [here](../../../documents/schema-validation/write-validation/write-validation_studio#the-schema-playground) a detailed explanation of the playground. + + + + + + + +You can define an **audit index** that validates documents against a schema during indexing and stores any validation error messages in the index. +* The audit index can then be queried to find documents by compliance status and specific validation error messages. +* Indexing is performed continuously in the background, preparing for validation queries that remain quick and efficient even with large datasets. + + +**Note** that disabling validation for the database or for the collection does **not** disable auditing by an index. + + + + +#### Define a validation schema + +The index will use the schema defined in its index configuration. +See how to add a schema to the index configuration [below](../../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_studio#create-the-audit-index). +* If no schema is defined in the index configuration, the index will use the [database-level schema](../../../documents/schema-validation/write-validation/write-validation_studio#creating-a-collection-schema). +* If no schema is defined either in the index configuration or at the database level, the index will not perform any validation. + + +**Note**: When the index uses a database-level schema, [disabling validation at the database level](../../../documents/schema-validation/write-validation/write-validation_studio#managing-and-testing-existing-schemas) will also disable auditing by the index. +However, when the index uses an index-level schema, disabling database-level validation will **not** disable auditing by the index. + + + + + + +#### Create the audit index + +To [create the index](../../../studio/database/indexes/create-map-index), open **Indexes** > **List of indexes** and click the **New index** button. + +* Name the index, and add to its map a call to the validation method: `Schema.GetErrorsFor()` for a static index, or `schema.getErrorsFor()` for a JavaScript map. + Assign the value returned by the validation method to a dedicated index field, e.g., `Errors`. + + + + + ```csharp + from doc in docs + let errors = Schema.GetErrorsFor(doc) + where errors != null && errors.Length > 0 + select new + { + Id = doc.Id, + Errors = errors + } + ``` + + + + ```javascript + map("@all_docs", (doc) => { + let errors = schema.getErrorsFor(doc); + if( errors != null && errors.length > 0) + return { + Id: id(doc), + Errors: errors + }; + }) + ``` + + + + +* Add the validation error messages field (defined in the map) to the index. + Set the [Store](../../../indexes/storing-data-in-index/) option for this field to **Yes**, to store actual error messages in the index rather than just their tokens. + + ![Create audit index](./assets/schema-validation_studio_audit_index_create-index.png) + +* If you prefer that the index use an index-level schema, add it in the **Schema definition** tab. + + Open the **Schema definition** tab and add a new schema definition. + ![Open index schema tab](./assets/schema-validation_studio_audit_index_schema-tab.png) + + + Select a collection to associate the schema with, and define the schema using JSON syntax. + ![Define index schema](./assets/schema-validation_studio_audit_index_schema-definition.png) + +* When you're done, remember to save the index. + + + + + +#### Query the index + +Query the index to find documents by compliance status and specific error messages. + +* While building or editing the index, you can use the **Test Index** bar to query the index and verify its functionality. + ![Run index test](./assets/schema-validation_studio_audit_index_test-index.png) + +* Run queries that filter documents based on the `Errors` field. + ![Query audit index](./assets/schema-validation_studio_audit_index_test-index_results.png) + + Click an item's preview (eye) icon to view the document details, including the validation error messages stored in the index. + ![See index entry](./assets/schema-validation_studio_audit_index_test-index_results_preview.png) + + + + + + + + diff --git a/docs/documents/schema-validation/schema-validation_configuration.mdx b/docs/documents/schema-validation/schema-validation_configuration.mdx new file mode 100644 index 0000000000..1fab10eda1 --- /dev/null +++ b/docs/documents/schema-validation/schema-validation_configuration.mdx @@ -0,0 +1,70 @@ +--- +title: "Schema validation configuration" +hide_table_of_contents: true +sidebar_label: "Configuration" +sidebar_position: 3 +--- + +import Admonition from '@theme/Admonition'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import CodeBlock from '@theme/CodeBlock'; +import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; +import LanguageContent from "@site/src/components/LanguageContent"; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from "@site/src/components/Panel"; + +# Schema validation configuration + + +This page aims to clarify the scope of settings and configurations related to schema validation. + +* In this article: + * [Enabling and disabling validation for the database](../../documents/schema-validation/schema-validation_configuration#enabling-and-disabling-validation-for-the-database) + * [Per-collection configuration](../../documents/schema-validation/schema-validation_configuration#per-collection-configuration) + * [Index configuration](../../documents/schema-validation/schema-validation_configuration#index-configuration) + + + + + + + +### Enabling and disabling validation for the database + +Writing validation can be enabled or disabled [for the database and per collection](../../documents/schema-validation/write-validation/write-validation_studio#managing-and-testing-existing-schemas ). + +When validation is **enabled for the database**, documents are validated when they are written if a schema is defined and enabled for their collection. +When validation is **disabled for the database**, documents are written without validation regardless of individual collection settings. + +* [Learn here](../../documents/schema-validation/write-validation/write-validation_studio#managing-and-testing-existing-schemas) how to enable and disable validation for the database using Studio. +* [Learn here](../../documents/schema-validation/write-validation/write-validation_api#2-create-the-schema-configuration) how to enable and disable validation for the database via the client API. + + + + +### Per-collection configuration + +Schemas can be defined **for collections** to ensure that documents are written only if they comply with the defined constraints. + +Define and enable a schema for a collection to validate the collection documents as they are written. Validation will take place only if it is also enabled for the database. + +* [Learn here](../../documents/schema-validation/write-validation/write-validation_studio#managing-and-testing-existing-schemas) how to enable and disable validation for the database using Studio. +* [Learn here](../../documents/schema-validation/write-validation/write-validation_api#2-create-the-schema-configuration) how to enable and disable validation for the database via the client API. + + + + + +### Index configuration + +Schemas can be defined for [audit indexes](../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_studio#audit-documents-compliance-by-index) to validate documents during indexing, store validation error messages in an index field, and can be queried to find documents by their compliance status. + +If a schema is **not** explicitly defined for an audit index, the index will use the schema defined **for the collection**. + +* [Learn here](../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_studio#create-the-audit-index) how to define a schema for an audit index using Studio. +* [Learn here](../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_api#define-a-validation-schema-optional) how to define a schema for an audit index via the client API. + + + + diff --git a/docs/documents/schema-validation/schema-validation_overview.mdx b/docs/documents/schema-validation/schema-validation_overview.mdx index 2df9780e08..acc9014d56 100644 --- a/docs/documents/schema-validation/schema-validation_overview.mdx +++ b/docs/documents/schema-validation/schema-validation_overview.mdx @@ -11,6 +11,8 @@ import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; import LanguageContent from "@site/src/components/LanguageContent"; +import ContentFrame from '@site/src/components/ContentFrame'; +import Panel from "@site/src/components/Panel"; # Schema validation overview diff --git a/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_document-schema-view.png b/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_document-schema-view.png index 69bbcaf51f..dc134ee666 100644 Binary files a/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_document-schema-view.png and b/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_document-schema-view.png differ diff --git a/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_manage-existing-schemas.png b/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_manage-existing-schemas.png index 0f7eafb7c2..80ff6ba281 100644 Binary files a/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_manage-existing-schemas.png and b/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_manage-existing-schemas.png differ diff --git a/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_playground.png b/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_playground.png new file mode 100644 index 0000000000..223402c3af Binary files /dev/null and b/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_playground.png differ diff --git a/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_playground_results.png b/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_playground_results.png new file mode 100644 index 0000000000..efc5b35c45 Binary files /dev/null and b/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_playground_results.png differ diff --git a/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_schema.png b/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_schema.png index 861eb41c74..f2cdfebbcf 100644 Binary files a/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_schema.png and b/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_schema.png differ diff --git a/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_test-button.png b/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_test-button.png new file mode 100644 index 0000000000..00b9c441ec Binary files /dev/null and b/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_test-button.png differ diff --git a/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_test-button_results.png b/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_test-button_results.png new file mode 100644 index 0000000000..7319a703a2 Binary files /dev/null and b/docs/documents/schema-validation/write-validation/assets/schema-validation_studio_test-button_results.png differ diff --git a/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_document-schema-view.snagx b/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_document-schema-view.snagx index 5652b7208a..422d7935d5 100644 Binary files a/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_document-schema-view.snagx and b/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_document-schema-view.snagx differ diff --git a/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_manage-existing-schemas.snagx b/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_manage-existing-schemas.snagx index 8a3756e83d..33aa98e7bb 100644 Binary files a/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_manage-existing-schemas.snagx and b/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_manage-existing-schemas.snagx differ diff --git a/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_playground.snagx b/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_playground.snagx new file mode 100644 index 0000000000..61b7be4fe0 Binary files /dev/null and b/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_playground.snagx differ diff --git a/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_playground_results.snagx b/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_playground_results.snagx new file mode 100644 index 0000000000..e77edeaf99 Binary files /dev/null and b/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_playground_results.snagx differ diff --git a/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_schema.snagx b/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_schema.snagx index a33ae1057a..122fff689e 100644 Binary files a/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_schema.snagx and b/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_schema.snagx differ diff --git a/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_test-button.snagx b/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_test-button.snagx new file mode 100644 index 0000000000..1423ad6d81 Binary files /dev/null and b/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_test-button.snagx differ diff --git a/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_test-button_results.snagx b/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_test-button_results.snagx new file mode 100644 index 0000000000..e795dfa2b0 Binary files /dev/null and b/docs/documents/schema-validation/write-validation/assets/snagit/schema-validation_studio_test-button_results.snagx differ diff --git a/docs/documents/schema-validation/write-validation/write-validation_api.mdx b/docs/documents/schema-validation/write-validation/write-validation_api.mdx index 469d29f092..36a6f3976e 100644 --- a/docs/documents/schema-validation/write-validation/write-validation_api.mdx +++ b/docs/documents/schema-validation/write-validation/write-validation_api.mdx @@ -25,86 +25,111 @@ import Panel from "@site/src/components/Panel"; * In this article: * [Creating a validation schema](../../../documents/schema-validation/write-validation/write-validation_api#creating-a-validation-schema) + * [Define your schema as a string](../../../documents/schema-validation/write-validation/write-validation_api#1-define-your-schema-as-a-string) + * [Create the schema configuration](../../../documents/schema-validation/write-validation/write-validation_api#2-create-the-schema-configuration) + * [Register the schema with the server](../../../documents/schema-validation/write-validation/write-validation_api#3-register-the-schema-with-the-server) + * [Validate documents](../../../documents/schema-validation/write-validation/write-validation_api#4-validate-documents) * [Example](../../../documents/schema-validation/write-validation/write-validation_api#example) * [Available constraints](../../../documents/schema-validation/write-validation/write-validation_api#available-constraints) * [Syntax](../../../documents/schema-validation/write-validation/write-validation_api#syntax) - + -To validate documents with a JSON schema: +To validate documents against a JSON schema: -1. **Define your [schema](https://json-schema.org/) as a string**: - Ensure that the schema sets all the constraints needed to validate your documents. + + +### 1. **Define your [schema](https://json-schema.org/) as a string**. +Ensure that the schema sets all the constraints needed to validate your documents. - **Example**: - ```csharp - // Ensure that a `Name` property exists. - string schemaJson = "{\"required\":[\"Name\"]}"; - ``` - - **Example**: - ```csharp - // Ensure that a `Name` property has a maximum length of 7 characters. - string schemaJson = "{\"properties\":{\"Name\":{\"maxLength\":7}}}"; - ``` - -2. **Create the schema configuration**: - Create a [SchemaValidationConfiguration](../../../documents/schema-validation/write-validation/write-validation_api#schemavalidationconfiguration-class) object and use it to enable validation and associate your schema with a document collection. - - E.g., to associate your schema with the `Users` collection: - ```csharp - // Create the configuration - var configuration = new SchemaValidationConfiguration - { - Disabled = false, // Enable validation - ValidatorsPerCollection = new Dictionary - { - { "Users", new SchemaDefinition { Schema = schemaJson } } - } - }; - ``` -3. **Register the schema with the server**: - Use [ConfigureSchemaValidationOperation](../../../documents/schema-validation/write-validation/write-validation_api#configureschemavalidationoperation-definition) to register the schema configuration with the server. - - E.g., - ```csharp - // Register the schema configuration with the server - await store.Maintenance.SendAsync(new ConfigureSchemaValidationOperation(configuration)); - ``` - - - Once a schema is enabled for a collection, collection documents are validated when they are saved directly, as well as when they are added or modified by operations such as patching or ETL tasks. (See a list of [operations that trigger validation](../../../documents/schema-validation/schema-validation_overview#a-list-of-operations-that-trigger-validation).) - - -4. **Validate documents**: - After registering the schema, any document saved to the associated collection will be validated against the schema constraints. +**Example**: +```csharp +// Ensure that a `Name` property exists. +string schemaJson = "{\"required\":[\"Name\"]}"; +``` + +**Example**: +```csharp +// Ensure that a `Name` property has a maximum length of 7 characters. +string schemaJson = "{\"properties\":{\"Name\":{\"maxLength\":7}}}"; +``` + + + + +### 2. **Create the schema configuration**. +Create a [SchemaValidationConfiguration](../../../documents/schema-validation/write-validation/write-validation_api#schemavalidationconfiguration-class) object and use it to enable validation and associate your schema with a document collection. - E.g., attempting to save a document that violates the `maxLength` constraint will result in a [SchemaValidationException](../../../documents/schema-validation/write-validation/write-validation_api#schemavalidationexception-message) such as: - ```plain - Raven.Client.Exceptions.SchemaValidation.SchemaValidationException: The length of the value 'TooLongValue' at 'Name' should not exceed 7, but its actual length is 12. - ``` - - To handle such exceptions, you can use a try-catch block when saving documents: - ```csharp - using (var session = store.OpenSession()) +E.g., to associate your schema with the `Users` collection: +```csharp +// Create the configuration +var configuration = new SchemaValidationConfiguration +{ + Disabled = false, // Enable validation for all collections + ValidatorsPerCollection = new Dictionary { - var user = new User { Name = "TooLongValue" }; // Violates maxLength constraint - session.Store(user); - try - { - session.SaveChanges(); // This will throw SchemaValidationException - } - catch (SchemaValidationException ex) - { - // Catch and handle the exception if the validation fails. + { "Users", new SchemaDefinition + { + Disabled = false, // Enable validation for the Users collection + Schema = schemaJson + } } } - ``` +}; +``` + + + + +### 3. **Register the schema with the server**. + +Use [ConfigureSchemaValidationOperation](../../../documents/schema-validation/write-validation/write-validation_api#configureschemavalidationoperation-definition) to register the schema configuration with the server. + +E.g., +```csharp +// Register the schema configuration with the server +await store.Maintenance.SendAsync(new ConfigureSchemaValidationOperation(configuration)); +``` + + +Once a schema is enabled for a collection, collection documents are validated when they are saved directly, as well as when they are added or modified by operations such as patching or ETL tasks. (See a list of [operations that trigger validation](../../../documents/schema-validation/schema-validation_overview#a-list-of-operations-that-trigger-validation).) + + + + + + +### 4. **Validate documents**. +After registering the schema, any document saved to the associated collection will be validated against the schema constraints. + +E.g., attempting to save a document that violates the `maxLength` constraint will result in a [SchemaValidationException](../../../documents/schema-validation/write-validation/write-validation_api#schemavalidationexception-message) such as: +```plain +Raven.Client.Exceptions.SchemaValidation.SchemaValidationException: The length of the value 'TooLongValue' at 'Name' should not exceed 7, but its actual length is 12. +``` + +To handle such exceptions, you can use a try-catch block when saving documents: +```csharp +using (var session = store.OpenSession()) +{ + var user = new User { Name = "TooLongValue" }; // Violates maxLength constraint + session.Store(user); + try + { + session.SaveChanges(); // This will throw SchemaValidationException + } + catch (SchemaValidationException ex) + { + // Catch and handle the exception if the validation fails. + } +} +``` + + - + In this example we validate documents saved in the `Users` collection by constraints set by a JSON schema. @@ -128,7 +153,12 @@ var configuration = new SchemaValidationConfiguration Disabled = false, ValidatorsPerCollection = new Dictionary { - { "Users", new SchemaDefinition { Schema = schemaJson } } + { "Users", new SchemaDefinition + { + Disabled = false, + Schema = schemaJson + } + } } }; @@ -162,9 +192,10 @@ await store.Maintenance.SendAsync(new ConfigureSchemaValidationOperation(configu ``` + - + Your validation schema can set the following constraints: @@ -172,11 +203,10 @@ Your validation schema can set the following constraints: ### `type` Enforces the data type (e.g., "string", "number", "object", "array", "boolean", "integer"). -**Example**: +**Example**: `Name` must be a string ```json { "properties": { - // 'Name' must be a string "Name": { "type": "string" } } } @@ -187,14 +217,13 @@ Enforces the data type (e.g., "string", "number", "object", "array", "boolean", ### `required` Enforces properties that must be present in the object. -**Example**: +**Example**: `name` must be a string, `age` must be an integer, and both are required. ```json { "properties": { - "name": { "type": "string" }, // Name must be a string - "age": { "type": "integer" } // Age must be an integer + "name": { "type": "string" }, + "age": { "type": "integer" } }, - // Both 'name' and 'age' are required "required": ["name", "age"] } ``` @@ -204,11 +233,10 @@ Enforces properties that must be present in the object. ### `const` Requires a property to have a specific value. -**Example**: +**Example**: `role` must always be `admin`. ```json { "properties": { - // 'role' must always be 'admin' "role": { "const": "admin" } } } @@ -219,11 +247,10 @@ Requires a property to have a specific value. ### `enum` Restricts a property to a set of allowed values. -**Example**: +**Example**: `status` must be `active`, `inactive`, or `pending`. ```json { "properties": { - // 'status' must be one of the specified values "status": { "enum": ["active", "inactive", "pending"] } } } @@ -234,11 +261,10 @@ Restricts a property to a set of allowed values. ### `minLength` Minimum number of characters for a string. -**Example**: +**Example**: `username` must be at least 3 characters long. ```json { "properties": { - // 'username' must be at least 3 characters long "username": { "type": "string", "minLength": 3 } } } @@ -249,11 +275,10 @@ Minimum number of characters for a string. ### `maxLength` Maximum number of characters for a string. -**Example**: +**Example**: `code` must be no more than 7 characters long. ```json { "properties": { - // 'code' must be no more than 7 characters long "code": { "type": "string", "maxLength": 7 } } } @@ -264,11 +289,10 @@ Maximum number of characters for a string. ### `pattern` String must match a regular expression. -**Example**: +**Example**: `email` must match a basic email pattern. ```json { "properties": { - // 'email' must match the given regular expression "email": { "type": "string", "pattern": "^[\\w.-]+@[\\w.-]+\\.\\w+$" } } } @@ -279,11 +303,10 @@ String must match a regular expression. ### `minimum` Minimum value for a number. -**Example**: +**Example**: `age` must be at least 18. ```json { "properties": { - // 'age' must be at least 18 "age": { "type": "integer", "minimum": 18 } } } @@ -294,11 +317,10 @@ Minimum value for a number. ### `maximum` Maximum value for a number. -**Example**: +**Example**: `score` must be no more than `100`. ```json { "properties": { - // 'score' must be at most 100 "score": { "type": "number", "maximum": 100 } } } @@ -309,11 +331,10 @@ Maximum value for a number. ### `minItems` Minimum number of items in an array. -**Example**: +**Example**: The `tags` array must have at least 2 items. ```json { "properties": { - // 'tags' array must have at least 2 items "tags": { "type": "array", "minItems": 2 } } } @@ -324,11 +345,10 @@ Minimum number of items in an array. ### `maxItems` Maximum number of items in an array. -**Example**: +**Example**: The `tags` array must have no more than 5 items. ```json { "properties": { - // 'tags' array must have at most 5 items "tags": { "type": "array", "maxItems": 5 } } } @@ -339,13 +359,11 @@ Maximum number of items in an array. ### `properties` Defines schemas for each property of an object. -**Example**: +**Example**: `firstName` and `lastName` must be strings. ```json { "properties": { - // 'firstName' must be a string "firstName": { "type": "string" }, - // 'lastName' must be a string "lastName": { "type": "string" } } } @@ -356,8 +374,7 @@ Defines schemas for each property of an object. ### `patternProperties` Defines validation rules for object properties whose names match a specified regular expression (regex) pattern. -**Example**: -All properties starting with "foo_" must be strings, and those starting with "bar_" must be integers. +**Example**: All properties starting with "foo_" must be strings, and those starting with "bar_" must be integers. ```json { "type": "object", @@ -374,13 +391,12 @@ All properties starting with "foo_" must be strings, and those starting with "ba ### `additionalProperties` Controls whether properties not defined in `properties` and `patternProperties` are allowed. -**Example**: +**Example**: Only `id` is allowed. ```json { "properties": { - "id": { "type": "string" } // Only 'id' is defined + "id": { "type": "string" } }, - // No properties other than 'id' are allowed "additionalProperties": false } ``` @@ -390,11 +406,10 @@ Controls whether properties not defined in `properties` and `patternProperties` ### `items` Defines the schema for items in an array. -**Example**: +**Example**: `scores` must be an array of integers. ```json { "properties": { - // 'scores' must be an array of integers "scores": { "type": "array", "items": { "type": "integer" } @@ -408,14 +423,13 @@ Defines the schema for items in an array. ### `dependencies` Specifies property dependencies. -**Example**: +**Example**: If `credit_card` is present, `billing_address` must also be present. ```json { "properties": { - "credit_card": { "type": "string" }, // Credit card number - "billing_address": { "type": "string" } // Billing address + "credit_card": { "type": "string" }, + "billing_address": { "type": "string" } }, - // If 'credit_card' is present, 'billing_address' must also be present "dependencies": { "credit_card": ["billing_address"] } @@ -428,31 +442,32 @@ Specifies property dependencies. ### `null` Allows a type or a property to be set to null (if supported by the implementation). -Setting `null` for variable types: +**Example**: Setting `null` for **variable types**. The value can be either a string or null. ```json { - // The value can be a string or null "type": ["string", "null"] } ``` +
-Setting `null` for **properties**: +**Example**: Setting `null` for **properties**. The `middleName` property can be either a string or null. ```json { "properties": { - // 'middleName' can be a string or null "middleName": { "type": ["string", "null"] } } } ``` + - + + -### `ConfigureSchemaValidationOperation` definition: +#### `ConfigureSchemaValidationOperation` definition: ```csharp // The schema validation configuration to apply. // If `configuration` is null, `ArgumentNullException` is generated. @@ -462,13 +477,14 @@ public ConfigureSchemaValidationOperation(SchemaValidationConfiguration configur -### `SchemaValidationConfiguration` class: +#### `SchemaValidationConfiguration` class: ```csharp public sealed class SchemaValidationConfiguration { // Indicates whether schema validation is globally disabled for the database. // When true, validation is disabled for all collections regardless of // individual collection settings. + // When false, validation is enabled based on the per-collection settings. public bool Disabled { get; set; } // Mapping of collection names to their corresponding schema definitions. @@ -482,7 +498,7 @@ public sealed class SchemaValidationConfiguration -### `SchemaDefinition` class: +#### `SchemaDefinition` class: ```csharp public class SchemaDefinition { @@ -499,7 +515,7 @@ public class SchemaDefinition -### `SchemaValidationException` message: +#### `SchemaValidationException` message: ```csharp // Initializes a new `SchemaValidationException` instance with an error message // that explains the reason for the schema validation failure. @@ -513,4 +529,5 @@ Raven.Client.Exceptions.SchemaValidation.SchemaValidationException: The required property 'Company' is missing. ``` + diff --git a/docs/documents/schema-validation/write-validation/write-validation_studio.mdx b/docs/documents/schema-validation/write-validation/write-validation_studio.mdx index 51471ac153..7711ebb21b 100644 --- a/docs/documents/schema-validation/write-validation/write-validation_studio.mdx +++ b/docs/documents/schema-validation/write-validation/write-validation_studio.mdx @@ -1,7 +1,7 @@ --- title: "Write validation: Studio" hide_table_of_contents: true -sidebar_label: Studio +sidebar_label: "Studio" sidebar_position: 2 --- @@ -11,6 +11,7 @@ import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; import LanguageSwitcher from "@site/src/components/LanguageSwitcher"; import LanguageContent from "@site/src/components/LanguageContent"; +import ContentFrame from '@site/src/components/ContentFrame'; import Panel from "@site/src/components/Panel"; # Write validation: Studio @@ -19,80 +20,169 @@ import Panel from "@site/src/components/Panel"; Manage [schema validation](../../../documents/schema-validation/schema-validation_overview) using the Studio **Document Schema** view, to condition documents storage on their compliance with a defined JSON schema. * In this article: - * [The Document Scheme view](../../../documents/schema-validation/write-validation/write-validation_studio#the-document-scheme-view) + * [The Document Schema view](../../../documents/schema-validation/write-validation/write-validation_studio#the-document-schema-view) * [Creating a collection schema](../../../documents/schema-validation/write-validation/write-validation_studio#creating-a-collection-schema) - * [Managing existing schemas](../../../documents/schema-validation/write-validation/write-validation_studio#managing-existing-schemas) + * [Managing and Testing existing schemas](../../../documents/schema-validation/write-validation/write-validation_studio#managing-and-testing-existing-schemas) + * [The schema playground](../../../documents/schema-validation/write-validation/write-validation_studio#the-schema-playground) - -To create a validation schema: + + +Use this view to create, test, and manage document schemas for your database collections. + ![Document schema view](./assets/schema-validation_studio_document-schema-view.png) 1. **Database settings** Click to open this view. 2. **Document schema** Click to open this view. +3. **Schema playground** + Click to create and test schemas in a [secluded playground](../../../documents/schema-validation/write-validation/write-validation_studio#the-schema-playground). 3. **Add new** - Click to create a new validation schema. - ![Add new schema](./assets/schema-validation_studio_add-new.png) + Click to create a [new schema](../../../documents/schema-validation/write-validation/write-validation_studio#creating-a-collection-schema) associated with one of your document collections. + - -To define a schema and associate it with a collection: + + +To create a validation schema, open: **Settings** > **Document schema** > **Add new** + ![Schema configuration](./assets/schema-validation_studio_schema.png) -1. **Collection selection** - Pick the collection whose documents you want to validate. - You can create one validation schema per collection, so if a schema already exists for a collection it will not be listed here. +1. **Select a collection** + Pick the collection for which you want to validate documents. + You can create one validation schema per collection. If a collection is already associated with a schema, the collection name will not be listed here. 2. **Define the schema using JSON syntax** - The depicted schema, for example, sets constraints for documents in the "Orders" collection. - A textual version is also provided below. Note that the comments included in it are meant to explain its structure and constraints, but comments are not supported in actual JSON syntax. + The depicted schema, for example, sets constraints for documents in the "Orders" collection: orders must include the string property "Company" and the positive numeric property "Freight". ```json { - // Enforce property types "properties": { - // Company ID (e.g., "companies/65-A") - "Company": { "type": "string" }, - - // Employee ID (e.g., "orders/1-A") - "Employee": { "type": "string" }, - - // Order time (e.g., "1998-05-06T00:00:00.0000000") - "OrderedAt": { "type": "string", "format": "date-time" } + "Company": { + "type": "string" + }, + "Freight": { + "type": "number", + "minimum": 0 + } }, - - // These properties must be present in every Order document - "required": ["Company", "Employee", "OrderedAt"], - - // Allow additional properties beyond those defined here - "additionalProperties": true + "required": [ + "Company", + "Freight" + ] } ``` + + + [See a list of available constraints.](../../../documents/schema-validation/schema-validation_overview#available-constraints) + + 3. **Save** Click to save and enable the schema. - When enabled, any document saved to the "Orders" collection will be validated against it. - E.g., attempting to save an "Order" document missing the "Company" field will result in a validation error: + When enabled, any document saved to the "Orders" collection will be validated against this schema. + E.g., attempting to save an "Order" document missing the "Company" field will generate a validation error: ```plain Raven.Client.Exceptions.SchemaValidation.SchemaValidationException: The required property 'Company' is missing. ``` + - Note that once a schema is enabled for a collection, collection documents are validated when they are saved directly, as well as when they are added or modified by operations such as patching or ETL tasks. (See a list of [operations that trigger validation](../../../documents/schema-validation/schema-validation_overview#a-list-of-operations-that-trigger-validation).) + Note that once a schema is enabled for a collection, collection documents are validated when they are saved directly, as well as when they are added or modified by operations such as patching or ETL tasks. + [See a list of operations that trigger validation](../../../documents/schema-validation/schema-validation_overview#a-list-of-operations-that-trigger-validation). + - -Created validation schemas are listed in the **Document schema** view, where you can edit,disable, enable, or delete them. + + +Existing schemas are listed in the Document Schema view, where you can disable, enable, test, edit, or delete schemas. + ![Manage existing schemas](./assets/schema-validation_studio_manage-existing-schemas.png) -1. **Disable/Enable validation per database** - Toggle to enable or disable schema validation for all collections. - This setting overrides per-collection schema settings. -2. **Disable/Enable validation per collection** - Toggle to enable or disable schema validation for this collection. - Note that if validation is disabled for the database, this setting will have no effect. -3. **Edit** - Click to edit an existing schema. -4. **Delete** - Click to delete an existing schema. +1. **Disable/Enable schema validation for all database collections.** + - **Disable** to deactivate validation for all collections regardless of individual collection settings (validation will **not** take place even when allowed locally for a collection). + - **Enable** to activate validation based on individual collection settings (validation will be enabled or disabled by per-collection configuration). + +2. **Disable/Enable schema validation for this collection.** + - **Disable** to deactivate validation for this collection (regardless of global settings). + - **Enable** to activate validation for this collection (providing validation is enabled for the database). + +3. **Test** + Click to test the validity of documents in the selected collection against the defined schema. + + ![Manage existing schemas](./assets/schema-validation_studio_test-button.png) + * **A. Test settings** + Validating a large number of documents against a schema can strain server resources. + Set **Max documents to scan** to the number of documents that you want to validate. E.g., set it to 100 to validate the first 100 documents in the collection. + Set **Max error messages to return** to the maximum number of validation errors to return. + * **B. Run test** + Click to start the validation test. + When the test is complete, you will get a summary of the results, including the number of invalid documents and the errors that were generated for each invalid document. + ![Manage existing schemas](./assets/schema-validation_studio_test-button_results.png) + +4. **Edit** + Click to edit this schema. +5. **Delete** + Click to delete this schema. + + + + +To create and test schemas in a secluded environment, without affecting your production data, open: **Settings** > **Document schema** > **Schema Playground** + +![Schema playground](./assets/schema-validation_studio_playground.png) + +1. **Add new** + Click to add a schema to the playground. + You can add multiple schemas to the playground, and test them all in a single run. + This can be helpful when, for example, you want to validate documents of multiple collections by checking for a specific field. + +2. **Define a schema** + + * **Collection** + Select the collection that the schema will be associated with. + * **Document schema** + Define the schema using JSON syntax. + In the depicted example we define two schemas: + The first, for the Orders collection, validates documents by field existence, type, and minimum value. + ```json + { + "properties": { + "Company": { "type": "string" }, + "Freight": { "type": "number", "minimum": 0 } + }, + "required": [ "Company", "Freight" ] + } + ``` + The second schema, for the Products collection, validates documents by the minimum value of the "UnitsOnOrder" field. + ```json + { + "properties": { + "UnitsOnOrder": { "minimum": 5 } + } + } + ``` + + +3. **Run test** + Click to run the validation test for all schemas defined in the playground. + The test will generate a summary of the results for each schema, including the number of invalid documents and the errors that were generated for each document. + + ![Schema playground results](./assets/schema-validation_studio_playground_results.png) + + * **A. Test results** + Scroll to view all error messages generated for each collection. + * **B. Test settings** + Validating a large number of documents against a schema can strain server resources. + Set **Max documents to scan** to the number of documents that you want to validate. E.g., set it to 100 to validate the first 100 documents of each collection participating in the test. + Set **Max error messages to return** to the maximum number of validation errors to return for each collection participating in the test. + * **C. Run test** + Click to re-run the test, e.g., after changing test settings. + + + + + + + +