-
Notifications
You must be signed in to change notification settings - Fork 143
Schema validation - compliance auditing via Studio #2199
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Schema validation - compliance auditing via Studio #2199
Conversation
| </Tabs> | ||
|
|
||
| * Add to the index the field defined in the map for validation errors, e.g., `Errors`. | ||
| Set the "Store" option for this field to **Yes**, indicating that querying the index will retrieve errors from the index rather than from indexed documents. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe worth mentioning the link?
https://docs.ravendb.net/6.0/indexes/storing-data-in-index/
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
…schema definition for Studio
| Learn how to define a schema for the index below. | ||
|
|
||
| <Admonition type="note" title=""> | ||
| * If you define only a collection schema or only an index schema, the index will use the one you did define to validate documents. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think the term "collection schema" is the right term here.
As I see it, "database schema" or equivalent is more accurate
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
==>
- If you define only a database-level schema, the index will use it.
- If you define only an index-level schema, the index will use it.
- If you define both, the index-level schema will take precedence.
| * Indexing is performed continuously in the background, preparing for validation queries that remain quick and efficient even with large datasets. | ||
|
|
||
| <Admonition type="note" title=""> | ||
| **Note** that disabling validation for the database or for the collection does **not** disable auditing by an index. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unless the index is using the definition from the database.
That is the case where you use Schema.GetErrorFor, but you don't define a schema for the index
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
==>
Note: When the index uses a database-level schema, disabling validation at the database level will also disable auditing by the index.
However, when the index uses an index-level schema, disabling database-level validation will not disable auditing by the index.
| * You can also define the schema as part of the index definition. | ||
| In this case, the index schema will take precedence over any schema defined for the collection. | ||
| If no schema is defined for the index, it will inherit the schema defined for the collection. | ||
| A **collection schema** is associated with the collection that you plan to audit. You can define it using the [Document Schema](../../../documents/schema-validation/write-validation/write-validation_studio#creating-a-collection-schema) view. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, here, the distinction, as I see it, should be between database schema and index schema
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#### Define a validation schema
The index will use the schema defined in its index configuration.
See how to add a schema to the index configuration [below](../../../documents/schema-validation/auditing-document-compliance/auditing-document-compliance_studio#create-the-audit-index).
* If no schema is defined in the index configuration, the index will use the [database-level schema](../../../documents/schema-validation/write-validation/write-validation_studio#creating-a-collection-schema).
* If no schema is defined either in the index configuration or at the database level, the index will not perform any validation.
<Admonition type="note" title="">
**Note**: When the index uses a database-level schema, [disabling validation at the database level](../../../documents/schema-validation/write-validation/write-validation_studio#managing-and-testing-existing-schemas) will also disable auditing by the index.
However, when the index uses an index-level schema, disabling database-level validation will **not** disable auditing by the index.
</Admonition>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
changed it the same way in the API page
| * Add to the index the field defined in the map for validation errors, e.g., `Errors`. | ||
| Set the "Store" option for this field to **Yes**, indicating that querying the index will retrieve errors from the index rather than from indexed documents. | ||
| * Add the validation errors field (defined in the map) to the index. | ||
| Set the [Store](../../../indexes/storing-data-in-index/) option for this field to **Yes**, indicating that querying the index will retrieve errors from the index rather than from indexed documents. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not accurate.
The Store option set to Yes tells the index to actually store the value.
The default is just keeping the tokens (not the actual value)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
==>
Set the Store option for this field to Yes, to store the actual error messages in the index rather than just their tokens.
|
|
||
|  | ||
|
|
||
| * If no schema is associated with the collection, or you prefer a different schema for the index, add an index schema. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It sounds too complex to me here.
I would write something like "If you like an index dedicated schema to use inside the index."
But this is minor.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
==>
- If you prefer that the index use an index-level schema, add it in the Schema definition tab.
| ```csharp | ||
| // Create a map for the index, including a call to Schema.GetErrorsFor() | ||
| const string OrdersValidationMap = @" | ||
| from doc in docs.Orders |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we change the examples to filter first out valid documents?
In this way, the number of entries in the index is the number of invalid documents.
Also, it means that most of the time, when all the data is vali,d the resource cost of the index is minimal (no entries)
from doc in docs
let errors = Schema.GetErrorsFor(doc)
where errors != null && errors.Length > 0
select new
{
Id = doc.Id,
Errors = errors
}
map("@all_docs", (doc) => {
let errors = schema.getErrorsFor(doc);
if( errors != null && errors.length > 0)
return {
Id: id(doc),
Errors: errors
};
})
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
related issues:
RDoc-3594 Explain schema validation auditing via Studio
RDoc-3603 Explain the schema validation playground
RDoc-3624 Index schema definition field added to Studio
RDoc-3622 Add a schema validation configuration page