Crawl columns descriptions across all table variants #213
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request centralizes and standardizes the column definitions for the
pages,requests, andparsed_csstables across the project. It introduces a single source of truth for column metadata inincludes/descriptions.jsand updates all relevant output and sample data definitions to reference these shared descriptions, ensuring consistency and easier maintenance. Additionally, new "latest" views are added for each table, selecting only the most recent data.Centralization of column definitions:
columnsobject toincludes/descriptions.jscontaining detailed column definitions forpages,requests, andparsed_css, including nested descriptions for complex fields.definitions/output/crawl,definitions/output/sample_data, and new view definitions indefinitions/output/latestto import and use the centralizedcolumnsdefinitions instead of duplicating column metadata inline.Addition of "latest" views:
pages,requests, andparsed_cssin thelatestschema, each selecting data from the most recent crawl date and referencing the centralized column definitions.These changes improve maintainability by ensuring all tables and views use consistent, up-to-date column descriptions and make it easier to update column metadata in the future.