Merge branch 'main' into ATSCALE-27878-m2m

Author: svetoslavpetkov

README.md

@@ -1,14 +1,22 @@
 ![logo](images/sml-logo-large.png)
 
+# SML version 1.2
+
+This is documentation for SML spec version `1.2`. For earlier versions browse the repository tags. Examples:
+
+- [SML version 1.0](https://github.com/semanticdatalayer/SML/tree/v1.0)
+- [SML version 1.1](https://github.com/semanticdatalayer/SML/tree/v1.1)
+
 # What is SML?
-Semantic Modeling Language, or SML for short, encompasses over a decade of hands-on development, solving use cases for hundreds of customers across industries such as finance, healthcare, retail, manufacturing, CPG, and more. SML covers more than just tabular use cases. At its core, it is a multidimensional semantic modeling language that supports metrics, dimensions, hierarchies,  semi-additive measures, many-to-many relationships, cell-based expressions, and much more. 
+
+Semantic Modeling Language, or SML for short, encompasses over a decade of hands-on development, solving use cases for hundreds of customers across industries such as finance, healthcare, retail, manufacturing, CPG, and more. SML covers more than just tabular use cases. At its core, it is a multidimensional semantic modeling language that supports metrics, dimensions, hierarchies, semi-additive measures, many-to-many relationships, cell-based expressions, and much more.
 
 SML delivers on the following requirements:
 
 1. **Object-oriented**: SML is an object-oriented language that promotes composability and inheritance. This allows semantic objects to be shared within other semantic objects and across organizations, supporting easy and consistent model-building.
 2. **Comprehensive**: SML is based on more than a decade of modeling experience across various industry verticals and use cases. SML handles multi-dimensional constructs and serves as a superset of all other existing semantic modeling languages.
 3. **Familiar**: SML is based on YAML, a widely adopted, human-readable, industry-standard syntax.
-4. **CI/CD Friendly**: SML is code, so it is compatible with Git and CI/CD practices for version control, automated deployment, and software lifecycle management. 
+4. **CI/CD Friendly**: SML is code, so it is compatible with Git and CI/CD practices for version control, automated deployment, and software lifecycle management.
 5. **Extensible**: SML syntax can be enhanced to support additional properties and features.
 6. **Open**: SML is Apache open-sourced to support community innovation and is free to use in any application or use case.
 
@@ -21,10 +29,11 @@ We are or will be open-sourcing the following:
 1. **A YAML-based Language Specification**: The SML specification is documented and encompasses tabular and multidimensional constructs.
 2. **Pre-built Semantic Models**: The GitHub repository contains pre-built semantic models that incorporate standard data models, such as TPC-DS, common training models like Worldwide Importers and AdventureWorks, and marketplace models like Snowplow and CRISP. We expect to add semantic models for SaaS applications such as Salesforce, Google Analytics, and Jira soon.
 3. **SML SDK**: An SDK that facilitates the programmatic reading and writing of SML syntax.
-4. **SML CLIs**: Command line interfaces (CLIs) for installing SML dependencies and validating SML syntax. This includes a reference CLI for deploying SML models to a proprietary semantic layer platform.
+4. **SML CLI**: Command line interface (CLI) for installing SML dependencies and validating SML syntax. This includes a reference CLI for deploying SML models to a proprietary semantic layer platform.
 5. **Semantic Converters**: A CLI for translating other semantic modeling languages to and from SML, including Snowflake Cortex semantic models, Databricks UC Metrics, and Power BI semantic models.
 
 ## SML Example
+
 The following is an example of an SML `model` object:
 
 ```
@@ -59,6 +68,7 @@ metrics:
 ```
 
 ## SML Object Hierarchy
+
 The following graphic illustrates the key SML objects and their relationships:
 
 ```mermaid
@@ -85,7 +95,7 @@ The following sections describe the different SML object types as well
 as the properties available for each:
 
 - [Catalog](sml-reference/catalog.md) - Defines the control file for a SML repository. It contains all repository-level definitions.
-- [Package](sml-reference/package.md) - Defines additional Git repositories references whose objects can be used in the current repository. 
+- [Package](sml-reference/package.md) - Defines additional Git repositories references whose objects can be used in the current repository.
 - [Model](sml-reference/model.md) - Defines the logical, business-friendly representation on top of the physical data.
 - [Dimension](sml-reference/dimension.md) - Defines the logical collection of attributes and hierarchies for supporting drill-down.
 - [Row Security](sml-reference/row-security.md) - Defines row-level data access rules for users and groups.
@@ -95,22 +105,27 @@ as the properties available for each:
 - [Connection](sml-reference/connection.md) - Defines a database and schema for connecting datasets to the physical data platform.
 - [Composite Model](sml-reference/composite-model.md) - Defines a model made up of multiple other models.
 
-## SML Converters
+## SML Tools and Converters
 
-[SML Converters and Tooling](https://github.com/semanticdatalayer/sml-converters) - Library of bi-directional SML converters and tooling for different semantic layer platforms.
+- SML SDK [(npm download)](https://www.npmjs.com/package/sml-sdk) - SDK for creating and updating SML objects. Used by converters.
+- SML CLI [(npm download)](https://www.npmjs.com/package/sml-cli) - Command line interface (CLI) for installing SML dependencies, validating SML syntax, and a reference implementation for deploying SML models to a proprietary semantic layer platform.
+- [SML Converters](https://github.com/semanticdatalayer/sml-converters) [(npm download)](https://www.npmjs.com/package/sml-converters)- Library of bi-directional SML converters for different semantic layer platforms.
 
 ## Model Library
 
 ### Tutorial Models
+
 1. [Internet Sales Model](https://github.com/semanticdatalayer/sml-models-tutorials-internet-sales) - a simple, single-fact model derived from the fictitious AdventureWorks retail dataset.
 2. [World Wide Importers Model](https://github.com/semanticdatalayer/sml-models-tutorials-ww-importers) - a more complex, multi-fact model representing a fictional wholesale and distribution company.
 3. [TPC-DS Model](https://github.com/semanticdatalayer/sml-models-tutorials-tpcds) - a complex, multi-fact model that encodes the [TPC-DS](https://www.tpc.org/tpcds/) benchmark model in SML.
 4. [TPC-H Model](https://github.com/semanticdatalayer/sml-models-tutorials-tpch) - a complex, multi-fact model that encodes the [TPC-H](https://www.tpc.org/tpch/) benchmark model in SML.
-5. [AdventureWorks2012 Model](https://github.com/semanticdatalayer/sml-models-tutorials-adventureworks2012) -  the standard Microsoft SSAS tutorial in SML.
+5. [AdventureWorks2012 Model](https://github.com/semanticdatalayer/sml-models-tutorials-adventureworks2012) - the standard Microsoft SSAS tutorial in SML.
 
 ### Data Warehouse Usage/Cost Models
+
 1. [Snowflake Usage Model](https://github.com/semanticdatalayer/sml-models-usage-snowflake) - a semantic model for analyzing Snowflake credit and data warehouse usage.
 
 ### Marketplace Models
+
 1. [Snowplow Digital Analytics Model](https://github.com/semanticdatalayer/sml-models-snowplow) - Snowplow empowers organizations to create a scalable, first-party data foundation so marketing and data teams can effectively analyze and tackle Customer 360 use cases.
 2. [CRISP CPG Retail and Distributor Data Model](https://github.com/semanticdatalayer/sml-models-crisp-cpg-retail) - Crisp connects to over 40 leading U.S. retailers and distributors.

sml-reference/catalog.md

@@ -14,6 +14,9 @@ label: SML Model Library
 version: 1.0
 aggressive_agg_promotion: false
 build_speculative_aggs: false
+hidden_models:
+  - Supply Model
+  - Sales Model
 ```
 
 # Entitity Relationships
@@ -28,6 +31,7 @@ classDiagram
       Boolean aggressive_agg_promotion
       Boolean build_speculative_aggs
       Object dataset_properties
+      Array~String hidden_models
     }
 ```
 
@@ -64,6 +68,15 @@ not need to be unique.
 
 The version of SML being used.
 
+## hidden_models
+
+- **Type:** array<string>
+- **Required:** Y
+- **Added in** v1.2
+- **Use case:** When deploying a composite model the referenced component models may not need to be accessible as separate models.
+
+A list of the models that will be excluded from the deploy. Cannot reference composite models, only regular models.
+
 ## aggressive_agg_promotion
 
 - **Type:** boolean
@@ -114,7 +127,7 @@ Supported properties:
 - `allow_peer_aggs`: Boolean, optional. Enables aggregation on data
   derived from datasets in data warehouses that are different from the
   source dataset.
-- `allow_preferred_aggs`: Boolean, optional. Allow aggregates to be built 
+- `allow_preferred_aggs`: Boolean, optional. Allow aggregates to be built
   in preferred storage.
 
 Specify the `unique_name` of the dataset followed by the properties and

sml-reference/connection.md

@@ -6,7 +6,9 @@ into your repository.
 
 Each connection file should define a single database connection *and*
 its schema. If you need to use additional schemas for the same database,
-each must be defined in a separate connection file.
+each must be defined in a separate connection file. If a dataset uses a query
+instead of tables, the database and schema do not need to be specified in the
+connection file.
 
 Sample `connection` file:
 
@@ -69,13 +71,13 @@ The name of the database connection itself, excluding the schema.
 ## database
 
 - **Type:** string
-- **Required:** Y
+- **Required:** N
 
-The source database used for this connection.
+The source database is required when used in the table datasets, but it is not necessary for query datasets.
 
 ## schema
 
 - **Type:** string
-- **Required:** Y
+- **Required:** N
 
-The source schema used for this connection.
+The source schema is required when used in the table datasets, but it is not necessary for query datasets.

sml-reference/dimension.md

@@ -270,6 +270,7 @@ namespace Dimensions{
       Boolean is_aggregatable
       Boolean exclude_from_fact_agg
       String time_unit
+      Int constraint_translation_rank
       Array~String~ allowed_calcs_for_dma
       CustomEmptyMember custom_empty_member
       String folder
@@ -309,12 +310,15 @@ namespace Dimensions{
       String label
       String description
       String folder
+      Number precedence
       Array~CalculatedMembers~ calculated_members
+      Boolean is_hidden
     }
     class CalculatedMembers{
       String unique_name
       String description
       String format
+      Boolean is_hidden
       String expression
       Boolean use_input_metric_format
       String template
@@ -549,13 +553,36 @@ Defines the individual calculated members in the group.
 
 A description of the calculation group.
 
+## is_hidden
+
+- **Type:** boolean
+- **Required:** N
+
+Determines whether the calculation group is visible in BI tools. 
+
+Supported values:
+
+- `false` (default)
+- `true`
+
 ## folder
 
 - **Type:** string
 - **Required:** N
 
 The name of the folder in which the calculation group is displayed in BI tools.
 
+## precedence
+
+- **Type:** number
+- **Required:** N
+
+Update to "Precedence" explicitly defines the order of Calculation Group evaluation, making it consistent across BI tools.
+
+Supported values:
+
+- Integer and floating point numbers
+
 # Calculated Members Properties
 
 ## unique_name
@@ -585,6 +612,18 @@ Supported templates:
 
 If you do not want to use a built-in template, you can define a custom expression using the `expression` property (see below).
 
+## is_hidden
+
+- **Type:** boolean
+- **Required:** N
+
+Determines whether the attribute is visible in BI tools. 
+
+Supported values:
+
+- `false` (default)
+- `true`
+
 ## expression
 
 - **Type:** string
@@ -1196,6 +1235,14 @@ If the key consists of one column, the values in that column must be
 unique. If the key is a compound key, the columns together must provide
 unique values.
 
+## constraint_translation_rank
+
+- **Type:** integer
+- **Required:** N
+- **Range:** should be a valid 32 bit integer
+
+Defines the translation of dimension filter constraints into fact table partition column constraints. This can significantly improve query performance for cases where fact-based aggregates are not used.
+
 ## shared_degenerate_columns
 
 - **Type:** array