[FEATURE]: Allow dialect-specific command-line arguments during `transpile`

[asnare]

Is there an existing issue for this?

• I have searched the existing issues

Category of feature request

Transpiler

Problem statement

The Bladebridge transpiler supports dialect-specific options, normally prompted during install-transpile. Unlike most other options, at present they cannot be specified as arguments during transpile.

This is particularly problematic for non-interactive situations:

• During install-transpile the configuration questionnaire will not take place, meaning they cannot be specified there.
• During transpile there is no way to provide them either.

This means that:

• The optional overrides-file for all BB dialects cannot be provided.
• ETL conversions are not possible: they cannot run without a target technology being specified.

Proposed Solution

Currently we only use the dialect-specific mechanism for Bladebridge:

• All dialects allow an overrides-file containing custom migration logic.
• The ETL converters need to target either PySpark or DBSQL.

Rather than support generic arbitrary options, the simplest solution right now is to support two additional arguments for transpile:

• --overrides-file: the path to an overrides file.
• --target-technology: the target technology to target for ETL conversions.

Additional Context

If BB-specific options are needed there are currently two workarounds available:

• databricks labs lakebridge install-transpile: this will prompt to update the default configuration, allowing the options to be specified.
• Edit ~/.lakebridge/config.yml directly, in the user's DB workspace.