You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
dbt "adapters" are responsible for adapting dbt's functionality to a given database. `dbt-oracle` implements dbt functionalities for the Oracle database. To learn more about building adapters, check
dbt "adapters" are responsible for adapting dbt's functionality to a given database. `dbt-oracle` implements dbt functionalities for the Oracle database.
8
7
9
8
> Prior to version 1.0.0, dbt-oracle was created and maintained by [Indicium](https://indicium.tech/) on [their GitHub repo](https://github.com/techindicium/dbt-oracle). Contributors in this repo are credited for laying the groundwork and maintaining the adapter till version 0.4.3.
10
9
From version 1.0.0, dbt-oracle is maintained and distributed by Oracle.
11
10
12
-
## What is dbt? <aname='what-is-dbt'></a>
13
-
14
-
dbt does the T in ELT (Extract, Load, Transform). To work with dbt you need a copy of your data already loaded in your warehouse.
15
-
16
-
### dbt features <aname='core-concepts'></a>
17
-
- With dbt, you can express all transforms with SQL select
18
-
- Different materialization strategies.
19
-
- view
20
-
- table
21
-
- incremental; selective rebuild for new rows
22
-
- ephemeral; Model 1 interpolated into Model 2 as a Common Table Expression (CTE)
23
-
- No need to write boilerplate code
24
-
- All code to create table or views is generated using macros.
25
-
- Idempotence; rerun models
26
-
- If your source data were to stop updating, successive runs of your transformations would still result in the same tables and views in your warehouse.
27
-
- If your production deployment of your transformations were interrupted, the next run of the transformations would result in the same tables and views as if the deployment had not been interrupted.
28
-
- If you manually triggered transformations between scheduled runs, the scheduled run would result in the same tables and views as if the manual runs had not been triggered.
29
-
- All transformation code is accessible and can be version controlled.
30
-
- Dependency resolution
31
-
- Use of ref() function ``select * from {{ ref('MODEL_NAME')}}``
32
-
- dbt automatically resolves dependencies in between models and builds a Directed Acyclic Graph (DAG).
33
-
Each path in the DAG can be independently executed using multiple threads.
34
-
- Interpolates the name of database schema
35
-
- Includes a built-in testing framework to ensure model accuracy
36
-
- not null
37
-
- unique
38
-
- contains accepted values
39
-
- relationships
40
-
- custom tests
41
-
- Generate documentation for your project and render it as a website.
42
-
- Use macros to write reusable SQL
43
-
44
-
### An example <aname='an-example'></a>
45
-
46
-
dbt model
47
-
```sql
48
-
--models/sales_internet_channel.sql
49
-
{{ config(materialized='table') }}
50
-
WITH sales_internet AS (
51
-
SELECT*FROM {{ source('sh_database', 'sales') }}
52
-
WHERE channel_id =4 )
53
-
SELECT*FROM sales_internet
54
-
```
55
-
dbt compiles the above SQL template to run the below DDL statement.
56
-
```sql
57
-
CREATETABLEdbt_test.sales_internet_channel AS
58
-
WITH sales_internet AS (
59
-
SELECT*fromsh.sales
60
-
WHERE channel_id =4 )
61
-
SELECT*FROM sales_internet
62
-
```
63
11
64
12
For dbt documentation, refer https://docs.getdbt.com/docs/introduction
65
13
66
-
## Installation <aname='install'></a>
67
-
68
-
dbt-oracle can be installed via the Python Package Index (PyPI) using pip
dbt-oracle requires the following 3 python packages.
85
-
86
-
`dbt-core`
87
-
88
-
- Open source framework for data transformation
89
-
- Jinja Templating and core SQL compilation logic
90
-
- Latest version of dbt-core is preferred; From version 1.0.0, dbt-core supports Python 3.7 or higher
91
-
- For Python 3.6, pip will fallback to version 0.21.1 of dbt-core
92
-
93
-
`cx-Oracle`
94
-
- Python driver for Oracle database
95
-
- Oracle client libraries should be installed on the system. For details check, https://cx-oracle.readthedocs.io/en/latest/user_guide/installation.html
96
-
97
-
`dataclasses; python_version < '3.7'`
98
-
- dataclasses package was introduced in the standard Python library from Python 3.7. This is conditional dependency and required only for Python 3.6
99
-
100
-
## Getting Started <aname='getting-started'></a>
101
-
102
-
Create a dbt project for oracle database using the `dbt init` command. The init command is interactive and will help you get started with a new project.
103
-
104
-
`dbt init` will:
105
-
106
-
* ask you the name of the project
107
-
* ask you the database adapter you are using i.e. oracle
108
-
* prompt to specify necessary connection details
109
-
110
-
This example shows initialization of test project `dbt_oracle_test_project`
111
-
112
-
```text
113
-
>> dbt init
114
-
115
-
Running with dbt=1.0.4
116
-
Enter a name for your project (letters, digits, underscore): dbt_oracle_test_project
service (service name in tnsnames.ora) [{{ env_var('DBT_ORACLE_SERVICE') }}]:
126
-
dbname (database name in which dbt objects should be created) [{{ env_var('DBT_ORACLE_DATABASE') }}]:
127
-
schema (database schema in which dbt objects should be created) [{{ env_var('DBT_ORACLE_SCHEMA') }}]:
128
-
threads (1 or more) [1]: 4
129
-
Profile dbt_oracle_test_project written to ~/.dbt/profiles.yml using target's profile_template.yml and your supplied values. Run 'dbt debug' to validate the connection.
130
-
Your new dbt project "dbt_oracle_test_project" was created!
131
-
132
-
```
133
-
134
-
Then dbt init command will:
135
-
136
-
1. Create the following folder with project name and sample files to get you started
137
-
```text
138
-
├── README.md
139
-
├── analyses
140
-
├── dbt_project.yml
141
-
├── macros
142
-
├── models
143
-
│ └── example
144
-
├── seeds
145
-
├── snapshots
146
-
└── tests
147
-
```
148
-
2. Create a connection profile on your local machine. The default location is `~/.dbt/profiles.yml`
14
+
## Installation
149
15
150
-
Next step, [configure connection][1] related parameters and test if dbt connection works using dbt debug command
151
-
152
-
```text
153
-
>> dbt debug
154
-
155
-
os info: macOS-11.6-x86_64-i386-64bit
156
-
Using profiles.yml file at ~/.dbt/profiles.yml
157
-
Using dbt_project.yml file at /dbt_oracle_test_project/dbt_project.yml
158
-
Configuration:
159
-
profiles.yml file [OK found and valid]
160
-
dbt_project.yml file [OK found and valid]
161
-
Required dependencies:
162
-
- git [OK found]
163
-
Connection:
164
-
user: ***
165
-
database: ga01d76d2ecd5e0_db202112221108
166
-
schema: ***
167
-
protocol: tcps
168
-
host: adb.us-ashburn-1.oraclecloud.com
169
-
port: 1522
170
-
service: <service_name>_high.adb.oraclecloud.com
171
-
connection_string: None
172
-
shardingkey: []
173
-
supershardingkey: []
174
-
cclass: None
175
-
purity: None
176
-
Connection test: [OK connection ok]
177
-
178
-
All checks passed!
179
-
```
16
+
For installation, read how you can set up [Oracle profile][1] for dbt
Copy file name to clipboardExpand all lines: dbt_adbs_test_project/README.md
+5-27Lines changed: 5 additions & 27 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,39 +9,15 @@ To test the integration with ADBS, you can use OCI's [Always Free Autonomous Dat
9
9
10
10
The database also provides a read-only Sales History data set. Any user can start querying the tables in this Sales History `sh` schema. Models in this test project refer the `sh` schema. You do not need to load any other dataset.
11
11
12
-
## Set the environment variables
12
+
## Setup the oracle profile
13
13
14
-
The following environment variables should be set to test integration with ADBS.
14
+
To setup [profiles.yml](profiles.yml) read [setup & installation instructions][1] on dbt docs website
15
15
16
-
```bash
17
-
# cx_oracle needs lib_dir parameter pointing to the folder
18
-
# containing the libraries from an unzipped Instant Client Basic or Basic Light package.
19
-
# If lib_dir is not passed client libraries are looked for in the Operating system search path
20
-
# or set in the following environment variables.
21
-
DYLD_LIBRARY_PATH # For MacOS
22
-
LD_LIBRARY_PATH # For Linux
23
-
24
-
# For ADBS, cx_oracle will need the path to the folder
25
-
# containing client wallet, sqlnet.ora and tnsnames.ora
26
-
TNS_ADMIN
27
-
28
-
# Database connection config - dbt specific variables
29
-
DBT_ORACLE_USER
30
-
DBT_ORACLE_HOST
31
-
DBT_ORACLE_PORT
32
-
DBT_ORACLE_SERVICE
33
-
DBT_ORACLE_PASSWORD
34
-
DBT_ORACLE_DATABASE
35
-
DBT_ORACLE_SCHEMA
36
-
```
37
-
Check [profiles.yml](profiles.yml) to understand how these environment variables are used.
16
+
Install dbt-oracle in your project's virtual environment.
38
17
39
-
Also read about [connecting to Oracle Database](https://cx-oracle.readthedocs.io/en/latest/user_guide/connection_handling.html) using cx_Oracle
40
18
41
19
## dbt project
42
20
43
-
Install dbt-oracle in your local development environment.
44
-
45
21
Next, run the `dbt compile` command in this project directory to compile the models and resolve dependencies.
46
22
```bash
47
23
dbt compile --profiles-dir ./
@@ -124,3 +100,5 @@ Following directory structure shows the 10 models, seed, test, analysis and snap
124
100
125
101
## Tests [TODO]
126
102
- Metrics - Experimental feature introduced in dbt-core==1.0.0
0 commit comments