Skip to content

Conversation

claramoreirag
Copy link
Contributor

@claramoreirag claramoreirag commented Nov 19, 2024

LLMstudio Tracker Version 1.0.3

What was done in this PR:

  • Added Tracker integration with BigQuery, using the sqlalchemy-bigquery to extend sqlalchemy support to BigQuery
  • Bigquery does not support some SQL features, so the Table model needed to be adapted for compatibility:
    - autoincrement on IDs is not supported, so log id is given a value based on current_timestamp
    - JSON columns are not supported, so a custom type JSONEncodedDict was created on db_utils.py to encode JSON objects in strings before writing and decoding on read.
    - timestamp defaults were changed from server_default = func.now() to being set directly in python

How it was tested:

  • Following the example in examples/04_bigquery_integration.ipynb

Additional notes:

Two dependencies were added to the pyproject.toml:

  • sqlalchemy-bigquery
  • google-cloud-bigquery-storage

@claramoreirag claramoreirag marked this pull request as ready for review November 19, 2024 14:23
Base automatically changed from develop to main November 19, 2024 16:30
Copy link
Collaborator

@diogoncalves diogoncalves left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's test the datetime functions

@claramoreirag claramoreirag self-assigned this Nov 21, 2024
@claramoreirag claramoreirag linked an issue Nov 21, 2024 that may be closed by this pull request
4 tasks
@diogoncalves diogoncalves self-requested a review November 21, 2024 19:49
@claramoreirag claramoreirag changed the base branch from main to develop November 27, 2024 09:22
@claramoreirag claramoreirag merged commit 8382042 into develop Nov 27, 2024
3 checks passed
@claramoreirag claramoreirag deleted the feat/bigquery-tracking branch November 27, 2024 10:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[tracking] Connect to BQ with SQL alchemy

2 participants