Skip to content

Conversation

@eltee1
Copy link
Contributor

@eltee1 eltee1 commented Dec 3, 2025

This PR adds a log table for logging import-table logdata and functionalily to fill it.
This table is filled during the database-build with some metadata from the imported import-files, like the name of the file, the import-time and the checksums of the table before and after the import. This is done by a altered function of "load_table".
This overwiew can be used for checks on the database build(are the correct import-files used during the build-proces?).
Also, often is the question what import-file is used for a particular table. This information is placed in the .html- and .rtf files, but these stand apart from the database, so if those can't be found, there is no way to know.
With the introduction of this log-table, the used import-files can be easy to find.

@eltee1 eltee1 requested a review from JacobusXIII December 3, 2025 13:13
Copy link
Contributor

@JacobusXIII JacobusXIII left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great first version! Now it needs some polishing up ;)

@eltee1 eltee1 requested a review from JacobusXIII December 3, 2025 14:52
Copy link
Contributor

@JacobusXIII JacobusXIII left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there!

Copy link
Contributor

@JacobusXIII JacobusXIII left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be more specific about naming? Like table_data_metadata instead of just metadata or so?
@BertScholten what do you think?

@eltee1 eltee1 requested a review from JacobusXIII December 3, 2025 16:02
Copy link
Contributor

@JacobusXIII JacobusXIII left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One tiny issue.

And I'm not sure if i like the (universal) term metadata. Can we come up with something better?
It's the metadata of the imported data.....

@eltee1
Copy link
Contributor Author

eltee1 commented Dec 4, 2025

One tiny issue.

And I'm not sure if i like the (universal) term metadata. Can we come up with something better? It's the metadata of the imported data.....
other options:

  • data_catalog
  • source_registry

@eltee1
Copy link
Contributor Author

eltee1 commented Dec 5, 2025

decided: metadata table will be renamed to "load_table_logs"

@eltee1 eltee1 requested a review from JacobusXIII December 5, 2025 14:01
Copy link
Contributor

@JacobusXIII JacobusXIII left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The action being performed is registering all load_tables, which result in een entry in the load_table_logs table. That's why I'm in favor of the name register_load_table as a function-, boolean- and contants-name.

@eltee1 eltee1 requested a review from JacobusXIII December 10, 2025 15:49
Copy link
Contributor

@JacobusXIII JacobusXIII left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Last few issues

@eltee1 eltee1 requested a review from JacobusXIII December 15, 2025 08:11
@eltee1 eltee1 changed the title add structure for registering metadata in the metadata table. add structure for registering loagdata during table imports. Dec 15, 2025
Copy link
Member

@BertScholten BertScholten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bit late to the party perhaps (went over open PRs to check if anything could be closed), but had a suggestion.

@eltee1 eltee1 changed the title add structure for registering loagdata during table imports. add structure for registering logdata during table imports. Jan 5, 2026
eltee1 added 2 commits January 6, 2026 10:12
…he default value of the should_register_load_table to TRUE. With this change, de build default is to register the metadata in the table log data.
Copy link
Member

@BertScholten BertScholten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants