Add pipeline run index attribute

[schustmi]

Describe changes

This PR adds an index attribute for each pipeline run, which specifies the index of the run within its pipeline.

Additionally, the pipeline attribute of the PipelineRunRequest was removed as the information is already included in the snapshot that is part of the request.

Limitations:

• If a run fails to be created (could be because a run with the same name exists or other reasons), the run number will still be bumped and that index will simply be unused
• When running a pipeline client-side, the following happens: We first create a pipeline run in the database, and then ask the orchestrator to submit the pipeline. If this submission fails, we currently delete the pipeline run. This will also cause a gap in the indexes of pipeline runs.

Pre-requisites

Please ensure you have done the following:

• I have read the CONTRIBUTING.md document.
• I have added tests to cover my changes.
• I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
• IMPORTANT: I made sure that my changes are reflected properly in the following resources:
  • ZenML Docs
  • Dashboard: Needs to be communicated to the frontend team.
  • Templates: Might need adjustments (that are not reflected in the template tests) in case of non-breaking changes and deprecations.
  • Projects: Depending on the version dependencies, different projects might get affected.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Other (add details above)

[github-actions]

Documentation Link Check Results

✅ Absolute links check passed
✅ Relative links check passed
_{Last checked: 2025-12-08 09:16:27 UTC}

[Json-Andriopoulos]

Normally I would advocate to execute data population for optional values in a post-deployment or post-migration step. Let's think about the worst case scenario in terms of execution time.

[schustmi]

Actually the index should be non-optional and the column type is changed at the end of the DB migration to reflect that, which is why we need some values for each existing row. I guess we could populate it with a placeholder and then later on replace by real values, not sure it's worth the effort here though?

[Json-Andriopoulos]

Nah let's no change the migration logic, let's just estimate the execution time for the backfill (worst case scenario). The general rule is migrations should complete quickly.

[stefannica]

Normally I would advocate to execute data population for optional values in a post-deployment or post-migration step

@Json-Andriopoulos I'm curious, why is it not acceptable to do these sort of changes in the alembic migration script ? what are the pros/cons ?

[Json-Andriopoulos]

Good one! I think this should be ok with the default isolation level in MySQL.

[schustmi]

Yep I think so! Also in my tests it seemed to work just fine. I also didn't add a unique constraint on pipeline_id, index, so even if this for some reason fails to generate unique values, we will not fail to create a run but instead only have a duplicate index.

[Json-Andriopoulos]

Looks good!