Split ProvidersManager to support client server separation

[amoghrajesh]

Body

The context here is that the ProvidersManager: https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/providers_manager.py creates tight coupling between Airflow core and task sdk making client server separation difficult:

What I mean by that is:

1. task sdk depends on core's ProvidersManager (5+ usages)
2. API server loads all provider code just for UI metadata(static data)
3. Single class handles runtime resources for task sdk, UI metadata, and in some cases infra discovery like CLI portion of providers

Due to such problems, I want to explore a strategy where we can split ProvidersManager architecturally for a runtime + metadata purpose.

I think the success criteria here is and driving principle is that providers are primarily for task execution. API server only needs static metadata for UI forms, which should not come from code execution.

1. API server can start without loading provider code for connections, hooks etc
2. Task sdk has no dependency on core's ProvidersManager class
3. UI metadata comes from static sources (yaml?)

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[amoghrajesh]

cc @ashb @potiuk @uranusjr @kaxil FYI

[potiuk]

+100 :)