Kombu typing improvements #207
Open
+6,544
−91
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Fix kombu.utils.json.loads/dumps: make optional params have defaults - Fix kombu.utils.json.object_hook: return Any instead of None - Add missing attributes and methods to kombu.transport.base.Transport - Fix kombu.transport.base.StdChannel to match runtime - Add kombu.transport.virtual module stubs (Transport, Channel, QoS, etc.) - Add kombu.utils.objects.cached_property stub - Add kombu.utils.scheduling.FairCycle stub - Add missing amqp exceptions (ChannelError, RecoverableConnectionError, etc.) - Export ChannelError from kombu.exceptions - Add __pycache__ to .gitignore
- Add __all__ exports to all modules - Replace Channel alias with StdChannel throughout - Expand Connection, Producer, Consumer with full signatures - Add ConnectionPool, ChannelPool, Resource classes - Fix parameter defaults and types to match runtime - Add missing attributes and methods across all modules - Create simple.pyi for SimpleQueue/SimpleBuffer - Use TypeAlias for forward references to avoid class attribute collisions
- Add missing amqp stubs: connection.pyi, channel.pyi, basic_message.pyi - Add __all__ exports to amqp modules - Fix amqp.promise class with full attributes and methods - Add AMQPError.method property - Fix Channel/Connection method signatures to match runtime - Use TypeAlias pattern to avoid class attribute name shadowing - Add missing kombu exports and __all__ declarations - Fix Mailbox.Node as method instead of class attribute - Fix PoolGroup, ProducerPool, set_limit signatures - Expand Entity, Resource, and exception stubs - Remove non-existent kombu.utils.encodings stub Remaining errors are internal modules without public API stubs and one kombu bug (reprkwargs in __all__ but undefined).
kombu.utils.__all__ includes reprkwargs but forgot to import it from kombu.utils.functional. The stub correctly re-exports it, fixing what kombu intended but failed to do.
AMQP (0 stubtest errors): - abstract_channel: AbstractChannel base class - method_framing: frame_handler/frame_writer functions - platform: LINUX_VERSION, SOL_TCP, KNOWN_TCP_OPTS - protocol: basic_return_t, queue_declare_ok_t namedtuples - sasl: SASL, PLAIN, AMQPLAIN, EXTERNAL, RAW, GSSAPI classes - serialization: GenericContent, Basic, dumps/loads functions - spec: Connection, Channel, Exchange, Queue, Basic, Tx, Confirm specs - transport: TCPTransport, SSLTransport, Transport factory - utils: promise, NullHandler, utility functions KOMBU (new module stubs): - asynchronous: Hub, Timer, Entry, semaphores, http client - clocks: LamportClock, timetuple - common: Broadcast, QoS, maybe_declare, eventloop, etc. - compat: Publisher, Consumer, ConsumerSet compatibility - compression: register, compress, decompress - log: LogMixin, Log, setup_logging - matcher: MatcherRegistry, match functions - transport backends: pyamqp, memory, redis, SQS, etc. - utils submodules: encoding, url, text, time, imports, debug, div, compat, collections, amq_manager, eventio 10 transport modules fail to import due to optional dependencies (botocore, azure, google, pymongo, etc.) - expected behavior.
This commit significantly improves the kombu type stubs by fixing
signature mismatches and missing exports identified by mypy stubtest.
Key changes:
- Fix asynchronous module stubs (hub, timer, semaphore, http)
- Fix common module (QoS class signature)
- Fix compat module (Publisher, Consumer, ConsumerSet classes)
- Fix log module (LogMixin, Log classes)
- Fix matcher module (MatcherRegistry class, MatcherFunction type)
- Fix transport module stubs:
- pyamqp: Proper inheritance from amqp classes
- SLMQ, consul, etcd, filesystem, memory, pyro, zookeeper:
Proper Channel inheritance from virtual.Channel
- qpid: Override methods with different signatures
- native_delayed_delivery: Now properly typed as utility module
- Fix utils module stubs (collections, compat, debug, eventio, url)
- Add missing exports and constants across modules
Results:
- AMQP stubs: 0 errors (previously 70)
- Kombu stubs: 28 real errors + 10 import failures from optional deps
(previously 430+ errors)
The 10 import failures are expected - they're from optional transport
backends (redis, mongodb, SQS, azure, etc.) that require additional
dependencies not installed.
- Fix transport stubs: redis, azureservicebus, mongodb, confluentkafka, azurestoragequeues, qpid, pyro, filesystem, zookeeper, memory, SLMQ, SQS, gcpubsub, native_delayed_delivery - Convert asynchronous/aws from single file to package structure with proper submodules for connection, ext, and sqs/ - Convert transport/sqlalchemy from single file to package with models - Add stubtest allowlist for unfixable issues (librabbitmq C extension, types-redis metaclass quirks, kombu bug in utils.__all__) - Install optional dependencies for comprehensive stubtest coverage
redis>=7.1.0 has native type hints (py.typed marker), so types-redis is no longer needed. This also resolves the stubtest metaclass issues for PrefixedStrictRedis and PrefixedRedisPipeline. Allowlist reduced from 5 to 3 unfixable items.
Write comprehensive stub based on kombu source code since librabbitmq is abandoned (last release 2018, Python 3.6 max) and cannot be installed. The stub now includes Message, Channel, Connection, and Transport classes with proper signatures matching the kombu transport implementation.
- Use Self return type for __enter__ and __new__ methods (PYI034) - Use object instead of Any for __exit__ *args (PYI036) - Assign values to __slots__ instead of annotations (PYI035) - Remove unused TypeVars (PYI018) - Sort imports and __all__ entries (I001, RUF022) - Import from collections.abc instead of typing (UP035) - Fix forward reference issues in qpid.pyi - Fix Mailbox.type shadowing builtin type in pidbox.pyi
- Import Self from typing_extensions for Python 3.10 compatibility - Export kombu.pools module for celery stubs - Configure basedpyright to allow legitimate variable/method overrides - Fix redis.client imports using direct imports - Widen memoize Cache parameter to accept dict
Replace global pyright check disables in pyproject.toml with inline # pyright: ignore[...] comments on specific lines. This follows the same pattern used in celery-stubs for handling legitimate type mismatches in stub files where subclasses override parent signatures. Changes: - Remove reportIncompatibleVariableOverride, reportIncompatibleMethodOverride, and reportImplicitOverride from pyproject.toml - Add inline pyright ignore comments to amqp-stubs, kombu-stubs, and celery-stubs where type overrides are intentional
This addresses reportImplicitOverride warnings by adding explicit @OverRide decorators from typing_extensions to all methods that override base class methods. This is consistent with the pattern used throughout celery-stubs. Also removes unnecessary pyright ignore comments where the override decorator makes them redundant. Reduces warnings from 135 to 15 (89% reduction).
- Add @OverRide decorators to amqp-stubs (sasl.pyi, utils.pyi) - Fix TypeVar warnings in kombu utils by using @overload for optional parameters that affect type inference - Add pyright ignore comments for missing external type stubs (confluent_kafka, google.cloud.pubsub_v1) This brings the warning count from 15 to 0.
- Remove quoted annotations in objects.pyi and scheduling.pyi (PYI020/UP037) - Fix pyright ignore comment placement after import sorting in confluentkafka.pyi
Contributor
|
It looks incredible! Thank you for your tremendous contribution. I think that when you need to write so many annotations, your approach works perfectly. I completely agree that minor inaccuracies are acceptable. We can always correct them if necessary. |
Contributor
|
Are you planning to add anything else? |
Contributor
Author
|
Give me a couple of days, I want to give this another pass... |
Pull latest kombu-stubs changes from main-ng, including: - New allowlist.txt for stubtest - New librabbitmq.pyi transport stub - Remove non-existent encodings.pyi module
Pull latest kombu-stubs changes including: - pyright ignore comments for transport method overrides - Updated allowlist for stubtest
- Add @OverRide decorators for implicit overrides to satisfy basedpyright - Remove unnecessary pyright ignore comment in azureservicebus.pyi
Contributor
Author
|
Ok. I have tested this by now with some of my Celery/Kombu projects, and also I used monkeytype to again check the stubs a bit more. This is everything from my end for the kombu-stubs, but I will submit other PRs probably soon. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've been working with Kombu directly in the last weeks, and tried to improve the type stubs along the way.
Overall I think the result is decent.
The workflow was basically just to run stubgen/stubtest to get first to some basic stubs, then mypy/basedpyright/ruff/tests to catch issues, everything semi-automated with AI, then some manual review. I've added the external deps which means the type checkers can actually resolve base classes and give useful feedback on overrides.
This is mainly here for some feedback. Personally I would go with this approach and just aim for full type stubs coverage, and then work out the kinks. But I can also see that this is not ideal.