Add support for PyMongo Async and N+1 Dereference/select_related using Pipeline #2904
+41,716
−5,781
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Refactored the core ORM to support PyMongo’s native async API - Unified sync and async code paths across documents, querysets, and transactions - Replaced legacy async implementations - Removed deprecated and compatibility code BREAKING CHANGE: - Removed legacy async behavior - Removed LazyReferenceField - Removed GenericLazyReferenceField - GenericReferenceField now requires `choices` - Dropped support for PyMongo < 4.14 - Dropped support for MongoDB < 4.2
BaseQuerySet is now defined only in the synchronous queryset implementation.
add: TestQuerysetLookupMatch
…uilder stages - Extract query normalization, match planning, lookup planning, stage building, and tail stages - Introduce clear aggregation pipeline architecture aligned with MongoDB stages - Reduce PipelineBuilder to a small orchestration layer - Improve readability, isolation, and long-term maintainability
…ilter-only lookups Body: • refactor StageBuilder traversal/handlers for readability • keep $lookup unfiltered for correct hydration; apply foreign predicates via local $filter • emit explicit _missing_reference markers so deref raises DoesNotExist • reduce duplication with shared helpers and structured dispatch
- Normalize refIds generation across scalar and container fields
- Use reduce-based flattening for ListField reference lookups
- Ensure missing references emit `{_missing_reference, _ref}` only
- Fix select_related pipelines to match MongoDB 4.2+ semantics
- Expand pipeline builder tests for nested and container references
…ort, updated installation steps, supported MongoDB versions, and improved examples.
…l image and init script.
…ster dep management
…date tox env deps
# Conflicts: # docs/requirements.txt
…project - Use uv for dependency management and builds - Simplify GitHub Actions matrix and MongoDB setup - Replace custom CI scripts with uv-based dependency management, actions - Match tox environment with GitHub Action matrix
…install target in Makefile
- Corrected version number in docs and package - disabled coveralls - fix tox not running selected env only - Parallel runner support for coveralls TODO: - Split github action
…registry cleanup - Update assertions to use `await` where needed for async compatibility. - Introduce `_DocumentRegistry.clear()` and `_CollectionRegistry.clear()` calls in test setups and teardowns. - Normalize test workflows to ensure proper registry state management across synchronous and asynchronous environments. - Simplify pipeline builder tests by removing unnecessary async and ensuring compatibility with recent updates.
…ts, and async example usage in `query_counter` and `async_query_counter`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contributes to #2902 , which tracks the ongoing effort to add native async support to MongoEngine using PyMongo’s official async API (>= 4.14).
It includes foundational changes toward async-first workflows, improvements to
select_relatedvia aggregation pipelines, and related updates across core internals, tests, CI, and documentation.Notes
📌 Details, motivation, and full scope are documented in the issue: #2902
🚧 Work is still in progress — feedback is welcome.