Add vendored Flatbuffers (v25.9.23) #1761
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add Google Flatbuffers as git submodule at deps/flatbuffers - Pinned to commit 95053e6a (v25.9.23-2, same as zlmdb) - Vendor Python runtime at root flatbuffers/ directory for packaging - Consistent with zlmdb approach (submodule + vendored copy) Part of crossbario#1760
- Add flatbuffers/* to ruff exclude list (vendored Python runtime) - Add deps/* to ruff exclude list (git submodules) - Prevents linting errors from upstream Google Flatbuffers code - Consistent with zlmdb approach Fixes code quality checks failing on E743 in flexbuffers.py
Implements RFC 7692-style Brotli compression for WebSocket messages: **New Features:** - NEW: autobahn/websocket/compress_brotli.py - Full permessage-brotli implementation - Follows exact same pattern as existing compress_snappy.py and compress_bzip2.py - Registered in PERMESSAGE_COMPRESSION_EXTENSION registry **Platform-Optimized Dependencies:** - CPython: Uses 'brotli' package (CPyExt, 40 binary wheels for 3.11-3.14) - PyPy: Uses 'brotlicffi' package (CFFI, 20 binary wheels for PyPy 3.11+) - Aligns with project CFFI policy for PyPy compatibility **Implementation Details:** - Extension name: 'permessage-brotli' - Supports client_no_context_takeover and server_no_context_takeover parameters - Uses brotli.Compressor() with process()/finish() API - Uses brotli.Decompressor() with process() API - Context takeover control for memory-efficient streaming **Compression Support Summary:** After this change, autobahn[compress] provides: - permessage-deflate: Always available (Python stdlib zlib) - permessage-bzip2: Optional (requires bz2 in Python build) - permessage-snappy: Optional (requires manual python-snappy install) - permessage-brotli: NEW - Included with binary wheels (CPython + PyPy) **Binary Wheel Coverage:** - brotli: 40 wheels (Linux/macOS/Windows x86_64+ARM64, CPython 3.11-3.14) - brotlicffi: 20 wheels (Linux/macOS/Windows, PyPy 3.11+) python-snappy remains optional (no binary wheels) - users install separately if needed. Part of crossbario#1760 - Batteries-included dependencies strategy
Enhanced 'WebSocket Acceleration and Compression' section: **Acceleration (Deprecated):** - Marked 'accelerate' as deprecated - Explains NVX (Native Vector Extensions) supersedes wsaccel - Points to NVX section for SIMD-accelerated WebSocket operations **Compression (Expanded):** - Comprehensive table of all compression methods: - permessage-deflate (always available, RFC 7692) - permessage-brotli (NEW, recommended, RFC 7932) - permessage-bzip2 (optional) - permessage-snappy (manual install) - Detailed Brotli section: - Platform-optimized: brotli (CPython) / brotlicffi (PyPy) - Advantages: superior compression, binary wheels, IETF standard - Full binary wheel coverage for all platforms - Resources with all relevant links: - RFC 7932 (Brotli spec) - Google Brotli GitHub - brotlicffi GitHub - PyPI: brotlicffi - WAMP protocol issue crossbario#555 - Note on Snappy explaining manual install requirement Part of crossbario#1760
- check-compressors: Verifies all WebSocket compression methods are available - check-serializers: Verifies all WAMP serializers are available - Both recipes print class mappings for each extension/serializer - Integrated into main 'check' recipe for CI/CD workflow Expected output: - Compressors: permessage-deflate, permessage-bzip2, permessage-brotli - Serializers: json, msgpack, cbor, ubjson, flatbuffers Resolves part of batteries-included serializers feature (crossbario#1760)
Contributor
Author
|
…ipes - Both recipes now accept optional 'expect' parameter with comma-separated list - Validates actual available items match expected list exactly - Shows clear error message with missing/extra items on mismatch - Exits with error code 1 if validation fails Usage examples: just check-compressors "" "permessage-deflate, permessage-bzip2, permessage-brotli" just check-serializers "" "json, msgpack, cbor, ubjson, flatbuffers" Error output example: Expected: cbor json msgpack Actual: cbor flatbuffers json msgpack ubjson Missing: Extra: flatbuffers ubjson Enables strict CI/CD validation of batteries-included features (crossbario#1760)
New section documents complete dependency strategy: Core Dependencies: - txaio, cryptography, hyperlink - all with excellent wheel coverage WAMP Serializers (Batteries Included): - All 5 serializers (json, msgpack, cbor2, ubjson, flatbuffers) documented - Flatbuffers confirmed as vendored (zero external dependency) - Platform-optimized: msgpack (CPython) vs u-msgpack-python (PyPy) WebSocket Compression: - permessage-deflate (stdlib zlib) - always available - permessage-brotli (brotli/brotlicffi) - 40+/20+ wheels, RFC 7932 - permessage-bzip2 (stdlib bz2) - always available - permessage-snappy (optional) - no wheels, manual install Optional Extras: - twisted: All pure Python/universal wheels - encryption: All with excellent coverage including PyPy - scram: argon2-cffi with CFFI wheels including PyPy - nvx: Our own CFFI-based SIMD implementation Platform Coverage: - Linux (glibc/musl), macOS, Windows - x86_64, ARM64 (Apple Silicon, AWS Graviton) - Python 3.11-3.14 (including free-threaded 3.14t) - CPython and PyPy 3.11+ Verdict: All goals achieved - batteries included, CFFI everywhere, comprehensive binary wheel coverage, zero system dependencies. Nothing more to optimize or wish for. Resolves documentation for crossbario#1760
3 tasks
3 tasks
oberstet
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
permessage-brotliRelated Issue(s)
Closes or relates to #1760
Checklist
the style guidelines of this project
is effective or that my feature works
updated the changelog
in this PR