feat: process-per-connection architecture with full proxy support #791
+1,317,646
−6,258
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Create the foundation for process-per-connection architecture using gen_statem. - New hackney_conn module implements connection state machine - States: idle -> connecting -> connected -> closed - Supports connect timeout and idle timeout - Owner process monitoring for cleanup on owner death - Reconnection support from closed state This is the base that will replace socket-based pooling with process-based connection management.
Add HTTP request/response support to the connection gen_statem. Features: - request/5,6 API to send HTTP requests - body/1,2 API to read full response body - stream_body/1 API to stream response chunks - New states: sending, receiving - HTTP parser integration using hackney_http - Support for HEAD, GET, POST and other methods - Connection lifecycle: connected -> sending -> receiving -> connected Tests: - Unit tests for state machine behavior - Integration tests for request/response cycle (require local server)
Implement async streaming for HTTP responses with two distinct modes: - Continuous mode (async=true): Streams body chunks automatically - Once mode (async=once): Waits for stream_next before each chunk New API functions: - request_async/6,7 - Start async request - stream_next/1 - Request next chunk (once mode) - stop_async/1 - Cancel async streaming - pause_stream/1, resume_stream/1 - Flow control Uses two gen_statem states (streaming, streaming_once) to handle the different modes cleanly, avoiding gen_statem enter callback limitations with next_event actions.
Add simple_one_for_one supervisor for hackney_conn connection processes: - hackney_conn_sup:start_conn/1 - Start new connection process - hackney_conn_sup:stop_conn/1 - Stop connection process gracefully - Temporary restart strategy (crashed connections are not restarted) - Integrated into hackney_sup supervision tree Tests verify supervisor lifecycle and child management.
Create test_http_resource.erl as a Cowboy handler to replace external httpbin dependency. Uses stdlib json module (OTP 27+). Supported routes: - GET /get, POST /post - request info - GET /status/:code - specific status codes - GET /redirect-to - redirects with optional status_code - GET /basic-auth/:user/:pass - basic authentication - GET /cookies/set, /cookies - cookie handling - GET /robots.txt, /connection-close - special responses
Replace external localhost:8000 dependency with embedded Cowboy server on port 8124 using test_http_resource. - Add setup/teardown for Cowboy server - Update URLs to use embedded server - Tests now fully self-contained
Replace external localhost:8000 dependency with embedded Cowboy server on port 8125 using test_http_resource. - Add setup_integration/teardown_integration for Cowboy server - Remove check_local_server() skip logic - tests always run - Update paths to use /get and /post routes - Tests now fully self-contained
- Add embedded Cowboy server on port 8126, removing httpbin dependency - Replace localhost:8000 URLs with url(Path) helper function - Switch from jsone to stdlib json module for JSON decoding - Fix connection pool pollution by disabling pooling for POST test - Simplify tests for process-per-connection model - Add timeout to receive_response helper
- Merge hackney_pool_new_tests into hackney_pool_tests - Remove jsone test dependency (use stdlib json module) - Upgrade Cowboy test dependency to 2.12.0 - Use embedded Cowboy server for all pool tests
- Restore SOCKS5 proxy support from master branch - Add ssl_opts/2 function to hackney_ssl.erl (moved from deleted hackney_connection.erl) - Update hackney_socks5.erl to use hackney_ssl:ssl_opts/2 instead of deleted modules
- Restore HTTP CONNECT proxy support from master branch - Export default_ua/0 from hackney.erl for proxy module use - Update hackney_http_connect.erl to use: - hackney_ssl:ssl_opts/2 instead of hackney_connection:ssl_opts/2 - hackney:default_ua/0 instead of hackney_request:default_ua/0
- Remove deprecated modules: hackney_connect, hackney_connection, hackney_connections, hackney_headers, hackney_pool_handler, hackney_request, hackney_response, hackney_stream - Add new pool supervisors: hackney_pool_sup, hackney_pools_sup - Refactor hackney_conn as gen_statem with full request/response handling - Simplify hackney_manager to connection registry only - Rewrite hackney_pool for process-per-connection model - Update hackney_multipart for new API
- Remove hackney_headers_tests (module removed) - Update Cowboy resource handlers for new connection API - Fix multipart and noclen tests
- Add comprehensive NEWS.md for 2.0.0 breaking changes - Document migration guide from 1.x to 2.0 - Update hackney.hrl with proxy env var definitions - Update .gitignore
Fixes #536: SSL record overflow when using HTTP CONNECT proxy. The proxy response may be split across TCP packets. The previous code did a single recv() and only checked the status line, leaving partial response data in the socket buffer. When SSL upgrade occurred, it read this leftover HTTP data as SSL records, causing handshake failures. Now we accumulate data until we find \r\n\r\n (end of HTTP headers) before checking status and proceeding to SSL upgrade.
Add hackney:parse_proxy_url/1 to extract host, port, and credentials from proxy URLs. Also extends hackney_url:parse_url/1 to support socks5:// scheme with default port 1080. Fixes #741
Add get_proxy_config/2 to parse proxy options into normalized format. Handles URL strings, tuples, and various proxy types (http, connect, socks5). Determines proxy type based on target scheme (http->http, https->connect).
Allow hackney_conn to start with an already-connected socket. If socket option is provided, start in connected state directly. This enables proxy support where tunneling happens before conn starts.
Add helper function to start hackney_conn with a pre-established socket.
This is used for proxy connections where the tunnel is established first.
- Handle both raw sockets and {Transport, Socket} tuples from proxy modules
- Normalize transport atoms (gen_tcp -> hackney_tcp, ssl -> hackney_ssl)
- Start hackney_conn directly in connected state
Add connect_via_connect_proxy/7 to tunnel HTTPS requests through HTTP proxy. Integrates with hackney_http_connect module for CONNECT handshake. - Route connect proxy config in maybe_proxy/5 - Pass target host, port, transport and auth to hackney_http_connect - Use start_conn_with_socket/5 with the tunneled socket
Add connect_via_socks5_proxy/7 to tunnel connections through SOCKS5 proxy. Integrates with hackney_socks5 module for SOCKS5 handshake. - Route socks5 proxy config in maybe_proxy/5 - Pass proxy host, port, auth and target transport to hackney_socks5 - Support socks5_resolve option for DNS resolution control - Add mock proxy servers for automated testing - Add integration tests using mock HTTP CONNECT and SOCKS5 proxies - Add documentation for local and CI proxy testing
Add connect_via_http_proxy/7 for HTTP requests through HTTP proxy. Uses absolute URLs instead of tunneling. - Connect directly to proxy server - Build absolute URLs (http://host:port/path) for requests - Add Proxy-Authorization header when credentials provided - Add mock HTTP proxy server for testing - Add integration test for HTTP proxy
Add automatic proxy detection from environment variables: - HTTP_PROXY/http_proxy for HTTP requests - HTTPS_PROXY/https_proxy for HTTPS requests - ALL_PROXY/all_proxy as fallback Implement NO_PROXY/no_proxy bypass support: - Exact hostname match (case-insensitive) - Suffix match (example.com matches www.example.com) - Leading dot for subdomain match (.example.com) - Wildcard (*) to bypass all hosts Update get_proxy_config to accept target host for NO_PROXY checking. Add unit tests for check_no_proxy/2 and integration test for bypass.
- Rename unused Transport parameter to _Transport in start_conn_with_socket/5 - Add dialyzer nowarn for check_no_proxy/2 binary clause (used in tests)
Vendor cow_ws and cow_deflate from cowlib as hackney_cow_ws and hackney_cow_deflate to provide WebSocket frame parsing and building. - hackney_cow_ws: WebSocket frame encode/decode, handshake key generation - hackney_cow_deflate: safe inflate helper with size limits - Add tests for key functions (23 tests) - Update NOTICE with cowlib attribution (ISC license) Files placed in src/vendor/ with hackney_ prefix to avoid conflicts if cowlib is also present as a dependency.
Add WebSocket URL scheme parsing to hackney_url: - ws:// maps to hackney_tcp transport, port 80 - wss:// maps to hackney_ssl transport, port 443 - Updated unparse_url/1 and normalize/2 for ws/wss schemes - Added 8 tests covering basic URLs, custom ports, query strings, credentials, IPv6, and round-trip parsing
Add WebSocket client functionality to hackney with a new gen_statem module (hackney_ws) that handles the WebSocket lifecycle: - hackney_ws.erl: gen_statem for WebSocket connections with states idle -> connected -> closing -> closed - Supports passive mode (blocking ws_recv) and active modes (true/once) - Uses vendored hackney_cow_ws for frame encoding/decoding - Handles ping/pong, close handshake, and fragmented messages New hackney API functions: - ws_connect/1,2: Connect to ws:// or wss:// URLs - ws_send/2: Send text, binary, ping, pong, or close frames - ws_recv/1,2: Receive frames (passive mode) - ws_setopts/2: Set active mode options - ws_close/1,2: Close connection gracefully Includes comprehensive tests with mock Cowboy WebSocket server.
Add HTTP CONNECT and SOCKS5 proxy support for WebSocket connections: - hackney.erl: Add get_ws_proxy_config/3 to convert proxy config for WebSocket (always uses tunnel mode, never simple HTTP proxy) - hackney_ws.erl: Add proxy field to state record, implement do_connect_via_http_proxy/9 and do_connect_via_socks5/9 - hackney_cow_ws.erl: Fix type spec for make_frame to allow undefined close_code for non-close frames Also includes: - Use trap_exit instead of monitor (simplify from previous commit) - Fix dialyzer warnings (unmatched expressions, dead code) - Add proxy integration tests for WebSocket
Expand WebSocket tests from 18 to 51 test cases covering: - Large messages (1KB, 64KB text and binary) - Query string handling in URLs - Custom headers in handshake - Close codes (1001, 1008, custom) - Active mode switching (passive <-> active) - Ping/pong with data, multiple pings, unsolicited pong - Concurrent connections and rapid connect/disconnect - Socket info (peername, sockname) - Controlling process transfer - Connect timeout - SOCKS5 proxy support - Empty messages, Unicode, binary with null bytes Enhanced mock_ws_handler to support: - Delayed responses - Header inspection - Query string inspection - Large message generation - Multiple replies
Add 15 new WebSocket tests covering: - Delayed server responses - Multiple messages from single command - Server-initiated ping handling in active mode - Conversation sequences (request/response patterns) - Rapid message exchange (100 messages) - Interleaved text/binary messages - Reconnection after server close - JSON-RPC style messaging simulation - Pub/sub messaging simulation - Binary protocol simulation (length-prefixed) Bug fixes: - Fix active mode to deliver bare ping frames to owner - Fix mock handler duplicate pong responses (Cowboy auto-responds)
- Add full lsquic engine integration for QUIC v1 (RFC 9000) - Implement TLS 1.3 handshake via BoringSSL - Add async I/O thread for packet handling - Support external socket FD from Erlang for pre-warming - Use dirty NIFs for all blocking operations - Add connection lifecycle: connect, open_stream, close - Send Erlang messages for connection events (connected, closed)
- Test NIF availability check - Test connection to cloudflare.com:443 - Test peername/sockname address retrieval - Test stream opening - Test get_fd for UDP socket FD extraction - Test error handling for invalid arguments
- Add send_headers (stub for now, needs header conversion) - Add send_data using lsquic_stream_write - Add reset_stream using lsquic_stream_close - Store lsquic_stream_t handle in QuicStream - Add find_stream helper to locate streams by ID
- Implement proper header conversion from Erlang to lsxpack_header
- Parse Erlang header list [{Name, Value}, ...]
- Build buffer with all header name/value pairs
- Use lsxpack_header_set_offset2 to set up headers
- Call lsquic_stream_send_headers to send
- Add HTTP/3 request headers test
- Fix error_code type warning in reset_stream
- Add hackney_http3 module for high-level HTTP/3 requests - Add UDP happy eyeballs support in hackney_happy (connect_udp/3,4) - Update hackney_ssl to handle http3 protocol in ALPN opts - hackney_http3 uses happy eyeballs to pre-warm UDP socket - Pass socket FD to QUIC NIF for connection
- Add hackney_h3 thin wrapper module for HTTP/3 operations - Add HTTP/3 fields to conn_data record (h3_conn, h3_streams, try_http3) - Add HTTP/3 connection and request handling in hackney_conn - Add QUIC message handlers for response processing - Fix UDP happy eyeballs socket ownership transfer bug - Simplify hackney_http3 to let lsquic manage UDP sockets - Add HTTP/3 integration tests
- Fix NIF to properly deliver HTTP/3 response headers and body to Erlang - Add lsquic_stream_flush() to ensure data is sent immediately - Add lsquic_stream_shutdown() to send FIN flag on request completion - Add stream_opened notification when new streams are created - Add ATOM_STREAM_OPENED to atoms for stream notifications - Add comprehensive test for full HTTP/3 request/response flow - Remove debug fprintf statements from production code The NIF now correctly: - Sends stream_opened message when streams are created - Delivers response headers via stream_headers message - Delivers response body via stream_data message - Properly signals end of stream with FIN flag
Add test/hackney_conn_http3_tests.erl with tests for: - HTTP/3 connection and request via hackney_conn - Protocol detection (get_protocol returns http3) - Response header parsing Verifies existing HTTP/3 integration in hackney_conn.erl works correctly.
- Remove unused msg_env field and quic_conn_send_to_owner function - Use atomic operations for should_stop flag to prevent data races
- Add HTTP/3 Support section to NEWS.md changelog - Add HTTP/2 and HTTP/3 guides to ex_doc configuration - Add lsquic and BoringSSL to NOTICE file
- Add `default_protocols` application env option (default: [http3, http2, http1])
- Add `hackney_util:default_protocols/0` to get the configured default
- HTTP/3 is now tried first by default when QUIC NIF is available
- Users can override globally via application:set_env/3
- Per-request {protocols, [...]} option still takes precedence
- Filter `protocols` option before passing to transport (fixes badarg)
- Update HTTP/2 tests to explicitly request HTTP/2 protocol
Add the QUIC/HTTP/3 dependencies directly to the repository to enable CI builds with full HTTP/3 support. - boringssl: Google's OpenSSL fork required by lsquic - lsquic: LiteSpeed QUIC library for HTTP/3 transport Note: Binary test data and policy documents excluded to keep only sources.
BoringSSL's generated assembly files use __has_feature(hwaddress_sanitizer) which is a Clang-specific preprocessor feature. Define it as 0 for GCC to allow ARM64 Linux builds with GCC.
- Add extern to atom declarations in atoms.h - Define atoms in hackney_quic_nif.c only - Enable CMAKE_POSITION_INDEPENDENT_CODE for lsquic/boringssl - Update to C17/C++20 standards
- Use ErlNifSInt64 instead of int64_t for stream_id (fixes pointer type warning) - Replace deprecated -flat_namespace -undefined suppress with -undefined dynamic_lookup on macOS
CMake doesn't support function-style preprocessor definitions on the command line, which caused the previous approach to fail on Linux CI. Instead, use a gcc_compat.h header file that defines __has_feature(x) as 0 for GCC, and force-include it via the -include compiler flag.
- Add Go dependency to all CI jobs for BoringSSL cmake build - Fix buffer overflow warning in cbs.c by adding len == 0 check - Remove legacy job for OTP versions < 27
- Updated from previous version to release 0.20251124.0 - Removed test files, documentation, and build system files not needed - Kept only essential source files for cmake build - Added c_src/DEPENDENCIES.md to document vendored dependency versions - Disabled pki and bssl targets in CMakeLists.txt (not needed for hackney) - Added BUILD_TESTING=OFF to disable test builds - Added priv/*.so and priv/*.dll to .gitignore
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Architecture
hackney_conngen_statem: one process per connectionhackney_conn_supsupervisor for connection processeshackney_managerreduced to connection registryhackney_poolrewritten for process-based poolingProxy Support
Bug Fixes
insecureoption searched inssl_options#786)Removed Modules
hackney_connect, hackney_connection, hackney_connections, hackney_headers, hackney_pool_handler, hackney_request, hackney_response, hackney_stream
Breaking Changes
See NEWS.md for migration guide from 1.x to 2.0.