Long-lived connections experience false disconnections due to timer reliability issues

[matiaslopezd]

First, I want to express my appreciation for Socket.IO. It has been instrumental in helping us build real-time features and has served us well for several years (4). The library's ease of use and feature set helped us get to market quickly.

The Issue

After running Socket.IO in production with connections that need to stay alive for hours or days (not just minutes), we've encountered systematic false disconnections that are impacting our business operations.

Environment:

• Socket.IO client: 3.x and 4.x
• Multiple deployment scenarios tested (including Web Workers)
• Stable network conditions verified through parallel WebSocket connections

Observed Behavior:
Our server logs show a pattern of disconnection/reconnection cycles every few hours, with reconnections happening within 2-4 seconds:

10:54:17 - reconnected
10:54:16 - disconnected 
09:44:13 - reconnected
09:44:11 - disconnected
08:41:12 - reconnected
08:41:10 - disconnected

These are false positives - the network connection remains stable throughout (verified by running parallel native WebSocket connections that maintain 99.9% uptime).

Critical Requirement: Connection Reliability

For our use case, connection reliability is paramount. We need connections that are only terminated by:

• Actual network failures
• Server-side decisions
• Explicit client disconnection

Connections should NEVER be disrupted by:

• Frontend code behavior (tab backgrounding, CPU load)
• Timer inaccuracies
• Browser throttling policies
• JavaScript event loop delays

In our domain, a false disconnection is worse than a delayed heartbeat. We need to trust that when a disconnection event fires, it represents a real connectivity issue that requires intervention.

Root Cause Analysis

The issue appears to stem from the heartbeat mechanism's reliance on setTimeout, which becomes unreliable in long-running applications due to:

1. Browser throttling (even in Web Workers)
2. JavaScript event loop delays under load
3. System-level timer coalescing
4. Accumulated timer drift over time

The client-side code is making decisions about connection health that should only be made based on actual network conditions.

Validation Through Native WebSocket Implementation

To confirm this analysis, we implemented a native WebSocket wrapper that:

1. Maintains Socket.IO protocol compatibility by implementing the message format:

   <packet type>[<# of binary attachments>-][<namespace>,][<acknowledgment id>][JSON-stringified payload without binary]
   

2. Uses RTT-based adaptive timeouts instead of fixed timers:

   // Timeout dynamically adjusted based on actual network conditions
   const timeout = baseTimeout * (1.5 + (measuredRTT / 1000));

3. Implements heartbeat without relying on setTimeout accuracy:

   • Monitors actual message activity
   • Uses performance.now() for drift-free time measurement
   • Validates connection through actual ping/pong exchange

Results:

• Native WebSocket wrapper: 99.9% uptime, 0 false disconnections
• Socket.IO client: ~67% perceived uptime due to false disconnections
• Both running on identical network conditions, same server

This confirms that the disconnections are not network-related but rather caused by the client-side timeout mechanism.

Business Impact

For applications requiring persistent connections (monitoring dashboards, trading platforms, real-time collaboration tools), these false disconnections cause:

• Message loss during reconnection windows
• State synchronization overhead
• Unnecessary server resource usage
• User experience degradation
• False alerts in monitoring systems
• Erosion of trust in connection status

Suggestion

While Socket.IO works excellently for short-to-medium duration connections (< 20 minutes), applications requiring long-lived, stable connections might benefit from:

1. Alternative heartbeat mechanisms that don't rely solely on setTimeout
2. RTT-based adaptive timeouts that adjust to actual network conditions
3. Option to disable client-side timeout detection entirely, relying only on server/network signals
4. Activity-based connection monitoring instead of timer-based
5. Documentation guidance on when native WebSockets might be more appropriate

Our Path Forward

After extensive testing, we've migrated to native WebSocket implementations for our long-lived connections while continuing to use Socket.IO for shorter-duration features. With native WebSockets and RTT-based monitoring, we achieve true connection reliability - disconnections only occur due to real network or server events, never due to client-side timer issues.

This isn't a criticism of Socket.IO, but rather a recognition that different use cases require different tools.

Appreciation

Socket.IO has been a crucial part of our journey. This issue is shared in the spirit of helping others who might face similar challenges with long-lived connections, and to provide feedback that might help shape the library's future direction.

Thank you for all the work you've put into this library. It has helped countless developers, myself included, build real-time applications.

[matiaslopezd]

As a follow-up to my previous comment about connection reliability, I wanted to share separate feedback about WebSocket standards compliance that could improve Socket.IO's interoperability and debugging experience.

Socket.IO uses its own disconnection reasons and error handling that don't always map cleanly to the established WebSocket standards. This creates challenges when:

1. Integrating with standard WebSocket tooling
2. Debugging connection issues
3. Implementing proper reconnection logic based on close codes
4. Working with infrastructure that expects standard WebSocket behavior

The IANA WebSocket Protocol Registries and RFC 6455 define standard close codes that provide clear semantics:

// Standard codes that could improve Socket.IO's clarity:
1000 - Normal Closure
1001 - Going Away  
1002 - Protocol Error
1003 - Unsupported Data
1006 - Abnormal Closure
1008 - Policy Violation
1011 - Internal Server Error
1012 - Service Restart
1013 - Try Again Later
1014 - Bad Gateway

Benefits of Standards Compliance

1. Better Debugging: Tools like Chrome DevTools, Wireshark, and WebSocket proxies already understand these codes

2. Clearer Reconnection Logic:

   // Standards-based decision making
   switch(closeCode) {
     case 1000: // Normal closure, don't reconnect
     case 1001: // Server going away, maybe reconnect
     case 1006: // Abnormal, definitely reconnect
     case 1008: // Policy violation, don't reconnect
     case 1013: // Try again later, reconnect with backoff
   }

3. Infrastructure Compatibility: Load balancers, proxies, and monitoring tools expect these codes

4. Cross-Library Interoperability: Easier migration paths and hybrid implementations

Practical Example

Currently in Socket.IO:

// Ambiguous reasons
socket.on('disconnect', (reason) => {
  // reason might be: "transport close", "ping timeout", "transport error"
  // Hard to map to specific handling logic
});

With standard codes:

socket.on('disconnect', (reason, code) => {
  // code: 1006 - Abnormal closure (network issue)
  // code: 1000 - Normal closure (server initiated)
  // code: 1008 - Policy violation (auth failed)
  // Clear, actionable information
});

[darrachequesne]

Hi!

First, thanks for the detailed analysis 👍

The heartbeat mechanism has indeed been a source of problems for years:

• its direction has been reversed in v3 (the server now inits the heartbeat), in order to reduce the number of false positives due to timer throttling

• the default value for the pingTimeout value has been decreased (60s => 5s in v2.1) to improve the detection time of an unclean disconnection, then increased again (5s => 20s in v4.0) to reduce the number of false positives

• actual traffic does not reset the timers (see also: be7b4e7), which can be problematic when sending a lot of data at once (as the ping/pong packets get queued)

Also related:

• setTimeout() blocking writes: socketio/engine.io-client@f30a10b
• sending packets over an expired connection: 8adcfbf

So we are really open to suggestions on this. Any change to the current mechanism would be quite a breaking change though...

Regarding your second message, we have currently 6 disconnection reasons on the client side:

• io server disconnect: when the server manually disconnects the client
• io client disconnect: when the client manually disconnects
• ping timeout: when the client detects a failure of the heartbeat mechanism
• transport close: when the low-level connection is closed (for example, when restarting the server with WebSocket)
• transport error: when the low-level connection is closed (for example, when restarting the server with HTTP long-polling)
• parse error: when the client receive an invalid payload

I guess we could indeed map those reasons to the WebSocket codes, something like:

• io server disconnect => 1000 Normal Closure
• io client disconnect => 1000 Normal Closure
• ping timeout => 1006 Abnormal Closure
• transport close => 1006 Abnormal Closure
• transport error => 1006 Abnormal Closure
• parse error => 1002 Protocol Error

Reference: https://socket.io/docs/v4/client-api/#event-disconnect

Possibly related: #3237