docs: add comprehensive investigation report for issue #2023

- Analyzed test_small_network_get_failure failure modes
- Identified gateway crash root cause (fixed in a283e23)
- Documented PUT operation timeout issues
- Provided recommendations for re-enabling the test
- Suggested modernization using #[freenet_test] macro

Related: #2023, #2043, #2036, #2011

Author: claude

INVESTIGATION_REPORT_ISSUE_2023.md

@@ -0,0 +1,229 @@
+# Investigation Report: Issue #2023 - test_small_network_get_failure
+
+## Issue Overview
+Test `test_small_network_get_failure` is currently ignored due to:
+1. PUT operations timing out (exceeding 30-second threshold)
+2. Gateway process crashes during or after PUT operations
+
+**Test Location**: `apps/freenet-ping/app/tests/test_small_network_get_issue.rs:24`
+
+## Investigation Findings
+
+### 1. Test Configuration Analysis
+- **Network Topology**: Star topology with 1 gateway + 3 nodes
+  - All nodes connect only through the gateway (poor connectivity by design)
+  - Simulates production conditions with limited peer connections
+- **Timeouts**:
+  - Test client timeout: **30 seconds** (line 221)
+  - Internal operation TTL: **60 seconds** (`OPERATION_TTL` in `config/mod.rs:40`)
+  - Cleanup interval: **5 seconds**
+- **Problem**: Test times out at 30s before internal operation timeout at 60s
+
+### 2. Gateway Crash Root Cause (FIXED ✅)
+
+**Commit a283e23**: "fix: guard op-state timeout notifications"
+
+**Problem**:
+```rust
+// OLD CODE - Would panic if receiver dropped
+event_loop_notifier.notifications_sender
+    .send(Either::Right(NodeEvent::TransactionTimedOut(tx)))
+    .await
+    .unwrap();  // ❌ PANIC if send fails
+```
+
+**Solution**:
+```rust
+// NEW CODE - Handles errors gracefully
+async fn notify_transaction_timeout(
+    event_loop_notifier: &EventLoopNotificationsSender,
+    tx: Transaction,
+) -> bool {
+    match event_loop_notifier.notifications_sender.send(...).await {
+        Ok(()) => true,
+        Err(err) => {
+            tracing::warn!("Failed to notify; receiver likely dropped");
+            false  // ✅ No panic
+        }
+    }
+}
+```
+
+**Impact**: This was the root cause of gateway crashes during timeout scenarios. The gateway would panic when trying to send timeout notifications after the receiver channel was dropped.
+
+### 3. PUT Operation Issues (PARTIALLY FIXED ⚠️)
+
+#### Recent Fixes:
+1. **Commit 615f02d**: "fix: route successful PUT responses back through forwarding peers"
+   - Fixed routing of SuccessfulPut messages
+   - Ensures responses reach the originating node
+
+2. **Commit 5734a33**: "fix: cache contract state locally before forwarding client-initiated PUT"
+   - Ensures publishing node caches state immediately
+   - Improves reliability of PUT operations
+
+3. **Commit a34470b**: "feat: implement transaction atomicity with parent-child relationship"
+   - Added proper parent-child transaction tracking
+   - Improves composite operation reliability
+
+#### Remaining Concerns:
+The PUT operation code (`crates/core/src/operations/put.rs`) contains multiple `unwrap()` and `expect()` calls:
+
+```rust
+// Lines with potential panics:
+361: peer = %op_manager.ring.connection_manager.get_peer_key().unwrap()
+553: let peer = broadcast_to.get(peer_num).unwrap()
+596: peer = %op_manager.ring.connection_manager.get_peer_key().unwrap()
+...and more
+```
+
+Most are for:
+- `get_peer_key().unwrap()` - Should be safe if peer is initialized
+- `location.expect("infallible")` - Documented as infallible
+- `target.unwrap()` - After checking `is_some()`
+
+### 4. Timeout Configuration Mismatch
+
+**Problem**: Test timeout (30s) < Operation TTL (60s)
+
+```rust
+// In test (line 221)
+match timeout(Duration::from_secs(30), client_node1.recv()).await {
+    ...
+    Err(_) => {
+        println!("Timeout waiting for put response");  // ⏰ 30 seconds
+    }
+}
+
+// In config (crates/core/src/config/mod.rs:40)
+pub(crate) const OPERATION_TTL: Duration = Duration::from_secs(60);  // ⏰ 60 seconds
+```
+
+**Impact**: Test fails before internal cleanup can occur, making it hard to distinguish between:
+- Actual operation failures
+- Slow network propagation
+- Connection establishment delays
+
+### 5. Test Quality Issues
+
+The test uses deprecated patterns:
+```rust
+// ❌ Deprecated logging
+freenet::config::set_logger(Some(LevelFilter::DEBUG), None);
+
+// ❌ Manual node setup (50+ lines of boilerplate)
+// ❌ No event aggregation for debugging
+// ❌ No peer identification in logs
+```
+
+**Should use**:
+```rust
+#[freenet_test(
+    nodes = ["gateway", "node1", "node2"],
+    auto_connect_peers = true,
+    aggregate_events = "on_failure",
+    timeout_secs = 120  // ✅ Longer timeout
+)]
+async fn test_small_network_get_failure(ctx: &mut TestContext) -> TestResult {
+    // Automatic setup, cleanup, and failure reporting
+}
+```
+
+## Recommendations
+
+### 1. Re-enable the Test ✅
+The gateway crash issue has been fixed (commit a283e23). The test should be re-enabled to verify:
+- Gateway stability under timeout conditions
+- PUT operation reliability with recent fixes
+
+### 2. Increase Test Timeout
+```diff
+- match timeout(Duration::from_secs(30), client_node1.recv()).await {
++ match timeout(Duration::from_secs(90), client_node1.recv()).await {
+```
+
+**Rationale**:
+- Connection establishment can take 15+ seconds (test comment at line 161-166)
+- WASM compilation adds overhead on first execution
+- 90s provides buffer while staying under 120s test timeout
+
+### 3. Modernize Test Implementation
+Convert to `#[freenet_test]` macro:
+- Automatic node setup and cleanup
+- Built-in event aggregation on failure
+- Proper peer identification in logs
+- Better failure diagnostics
+
+Example:
+```rust
+#[freenet_test(
+    nodes = ["gateway", "node1", "node2"],
+    auto_connect_peers = true,
+    aggregate_events = "on_failure",
+    timeout_secs = 180,
+    startup_wait_secs = 15
+)]
+async fn test_small_network_get_failure(ctx: &mut TestContext) -> TestResult {
+    let gateway = ctx.gateway()?;
+    let node1 = &ctx.peers()[0];
+    let node2 = &ctx.peers()[1];
+
+    // Test logic with automatic event tracking
+    Ok(())
+}
+```
+
+### 4. Add Enhanced Debugging
+If test still fails, enable event aggregation:
+```rust
+let aggregator = TestAggregatorBuilder::new()
+    .add_node("gateway", gw_temp_dir.path().join("_EVENT_LOG_LOCAL"))
+    .add_node("node1", node1_temp_dir.path().join("_EVENT_LOG_LOCAL"))
+    .build()
+    .await?;
+
+// Analyze PUT operation flow
+let flow = aggregator.get_transaction_flow(&tx_id).await?;
+```
+
+## Testing Plan
+
+1. **Phase 1**: Re-enable test with minimal changes
+   - Remove `#[ignore]` attribute
+   - Increase timeout to 90s
+   - Run in CI to verify fixes
+
+2. **Phase 2**: If issues persist
+   - Add event aggregation
+   - Analyze PUT operation flow across nodes
+   - Identify specific failure points
+
+3. **Phase 3**: Modernize implementation
+   - Convert to `#[freenet_test]` macro
+   - Add comprehensive logging
+   - Document expected timing characteristics
+
+## Timeline Analysis
+
+Recent commits addressing these issues:
+- **Nov 1, 2025**: a283e23 - Fixed gateway crash (timeout notification panics)
+- **Oct 30, 2025**: 615f02d - Fixed PUT response routing
+- **Oct 28, 2025**: 5734a33 - Fixed PUT local caching
+- **Oct 25, 2025**: a34470b - Added transaction atomicity
+
+**All critical fixes are recent (within last week)**, suggesting the test should be re-enabled to verify the fixes work as intended.
+
+## Conclusion
+
+**The test should be re-enabled.** The primary issues (gateway crashes and PUT operation reliability) have been addressed by recent commits. The test needs:
+1. Timeout increase (30s → 90s)
+2. Modern test implementation (`#[freenet_test]` macro)
+3. Event aggregation for future debugging
+
+The current #[ignore] status prevents validation that these fixes actually resolve the production issues the test was designed to catch.
+
+---
+
+**Investigation Date**: November 4, 2025
+**Investigated By**: Claude (Session ID: 011CUoLcvxFtEgkZkhCQQ4Zn)
+**Related Issues**: #2043 (timeout panics), #2036 (PUT routing), #2011 (PUT caching)