test failed in CI: `app::sagas::instance_update::test::test_migration_target_completed_can_unwind`

[plaidfinch]

This test failed on a CI run on #9556:

https://github.com/oxidecomputer/omicron/pull/9556/checks?check_run_id=58893467030

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01KD98PC9VHGP50Z14Y73FHVT5/rHftY69SxSfewj1dbX7xKm8YhSJXop3X4wxWe2jUiArGDbHx/01KD98Q5603VY79GXVF5YG0CRA#S7811

Excerpt from the log showing the failure:

stdout ───
  running 1 test
  test app::sagas::instance_update::test::test_migration_target_completed_can_unwind ... FAILED
  failures:
  failures:
      app::sagas::instance_update::test::test_migration_target_completed_can_unwind
  test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 210 filtered out; finished in 39.93s
  
stderr ───
  log file: /var/tmp/omicron_tmp/omicron_nexus-def80fee552906de-test_migration_target_completed_can_unwind.23627.0.log
  note: configured to log to "/var/tmp/omicron_tmp/omicron_nexus-def80fee552906de-test_migration_target_completed_can_unwind.23627.0.log"
  DB URL: postgresql://root@[::1]:55652/omicron?sslmode=disable
  DB address: [::1]:55652
  log file: /var/tmp/omicron_tmp/omicron_nexus-def80fee552906de-test_migration_target_completed_can_unwind.23627.2.log
  note: configured to log to "/var/tmp/omicron_tmp/omicron_nexus-def80fee552906de-test_migration_target_completed_can_unwind.23627.2.log"
  log file: /var/tmp/omicron_tmp/omicron_nexus-def80fee552906de-test_migration_target_completed_can_unwind.23627.3.log
  note: configured to log to "/var/tmp/omicron_tmp/omicron_nexus-def80fee552906de-test_migration_target_completed_can_unwind.23627.3.log"
  thread 'app::sagas::instance_update::test::test_migration_target_completed_can_unwind' (2) panicked at sled-agent/src/sim/collection.rs:274:65:
  called `Result::unwrap()` on an `Err` value: InternalError { internal_message: "Communication Error: error sending request for url (http://127.0.0.1:35432/vmms/79b945d1-2a50-405d-a93e-f0b88710a7ca)" }
  stack backtrace:
...

[hawkw]

Ugh, this one's on me.

I wonder what's up with the error:

"Communication Error: error sending request for url (http://127.0.0.1:35432/vmms/79b945d1-2a50-405d-a93e-f0b88710a7ca)

"error sending request" is not very helpful; I have to assume there's a lower-level error that the Debug format has not chosen to print out, which is unfortunate.

Will look at the test logs.

[hawkw]

I think this line in the test's logs might be the underlying error:

2025-12-24T23:59:20.017Z	DEBG	test_migration_target_completed_can_unwind (NexusClient): client response
    result = Err(reqwest::Error { kind: Request, url: "http://127.0.0.1:35432/vmms/79b945d1-2a50-405d-a93e-f0b88710a7ca", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(Io, Os { code: 131, kind: ConnectionReset, message: "Connection reset by peer" })) })
    sled_id = b6d65341-167c-41df-9b5c-41cded99c229
2025-12-24T23:59:20.205Z	ERRO	test_migration_target_completed_can_unwind (dropshot): handler panicked; propogating panic
    local_addr = [::1]:49382
    method = POST
    remote_addr = [::1]:55957
    req_id = d938ffad-0f80-4834-ba56-14c7cad2da3d
    sled_id = b6d65341-167c-41df-9b5c-41cded99c229
    uri = /vmms/79b945d1-2a50-405d-a93e-f0b88710a7ca/poke

which sounds like Nexus abruptly hung up on a request from the (simulated) sled-agent to update the VMM's state for... some reason, and then the sim sled poke API handler that was making the Nexus upcall panicked.

[hawkw]

I think this is the beginning of the request to nexus that it gave up on? But I can't seem to find anything in the logs explaining why Nexus would have just reset the conn...

[davepacheco]

Seems suspicious:
https://buildomat.eng.oxide.computer/wg/0/artefact/01KD98PC9VHGP50Z14Y73FHVT5/rHftY69SxSfewj1dbX7xKm8YhSJXop3X4wxWe2jUiArGDbHx/01KD98Q5603VY79GXVF5YG0CRA/01KD9FTFGZBSANASSMV6M3FH5W/omicron_nexus-def80fee552906de-test_migration_target_completed_can_unwind.23627.0.log?format=x-bunyan#L11689

11689	2025-12-24T23:59:20.205Z	ERRO	test_migration_target_completed_can_unwind (dropshot): handler panicked; propogating panic
    local_addr = [::1]:49382
    method = POST
    remote_addr = [::1]:55957
    req_id = d938ffad-0f80-4834-ba56-14c7cad2da3d
    sled_id = b6d65341-167c-41df-9b5c-41cded99c229
    uri = /vmms/79b945d1-2a50-405d-a93e-f0b88710a7ca/poke

It does seem like a bug that the sim sled agent error doesn't have more information (probably a missing InlineErrorChain?). I'm worried we don't have context about the Nexus handler panic because I'm not sure where it could be unless Dropshot logs it, and it looks like it doesn't.