100 sequential DNS queries over TCP consistently result in 1-2 failures only on macOS

[MahdiBM]

I haven't investigated this too thoroughly myself. I'll be doing that later, and I'll appreciate if anyone here has any pointers to anything that could help my debugging. Or I could help you debug through the library, that's also one way.

Expected behavior

100 sequential-only DNS NS queries in swift-dns, on macOS only, over TCP only, using PostgresNIO's connection-pool module, should result in 0 failures, but don't.

Actual behavior

This specific test in swift-dns has been consistently failing over the past few months.
I didn't report this because I wanted to make sure I have connection unit tests first.

The failures are:

• Almost always the query for dropbox.com
• And then occasionally another domain as well.

dropbox.com usually fails with either channelInactive or queryTimeout (both are thrown by swift-dns).

Doing a single dropbox NS query doesn't ever fail by itself.
There is an integration test to do concurrent requests, that doesn't fail either.
There are a bunch of conn unit test to do concurrent/sequential queries and all of those are passing as well.

Linux is doing fine which makes me think the issue is due to the difference in ELGs used on different platforms in swift-dns (though this is common in swift-nio based pacakges), or the bootstraps.

Steps to reproduce

Clone https://github.com/MahdiBM/swift-dns.
Revision doesn't matter. Just use the main branch.
Run IntegrationTests.query100DomainsSequentially.
Like: swift test --filter IntegrationTests.query100DomainsSequentially.

If possible, minimal yet complete reproducer code (or URL to code)

Mentioned links above. I also have these packet captures that the CI archives:

swift-dns_dns_capture_2025-10-03-09-29-12_macos-26_6.2_debug_latest-stable_0a7eed3_failure.zip
You can find the dropbox query with this Wireshark filter: dns.id == 0xa95e.
The other failing query, that doesn't always happen, but does most of the times: dns.id == 0x1ea9 (edgekey.net).
This is a capture of the whole integration tests run, so it does contain more than just that one test.
Looking at the packet capture, the queries look to have been timely fulfilled and received a response. So this packet-capture might not be too helpful.

SwiftNIO version/commit hash

2.86.2

System & version information

macOS 15/26. The library requires Swift 6.2.

Please provide at the very least your operating system and Swift version
information.

Ideally, run

scripts/nio-diagnose -o nio-diagnose.md PID_OF_YOUR_NIO_PROGRAM

and attach (or paste) the resulting file nio-diagnose.md into this bug report
or send it to the SwiftNIO maintainers privately.

[MahdiBM]

I took a look and got the nio-diagnose report, but it contains a bit lot too much info.
You could try to reproduce it quickly too I guess, like:

bash -c '                                                                                                                                                                                                                                                         
  cd /path/to/swift-dns
  swift test --filter IntegrationTests.query100DomainsSequentially &
  PID=$!
  /path/to/nio-diagnose -o nio-diagnose.md $PID
'

[Lukasa]

Thanks for filing this Mahdi! I’ll take a look next week, the whole team is at serversideswift.info this week.

[camunro]

In the meantime...
Can you shuffle order of top100, same results?
Change DNS server from 8.8.4.4 to 1.1.1.1 ?
You might be triggering rate limiting (or DDoS) all coming from same source.
Try UDP ?
https://developers.google.com/speed/public-dns/docs/troubleshooting
https://dns.google/resolve?name=dropbox.com&type=NS&do=true

[MahdiBM]

I havent had the time to investigate this myself, but I will at the first opportunity.

@camunro those are valid ideas. I doubt this is a ratelimit issue though. Like I mentioned, the packet capture does contain the Dropbox response. Just that the dns client doesn't end up receiving it for some reason. So more like the packet is lost at some point? Also this only happens on macOS, not Linux.

[Lukasa]

@MahdiBM can you clarify what a failure output looks like on the terminal? I don't think I've been able to make this fail.

[MahdiBM]

Out right now, but the test doesnt actually fail because I made it tolerate some failures. The dns request does actually fail though.

[Lukasa]

Ah shoot, ok. When you get a minute (not urgent) it'd be nice to have an idea of what commit I can revert to get it to fail again.

[MahdiBM]

As you requested, here's a branch that simply reports all failures: mmbm-debug-seq-dropbox-failure. However right now I haven't been able to make it fail locally (don't recall if I could before, or not).
EDIT: Ok I can reproduce locally now, again. Basically skipping some dns cache I have locally on the router in the tests. Thought I already had that covered.

I'm taking another look at this. I added some more debugging-helpers in swift-dns to integration tests as well so that's helping too.

On CI it fails on macOS 15/26 and both debug and release build configs. CI (re)installs swift via swiftly, and although we noticed an issue with swiftly toolchains a week or two ago, this particular issue doesn't look something that could be related.

This is the CI output, mentioning a problem occurred with the dropbox.com query.
This is a print that the test itself does just to keep track of failures. It only fails if there are more than 4 failures.

◇ Test query100DomainsSequentially(client:) started.
◇ Test case passing 1 argument client → DNSClient(transport: DNSClient.DNSClient.Transport.preferUDPOrUseTCP(DNSClient.PreferUDPOrUseTCPDNSClientTransport)) to query100DomainsSequentially(client:) started.
◇ Test case passing 1 argument client → DNSClient(transport: DNSClient.DNSClient.Transport.tcp(DNSClient.TCPDNSClientTransport)) to query100DomainsSequentially(client:) started.
query100DomainsSequentially(client:) encountered these errors:
[("dropbox.com", channelInactive)]

Here's a new tcpdump capture that should be more helpful, from a CI run a few minutes ago. This is on macOS 26 with debug build config.

swift-dns_dns_capture_2025-10-12-11-00-54_success_macos-26_6.2_debug_latest-stable_bdc9624.zip

Using the debug stuff I added, I can say for sure that this Wireshark filter will show what is related to this test:
tcp and frame.number >= 951 and frame.number <= 1409.

Here you can see that a query from dropbox has indeed been received on the machine, just that swift-dns didn't receive it, presumably because the channel was already closed:

From memory, and like I've mentioned before, I think sometimes the query fails due to swift-dns's query timeout. But it appears most of the time the issue is a channel closure. And again, it's pretty much always dropbox.com that's having an issue.

Some more info:

• Each test first runs using the udp transport, then using the tcp transport. Using swift-testing's test arguments to pass different DNSClients to the test.
• I added some marker packets just so it's easier to assert test boundaries when looking at CI packet captures. These are not visible if you use the frame filter I mentioned above, so this is just a FYI.
• The marker packets always use marker.packet.local domain. e.g. test.start.marker.packet.local or test.end.with.success.marker.packet.local.
• The EDNS part of the packet contains the test's name when read as ASCII.
• Due to how swift-testing works, these marker packets are only sent once even if there are multiple test arguments with e.g. different DNSClients.

[MahdiBM]

This is from the same packet capture above:

Locally though, I see a TCP RST + no dns response, which would make some sense at least. This local run results in the same [("dropbox.com", channelInactive)]:

This is the CI capture command (so related to the first pic, not the second): tcpdump -w dns.pcap 'port 53 and (host 8.8.4.4 or host 240.1.2.3)', which should be fine?