Deadlock(s)

[davidlattimore]

This crate shows some great promise! I love the idea of providing a rayon compatibility layer! Makes it so much easier to experiment.

I tried it in the Wild linker using the rayon-compat crate. It wasn't a drop-in replacement for rayon, but was close enough. I was able to link a trivial program and it showed a good performance improvement. I'd love to see how it performs when linking larger programs, but unfortunately I seem to hit a deadlock where a scope appears to not shut down despite not having any active work.

I've since tried using forte directly from a small test program and the following code failed to terminate:

use forte::ThreadPool;

static THREAD_POOL: ThreadPool = ThreadPool::new();

fn main() {
    THREAD_POOL.resize_to(2);

    THREAD_POOL.resize_to(0);
}

Running cargo test at head also seems to result in some deadlocks. e.g. thread_pool::tests::join_basic seems to deadlock. This looks to be the same deadlock as in the example code above.

It seems like the main thread is waiting for the heartbeat thread to shut down while holding a lock and the heartbeat thread is waiting for the lock. I tried fixing this by dropping the mutex guard before trying to join the heartbeat thread, but then hit a different deadlock waiting for the worker threads to shut down. Possibly related to this commented-out line.

I'm not sure if the deadlock I see when using this crate in the wild linker is the same or different to either of the above.

[NthTensor]

As I mentioned in the review of #38, I have some local work I need to publish that should hopefully resolve many of deadlocks. But even on that branch, there is at least one very persistent deadlock to do with worker-sleeping which I have not yet resolved (and which I was hoping to fix before publishing).

I will prioritize getting up the remainder of those tests and fixes now.

I love the idea of providing a rayon compatibility layer! Makes it so much easier to experiment.

This is useful feedback. Unfortunately bevy and rayon use incompatible type signature for scopes, so was considering ditching the rayon_compat layer all together to simplify my bevy integration work. But if people are finding it useful, I will have a go at including both versions of scope, and having a feature to switch between them, letting me keep the existing rayon_compat stuff.

Eventually I sense that I may need to fork rayon directly and make some minimal adjustments, instead of overriding rayon_core.

I was able to link a trivial program and it showed a good performance improvement.

This is very encouraging!

I'd love to see how it performs when linking larger programs, but unfortunately I seem to hit a deadlock where a scope appears to not shut down despite not having any active work.

Could you direct me to a test-case to run with wild and forte? I'm eager to profile some real-world examples, and if you can give me a locking example I'll happily work to resolve it.

[davidlattimore]

The deadlocks were fixed by the two PR that you merged. With those, I was able to get Wild to work with Forte. Unfortunately the good performance didn't scale beyond non-trivial inputs. With larger inputs, the Forte-based version of wild performed worse than the rayon version.

If you'd like to look at that, I've uploaded patches to wild and to forte that make them work together. The forte patch adds support for spawn_broadcast, which wild uses. It also makes Scope implement Copy and Clone. I don't recall why I did that. It also renames forte-rayon-compat to rayon-core, since I had trouble patching it without doing that for some reason.

The patches are here. The forte patch is relative to 570defa. The wild patch is relative to 3927cb936cee7157e7c42bfb96c5f04cf902d21b. You should be able to apply the patches onto their respective parent commits with patch -Np1 -i filename.patch.

To benchmark wild, there are instructions in BENCHMARKING.md in the wild repo. Benchmarking wild linking itself is probably one of the easier benchmarks to run. The results for that look like this:

B=wild F="--threads=32 --strip-debug" R=/home/david/save/$B/run-with bench poop -d 5000  "$R $HOME/wild-builds/2025-12-27 $F" "$R target/release/wild $F"
  Temperature: 44.9 C
Benchmark 1 (159 runs): /home/david/save/wild/run-with /home/d/wild-builds/2025-12-27 --threads=32 --strip-debug
  measurement          mean ± σ            min … max           outliers         delta
  wall_time          30.8ms ± 8.62ms    28.7ms …  138ms          5 ( 3%)        0%
  peak_rss           7.13MB ± 7.08KB    7.06MB … 7.15MB         15 ( 9%)        0%
  cpu_cycles          695M  ± 18.0M      658M  …  850M           3 ( 2%)        0%
  instructions        695M  ± 13.1M      672M  …  823M           3 ( 2%)        0%
  cache_references   21.3M  ±  428K     19.5M  … 23.8M           2 ( 1%)        0%
  cache_misses       4.48M  ± 96.9K     4.33M  … 5.36M           1 ( 1%)        0%
  branch_misses      1.90M  ± 25.2K     1.86M  … 2.15M           2 ( 1%)        0%
Benchmark 2 (68 runs): /home/david/save/wild/run-with target/release/wild --threads=32 --strip-debug
  measurement          mean ± σ            min … max           outliers         delta
  wall_time          73.1ms ± 3.43ms    63.9ms … 80.5ms          1 ( 1%)        💩+137.3% ±  6.9%
  peak_rss           7.13MB ± 8.66KB    7.09MB … 7.15MB         10 (15%)          -  0.0% ±  0.0%
  cpu_cycles          311M  ± 4.16M      297M  …  320M           2 ( 3%)        ⚡- 55.3% ±  0.6%
  instructions        533M  ± 98.2K      532M  …  533M           2 ( 3%)        ⚡- 23.4% ±  0.4%
  cache_references   15.9M  ±  120K     15.6M  … 16.2M           0 ( 0%)        ⚡- 25.1% ±  0.5%
  cache_misses       2.68M  ± 33.9K     2.60M  … 2.75M           0 ( 0%)        ⚡- 40.1% ±  0.5%
  branch_misses      1.48M  ± 10.2K     1.45M  … 1.50M           5 ( 7%)        ⚡- 21.9% ±  0.3%

Here's the results for linking the zed editor:

Benchmark 1 (20 runs): /home/david/save/zed/run-with /home/d/wild-builds/2025-12-27 --threads=32 --strip-debug
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           500ms ±  158ms     456ms … 1.17s           2 (10%)        0%
  peak_rss           7.13MB ± 3.73KB    7.13MB … 7.15MB          2 (10%)        0%
  cpu_cycles         22.0G  ±  402M     20.6G  … 22.4G           1 ( 5%)        0%
  instructions       20.0G  ± 19.1M     20.0G  … 20.1G           0 ( 0%)        0%
  cache_references    511M  ± 3.85M      502M  …  517M           0 ( 0%)        0%
  cache_misses        127M  ± 1.03M      125M  …  129M           0 ( 0%)        0%
  branch_misses      40.0M  ±  164K     39.8M  … 40.4M           0 ( 0%)        0%
Benchmark 2 (9 runs): /home/david/save/zed/run-with target/release/wild --threads=32 --strip-debug
  measurement          mean ± σ            min … max           outliers         delta
  wall_time          1.13s  ± 44.6ms    1.05s  … 1.20s           0 ( 0%)        💩+126.3% ± 22.2%
  peak_rss           7.13MB ±    0      7.13MB … 7.13MB          0 ( 0%)          -  0.0% ±  0.0%
  cpu_cycles         17.8G  ±  178M     17.6G  … 18.1G           0 ( 0%)        ⚡- 19.2% ±  1.3%
  instructions       19.3G  ± 10.8M     19.3G  … 19.3G           1 (11%)        ⚡-  3.8% ±  0.1%
  cache_references    492M  ±  966K      490M  …  494M           1 (11%)        ⚡-  3.8% ±  0.5%
  cache_misses        114M  ±  617K      113M  …  115M           0 ( 0%)        ⚡- 10.0% ±  0.6%
  branch_misses      39.5M  ±  126K     39.3M  … 39.6M           0 ( 0%)        ⚡-  1.3% ±  0.3%

Wild has support for writing perfetto traces that show what executed when. Details are in the benchmarking document. From a look at a perfetto trace of linking zed, it looks like possibly the stages that use scopes aren't parallelising well. e.g. they only have 3 or 4 active threads, rather than all 32.

With regular rayon:

With forte:

The screenshots don't show all the threads, so for the forte screenshot some of the stages look like they've only got one active thread, when actually there are a couple more if you scroll down. It's probably best to look at the actual traces in a browser.

[NthTensor]

Yeah, those are some pretty significant regressions!

From a look at a perfetto trace of linking zed, it looks like possibly the stages that use scopes aren't parallelising well. e.g. they only have 3 or 4 active threads, rather than all 32.

This isn't super surprising to me, for two reasons:

1. I’ve not actually invested much time in benchmarking and optimizing scopes yet.
2. While lazy scheduling works pretty well for deeply recursive fork/join stuff, where a single shared task can represent a large part of the total workload, it performs much much worse on "flat" workloads (which are more often represented using scopes).

I've heard similar things from a few other early testers, including physics sims and bvh-construction. It's definitely a general problem with the lib as a whole currently.

For more flat workloads, where you only really care about latency and can live with higher cpu use, what you probably want is an "eager scope". Basically, while the scope is active, work is shared eagerly instead of on a heartbeat, and the worker threads spin instead of sleeping. A similar design is discussed in this talk.

I'll have a go at implementing a version of this and will see if it improves the situation with wild at all. Under-utilizing the available parallelism is one of the tradeoffs forte chooses, but I'm confident we can significantly improve thread use here.