[Bug] smallsort panic in tokio task

[decisionBasis]

Bug report

Getting these logs on latest version of node, node is crashlooping. I can't pinpoint particular subgraph because it never has same logs preceeding the panic.

Relevant log output

thread 'tokio-runtime-worker' panicked at library/core/src/slice/sort/shared/smallsort.rs:860:5:
user-provided comparison function does not correctly implement a total order
stack backtrace:
   0:     0x564e9f20f032 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hc04c8f544ab24d66
   1:     0x564e9f239b03 - core::fmt::write::hfe57b7174b7d8eab
   2:     0x564e9f20ae13 - std::io::Write::write_fmt::h154385efa8565236
   3:     0x564e9f20ee82 - std::sys::backtrace::BacktraceLock::print::h0c8f24e22f5873a8
   4:     0x564e9f20fee2 - std::panicking::default_hook::{{closure}}::hd07d57e6a602c8e4
   5:     0x564e9f20fce5 - std::panicking::default_hook::h63d12f7d95bd91ed
   6:     0x564e9f210882 - std::panicking::rust_panic_with_hook::h33b18b24045abff4
   7:     0x564e9f2105f6 - std::panicking::begin_panic_handler::{{closure}}::hf8313cc2fd0126bc
   8:     0x564e9f20f539 - std::sys::backtrace::__rust_end_short_backtrace::h57fe07c8aea5c98a
   9:     0x564e9f2102bd - __rustc[95feac21a9532783]::rust_begin_unwind
  10:     0x564e9daa12e0 - core::panicking::panic_fmt::hd54fb667be51beea
  11:     0x564e9daa174b - core::slice::sort::shared::smallsort::panic_on_ord_violation::h4d7d0ae5a308eb96
  12:     0x564e9e5f2789 - core::slice::sort::shared::smallsort::bidirectional_merge::h1044bbad7b7beb44
  13:     0x564e9e5f3dc7 - core::slice::sort::shared::smallsort::small_sort_general_with_scratch::h43437cc608b24fb8
  14:     0x564e9e5d67ca - core::slice::sort::stable::quicksort::quicksort::h6680c382a9895184
  15:     0x564e9e5c292a - core::slice::sort::stable::drift::sort::hd5b4358f62f7bc88
  16:     0x564e9e6817a3 - core::slice::sort::stable::driftsort_main::hb989dc57d61dcb54
  17:     0x564e9e5c31f3 - graph::blockchain::block_stream::BlockWithTriggers<C>::new_with_triggers::h382c15402379671a
  18:     0x564e9e5c3377 - graph::blockchain::block_stream::BlockWithTriggers<C>::new::h9f977ef62ae27e9e
  19:     0x564e9e627127 - <alloc::vec::into_iter::IntoIter<T,A> as core::iter::traits::iterator::Iterator>::try_fold::h16cd43443c2189d5
  20:     0x564e9e6f22e0 - <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::try_fold::h8517dd5b89e876ea
  21:     0x564e9e68388f - alloc::vec::Vec<T,A>::extend_desugared::h022b117deec25a99
  22:     0x564e9e675dbb - <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter::h082fdbcf4f5694b8
  23:     0x564e9e6f7aae - core::iter::adapters::try_process::h524dee65de37f739
  24:     0x564e9e6e073c - graph_chain_ethereum::ethereum_adapter::blocks_with_triggers::{{closure}}::hbcddd29392b358df
  25:     0x564e9e6e8721 - <graph_chain_ethereum::chain::TriggersAdapter as graph::blockchain::block_stream::TriggersAdapter<graph_chain_ethereum::chain::Chain>>::scan_triggers::{{closure}}::hd19e36313c94865c
  26:     0x564e9e5fcf4e - <graph::blockchain::polling_block_stream::PollingBlockStream<C> as futures_core::stream::Stream>::poll_next::{{closure}}::hd37678890ff68cbd
  27:     0x564e9e5fb564 - <graph::blockchain::polling_block_stream::PollingBlockStream<C> as futures_core::stream::Stream>::poll_next::h4b4aca241f62d2f1
  28:     0x564e9de02fea - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::future::future::Future>::poll::h8b4edb2df8bc6275
  29:     0x564e9db12930 - std::panic::catch_unwind::h366e6067c275c748
  30:     0x564e9de602ba - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::hb6404ec87feab6b4
  31:     0x564e9db9415b - tokio::runtime::task::core::Core<T,S>::poll::hdbda6337e5b585e6
  32:     0x564e9db13f24 - std::panic::catch_unwind::h686a3f3ae1ca0632
  33:     0x564e9dec2f15 - tokio::runtime::task::harness::Harness<T,S>::poll::h624f0d6501408bad
  34:     0x564e9f09c9e6 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::hd4150d71a835da8f
  35:     0x564e9f09c260 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h4ca0d2f167d185e8
  36:     0x564e9f0942ad - tokio::runtime::context::scoped::Scoped<T>::set::h6bb32a3dcf532dbc
  37:     0x564e9f096ae1 - tokio::runtime::context::runtime::enter_runtime::h73863383e535eab2
  38:     0x564e9f09bd6b - tokio::runtime::scheduler::multi_thread::worker::run::he18d531dba0987c7
  39:     0x564e9f099835 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h8649ce37c791bb9d
  40:     0x564e9f08c0a1 - tokio::runtime::task::core::Core<T,S>::poll::hdb1e4176d9a11895
  41:     0x564e9f086e51 - std::panic::catch_unwind::h2168414212ee0f61
  42:     0x564e9f086b5e - tokio::runtime::task::harness::poll_future::h4dd9e57c4da4e573
  43:     0x564e9f085f48 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::hf0009ae66083b8b7
  44:     0x564e9f085cc4 - tokio::runtime::task::harness::Harness<T,S>::poll::h1ea3c48f1483ded5
  45:     0x564e9f089b2c - tokio::runtime::task::UnownedTask<S>::run::h581e0dde7d69b0a4
  46:     0x564e9f08afeb - tokio::runtime::blocking::pool::Inner::run::h5016cef15e1d031a
  47:     0x564e9f0874ae - std::sys::backtrace::__rust_begin_short_backtrace::h17d92e3a31e8f073
  48:     0x564e9f08f6db - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc9533e2ba1f167ac
  49:     0x564e9f21291b - std::sys::pal::unix::thread::Thread::new::thread_start::h2717d4f9aca577e6
  50:     0x7f3ef43051c4 - <unknown>
  51:     0x7f3ef4384ac0 - __clone
  52:                0x0 - <unknown>
Panic in tokio task, aborting!

IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out

• Tick this box if this bug is caused by a regression found in the latest release.
• Tick this box if this bug is specific to the hosted service.
• I have searched the issue tracker to make sure this issue is not a duplicate.

OS information

Linux