clustification v1

[sreichl]

Collect below all issues related to the proposed ML-based clustering approach: clustification

• Plan below's tasks
• clustification: add visualization/diagnostics of performance/convergence over time #28
• clustification: implement XGBoost #47
• clustification: train/predict on approximated manifold instead of original space #48
• clustification: Benchmark clf-based clustering approach #12

[sreichl]

https://www.nature.com/articles/s41588-025-02148-8
Bascially the same idea as clustification made it into Nature Genetics as methods paper. Overcluster, then merge if a random forest cannot separate them. Don’t use misclassification but label shuffling and p-values, and have a few extra tricks.
Validation for example in simulated data with gaussian blobs, and then with real data where there are 150 cancer cell lines and CHOIR is good in separating them.
Cool experiments: Show that why they misclassify some cell lines, e.g., based on proliferation scores. Also downsample clusters and show that choir still finds these separately even though they have only 50 cells, while other methods miss them (so it can do multiple resolutions).