Speed improvements

[jpromeror]

This PR includes a series of commits intended to reduce the computational time when running the algorithm in doublet mode. Mostly intended for high definition assays (big number of spots), but usable with all other methods.

Main changes:

1. Change parallelization approach

• Use foreach + DoParallel for the multicore implementation. This prevents launching multiple R sessions and we also included a progress bar for a cleaner UI.

2. Speed up gather_results

• Bottle neck in the algorithm. Fully vectorized the function to remove unnecessary loops. Added progress bar as well

3. General speed up

• Modified existing functions to improve overall performance.

4. Add MIN_OBS as parameter to create.RCTD

• Adds MIN_OBS as a variable to allow running in specific scenarios. Allows more control and customized running, but user should be aware of the drawbacks (i.e. sampling noise). Kept the original default value.

To do list:

• Adapt new parallelization approach to other modes (full & multi)
• Improve screen messages for easier progress tracking

[csangara]

I think there is a typo of 'second_type' here

[jpromeror]

Thanks for catching that!

[dpaysan]

I have tried running the branch but in my case the function choose_sigma_c.R now runs substantially slower: While with the version from dmcable on my Visium HD data the 8 epochs complete within one hour using 28 cores, not even one epoch is processed within that time using the updated version proposed in this pull request. Could that be due to the fact that no cluster is created with makeClusters ? I also see very little CPU usage in general with most threads being in the S state and not running.

[jpromeror]

Hi @dpaysan! It is a bit difficult to identify the issue without any code. Here are some suggestions to see if we can get it to work as intended:

1. We only updated the algorithm when running in "doublet mode". Can you confirm you are using this mode?

2. We changed the parallel approach and we don't use makeClusters anymore (instead registerDoParallel(cores=max_cores) ), if your multicore session in enabled you should see a message like this: "Multicore enabled using ", max_cores," cores". Can you confirm this is the case?