Copy the Cell Ranger reference locally

[arteymix]

Our Cells Ranger tasks spend considerable amount of time copying the reference transcriptome over NFS.

This is mitigated in the bulk RNA-Seq pipeline by using shared memory and unloading the segment only if no other jobs are using it.

This is not possible with Cell Ranger as calling STAR is an internal implementation detail.

Thus, a way of mitigating the networking cost would be to copy the reference locally. Care must be taken to coordinate copies since this is a shared resource.

• add a wrapper script that rewrites the --transcriptome argument to refer to a local file
• copy the reference to a local scratch directory
• add a lockfile to coordinate the copy
• use os.execv to delegate the process to Cell Ranger with the file locked in shared mode
• make this optional and make the scratch directory configurable

[arteymix]

Important note: Python started setting O_CLOEXEC by default for opened files in 3.4.

The lockfile can be made explicitly inheritable with is.set_inheritable. That should protect the reference for the lifetime of the Cell Ranger execution.

With that in place, we could consider a mechanism for invalidating the reference or remove unused copies by trying to acquire an exclusive lock.

https://docs.python.org/3/library/os.html#os.set_inheritable

[arteymix]

We should apply the same principle to the RSEM wrapper to copy over the STAR reference.

[arteymix]

Did the same for bulk RNA-Seq.