threaded algorithms?

[bjarthur]

i've just written a threaded version of a DiskArray-aware unique method:

uniqueT(v::AbstractDiskArray) = uniqueT(identity, v)
function uniqueT(f, v::AbstractDiskArray)
    u = Vector{Vector{eltype(v)}}(undef, length(eachchunk(v)))
    Threads.@threads :greedy for (i,c) in enumerate(eachchunk(v))
        u[i] = unique(f, v[c...])
    end
    reduce(u) do acc, t
        unique!(f, append!(acc, t))
    end
end

for a Zarr of size (666, 766, 3233) with 6732 chunks it is 4-5x faster using ~64 threads. probably limited by the i/o of the mounted fileshare.

am i correct in thinking that to include such a method in DiskArrays.jl we'd need to know whether the storage format that backs the DiskArray is thread safe when read? maybe with a type parameter?

has including threaded algorithms been discussed before?

happy to submit a PR if interested.

[rafaqz]

Sounds good.

I think we will need a threading trait, that would be no threading by default, then backends can opt in.

We also probably need a way to turn it off globally like how allowscalar works with a Ref{Bool} const.