out from workers not show in real time

[Lightup1]

Even when I add flush(stdout) after all printlns executed by workers.
Ther slurm-xxx.out still does not show the output in real time.

[Lightup1]

One can try some thing like

using Distributed, SlurmClusterManager
addprocs(SlurmManager(),exeflags=["--project=.", "-t$(ENV["SLURM_CPUS_PER_TASK"])", "--color=yes", "--startup-file=no"])

@everywhere function sleep_println(s)
    sleep(10)
    println("hello world")
    flush(stdout)
end
pmap(sleep_println,1:parse(Int64,ENV["SLURM_NTASKS"]))

[DilumAluthge]

Just to clarify: Once the Slurm job has finished, does the slurm-nnn.out file then contain the expected output?

[RefatIsmail96]

I have the same issue. The output file does have expected output but only once the Slurm job is finished/terminated. I have tried to use flush(stdout) too but it does not help.

[kleinhenz]

srun has a --unbuffered option which might help with this. Does adding this to the srun command in slurmmanager.jl fix the issue? We could make this an option.

[DilumAluthge]

Alternatively, we could try setting the SLURM_UNBUFFEREDIO environment variable (which might be easier than needing to add a command-line flag)?

@Lightup1 @RefatIsmail96 Could you try the following out and see if it helps?

using Distributed: addprocs
using SlurmClusterManager: SlurmManager

const env = Dict("SLURM_UNBUFFEREDIO" => "1")

addprocs(SlurmManager(); env=env)

Note: You'll need to be using SlurmClusterManager v0.1.3 or later.