understanding timeouts

[mcol]

From my testing, it appears that a .timeoutset when calling mirai() starts from the moment mirai() returns, independently of when the actual processing starts. For example, in the following case all processes time out, even though the last processes would have lasted less than the given timeout:

mirai::daemons(4)
res <- list()
for (i in 1:10) {
  res[[i]] <- mirai::mirai(Sys.sleep(secs), .timeout = 3000,
                           .args = list(secs = ifelse(i <= 5, 10, 1)))
}

Is there a way to set a timeout so that a process is stopped only when it has been actively working for more than a certain amount of time (independently of how long it may have waited in the queue)?

[shikokuchuo]

Yes, the timeout is set from the point of view of the caller by design. If you want to limit how long your code takes then that's comparatively easier to add to your inner code.

[mcol]

How would I implement that even just in the case of the example I posted above? Effectively, my setup is that I'm calling a function that may be rather quick but in exceptional cases may get stuck (hence the timeout from the time the actual computation starts) and I have control of the mirai block but I can't modify the code of the function called inside of it.

[shikokuchuo]

I would in turn ask how you would do this normally without it being in a daemon process? If it's something specific to mirai then I'd be able to look into it, but if it's general capability that's missing from base R then possibly it's for another package to address.

[mcol]

Well, I don't know how it's possible to kill a running function "normally". With callr I could get something that sorts of works (just a proof of concept!):

bg.rs <- lapply(1:10, function(dummy) callr::r_session$new())
res <- list()
for (i in 1:10) {
  res[[i]] <- bg.rs[[i]]$call(function(time) {
    Sys.sleep(time)
    return("OK")
  }, args = list(ifelse(i <= 5, 10, 1)))
}

timeout_secs <- 2
results <- list()
for (i in 1:10) {
  if (bg.rs[[i]]$poll_process(timeout_secs * 1000) == "timeout") {
    bg.rs[[i]]$close()
    bg.rs[[i]] <- callr::r_session$new()
    results[[i]] <- NA
  } else {
    results[[i]] <- bg.rs[[i]]$read()$result
  }
}
unlist(results)

# [1] NA   NA   NA   "OK" "OK" "OK" "OK" "OK" "OK" "OK"

I admit that I don't fully understand why jobs 4 and 5 (which should have timed out) return successfully, but at least the final short jobs are not timed out.

I suppose what I would need in mirai is a way to tell that the timeout should start whey the dispatcher actually dispatches a job to a daemon, not when the mirai is started. Thanks for your patience!

[mcol]

I 've worked around the issue by writing my own custom dispatcher so that jobs are launched only when there are idle daemons:

idle_daemons <- function() {
  info <- mirai::info()
  info[["connections"]] - info[["executing"]]
}
mirai_dispatch <- function(expr, env, .args, .timeout) {
  idle_daemons() && return(mirai::mirai(expr, env,
                                        .args = .args,
                                        .timeout = .timeout))
  ## simple polling loop
  while (idle_daemons() == 0L)
    Sys.sleep(0.1)
  mirai::mirai(expr, env, .args = .args, .timeout = .timeout)
}

res <- list()
for (i in 1:10) {
  res[[i]] <- mirai_dispatch(quote(ff(secs)), env, .timeout = 3000,
                           .args = list(secs = ifelse(i <= 5, 10, 1)))
}
unlist(mirai::collect_mirai(res))

# [1] "5"  "5"  "5"  "5"  "5"  "OK" "OK" "OK" "OK" "OK"

I wonder if there's a better way of counting how many daemons are idle... Anyway, if you think that having something like this as a vignette, I could open a PR with it.

[mcol]

The disadvantage of this approach is that I can only collect the results at the end, which is a bummer as in my application I used .progress in mirai_collect() to report the progress.