Shutdown daemons when they are not used

[raphaelbetschart]

Hi, I'm playing around using mirai for my Shiny app. For illustration, I adapted the following example from, and modified it for remote submission through SLURM https://mirai.r-lib.org/articles/v02-promises.html#shiny-extendedtask-introduction.

library(shiny)
library(bslib)
library(mirai)

ui <- page_fluid(
  p("The time is ", textOutput("current_time", inline = TRUE)),
  hr(),
  numericInput("n", "Sample size (n)", 100),
  numericInput("delay", "Seconds to take for plot", 5),
  input_task_button("btn", "Plot uniform distribution"),
  plotOutput("plot")
)

server <- function(input, output, session) {
  output$current_time <- renderText({
    invalidateLater(1000)
    format(Sys.time(), "%H:%M:%S %p")
  })

  task <- ExtendedTask$new(
    function(...) mirai({Sys.sleep(y); runif(x)}, ...)
  ) |> bind_task_button("btn")

  observeEvent(input$btn, task$invoke(x = input$n, y = input$delay))

  output$plot <- renderPlot(hist(task$result()))

}

config <- cluster_config(
  command = "sbatch",
  options = "#SBATCH --job-name=mirai
             #SBATCH --cpus-per-task=1
             #SBATCH --mem-per-cpu=2G
             #SBATCH --output=job.out
             ml R/4.5.1",
  rscript = file.path(R.home("bin"), "Rscript")
)

# run app using 4 SLURM daemons
daemons(n = 4, url = host_url(), remote = config)

# automatically shutdown daemons when app exits
onStop(function() daemons(0))

shinyApp(ui = ui, server = server)

This works, but it always occupies 4 jobs, no matter if anything needs to get submitted or not.
How would I change the code, so that the daemons only spin up when necessary (e.g. in the task)?
Seems like an easy fix, but I really can't figure it out.

When using future.batchtools, jobs are only submitted when needed:

library(future.batchtools)
plan(
  batchtools_slurm(
    resources = list(
      asis = c(
        "--mem-per-cpu=2G",
        "--ntasks=4",
        "--cpus-per-task=1"
      ),
      modules = c("R/4.5.1")
    )
  )
)

[shikokuchuo]

Daemons are usually persistent processes, but they're quite flexible in supporting the kind of on-demand scaling you need.

I think you can achieve what you want with a pattern like this:

# Set up daemons
daemons(url = host_url())

Then within your server function:

task <- ExtendedTask$new(
  function(...) {
    # Start up a single Slurm worker that does one task and then exits
    launch_remote(1, remote = config, maxtasks = 1)
    mirai({Sys.sleep(y); runif(x)}, ...)
  }
) |> bind_task_button("btn")

[raphaelbetschart]

Perfect, works like a charm.