Improve documentation for Task.Anon (or improve its functionality)

[arminsumo]

Current documentation: https://mill-build.org/mill/fundamentals/tasks.html#_anonymous_tasks

---

You can define anonymous tasks using the Task.Anon {...​} syntax. These are not runnable from the command-line, but can be used to share common code you find yourself repeating in Tasks and Commands.

Anonymous task’s output does not need to be JSON-serializable, their output is not cached, and they can be defined with or without arguments. Unlike Tasks or Commands, anonymous tasks can be defined anywhere and passed around any way you want, until you finally make use of them within a downstream task or command.

---

From this description, it is not clear why this does not work

  def taskA = Task {
    val a = 1+2
    taskB(a)() // instead, doing taskB(1+2)() does work. taskB(taskC())() also does not work.
  }

  def taskB(n: Int) = Task.Anon {
    println(Task.dest) // prints taskA.dest as expected, if there is no compilation error
    println(n)
  }

  def taskC = Task{
    1 + 2
  }

results in error
Task#apply() call cannot use val a defined within the Task{...} block

It seems like this limitation somehow defeats the purpose of Anonymous tasks as helper functions for reusable pieces of code, as it's too limiting to not use any locally defined variables. This leads to a lot of code duplication, because there doesn't seem to be a good way to split a Task into smaller pieces. It would be great if Anon task can be updated to reflect this, and also to document what the best practices are for splitting Tasks up into smaller ones when the logic is not trivial.

[lefou]

In your example, in taskA you create an instance of taskB which depends on a. This value is only defined inside the task, so it's only known at evaluation time. But for evaluation, all tasks need already be defined. One way to fix it is to move the definition of value a out of the Task body.

  def taskA = {
    val a = 1+2
    Task {
      taskB(a)() // instead, doing taskB(1+2)() does work. taskB(taskC())() also does not work.
    }
  }

  def taskB(n: Int) = Task.Anon {
    println(Task.dest) // prints taskA.dest as expected, if there is no compilation error
    println(n)
  }

  def taskC = Task{
    1 + 2
  }

[lefou]

Another way to fix this is to make the parameter of taskB itself a task.

[lefou]

This info hopefully helps to

• enhance your understanding of annonymous tasks
• come up with a better wording that you understand
• open a PR for the documentation

[arminsumo]

This info hopefully helps to

* enhance your understanding of annonymous tasks

* come up with a better wording that you understand

* open a PR for the documentation

Definitely I will be happy to open a pr for that, but I think my example didn't capture the full case I was trying to achieve.

what if we have something like this?

  def sourceTask = Task.Source("a.yml")

  def readWriteFile = Task {
    val file = sourceTask().path
    val fileContent = os.read(file) + "something"
    val name = file.last
    writeFileWithExtraContent(fileContent, name)()
    println("wrote file")
  }
  
  def source2Task = Task.Source("b.yml")

  def writeFileWithExtraContent(fileContent: String, name: String) = Task.Anon {
    val extraContent = os.read(source2Task().path)
    os.write(Task.dest / name, fileContent + extraContent)
    println("wrote file from anon task")
  }

here we can't do the trick you mentioned, as we need to call and process data from sourceTask(), which must be done inside a Task. writeFile must also be a Anon Task since it needs Task.dest. I'm aware we could theoretically pass in Task.dest and source2Task into the writeFileWithExtraContent and make it a normal function, but in a more complicated case this can create a lot of duplicated code. I'm not sure how to reconcile this now, since it seems like both tasks need to process data that require a Task Context, but only one of the can really do it.

[arminsumo]

Another different but related example
let's say I have some trait with a generate method eg

and I want to perform some post processing on it's output like this.. but this is not possible since the Task has a different folder name than it's super and we end up getting errors like Normal Tasks can only write to files within their Task.dest folder, only Task.Commands can write to other arbitrary files

I wonder if this is a case that can be solved with Anon Tasks? Or what would be the best practice? Sorry for all the questions, I'm coming from a gradle world where things are quite different, trying to learn and figure out what it might take to migrate.

  override def generate: T[PathRef] = Task {
    val generatedPath = super.generate()
    val generatedDir = generatedPath.path
    if (os.exists(generatedDir)) {
      os.walk(generatedDir)
        .foreach { javaFile =>
          val content = os.read(javaFile)
          val sanitized = content
            .replaceAll("import javax\\.annotation\\.Nullable;\\s*", "")
          os.write.over(javaFile, sanitized)
        }
    }
    generatedPath
  }

[lefou]

Tasks are all about dependencies. Instead of imperatively calling writeFileWithExtraContent from readWriteFile (which would be better suited by using a normal def, maybe with an implicit ctx: TaskCtx parameter), you want to have a task that gives you the content of the file. And another one, that enhances the content. That way, you can depend on the results of other tasks. What you currently try is to control the inputs of a task.

What really helps in designing your tasks is naming. Instead of naming your tasks like an action (write, read) you want them to represent results (fileContent, compiledCode).

Regarding your other generate example, the overriding generate tasks should not try to modify it's super task. Instead it should write a new file into it's own dest location and return that instead.

It's really important that task are immutable, at least from the outside. If you write to the dest location of another task, you break the whole integrity of Mill. Practically, the caching will break, since you modified super.generate, but Mill doesn't know anything about it. That's why Mill tries to help you with mentioned error message.

If you want to modify the result of super.generate, just copy it into your Task.dest and then modify it. Or read the original, but write the new content into the new location.

[arminsumo]

Added the pr (#5816) for documentation improvements

[arminsumo]

Not sure if this is a bug or intended, but I realized you can get around the "Tasks only write to their own dest" rule by using a JVM process... eg

  def destTask = Task{Task.dest}
  def hello = Task {"hello"}
  def generate = Task { os.write(destTask() / "hello.txt", hello()) } // fails as expected because it's trying to write to destTask.dest
  def generate2 = Task { this.runner().run(Array((destTask() / "hello.txt").toString, hello()), "foo.Foo") } // able to write to destTask.dest with no issue

package foo
import java.nio.file.{Paths, Files}
import java.nio.charset.StandardCharsets

object Foo {
  def main(args: Array[String]): Unit = {
    val List(dest, content) = args.toList
    Files.write(Paths.get(dest), content.getBytes(StandardCharsets.UTF_8))
  }
}

I discovered this when I was trying to write a helper for running jvm tasks like this, and the Swagger generator was able to use the output.dest with no issue.. I assume this can also cause caching issues, and is just not caught by the compiler?

trait ClassLoaderHelper extends JavaModule {
  protected def dependencies: T[Seq[Dep]]
  protected def classToRun: T[String]
  protected def methodToRun: T[String]
  protected def inputs: T[Seq[Any]]

  private def classpath: Task[Seq[PathRef]] = Task {
    defaultResolver().classpath(dependencies())
  }
  private def worker: Worker[java.net.URLClassLoader] = Task.Worker {
    Jvm.createClassLoader(classpath().map(_.path).toSeq)
  }

  def run: Task[Path] = Task {
    val in = inputs()
    mill.api.ClassLoader.withContextClassLoader(worker()) {
      worker()
        .loadClass(classToRun())
        .getMethod(methodToRun(), in.map(_.getClass)*)
        .invoke(
          null,
          in*
        )
    }
    Task.dest
  }
}


trait Swagger extends JavaModule {
  def source: Task[PathRef]
  def generatorName: T[String] = "java"
  def version: T[String] = "2.2.2"
  def extraArgs: T[Array[String]]

  def generate: Task[PathRef] = Task {
    compiler.run()
    output()
  }

  private def output: Task[PathRef] = Task{ PathRef(Task.dest) }
  private object compiler extends build.plugins.util.ClassLoaderHelper {
    protected def dependencies: T[Seq[Dep]] = Seq(mvn"io.swagger:swagger-codegen-cli:${version()}")
    protected def classToRun: T[String] = "io.swagger.codegen.SwaggerCodegen"
    protected def methodToRun: T[String] = "main"
    protected def inputs: T[Seq[Any]] = Task {
      Seq(
        Array("generate", "--lang", "java", "--input-spec", source().path.toString, "--output", output().path.toString) ++ extraArgs()
      )
    }
  }
}

[lihaoyi]

@arminsumo It's expected the sandbox can be easily bypassed. as mentioned in the docs (https://mill-build.org/mill/depth/sandboxing.html):

Mill sandboxing is not meant to be fully hermetic, and is instead just a set of guardrails meant to nudge Mill users towards Mill’s best practices and coding conventions. Every Mill sandboxing guardrail has an escape-hatch to bypass it (at your own risk!) if really necessary.

[lihaoyi]

Going to call this a wontfix