Memory logging for workers/daemons?

[wlandau]

@shikokuchuo, as you may have seen in wlandau/crew#178, I am thinking about implementing memory logging for crew workers. I want to implement this in a general way for all crew plugins, even the cloud, where logs are stored remotely in e.g. CloudWatch instead of local physical files. I also want to watch memory usage over the progression of an individual task, which rules out simply returning memory diagnostics as part of the task's output. That leads me to my current idea: spawn a C thread which prints memory diagnostic messages (just RSS should be fine) to stdout (or I guess stderr if we need to use REprintf()) every few seconds. If the messages are structured and semantic, users could grep for them and plot the date like in wlandau/crew#179. (I will probably want to add proper timestamps crew's dispatcher memory messages so different data sources can be aggregated and plotted together.)

I would like this feature to be available in crew somehow, but I wanted to check with you first in case you have any interest in implementing it in mirai or even nanonext. Adding C code to crew would be a major change to support an ancillary feature.

[shikokuchuo]

What are your thoughts on why it should be in this stack versus a standalone package?

[wlandau]

A standalone package is an even better idea! I didn't think of that.

[wlandau]

FYI I just open-sourced https://github.com/wlandau/autometric and submitted it to CRAN.