golang编译的so库使用时会导致resty-shell插件无法正常工作，error.log报告unknown process 140767 exited with code 0

[yuzhou-nj]

前辈，你好！

我用了go的example测试了下，发现会影响resty.shell插件（执行shell脚本），nginx重启后，访问/testshell不会有问题，但只要访问了/echo（内部走libffi_go_echo.so）之后就不正常了，无法得到执行结果。nginx的error.log报错【unknown process 】

具体测试详情如下：
openresty1.21.4.1、lua-resty-ffi为最新版本。

nginx.conf中配置：

    server {
        listen 20000;

        location /echo {
            content_by_lua_block {
                local demo = ngx.load_ffi("ffi_go_echo")
                local ok, res = demo:echo("foobar")
                assert(ok)
                assert(res == "foobar")

                demo:__unload()
                ok, res = demo:echo("foobar")
                assert(not ok)
                ngx.log(ngx.ERR, res)

                ngx.say("ok")
            }
        }
        location /testshell {
            content_by_lua_block {
                local shell = require("resty.shell")
                local stdin = nil
                local ex_timeout = 1000
                local ex_max_size = 4096
                local ok2, stdout, stderr, reason, status = shell.run("date", stdin, ex_timeout, ex_max_size)

                ngx.say("shell.run=" .. tostring(ok2) .. ", out=" .. tostring(stdout) .. ", stderr=" .. tostring(stderr) .. ", reason=" .. tostring(reason) .. ", status=" .. tostring(status))
            }
        }    

测试访问/echo，：

# date; time curl "http://127.0.0.1:20000/echo"
Tue Dec  3 11:47:58 CST 2024
ok

real    0m0.011s
user    0m0.004s
sys     0m0.005s

但再访问/testshell就会失败，得不到shell执行结果（重启nginx后直接访问/testshell就没问题）：

# date; time curl "http://127.0.0.1:20000/testshell"
Tue Dec  3 11:48:02 CST 2024
shell.run=nil, out=Tue Dec  3 11:48:02 CST 2024
, stderr=, reason=failed to wait for process: timeout; failed to kill process 140767: No such process, status=nil

real    0m1.011s
user    0m0.004s
sys     0m0.006s

nginx的error.log日志：

2024/12/03 11:47:58 start go echo runtime
2024/12/03 11:47:58 [error] 128368#128368: *62 [lua] content_by_lua(nginx.conf:209):10: task queue is finished, client: 127.0.0.1, server: , request: "GET /echo HTTP/1.1", host: "127.0.0.1:20000"
2024/12/03 11:47:58 exit go echo runtime
2024/12/03 11:47:58 [info] 128368#128368: *62 client 127.0.0.1 closed keepalive connection
2024/12/03 11:48:02 [notice] 128369#128369: signal 17 (SIGCHLD) received from 140767
2024/12/03 11:48:02 [notice] 128369#128369: unknown process 140767 exited with code 0
2024/12/03 11:48:03 [info] 128369#128369: *63 client 127.0.0.1 closed keepalive connection

谢谢！

[yuzhou-nj]

好像规律是：nginx的每个worker只要访问过一次/echo，再次访问到这个worker的/testshell就会有异常（取不到shell执行结果，但实际命令是执行成功的）

比如我这2个worker，在nginx -s reload后：

访问/testshell正常：
# time curl "http://127.0.0.1:20000/testshell"
shell.run=true, out=Tue Dec  3 15:26:44 CST 2024
, stderr=, reason=exit, status=0

real    0m0.015s
user    0m0.004s
sys     0m0.006s

访问/echo：
# time curl "http://127.0.0.1:20000/echo"
ok

real    0m0.011s
user    0m0.005s
sys     0m0.005s

再访问/testshell异常：
# time curl "http://127.0.0.1:20000/testshell"
shell.run=nil, out=Tue Dec  3 15:26:55 CST 2024
, stderr=, reason=failed to wait for process: timeout; failed to kill process 69220: No such process, status=nil

real    0m1.012s
user    0m0.005s
sys     0m0.005s

再访问/testshell又正常（可能到另外一个worker了）：
# time curl "http://127.0.0.1:20000/testshell"
shell.run=true, out=Tue Dec  3 15:26:59 CST 2024
, stderr=, reason=exit, status=0

real    0m0.015s
user    0m0.002s
sys     0m0.007s

再访问/testshell又异常：
# time curl "http://127.0.0.1:20000/testshell"
shell.run=nil, out=Tue Dec  3 15:27:02 CST 2024
, stderr=, reason=failed to wait for process: timeout; failed to kill process 69310: No such process, status=nil

real    0m1.011s
user    0m0.003s
sys     0m0.007s

[kingluo]

yes, this issue is also reproducible in my env.
I'll fix it soon later, please stay tuned. Thanks for your bug report.

[kingluo]

@yuzhou-nj This is due to Golang runtime interference (something went wrong between the cgo function and fork()). Further research is needed. If you really need to use lua-resty-shell, it is recommended to use other languages ​​​​for the time being, such as Python or Rust.

[yuzhou-nj]

Thank you for your prompt response and analysis! Lua-resty-ffi is a fantastic plugin, but it would be a pity if we couldn't use libraries written in Go. After all, there are many excellent components in Go that can enhance the capabilities of NGINX with Lua.

I attempted to write a new C dynamic library that internally calls a dynamic library generated by Go, but the issue still persists.

[kingluo]

Yes, but the reason here is that lua-resty-shell uses fork() to run external programs in subprocesses. If not, you can use Go with other extensions via lua-resty-ffi .

I'll try to figure out some solution later (maybe this weekend).

As a workaround, I suggest you use https://pkg.go.dev/os/exec to mimic the functionality of lua-resty-shell.

[yuzhou-nj]

Yes, but the reason here is that lua-resty-shell uses fork() to run external programs in subprocesses. If not, you can use Go with other extensions via lua-resty-ffi .

I'll try to figure out some solution later (maybe this weekend).

As a workaround, I suggest you use https://pkg.go.dev/os/exec to mimic the functionality of lua-resty-shell.

I tried using os.exec in Go, but the issue still persists.

	//cmd := exec.CommandContext(ctx, "sh", "-c", req.Cmd)  
	cmd := exec.CommandContext(ctx, req.Cmd)
	out, err := cmd.CombinedOutput()
	if err != nil {
		return err //here return: waitid: no child processes
	}

When I added the code below in libffi_init, os.exec returned normally, but resty.shell still doesn't get results when executing shell scripts. Additionally, resty.shell internally uses lightweight threads created with ngx.thread.spawn， not fork(), so I'm concerned that other code using this method might also encounter issues.

	c := make(chan os.Signal)
	signal.Notify(c, syscall.SIGPIPE, syscall.SIGCHLD)

	go func() {
		for s := range c {
			switch s {
			default:
				logger.Warn("recv signal=%s", s.String())  
			}
		}
	}()

[kingluo]

Just to remind, I have some accidents myself these two weeks, so I have to postpone the fix. Sorry.

---

@yuzhou-nj don't use resty.shell for the time being. It does fork(), using ngx.pipe.

With os.exec, I mean you can tell your go runtime (like echo.go) to fork and exec an external program and return the result.

For example:

local demo = ngx.load_ffi("ffi_go_os_exec")
local ok, res = demo:exec("date")
if ok then
        print(res.stdout)
        print(res.stderr)
end

See, you can write your go code, with os.exec, to simulate the functionalities of resty.shell, which should be not difficult.

You can refer to below lua-resty-ffi-go example libraries:

lua-resty-ffi-etcd
lua-resty-ffi-kafka
lua-resty-ffi-req

Hopefully this helps.

[yuzhou-nj]

Just to remind, I have some accidents myself these two weeks, so I have to postpone the fix. Sorry.

@yuzhou-nj don't use resty.shell for the time being. It does fork(), using ngx.pipe.

With os.exec, I mean you can tell your go runtime (like echo.go) to fork and exec an external program and return the result.

For example:

local demo = ngx.load_ffi("ffi_go_os_exec")
local ok, res = demo:exec("date")
if ok then
        print(res.stdout)
        print(res.stderr)
end

See, you can write your go code, with os.exec, to simulate the functionalities of resty.shell, which should be not difficult.

You can refer to below lua-resty-ffi-go example libraries:

lua-resty-ffi-etcd lua-resty-ffi-kafka lua-resty-ffi-req

Hopefully this helps.

Thank you very much! I referred to echo.go to write exec.go, implementing the same functionality as resty-shell. I just need to add the following code in libffi_init.

 c := make(chan os.Signal)
 signal.Notify(c, syscall.SIGCHLD)

[kingluo]

golang does not work well with the native fork system call, and there seems to be no official solution yet, so if a resty library uses fork(), please rewrite it in the golang library for lua-resty-ffi using os/exec. This PR will not be fixed.

golang/go#14767

[yuzhou-nj]

golang 它与原生的 fork 系统调用配合得不太好，目前似乎还没有官方的解决方案，所以如果 Resty 库使用了 fork()，请使用 os/exec 在 lua-resty-ffi 的 golang 库中重写。此PR不会被修复。

golang/go#14767

感谢大侠回复！

我小结一下哈：

用golang写的echo库，内部没有调用fork，但也影响了resty-shell的调用返回，后者内部是thread.spawn轻线程，应该不是fork。

貌似是nginx中调用golang写的so动态库，处理SIGCHLD信号时与nginx的冲突，有20%的概率是未被go动态库捕获，导致nginx打印了“signal 17 (SIGCHLD) received from XXX”这种错误。

临时规避：用golang写动态库，初始化时执行signal.Reset(syscall.SIGCHLD)，然后接口函数中调用os/exec库来实现，当c.Run()的err为nil，或者虽然报错但描述中包含 “no child process”，且c.Stderr不含错误信息，都认为成功。