Provide ARM release to allow support for FIL backend on Jetson and other ARM platforms

[blthayer]

Hello,

The FIL backend installation instructions indicate that:

The FIL backend is a part of Triton and can be installed via the methods described in the main Triton documentation.

However, when I download the latest Jetson release (tritonserver2.35.0-jetpack5.1.2.tgz) for Triton, it does not appear to have the FIL backend:

$ tar --list tritonserver/backends -f tritonserver2.35.0-jetpack5.1.2.tgz
...
tritonserver/backends/identity/
...
tritonserver/backends/tensorflow/
...
tritonserver/backends/onnxruntime/
...
tritonserver/backends/python/
...
tritonserver/backends/pytorch/
...
tritonserver/backends/tensorrt/
...

(I also checked the tritonserver2.26.0-jetpack5.0.2.tgz release, and had to use --list ./backends instead of --list tritonserver/backends and reached the same conclusion: FIL is not included.)

Does a pre-built FIL backend for Jetson exist? Note that I do not wish to use NGC Docker containers as those come with the kitchen sink of built-in tools which is not appropriate for my IoT application.

PS - I'm not sure if this issue belongs in this repository, or the server repo so I'm cross-posting this issue and will close one once I identify the appropriate location for the issue/ticket.

[wphicks]

The FIL backend has not yet been released for ARM due to a very rare and hard-to-reproduce bug that occasionally causes incorrect results to be returned. Even though the bug is extremely rare, we do not want to ship anything that could silently return incorrect output under any circumstances, so we have held off on release so far.

That being said, we do very much want to include the FIL backend with an upcoming ARM release, and we will likely prioritize this very soon.

[blthayer]

Thanks for your response, @wphicks! I look forward to the ARM release. Are there any issues I can track/subscribe to related to this topic? Or will this ticket serve as the "ARM release placeholder/proxy" ticket?

[wphicks]

Or will this ticket serve as the "ARM release placeholder/proxy" ticket?

Let's make it that! If it's all right, I'm going to go ahead and adjust the title accordingly. Others, please do upvote if having the FIL backend on ARM is important to you. This is already pretty high priority, but additional interest always helps.

[wphicks]

Status update: We have put this issue at high priority and will likely begin work on it middle of next week. We can't commit to any specific timeline for resolving it, since the bug is so difficult to reproduce, but we intend to get it figured out as soon as possible. Thanks very much to everyone who let us know that this is an important issue for you!

[blthayer]

Thank you, @wphicks! Looking forward to updates as this progresses 😄

[wphicks]

I wanted to quickly update this thread, since we have not in awhile. Despite the radio silence, we have been actively working on this; it just takes some time to investigate such a rare and flaky bug. I still cannot promise any specific timeline, but this is our top priority issue for the FIL backend at the moment. I'll continue to update this thread with progress or lack thereof as we go.

[blthayer]

Thanks for the update, @wphicks!

[blthayer]

@wphicks - do you have any updates on this issue at this time? Thanks!

[wphicks]

@blthayer Indeed I do! We're wrapping up the final round of testing on ARM. With millions of runs under our belt and no sign of the previous issue, we're feeling pretty good about moving forward. I'll be speaking to one of the other members of the Triton team at the end of this week, and assuming there is consensus, we will try to get the FIL backend into the next ARM release.

I know your interest is specifically in Jetson, and I cannot promise that it will specifically be included in the very next Jetson release, but the major blocker to that will be gone. I'll update this thread with more specific Jetson timeline and information as soon as I've got that info.

[blthayer]

@wphicks - thanks for the update! Looking forward to trying out the Jetson release when it's available. Thanks!

[guptap11]

@wphicks just to confirm this would be fixed in 23.10 triton server release ?

[wphicks]

Unfortunately, there's still some discussion ongoing internally about exactly when it will be released. I'll update here as soon as I have a firm timeline. Apologies for the delay!

[wphicks]

I've received the go-ahead to move forward with the ARM release. I'm going to work hard to get it onto at least some ARM platforms for 23.11, but I cannot guarantee that Jetson will be one of them. Code freeze for the 23.11 release of the FIL backend is 10/31, so I'll do as much as I can before then.