PaddleX/3.0-rc/pipeline_deploy/serving

[giscus[bot]]

PaddleX/3.0-rc/pipeline_deploy/serving

https://paddlepaddle.github.io/PaddleX/3.0-rc/pipeline_deploy/serving.html

[yaoguang]

我在测试‘高稳定性服务化部署’，使用通用OCR SDK，本地docker启动成功，client.py测试GRPCInferenceService通过，Metrics Service访问成功，但是HTTPService请求失败，一直报错400 Bad Request。想问一下HTTPService的接口调用文档有没有，请求参数是什么，怎么才能正确访问HTTPService

[ethlucky]

文档在哪里呢？ 没看到，给个连接啊

[ethlucky]

https://paddlepaddle.github.io/PaddleX/latest/pipeline_usage/tutorials/ocr_pipelines/OCR.html#3 是这个文档中的api参考和和多语言调用服务示例部分吗

[Bobholamovic]

https://paddlepaddle.github.io/PaddleX/3.1/pipeline_deploy/serving.html#242-http

[ethlucky]

好的，非常感谢

[bmmpp]

sudo docker run -e PADDLEX_HPS_DEVICE_TYPE=cpu -v /usr/local/paddleocr:/app -w /app --rm --init --network host --shm-size 8g ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlex/hps:paddlex3.0.3-cpu /bin/bash server.sh
/bin/bash: server.sh: No such file or directory 为什么我run的时候报不存在呢 ？ 到容器里看确实不存在呀

[asharca]

请问能否同时部署多个产线

[Bobholamovic]

因为一条产线本身可能不止一个endpoint，为了避免过于复杂的架构，一次只支持启动一个服务。如果需要部署多条产线，可以启动多个服务，绑定到不同的端口

[zhanghongtian]

执行/bin/bash server.sh报一下错误，请问有人遇到吗？怎么解决的呢？
failed to load model 'ocr': at least one version must be available under the version policy of model 'ocr'

[zhanghongtian]

是自己的失误，解决了

[mzzya]

很强，希望能提供高性能的其他语言调用示例

[Bobholamovic]

方便的话可以提个feature request issue，我们将讨论这些需求～

[Bobholamovic]

你好，最新的文档中提供了手动构造HTTP请求的方式，这种方式对client的语言没有限制，可以试试～

[bmmpp]

容器里 server.sh 不存在呀， 2.3 直接跑不了吗？ 是要把sdk 里的 sh，和yml 配置抄到挂载目录吗？

sudo docker run -e PADDLEX_HPS_DEVICE_TYPE=cpu -v /usr/local/paddleocr:/app -w /app --rm --init --network host --shm-size 8g ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlex/hps:paddlex3.0.3-cpu /bin/bash server.sh
/bin/bash: server.sh: No such file or directory

执行后 docker run -it --entrypoint /bin/bash ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlex/hps:paddlex3.0.3-cpu
root@aff23e4814d0:/# ls
bin dev home lib32 libx32 mnt paddlex root sbin sys usr
boot etc lib lib64 media opt proc run srv tmp var
root@aff23e4814d0:/#

[Bobholamovic]

paddle应该不会自动安装nvcc。如果你是在安装docker之后安装nvidia driver的，请重启docker服务。

[ethlucky]

我重启下docker服务问题解决了，如果不使用高性能推理的话，服务可以正常运行起来了，看到服务启动端口
类似以下信息
I1216 11:37:21.601943 35 grpc_server.cc:4117] Started GRPCInferenceService at 0.0.0.0:8001

I1216 11:37:21.602333 35 http_server.cc:2815] Started HTTPService at 0.0.0.0:8000

I1216 11:37:21.643494 35 http_server.cc:167] Started Metrics Service at 0.0.0.0:8002

docker run
-it
-e PADDLEX_HPS_DEVICE_TYPE=gpu
-e PADDLEX_HPS_USE_HPIP=1
-v "$(pwd)":/app
-w /app
--rm
--gpus all
--init
--network host
--shm-size 8g
ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlex/hps:paddlex3.0.3-gpu
/bin/bash server.sh
但是 -e PADDLEX_HPS_USE_HPIP=1 开启高性能推理后一直在以下日志位置不动了，并没有看到的启动端口

Tue Jul 15 01:49:33-INFO: set shape of Value(define_op_name=pd_op.data, index=0, dtype=tensor<-1x3x-1x-1xf32>, stop_gradient=False), op is: Operation((%0) = "pd_op.data" () {dtype:float32,name:"x",place:Place(undefined:0),shape:[-1,3,-1,-1],stop_gradient:[false]} : () -> tensor<-1x3x-1x-1xf32>)
Tue Jul 15 01:49:33-INFO: set min_shape of Value(define_op_name=pd_op.data, index=0, dtype=tensor<-1x3x-1x-1xf32>, stop_gradient=False) as [1, 3, 32, 32]
Tue Jul 15 01:49:33-INFO: set opt_shape of Value(define_op_name=pd_op.data, index=0, dtype=tensor<-1x3x-1x-1xf32>, stop_gradient=False) as [1, 3, 736, 736]
Tue Jul 15 01:49:33-INFO: set max_shape of Value(define_op_name=pd_op.data, index=0, dtype=tensor<-1x3x-1x-1xf32>, stop_gradient=False) as [1, 3, 4000, 4000]
Tue Jul 15 01:49:33-INFO: Run Paddle-TRT FP16 mode

[Bobholamovic]

高性能推理第一次启动的时候可能需要执行优化，可能需要花费较长时间，从第二次开始有缓存就不会这么慢了。如果采用上面的方式启动，同时希望保留缓存，请考虑建立模型下载目录和宿主机目录之间的映射，使缓存能够被持久地存储而不会随着容器销毁。或者，去掉--rm，使其保存在容器中，下次启动服务直接重启容器。

[ethlucky]

哦，那就是还没执行完是吗，的等着他执行完就行了,

模型下载目录和宿主机目录之间的映射， 这个有哪些参数我可以配置呢，文档在哪里呢

[Bobholamovic]

这取决于你把模型存在什么位置，在执行日志中也可以看到模型的下载目录。可以通过docker run的-v/--volume选项设置目录映射。这部分属于docker的用法，我们的文档中就没有体现了，具体可以参考docker官方文档哈～

[59592-9]

怎么指定2，3号gpu?

[ethlucky]

docker pull ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlex/hps:paddlex3.0.3-gpu

文档写的是这个啊？ 3.1版本的在哪里？

[Bobholamovic]

请参考最新的文档：https://paddlepaddle.github.io/PaddleX/latest/pipeline_deploy/serving.html#22

[bmmpp]

大佬 能不能指导一下 其它语言 grpc 怎么调用的

[Bobholamovic]

大佬 能不能指导一下 其它语言 grpc 怎么调用的

其他语言的grpc调用建议依赖对应语言的Triton client开发，但Triton官方并没有为每个语言都提供client～

[59592-9]

我跑在4张4090上的为什么会显示内存不足，是不是显存池太小才64mb,设置大一点的命令是什么？

[59592-9]

您好我在config_gpu内指定是2，3号gpu
为什么我看到分配显存池的时候还是给0，1号gpu分配显存
而且我的gpu全是4090为什么会出现显存不足的报错
0717 05:38:46.416882 17685 pb_stub.cc:352] Failed to initialize Python stub: OSError: (External) CUDA error(2), out of memory.
[Hint: 'cudaErrorMemoryAllocation'. The API call failed because it was unable to allocate enough memory to perform the requested operation. ] (at ../paddle/phi/backends/gpu/cuda/cuda_info.cc:206)

At:
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/models/common/static_infer.py(433): _create
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/models/common/static_infer.py(274): init
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/models/base/predictor/base_predictor.py(240): create_static_infer
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/models/image_classification/predictor.py(82): _build
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/models/image_classification/predictor.py(49): init
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/models/init.py(77): create_predictor
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/base.py(107): create_model
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/doc_preprocessor/pipeline.py(67): init
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/_parallel.py(158): _create_internal_pipeline
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/_parallel.py(103): init
/paddlex/py310/lib/python3.10/site-packages/paddlex/utils/deps.py(195): _wrapper
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/init.py(165): create_pipeline
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/base.py(140): create_pipeline
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/ocr/pipeline.py(73): init
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/_parallel.py(158): _create_internal_pipeline
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/_parallel.py(103): init
/paddlex/py310/lib/python3.10/site-packages/paddlex/utils/deps.py(195): _wrapper
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/init.py(165): create_pipeline
/paddlex/py310/lib/python3.10/site-packages/paddlex_hps_server/base_model.py(134): _create_pipeline
/paddlex/py310/lib/python3.10/site-packages/paddlex_hps_server/base_model.py(62): initialize
/paddlex/var/paddlex_model_repo/ocr/1/model.py(18): initialize

E0717 05:38:54.064662 7 model_repository_manager.cc:1186] failed to load 'ocr' version 1: Internal: OSError: (External) CUDA error(2), out of memory.
[Hint: 'cudaErrorMemoryAllocation'. The API call failed because it was unable to allocate enough memory to perform the requested operation. ] (at ../paddle/phi/backends/gpu/cuda/cuda_info.cc:206)

At:
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/models/common/static_infer.py(433): _create
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/models/common/static_infer.py(274): init
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/models/base/predictor/base_predictor.py(240): create_static_infer
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/models/image_classification/predictor.py(82): _build
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/models/image_classification/predictor.py(49): init
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/models/init.py(77): create_predictor
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/base.py(107): create_model
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/doc_preprocessor/pipeline.py(67): init
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/_parallel.py(158): _create_internal_pipeline
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/_parallel.py(103): init
/paddlex/py310/lib/python3.10/site-packages/paddlex/utils/deps.py(195): _wrapper
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/init.py(165): create_pipeline
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/base.py(140): create_pipeline
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/ocr/pipeline.py(73): init
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/_parallel.py(158): _create_internal_pipeline
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/_parallel.py(103): init
/paddlex/py310/lib/python3.10/site-packages/paddlex/utils/deps.py(195): _wrapper
/paddlex/py310/lib/python3.10/site-packages/paddlex/inference/pipelines/init.py(165): create_pipeline
/paddlex/py310/lib/python3.10/site-packages/paddlex_hps_server/base_model.py(134): _create_pipeline
/paddlex/py310/lib/python3.10/site-packages/paddlex_hps_server/base_model.py(62): initialize
/paddlex/var/paddlex_model_repo/ocr/1/model.py(18): initialize

[59592-9]

我想换到2，3号gpu上因为0，1有人在使用，指定gpu的指令规范是什么？

[Bobholamovic]

可以在你现在配置的基础上，把instance_group里的count设置成1。

[59592-9]

全是1了现在，为什么写两块gpu就失败只用一块gpu就成功

[Bobholamovic]

之前的配置是在2块GPU上各启动4个实例，可能因为资源用超了所以失败。2块GPU上各启动1个实例（count为1，gpus为[2, 3]）应该是没问题的，请问目前是这样吗？

[59592-9]

可以了4090上只能跑一个实例，设置为2就报错