Support other architectures

[unixfox]

EDIT: Fixed in https://github.com/searxng/searxng-docker if anyone is interested.

---

If #26 get merged then it's one step towards supporting other architectures because the Docker image lucaslorentz/caddy-docker-proxy supports ARM devices.
filtron and morty can be compiled in ARM too because Golang support it. And searx and searx-checker are using python so they can be switched to use the alpine ARM base image.

I would like support for PPC64LE as well

[SuperSandro2000]

filtron and morty can be compiled in ARM too because Golang support it. And searx and searx-checker are using python so they can be switched to use the alpine ARM base image.

I would suggest to use Debian because it has more prebuilts which makes building easier.

[samuel-p]

I just could successfully build the Searx Docker Image on arm without any modifications:

git clone https://github.com/searx/searx.git
cd searx/
./manage.sh docker_build

So I think we just need to update the CI Docker builds to also build it for arm (or others) and publish it to Docker-Hub.

[dalf]

Reading https://www.docker.com/blog/multi-arch-build-what-about-travis/

I have this script for travis:

# install the last docker verison (for travis, according to the article, not tested)
sudo rm -rf /var/lib/apt/lists/*
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) edge"
sudo apt-get update
sudo apt-get -y -o Dpkg::Options::="--force-confnew" install docker-ce

# install & setup buildx
mkdir -vp ~/.docker/cli-plugins/
curl --silent -L "https://github.com/docker/buildx/releases/download/v0.4.2/buildx-v0.4.2.linux-amd64" > ~/.docker/cli-plugins/docker-buildx
chmod a+x ~/.docker/cli-plugins/docker-buildx
# to avoid "invalid non-bool value for BUILDX_NO_DEFAULT_LOAD:" :
# docker buildx create --use --name multi-arch --driver-opt image=moby/buildkit:master
docker buildx create --use
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

# build searx image
docker buildx build --platform linux/amd64,linux/arm64,linux/ppc64le --tag dalf/searx:buildx-latest .

# it fails with linux/arm

The docker buildx command is only for the test, it should be here https://github.com/searx/searx/blob/fd5fe369844e481aecc0d731b08ee8b29c9b47e6/manage.sh#L154

[SuperSandro2000]

I just use the following setup for buildx in travis:

.travis.yaml

env:
  global:
    - DOCKER_CLI_EXPERIMENTAL=enabled
    - DOCKER_BUILDKIT=1
...
      - mkdir -p /etc/docker/
      - |
        echo -e '{"experimental": true}' | sudo tee /etc/docker/daemon.json
      - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
      - sudo add-apt-repository "deb https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
      - sudo apt-get install -qy docker-ce
      # log into Docker HUB
      - echo ${DOCKER_PASSWORD} | docker login -u ${DOCKER_USERNAME} --password-stdin
      # enable multiarch
      - docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
      # create builder for buildx
      - docker buildx create --use --name build --node build --driver-opt network=host

[dalf]

Thank you for the configuration.

I have create dalf/searx@docker-buildx branch:

• .travis.yml
• manage.sh

Output here:
https://travis-ci.org/github/dalf/searx/jobs/732563796

Unfortunately, the build takes forever and timeout.

Most probably pyopenssl can be removed from the dependencies (since searx/searx#2199 ), but anyway even lxml is a problem (and this one is mandatory).

---

[EDIT] There is https://docs.travis-ci.com/user/multi-cpu-architectures/

[SuperSandro2000]

You probably need to find a pypi repository which has this as a prebuilt. Alternatively you would need to setup a pipeline where you prebuilt this package and upload it to a pypi server.

I am doing the same for arm for this Image https://github.com/SuperSandro2000/docker-images/blob/master/reddiscord/Dockerfile#L45 https://github.com/SuperSandro2000/docker-images/blob/master/reddiscord/files/pip.conf.

I am using this repo to built the wheels and this pypiserver to host them.

This would probably take quite some time if there is nothing like https://www.piwheels.org/ for ppc64le.

If you need help with any of the above feel free to reach out to me.

[avpnusr]

I'm not as deep in CI/CD workflows, as you guys seem to be :)

However, I could successfully build the docker-images for armv6, armv7, arm64 (ppc is building at the moment), using the github workflow integrations: https://github.com/avpnusr/searx/blob/master/.github/workflows/update-build.yml
As @dalf said, it runs forever (more than 1 hour), but it finishes with a working docker-image.
https://hub.docker.com/r/avpnusr/searx/tags

As far as i can see from your links above, Travis does not support armv6 armv7, but only arm64 and ppc64.

The image from github workflow works, as I run it on a RaspberryPi 3B+ .

Perhaps this helps in any way?

[unixfox]

Yes with docker buildx you can build for a lot of architectures. Searx could switch to it using Github actions.

[dalf]

Looking at

• https://github.com/searx/searx/blob/master/.github/workflows/integration.yml#L86-L117
• https://github.com/searx/searx/blob/5fb9a5c7c10252b21a23fe66e1625ed9f8691b4a/manage.sh#L154-L163

few things are missing.

[dalf]

It is possible to speed lxml install / compilation:
pip install lxml --global-option="build_ext" --global-option="-j3"

Moreover, there is no need to install pyopenssl : see https://github.com/psf/requests/blob/master/HISTORY.md#2250-2020-11-11 ( requests[security] is deprecated ).

[avpnusr]

Looking at

* https://github.com/searx/searx/blob/master/.github/workflows/integration.yml#L86-L117

* https://github.com/searx/searx/blob/5fb9a5c7c10252b21a23fe66e1625ed9f8691b4a/manage.sh#L154-L163

few things are missing.

Sorry, took me a while, to get back to this :)

As far as I can see, you mean the missing build variables in the dockerx command.
I tried to integrate them into the github actions workflow and testing a build at the moment.

Regarding the installation of the lxml via pip - do you know, where i have to comment the lxml build out?
Is it enough to comment out in the requirements.txt and insert the "pip install lxml [...]" command in the Dockerfile, or does this dependency come from another place?