Install error - error while loading shared libraries

[kdkorthauer]

I'm trying to install Rhdf5lib from Bioc 3.11 and am running into an error. I'm pasting below the last chunk of output which contains some error messages. Full configuration output attached as well.

LD_LIBRARY_PATH="$LD_LIBRARY_PATH`echo  |                  \
	sed -e 's/-L/:/g' -e 's/ //g'`"                               \
 ./H5make_libsettings > H5lib_settings.c  ||                               \
    (test $HDF5_Make_Ignore && echo "*** Error ignored") ||          \
    (rm -f H5lib_settings.c ; exit 1)
./H5make_libsettings: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory
make[2]: *** [H5lib_settings.c] Error 1
make[2]: Leaving directory `/tmp/RtmpkjM40i/R.INSTALL31ccd7a1c3035/Rhdf5lib/src/hdf5/src'
make[1]: *** [lib] Error 1
make[1]: Leaving directory `/tmp/RtmpkjM40i/R.INSTALL31ccd7a1c3035/Rhdf5lib/src/hdf5/src'
make: *** [lib] Error 1
configure:    HDF5_INCLUDE=hdf5/src
configure:    HDF5_CXX_INCLUDE=hdf5/c++/src
configure:    HDF5_HL_INCLUDE=hdf5/hl/src
configure:    HDF5_HL_CXX_INCLUDE=hdf5/hl/c++/src
configure:    HDF5_LIB=hdf5/src/.libs/libhdf5.a
configure:    HDF5_CXX_LIB=hdf5/c++/src/.libs/libhdf5_cpp.a
configure:    HDF5_HL_LIB=hdf5/hl/src/.libs/libhdf5_hl.a
configure:    HDF5_HL_CXX_LIB=hdf5/hl/c++/src/.libs/libhdf5_hl_cpp.a
configure:    SZIP_LIB=hdf5/szip/src/.libs/libsz.a
configure: creating ./config.status
config.status: creating src/Makevars
** libs
mkdir -p "/arc/home/kdkortha/Rlibs/00LOCK-Rhdf5lib/00new/Rhdf5lib/include"
cp "hdf5/src/"*.h "/arc/home/kdkortha/Rlibs/00LOCK-Rhdf5lib/00new/Rhdf5lib/include"
cp "hdf5/c++/src/"*.h "/arc/home/kdkortha/Rlibs/00LOCK-Rhdf5lib/00new/Rhdf5lib/include"
cp "hdf5/hl/src/"*.h "/arc/home/kdkortha/Rlibs/00LOCK-Rhdf5lib/00new/Rhdf5lib/include"
cp "hdf5/hl/c++/src/"*.h "/arc/home/kdkortha/Rlibs/00LOCK-Rhdf5lib/00new/Rhdf5lib/include"
mkdir -p "/arc/home/kdkortha/Rlibs/00LOCK-Rhdf5lib/00new/Rhdf5lib/lib/"
cp "hdf5/src/.libs/libhdf5.a" "/arc/home/kdkortha/Rlibs/00LOCK-Rhdf5lib/00new/Rhdf5lib/lib/"
cp: cannot stat ‘hdf5/src/.libs/libhdf5.a’: No such file or directory
make: *** [copying] Error 1
ERROR: compilation failed for package ‘Rhdf5lib’
* removing ‘/arc/home/kdkortha/Rlibs/Rhdf5lib’
* restoring previous ‘/arc/home/kdkortha/Rlibs/Rhdf5lib’

All other packages are up-to-date - BiocManager::valid() only flags 1 out-of-date package which is an earlier version of Rhdf5lib built with R 3.6.2.

Any suggestions are greatly appreciated.

[grimbough]

The crucial part of the error is error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory, everything that follows is a knock on effect of that.

I'm a bit confused by why the error occurs. Rhdf5lib doesn't link against libssl directly, but it does now link to libcrypto and libcurl, and it must be one of these that's trying to use libssl. For me it's libcurl:

ldd /usr/lib/x86_64-linux-gnu/libcurl.so | grep ssl 
	libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007fdd62cd9000)

According to the config log both openssl and curl are found, which should be sufficient for those flags to work, but there seem to be quite a few posts online with the same error under similar circumstances e.g. 1, 2, 3.

Perhaps you can do locate libcurl to find where the curl library is and then us ldd to check that where the issue lies. I'd expect to see something like

libssl.so.1.1 => not found

If that's the case then you probably need to set the LD_LIBRARY_PATH to the correct location, but lets verify this is the issue first.

[kdkorthauer]

Thanks, Mike.

Here's what I get from ldd:

(base) [kdkortha@login03 ~]$ which openssl
/arc/software/spack/opt/spack/linux-centos7-x86_64/gcc-5.4.0/miniconda3-4.6.14-f4hr756q34tvp7nsjn7hovq46fomaww6/bin/openssl
(base) [kdkortha@login03 ~]$ ldd /arc/software/spack/opt/spack/linux-centos7-x86_64/gcc-5.4.0/miniconda3-4.6.14-f4hr756q34tvp7nsjn7hovq46fomaww6/bin/openssl
	linux-vdso.so.1 =>  (0x00002aaaaaacd000)
	libssl.so.1.1 => /arc/software/spack/opt/spack/linux-centos7-x86_64/gcc-5.4.0/miniconda3-4.6.14-f4hr756q34tvp7nsjn7hovq46fomaww6/bin/../lib/libssl.so.1.1 (0x00002aaaaab8c000)
	libcrypto.so.1.1 => /arc/software/spack/opt/spack/linux-centos7-x86_64/gcc-5.4.0/miniconda3-4.6.14-f4hr756q34tvp7nsjn7hovq46fomaww6/bin/../lib/libcrypto.so.1.1 (0x00002aaaaaccf000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaaafc7000)
	libc.so.6 => /lib64/libc.so.6 (0x00002aaaab1e3000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaab5b1000)
	/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)

Looks like libssl.so.1.1 is found.

[grimbough]

Thanks for the output. I wonder if my configure script is finding a copy in somewhere more 'usual' like /lib64 but that's not the version that's actually being used. Both spack and conda are package managers in addition to the usual CentOS one, so I guess there's some scope for it to get confused.

For the moment I'll add a flag that lets you turn off the need for this so you can at least install the package, but I might ask you to install some experimental versions so I can try and get a handle on this if that's ok.

[kdkorthauer]

Happy to install future experimental versions to help troubleshoot.

Let me know when you've added the flag (no rush). Thanks!

[grimbough]

Hi Keegan, can you give this a try:

BiocManager::install('grimbough/Rhdf5lib', ref="curl-off", configure.args = "--disable-ros3-vfd")

Hopefully that will at least allow you to install an up-to-date version of the package. Let me know if it works, but it's not really satisfactory for you to have to do manual configuration every time the package needs installing.

I haven't made much progress at my end on figuring out why the config script doesn't detect the problem before doing all the compilation. I'll make an (even more) verbose config script that might help us see what it thinks it finds so we can compare with the paths you found from ldd.

[kdkorthauer]

That worked - I was able to install with that command. Thanks!

Again, happy to help troubleshoot further. Cheers!

[grimbough]

Great, good to know that's a temporary work around. I'm trying to emulate your setup with a virtual machine so I can debug this, can you run the following in a terminal so I can get a better idea of how it's set up?

which R ## or let me know how you installed R if you control that
ldconfig -v 2>/dev/null | grep -v ^$'\t'
echo $LD_LIBRARY_PATH

Thanks!

[kdkorthauer]

Here is the output for those commands:

(base) [kdkortha@login02 ~]$ which R ## or let me know how you installed R if you control that
/arc/software/spack-0.14.0-110/opt/spack/linux-centos7-skylake_avx512/gcc-9.2.0/r-4.0.1-crhtygajcebl3ck3l32ptcocthffwerh/bin/R
(base) [kdkortha@login02 ~]$ ldconfig -v 2>/dev/null | grep -v ^$'\t'
/usr/lib64/atlas:
/usr/lib64//bind9-export:
/opt/mellanox/hcoll/lib:
/opt/ibutils/lib64:
/usr/lib64/mysql:
/opt/mellanox/mxm/lib:
/cm/local/apps/mysql++/current/lib:
/cm/local/apps/cmd/lib:
/opt/mellanox/sharp/lib:
/opt/dell/srvadmin/lib64:
/opt/dell/srvadmin/lib64/openmanage:
/opt/dell/srvadmin/lib64/srvadmin-isvc:
/lib:
/lib64:
/lib/sse2: (hwcap: 0x0000000004000000)
/lib/i686: (hwcap: 0x0008000000000000)
/lib64/tls: (hwcap: 0x8000000000000000)
/lib64/sse2: (hwcap: 0x0000000004000000)
(base) [kdkortha@login02 ~]$ echo $LD_LIBRARY_PATH
/arc/software/spack-0.14.0-110/opt/spack/linux-centos7-skylake_avx512/gcc-9.2.0/perl-5.30.1-zqs5oucmrylqadpnu42hzxq45sgtxhgv/lib:/arc/software/spack-0.14.0-110/opt/spack/linux-centos7-skylake_avx512/gcc-9.2.0/r-4.0.1-crhtygajcebl3ck3l32ptcocthffwerh/rlib/R/lib:/arc/software/spack-0.14.0-110/opt/spack/linux-centos7-skylake_avx512/gcc-9.2.0/curl-7.68.0-ydwzakwwijrhestknszzrl4elhy2li7c/lib:/arc/software/spack-0.14.0-110/opt/spack/linux-centos7-skylake_avx512/gcc-9.2.0/libxmu-1.1.2-tdwivwxis76qurw63cyktt7ghehvhvoh/lib:/arc/software/spack-0.14.0-110/opt/spack/linux-centos7-skylake_avx512/gcc-9.2.0/libxt-1.1.5-3qm7qafclzdounvgprg2yyxc6l4aldee/lib:/arc/software/spack-0.14.0-110/opt/spack/linux-centos7-skylake_avx512/gcc-9.2.0/libx11-1.6.7-unmd3vcqo65o2yhws33wblaliam6i2rs/lib:/arc/software/spack-0.14.0-110/opt/spack/linux-centos7-skylake_avx512/gcc-9.2.0/freetype-2.10.1-nfb62h7cybxvusypjtrev7iavorplusl/lib:/arc/software/spack-0.14.0-110/opt/spack/linux-centos7-haswell/gcc-4.8.5/gcc-9.2.0-qdesip6wi3mdfph6mj4l4obbukn5342n/lib64:/arc/software/spack-0.14.0-110/opt/spack/linux-centos7-haswell/gcc-4.8.5/gcc-9.2.0-qdesip6wi3mdfph6mj4l4obbukn5342n/lib:/arc/software/spack-0.14.0-110/opt/spack/linux-centos7-skylake_avx512/gcc-9.2.0/cuda-10.2.89-xj3nb45c3nofc5kv4zpomcxiju4umnpc/lib64:/cm/shared/apps/pbspro-ce/19.1.3/lib

R 4.0.1 is loaded using the module system (recently installed by sysadmin). I was previously using an older version of R (<4), hence the need to update this package.

[grimbough]

Hi Keegan,

I wasn't able to replicate the issue you are seeing on CentOS - I guess there's too many unknowns in the cluster setup - but @kevinrue reported something very similar in the issue above using OSX and conda, and I think I've got a better idea about what's going on.

If you have time, can you try the following install and see if it completes successfully:

BiocManager::install('grimbough/Rhdf5lib', ref = "ld-flags")

[kevinrue]

Linking #32 (comment) where I declare having run successfully

BiocManager::install('grimbough/Rhdf5lib', ref = "ld-flags")

on a CentOS cluster, within a Conda environment created using

conda create --name r-env -c bioconda -c conda-forge r-base=4.0.2 r-renv=0.11.0

Note that in that environment, before I can run BiocManager::install() I also need to run

install.packages(c('BiocManager', 'remotes'))

[kdkorthauer]

I ran the following and unfortunately it did not complete successfully:

(base) [kdkortha@login03 ~]$ Rscript -e 'BiocManager::install("grimbough/Rhdf5lib", ref = "ld-flags")' > /tmp/config_output.txt
Bioconductor version 3.11 (BiocManager 1.30.10), R 4.0.1 (2020-06-06)
Installing github package(s) 'grimbough/Rhdf5lib'
Downloading GitHub repo grimbough/Rhdf5lib@ld-flags
* installing *source* package ‘Rhdf5lib’ ...
** using staged installation
./H5make_libsettings: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory
make[2]: *** [H5lib_settings.c] Error 1
make[1]: *** [lib] Error 1
make: *** [lib] Error 1
** libs
cp: cannot stat ‘hdf5/src/.libs/libhdf5.a’: No such file or directory
make: *** [copying] Error 1
ERROR: compilation failed for package ‘Rhdf5lib’
* removing ‘/arc/home/kdkortha/Rlibs/Rhdf5lib’
* restoring previous ‘/arc/home/kdkortha/Rlibs/Rhdf5lib’
Error: Failed to install 'Rhdf5lib' from GitHub:
  (converted from warning) installation of package ‘/tmp/RtmpdWHvpY/file28cd83b4ba5c5/Rhdf5lib_1.11.2-1.tar.gz’ had non-zero exit status
Execution halted

Here is the full configuration output

Best,
Keegan

[kevinrue]

Could it be about adding LDFLAGS in the Makevars like here? #32 (comment)

Oh no wait, that was for Mac.