spack package

[ax3l]

Update: Development updates are now in https://github.com/ComputationalRadiationPhysics/spack-repo

---

Here is a draft of a spack package.
https://github.com/ax3l/spack/tree/topic-graybat

The main things:

• signaling target:
  • make install fails, since signaling is always tried to be installed, even if not build
  • does not add the gRPC libs and headers yet
  • make it so: build it and add it to install targets if gRPC is found
• several dependencies are not yet on a stable release but work in current develop (master) versions

ccing @erikzenker @fabian-jung

Snapshot

(branch above will be regularly rebased if necessary)

from spack import *


class Graybat(CMakePackage):
    """Graph Approach for Highly Generic Communication Schemes Based on Adaptive Topologies"""

    homepage = "https://github.com/ComputationalRadiationPhysics/graybat"
    url      = "https://github.com/ComputationalRadiationPhysics/graybat/archive/1.2.0.tar.gz"

    # version('develop', branch='remove_grpc',
    #         git='https://github.com/fabian-jung/graybat.git')
    version('develop', branch='dev',
            git='https://github.com/ComputationalRadiationPhysics/graybat.git')
    version('master', branch='master',
            git='https://github.com/ComputationalRadiationPhysics/graybat.git')
    # todo: only add first working version
    version('1.2.0', 'd47200e99d712d33e6e6a5f0ad946c6a')

    # this should not be necessary when all variants are properly optional
    build_targets = ['all', 'signaling']

    # C++14
    conflicts('%gcc@:5')
    conflicts('%clang@:3.4')

    variant('mpi', default=True,
            description='Enable MPI communication policy')
    variant('zeromq', default=True,
            description='Enable ZeroMQ communication policy')
    variant('metis', default=False,
            description='Enable graph partitioning mapping')
    variant('grpc', default=True,
            description='Enable gRPC signaling')

    depends_on('cmake@3.0.2:', type='build')
    depends_on('boost@1.61.0:1.62.0~mpi', type='link', when='~mpi')
    depends_on('boost@1.61.0:1.62.0+mpi', type='link', when='+mpi')
    depends_on('mpi', type='link', when='+mpi')
    depends_on('cppzmq@4.2.2:', type='link', when='+zeromq')
    depends_on('metis@5.1.0:', type='link', when='+metis')
    depends_on('protobuf@3.4.0:')
    depends_on('grpc@1.6.0+shared', when='+grpc')

[erikzenker]

Cool, thank you very much!!!

I just tested the package and you are right it does not succeed. On my computer it fails on the linking step do you have the same issue?

Nevertheless, I will provide a version with optional zmq, gprc, protobuf and signaling. Such that, it will be possible to build with mpi only.

[ax3l]

Nevertheless, I will provide a version with optional zmq, gprc, protobuf and signaling. Such that, it will be possible to build with mpi only.

Yes, that would be excellent! ✨
You could even make MPI itself optional, too if that is possible. (Isn't it already optional? :) )

I just tested the package and you are right it does not succeed. [...]

I have currently the following patches done in upstream projects:

• cppzmq: install fixed if libzmq was installed with autotools: CMake: pkg-config Fallback zeromq/cppzmq#133 (might did go into a 4.2.2) (spack)
• protobuf: add pkgconfig files in CMake based installs which are used in several downstream projects: CMake: Install .pc Files protocolbuffers/protobuf#3403 (probably goes into 3.4) (spack)
• gRPC: build with all-external dependencies C-Ares: Remove Internal Header grpc/grpc#11903 (might go into 1.5.0) (spack package todo)

When the protobuf and gRPC fixes are merged I still need to publish the spack gRPC package.
For graybat, I would make all the above mentioned dependencies "variants" of the graybat spack package as soon as they are optional.

[ax3l]

@erikzenker @fabian-jung just a note: I moved our in-development spack packages to a central repo in https://github.com/ComputationalRadiationPhysics/spack-repo

Please see the instructions therein, so you don't need to depend on some weird branches of mine.

[ax3l]

grpc@develop dependency in spack-repo updated:

• grpc@develop now depends on protobuf 3.4.0+
• protobuf 3.4.0 added to known versions in mainline spack: Protobuf: 3.4 spack/spack#5177

(there is still no gRPC 1.5.0 release we can put a fix on)

[ax3l]

currently I also experience a missing zlib when trying to build graybat@develop:

libz.so.1: error adding symbols: DSO missing from command line

[ax3l]

ccing @fabian-jung @erikzenker

[ax3l]

@fabian-jung @erikzenker I just managed to build graybat 1.2.0 and dev with spack and gRPC 1.6.0.

Like this (using gcc 6.3.0: %gcc@6.3.0 as a suffix in spack install graybat)

# the standard spack install
git clone https://github.com/llnl/spack.git $HOME/src/spack
. $HOME/src/spack/share/spack/setup-env.sh

# our graybat recipe
git clone https://github.com/ComputationalRadiationPhysics/spack-repo.git $HOME/src/spack-repo
spack repo add $HOME/src/spack-repo

# now build graybat
spack install graybat@1.2.0
spack install graybat@develop

@erikzenker any progress on #145? :)

[erikzenker]

#145 is in good shape. I will test it with the current spack version.

[ax3l]

perfect, as soon as #145 is in we can add proper spack variants

[erikzenker]

I failed to install protobuf. I will continue to investigate after my vacations in 2 weeks.

==> Installing protobuf
==> Fetching https://github.com/google/protobuf/archive/v3.4.0.tar.gz
######################################################################## 100,0%
==> Error: ChecksumError: md5 checksum failed for /home/erik/projects/spack/var/spack/stage/protobuf-3.4.0-vp2si5mlgalhsqre64yw35vaga62ge6k/v3.4.0.tar.gz
    Expected 4f47de212ef665ea619f5f97083c6781 but got 1d077a7d4db3d75681f5c333f2de9b1a
ChecksumError: ChecksumError: md5 checksum failed for /home/erik/projects/spack/var/spack/stage/protobuf-3.4.0-vp2si5mlgalhsqre64yw35vaga62ge6k/v3.4.0.tar.gz
    Expected 4f47de212ef665ea619f5f97083c6781 but got 1d077a7d4db3d75681f5c333f2de9b1a

/home/erik/projects/spack/lib/spack/spack/package.py:962, in do_fetch:
     29            self._fetch_time = time.time() - start_time
     30    
     31            if spack.do_checksum and self.version in self.versions:
  >> 32                self.stage.check()
     33    
     34            self.stage.cache_local()

[ax3l]

oh, true. that's a weird issue currently in spack with the way it discovers potential URL candidates for versions: spack/spack#5356

the only thing you can do right now is: open the (/home/erik/projects/spack/) var/spack/repos/builtin/packages/protobuf/packages.py file and just put in the md5sum you see on the terminal behind "but got ...". this will let it pass for now until the issue is fixed upstream

[ax3l]

@erikzenker I have a PR open in spack/spack#5373 that should fix your fetch/md5 issue in protobuf :)
update: merged :)