pytest-pyvista

Plugin to test PyVista plot outputs.

---

This pytest plugin was generated with Cookiecutter along with @hackebrot's cookiecutter-pytest-plugin template.

Features

This plugin facilitates the comparison of the images produced by PyVista. It generates a cache of images from the tests, using the PyVista plotting function in its first execution. Then, further executions will compare its results against this cache, so if there are any changes in the code that break the image generation, the comparison against the cache will notice it. Note that there is an error tolerance in the comparison, so minor differences won't fail.

Requirements

You must have a Python version >= 3.10, as well as PyVista installed in your environment.

pyvista>=0.37.0 and vtk>=9.2.2 are required.

Installation

You can install pytest-pyvista via pip from PyPI

pip install pytest-pyvista

Alternatively, you can also install via conda or mamba from conda-forge

mamba install -c conda-forge pytest-pyvista

Usage

The plugin has two main use cases:

1. Evaluate images generated by calling pl.show() in unit tests.
2. Evaluate images generated by the Sphinx PyVista Plot Directive when building documentation.

Specifying multiple cache images

The cache directory is typically flat with no sub-directories. However, it is possible to specify multiple cache images for a single test by including a sub-directory with the same name as the test, and including multiple "valid" cache images in the sub-directory. For example, a single cached image:

cache/my_image.jpg

Can be replaced with multiple versions of the image:

cache/my_image/0.jpg
cache/my_image/1.jpg

Note

• The sub-directory name should match the name of the test.
• The image names in sub-directories can be arbitrary, e.g. 0.jpg or foo.jpg.
• Nested sub-directories are also supported, and their names can also be arbitrary.
• Use the --generate_subdirs flag to automatically generate test images in a sub-directory format.

When there are multiple images, the test will initially compare the test image to the first cached image. If that comparison fails, the test image is then compared to all other cached images for that test. The test is successful if one of the comparisons is successful, though a warning is still issued if it initially failed.

Both use cases (i.e. unit tests and documentation tests) support specifying multiple cache images.

Unit tests

Once installed, you only need to use the command pl.show() in your test. The plugin will automatically manage the cache generation if it does not exist, and the image comparison itself. Make sure you enable pv.OFF_SCREEN when loading PyVista, so the pl.show() doesn't pop up any window while testing. By default, the verify_image_cache fixture should be used for each test for image comparison:

import pyvista as pv

pv.OFF_SCREEN = True


def test_succeeds(verify_image_cache):
    pl = pyvista.Plotter()
    pl.add_mesh(pyvista.Sphere(), show_edges=True)
    pl.show()

If most tests utilize this functionality, possibly restricted to a module, a wrapped version could be used

@pytest.fixture(autouse=True)
def wrapped_verify_image_cache(verify_image_cache):
    return verify_image_cache

If you need to use any flag inside the tests, you can modify the verify_image_cache object in the test:

import pyvista as pv

pv.OFF_SCREEN = True


def test_succeeds(verify_image_cache):
    verify_image_cache.windows_skip_image_cache = True
    pl = pyvista.Plotter()
    pl.add_mesh(pyvista.Sphere(), show_edges=True)
    pl.show()

Documentation image tests

Unlike the unit tests, which use the verify_image_cache fixture to evaluate test images during each test's teardown, the documentation tests do not use a fixture. Instead, the documentation tests are invoked with the --doc_mode flag, and requires specifying:

1. A target directory which contains all images to be tested. The directory is specified using the --doc_images_dir flag.
2. A cache directory containing all reference images to compare with. The directory is specified using the --image_cache_dir flag.

Since all images must be available for testing, the documentation tests are typically executed after building documentation with Sphinx or some other build process. To test build images against images in a cache directory use:

pytest --doc_mode --doc_images_dir images --image_cache_dir cache

where images is the target directory of images to test, and cache is the cache directory.

When executed, the test will first pre-process the build images. The images are:

1. Collected from the images directory (including images in nested directories).

2. Resized to a maximum of 400x400 pixels.

3. Renamed so that each file's parent directories are included in the name.

4. Saved as images inside a temporary directory. The directory is flat with no sub-directories.

   Note

   These temporary images may be saved using the --generated_image_dir flag.

Next, the pre-processed images are compared to the cached images. The tests have three main modes of failure:

1. An image is in the cache but is missing from the build.
2. An image is in the build but is missing from the cache.
3. The error threshold when comparing two images is exceeded.

Note

Use the --failed_image_dir flag to save copies of the images for failed tests.

Global flags

These are the flags you can use when calling pytest in the command line.

Common flags

These flags may be used with regular unit testing or documentation testing with --doc_mode enabled.

• --image_cache_dir <DIR> sets the image cache directory, relative to pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>. This will override any configuration, see below.
• --generated_image_dir <DIR> dumps all generated test images into the provided directory, relative to pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>. This will override any configuration, see below.
• --failed_image_dir <DIR> dumps copies of cached and generated test images when there is a warning or error raised. This directory is useful for reviewing test failures. It is relative to pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>. This will override any configuration, see below.
• --generate_subdirs saves generated test images in separate sub-directories instead of saving them directly to the generated_image_dir. Without this option, generated images are saved as generated_image_dir/<test_name>.png; with this option enabled, they are instead saved as <generated_image_dir>/<test_name>/<image_name>.png, where the image name has the format <os-version>_<machine>_<gpu-vendor>_<python-version>_<pyvista-version>_<vtk-version>_<using-ci>. This can be useful for providing context about how an image was generated. See the Customizing test cases section for customizing the info.
• Use --image_format to save test images in either png or jpg format. png files are saved by default. Use jpg to reduce the image file size. This will override any configuration, see below.
• Use --max_image_size to specify a single integer that limits the maximum width or height (whichever is larger) of generated images to this value or smaller. By default, images have dimensions (1024, 768) as determined by the Plotter.window_size or pyvista.global_theme.window_size. If enabled, images are resized while preserving the aspect ratio. Images are only resized to be smaller, and are not enlarged.

Unit testing flags

These flags are specific to the unit tests. They cannot be used with --doc_mode enabled.

• --reset_image_cache creates a new image for each test in tests/plotting/test_plotting.py and is not recommended except for testing or for potentially a major or minor release.
• You can use --ignore_image_cache if you want to temporarily ignore regression testing, e.g. on a particular CI action.
• --add_missing_images adds any missing images from the test run to the cache.
• --reset_only_failed reset the image cache of the failed tests only.
• Use --allow_unused_generated to prevent an error from being raised when a test image is generated but not used. A test image is considered "used" if it has a corresponding cached image to compare against, or is used to reset or update the cache (e.g. if using --add_missing_images). Otherwise, an error is raised by default.
• --disallow_unused_cache report test failure if there are any images in the cache which are not compared to any generated images.
• Use --allow_useless_fixture to prevent test failure when the verify_image_cache fixture is used but no images are generated. If no images are generated (i.e. there are no calls made to Plotter.show() or mesh.plot()), then these tests will fail by default. Set this CLI flag to allow this globally, or use the test-specific flag by the same name below to configure this on a per-test basis.

Documentation testing flags

These flags are specific to documentation tests. They cannot be used with regular unit tests.

• --doc_mode is a required flag for testing documentation images. It configures pytest to only collect tests relevant for the image testing.

• --doc_images_dir <DIR> sets the target directory of images to be tested.

  Note

  With Sphinx, build images are typically saved to doc/_build/html/_images.

• Use --include_vtksz to include tests for interactive plots which are generated by PyVista's plot-directive as vtksz files. When enabled, each vtksz file inside the doc_images_dir is first converted to HTML, loaded in a web browser, and rendered with the same window size as its corresponding static image. A screenshot is captured, then compared with the cached image like normal.

  Note

  This option requires installing additional packages, as well as installing the chromium browser with playwright:

  pip install pytest-pyvista[vtksz]
  playwright install chromium

  Note

  Enabling this option may increase overall test time considerably due to the additional file processing required.

• Use --max_vtksz_file_size to include tests to limit the size of vtksz files. All vtksz files inside the doc_images_dir are collected and compared to the specified value. The value should be specified in megabytes (MB). 20 is a good initial value to set. The value can be customized on a per-test (i.e. per-file) basis, see Customizing test cases for details. Setting this value overrides any configuration, see below.

  Note

  This option is completely independent from the --include_vtksz option. File sizes may be tested without any additional installation.

Customizing test cases

Both the regular unit tests and documentation tests allow for some level of customization.

Customizing unit tests

These are attributes of verify_image_cache. You can set them as True if needed in the beginning of your test function.

• high_variance_test: If necessary, the threshold for determining if a test will pass or not is incremented to another predetermined threshold. This is currently done due to the use of an unstable version of VTK, in stable versions this shouldn't be necessary.

• windows_skip_image_cache: For test where the plotting in Windows is different from MacOS/Linux.

• macos_skip_image_cache: For test where the plotting in MacOS is different from Windows/Linux.

• skip: If you have a test that plots a figure, but you don't want to compare its output against the cache, you can skip it with this flag.

• allow_useless_fixture: Set this flag to True to prevent test failure when the verify_image_cache fixture is used but no images are generated. The value of this flag takes precedence over the global flag by the same name (see above).

• env_info: Dataclass for controlling the environment info used to name the generated test image(s) when the --generate_dirs option is used. The info can be test-specific or can be modified globally by wrapping the verify_image_cache fixture, e.g.:

  @pytest.fixture(autouse=True)
  def wrapped_verify_image_cache(verify_image_cache):
      # Customize the environment info (NOTE: Default values are shown)
      info = verify_image_cache.env_info
      info.prefix: str = ""  # Add a custom prefix
      info.os: bool = True  # Show/hide the os version (e.g. ubuntu, macOS, Windows)
      info.machine: bool = True  # Show/hide the machine info (e.g. arm64)
      info.gpu: bool = True  # Show/hide the gpu vendor (e.g. NVIDIA)
      info.python: bool = True  # Show/hide the python version
      info.pyvista: bool = True  # Show/hide the pyvista version
      info.vtk: bool = True  # Show/hide the vtk version
      info.ci: bool = True  # Show/hide if generated in CI
      info.suffix: str = ""  # Add a custom suffix
  
      # Alternatively, set a custom string
      verify_image_cache.env_info = 'my_custom_string'
  
      return verify_image_cache

Customizing documentation tests

Similar to how the unit tests may be customized using the verify_image_cache fixture, the documentation tests can be customized using a doc_verify_image_cache object. Instead of a fixture, a pytest hook function is used.

In your conftest.py file, define a hook function named pytest_pyvista_doc_mode_hook with doc_verify_image_cache and request as arguments. The doc_verify_image_cache object can then be modified directly on a per-test (i.e. per-image) basis.

For example, a test comparing the build image images/foo.png to the cached image cache/foo.png will have the test name foo, and can be modified as shown below. Currently, only the env_info can be customized. Refer to the test-specific flags for unit tests above for more details about customizing the env_info string.

def pytest_pyvista_doc_mode_hook(doc_verify_image_cache, request):
    if doc_verify_image_cache.test_name == 'foo':
        doc_verify_image_cache.env_info = 'my_custom_string'
    return doc_verify_image_cache

Note

Customizing the env_info will affect the generated image's filename, and is only relevant if the --generate_subdirs option is enabled.

Since the regular pytest request fixture is also exposed by the hook, users can further modify the test properties based based on node, markers, or other fixtures.

If the --max_vtksz_file_size option is used, these tests may similarly be customized with a pytest_pyvista_max_vtksz_file_size_hook. For example, to set the max allowed file size to 50 for the foo.vtksz file:

def pytest_pyvista_max_vtksz_file_size_hook(test_case, request):
    if test_case.test_name == 'foo':
        test_case.max_vtksz_file_size = 50
    return test_case

Configuration

If using pyproject.toml or any other pytest configuration section, consider configuring your test directory location to avoid passing command line arguments when calling pytest, for example in pyproject.toml:

[tool.pytest.ini_options]
image_cache_dir = "tests/plotting/image_cache"

Additionally, to configure the directory that will contain the generated test images:

[tool.pytest.ini_options]
generated_image_dir = "generated_images"

Similarly, configure the directory that will contain any failed test images:

[tool.pytest.ini_options]
failed_image_dir = "failed_images"

Configure directories for when --doc_mode is used:

[tool.pytest.ini_options]
doc_failed_image_dir = "failed_test_images"
doc_generated_image_dir = "generated_test_images"
doc_image_cache_dir = "tests/doc/doc_image_cache"
doc_images_dir = "doc/_build/html/_images"

Note that these directories are relative to pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>.

Include additional vtksz file tests.

[tool.pytest.ini_options]
include_vtksz = true
max_vtksz_file_size = 20

Configure the image format to be jpg for both unit tests and when using --doc_mode. Limit the image dimensions to 400 pixels as well.

[tool.pytest.ini_options]
image_format = "jpg"
max_image_size = 400

Or, set them to use different image formats and sizes:

[tool.pytest.ini_options]
image_format = "png"
max_image_size = 400
doc_image_format = "jpg"
doc_max_image_size = 600

Enable the generation of test images inside of sub-directories for both unit tests and when using --doc_mode.

[tool.pytest.ini_options]
generate_subdirs = true

Or, set them to different values:

[tool.pytest.ini_options]
generate_subdirs = true
doc_generate_subdirs = false

Contributing

Contributions are always welcome. Tests can be run with tox, please ensure the coverage at least stays the same before you submit a pull request.

License

Distributed under the terms of the MIT license, pytest-pyvista is free and open source software.

Issues

If you encounter any problems, please file an issue along with a detailed description.