Skip to content

Plugin Module

The plugin module provides the pytest integration for pytest-gremlins. It implements pytest hooks that enable mutation testing as part of the test lifecycle.

Overview

The plugin handles:

  1. Command-line options - Adding --gremlins and related flags
  2. Configuration - Loading settings from pyproject.toml and merging with CLI
  3. Source discovery - Finding Python files to mutate
  4. Instrumentation - Transforming source code with embedded mutations
  5. Test execution - Running tests against each gremlin
  6. Result reporting - Displaying mutation scores and survivors

Command-Line Options

Option Default Description
--gremlins False Enable mutation testing
--gremlin-operators All Comma-separated list of operators to use
--gremlin-report console Report format: console, html, json (repeatable)
--gremlin-targets None (auto-discovered) Comma-separated source directories/files
--gremlin-exclude None Glob patterns to exclude from mutation (repeatable)
--gremlin-cache False Enable incremental analysis cache
--gremlin-clear-cache False Clear cache before running
--gremlin-parallel False Enable parallel execution
--gremlin-workers CPU count Number of parallel workers
--gremlin-batch False Enable batch execution mode
--gremlin-batch-size 10 Gremlins per batch
--gremlins-html-dir None Output directory for HTML reports
--strict-pardons False Treat pardoned gremlins as CI failures (exit non-zero if any exist)
--gremlin-audit-pardons False Audit pardon pragma usage
--gremlin-max-pardons-pct None Maximum percentage of pardoned gremlins
--max-pardons None Maximum absolute number of pardoned gremlins

Usage Examples

Basic Usage

Bash
# Enable mutation testing
pytest --gremlins

# Target specific directory
pytest --gremlins --gremlin-targets=mypackage/

# Generate HTML report
pytest --gremlins --gremlin-report=html

With Caching

Bash
# Enable incremental caching (faster subsequent runs)
pytest --gremlins --gremlin-cache

# Clear cache and start fresh
pytest --gremlins --gremlin-cache --gremlin-clear-cache

Parallel Execution

Bash
# Run with parallel workers (auto-detects CPU count)
pytest --gremlins --gremlin-parallel

# Specify worker count
pytest --gremlins --gremlin-parallel --gremlin-workers=8

# Use batch mode for reduced subprocess overhead
pytest --gremlins --gremlin-batch --gremlin-batch-size=20

Selective Operators

Bash
# Use only comparison and boundary operators
pytest --gremlins --gremlin-operators=comparison,boundary

# Use only arithmetic operator
pytest --gremlins --gremlin-operators=arithmetic

Configuration

Configuration can be specified in pyproject.toml under [tool.pytest-gremlins]:

TOML
[tool.pytest-gremlins]
# Mutation operators to enable
operators = ["comparison", "arithmetic", "boolean"]

# Source paths to mutate
paths = ["src/mypackage"]

# Glob patterns to exclude from mutation
exclude = ["**/migrations/**"]

# Number of parallel workers ("auto" or an integer)
workers = "auto"

# Enable incremental analysis cache
cache = true

# Report formats
report = ["html", "json"]

# Gremlins per batch in batch mode
batch_size = 20

# Pardon budget (percentage and absolute cap)
max-pardons-pct = 5.0
max_pardons = 10

Configuration Precedence

  1. CLI arguments (highest priority)
  2. pyproject.toml [tool.pytest-gremlins] section
  3. Built-in defaults (lowest priority)

GremlinConfig

GremlinConfig dataclass

Python
GremlinConfig(operators=None, paths=None, exclude=None, workers=None, cache=None, report=None, batch_size=None, max_pardons_pct=None, max_pardons=None)

Configuration for pytest-gremlins.

All fields are optional and default to None, meaning the plugin will use CLI defaults or built-in defaults.

Attributes:

Name Type Description
operators list[str] | None

List of mutation operator names to enable.

paths list[str] | None

List of paths to scan for source files to mutate.

exclude list[str] | None

List of glob patterns for files to exclude from mutation.

workers int | str | None

Number of parallel workers, "auto" for CPU count, or None.

cache bool | None

Whether to enable incremental analysis cache.

report list[str] | None

List of report formats (e.g. ["html", "json"]).

batch_size int | None

Number of gremlins per batch in batch mode.

operators class-attribute instance-attribute

Python
operators = None

paths class-attribute instance-attribute

Python
paths = None

exclude class-attribute instance-attribute

Python
exclude = None

workers class-attribute instance-attribute

Python
workers = None

cache class-attribute instance-attribute

Python
cache = None

report class-attribute instance-attribute

Python
report = None

batch_size class-attribute instance-attribute

Python
batch_size = None

max_pardons_pct class-attribute instance-attribute

Python
max_pardons_pct = None

max_pardons class-attribute instance-attribute

Python
max_pardons = None

Configuration Functions

load_config

Python
load_config(rootdir)

Load configuration from pyproject.toml.

Reads the [tool.pytest-gremlins] section from pyproject.toml in the given directory. Returns default configuration if the file or section does not exist.

Parameters:

Name Type Description Default
rootdir Path

Directory containing pyproject.toml.

required

Returns:

Type Description
GremlinConfig

GremlinConfig with values from pyproject.toml or defaults.

Source code in src/pytest_gremlins/config.py
Python
def load_config(rootdir: Path) -> GremlinConfig:  # noqa: C901, PLR0912, PLR0915
    """Load configuration from pyproject.toml.

    Reads the [tool.pytest-gremlins] section from pyproject.toml in the
    given directory. Returns default configuration if the file or section
    does not exist.

    Args:
        rootdir: Directory containing pyproject.toml.

    Returns:
        GremlinConfig with values from pyproject.toml or defaults.
    """
    pyproject_path = rootdir / 'pyproject.toml'

    if not pyproject_path.exists():
        return GremlinConfig()

    try:
        with pyproject_path.open('rb') as f:
            pyproject_content = tomllib.load(f)
    except TOMLDecodeError:
        logger.warning('Skipping gremlin config: malformed TOML in %s', pyproject_path)
        return GremlinConfig()

    tool_config = pyproject_content.get('tool', {}).get('pytest-gremlins', {})

    workers_raw = tool_config.get('workers')
    try:
        _resolve_workers(workers_raw)  # validate early; "auto" stays as str, resolved lazily in merge_configs
    except ValueError:
        logger.warning('Invalid workers in %s: got %r', pyproject_path, workers_raw)
        raise

    batch_size_raw = tool_config.get('batch_size')
    if batch_size_raw is not None and (not isinstance(batch_size_raw, int) or isinstance(batch_size_raw, bool)):
        logger.warning('Invalid batch_size in %s: expected positive integer, got %r', pyproject_path, batch_size_raw)
        raise ValueError(
            f'[tool.pytest-gremlins].batch_size must be a positive integer (e.g. batch_size = 50), '
            f'got {batch_size_raw!r}'
        )
    if isinstance(batch_size_raw, int) and not isinstance(batch_size_raw, bool) and batch_size_raw <= 0:
        logger.warning('Invalid batch_size in %s: expected positive integer, got %r', pyproject_path, batch_size_raw)
        raise ValueError(
            f'[tool.pytest-gremlins].batch_size must be a positive integer (e.g. batch_size = 50), '
            f'got {batch_size_raw!r}'
        )

    cache_raw = tool_config.get('cache')
    if cache_raw is not None and not isinstance(cache_raw, bool):
        logger.warning('Invalid cache in %s: expected boolean, got %r', pyproject_path, cache_raw)
        raise ValueError(f'[tool.pytest-gremlins].cache must be a boolean (e.g. cache = true), got {cache_raw!r}')

    report_raw = tool_config.get('report')
    if report_raw is not None:
        if isinstance(report_raw, str):
            report_raw = [fmt.strip() for fmt in report_raw.split(',')]
        elif isinstance(report_raw, list):
            if not all(isinstance(fmt, str) for fmt in report_raw):
                logger.warning('Invalid report in %s: expected list of strings, got %r', pyproject_path, report_raw)
                raise ValueError(
                    f'[tool.pytest-gremlins].report must be a string or list of strings '
                    f'(e.g. report = "html" or report = ["html", "json"]), got {report_raw!r}'
                )
            report_raw = [fmt.strip() for fmt in report_raw]
        else:
            logger.warning('Invalid report in %s: expected string or list, got %r', pyproject_path, report_raw)
            raise ValueError(
                f'[tool.pytest-gremlins].report must be a string or list of strings '
                f'(e.g. report = "html" or report = ["html", "json"]), got {report_raw!r}'
            )
        # Filter empty strings (trailing/leading/double commas) and deduplicate
        seen: set[str] = set()
        deduped: list[str] = []
        for fmt in report_raw:
            if fmt and fmt not in seen:
                seen.add(fmt)
                deduped.append(fmt)
        report_raw = deduped
        if not report_raw:
            raise ValueError(
                '[tool.pytest-gremlins].report must contain at least one valid format '
                f'(e.g. report = "html"). Valid formats: {sorted(VALID_REPORT_FORMATS)}'
            )
        invalid = set(report_raw) - VALID_REPORT_FORMATS
        if invalid:
            raise ValueError(
                f'Unknown report format(s): {sorted(invalid)}. Valid formats: {sorted(VALID_REPORT_FORMATS)}'
            )

    max_pardons_pct_raw = tool_config.get('max-pardons-pct')
    if max_pardons_pct_raw is not None:
        if isinstance(max_pardons_pct_raw, bool) or not isinstance(max_pardons_pct_raw, (int, float)):
            logger.warning(
                'Invalid max-pardons-pct in %s: expected float in [0, 100], got %r',
                pyproject_path,
                max_pardons_pct_raw,
            )
            raise ValueError(
                f'[tool.pytest-gremlins].max-pardons-pct must be a number between 0 and 100 '
                f'(e.g. max-pardons-pct = 5.0), got {max_pardons_pct_raw!r}'
            )
        if not (0 <= max_pardons_pct_raw <= 100):  # noqa: PLR2004
            logger.warning(
                'Invalid max-pardons-pct in %s: expected float in [0, 100], got %r',
                pyproject_path,
                max_pardons_pct_raw,
            )
            raise ValueError(
                f'[tool.pytest-gremlins].max-pardons-pct must be between 0 and 100 '
                f'(e.g. max-pardons-pct = 5.0), got {max_pardons_pct_raw!r}'
            )

    max_pardons_raw = tool_config.get('max_pardons')
    if max_pardons_raw is not None:
        if isinstance(max_pardons_raw, bool) or not isinstance(max_pardons_raw, int):
            logger.warning(
                'Invalid max_pardons in %s: expected non-negative integer, got %r',
                pyproject_path,
                max_pardons_raw,
            )
            raise ValueError(
                f'[tool.pytest-gremlins].max_pardons must be a non-negative integer '
                f'(e.g. max_pardons = 5), got {max_pardons_raw!r}'
            )
        if max_pardons_raw < 0:
            logger.warning(
                'Invalid max_pardons in %s: expected non-negative integer, got %r',
                pyproject_path,
                max_pardons_raw,
            )
            raise ValueError(
                f'[tool.pytest-gremlins].max_pardons must be >= 0 (e.g. max_pardons = 5), got {max_pardons_raw!r}'
            )

    return GremlinConfig(
        operators=tool_config.get('operators'),
        paths=tool_config.get('paths'),
        exclude=tool_config.get('exclude'),
        workers=workers_raw,
        cache=cache_raw,
        report=report_raw,
        batch_size=batch_size_raw,
        max_pardons_pct=max_pardons_pct_raw,
        max_pardons=max_pardons_raw,
    )

merge_configs

Python
merge_configs(file_config, cli_operators=None, cli_targets=None, cli_exclude=None, cli_workers=None, cli_cache=None, cli_report=None, cli_batch_size=None, cli_max_pardons_pct=None, cli_max_pardons=None)

Merge CLI arguments with file configuration.

CLI arguments take precedence over pyproject.toml configuration. Empty strings are treated as not provided.

Parameters:

Name Type Description Default
file_config GremlinConfig

Configuration loaded from pyproject.toml.

required
cli_operators str | None

Comma-separated operator names from CLI (--gremlin-operators).

None
cli_targets str | None

Comma-separated target paths from CLI (--gremlin-targets).

None
cli_exclude list[str] | None

Glob patterns from CLI (--gremlin-exclude, repeatable).

None
cli_workers int | None

Worker count from CLI (--gremlin-workers), already resolved to int.

None
cli_cache bool | None

Cache flag from CLI (--gremlin-cache).

None
cli_report list[str] | None

Report format list from CLI (--gremlin-report).

None
cli_batch_size int | None

Batch size from CLI (--gremlin-batch-size).

None
cli_max_pardons_pct float | None

Max pardoned % from CLI (--gremlin-max-pardons-pct).

None
cli_max_pardons int | None

Max absolute pardon count from CLI (--max-pardons).

None

Returns:

Type Description
GremlinConfig

GremlinConfig with CLI values overriding file config where provided.

Source code in src/pytest_gremlins/config.py
Python
def merge_configs(
    file_config: GremlinConfig,
    cli_operators: str | None = None,
    cli_targets: str | None = None,
    cli_exclude: list[str] | None = None,
    cli_workers: int | None = None,
    cli_cache: bool | None = None,
    cli_report: list[str] | None = None,
    cli_batch_size: int | None = None,
    cli_max_pardons_pct: float | None = None,
    cli_max_pardons: int | None = None,
) -> GremlinConfig:
    """Merge CLI arguments with file configuration.

    CLI arguments take precedence over pyproject.toml configuration.
    Empty strings are treated as not provided.

    Args:
        file_config: Configuration loaded from pyproject.toml.
        cli_operators: Comma-separated operator names from CLI (--gremlin-operators).
        cli_targets: Comma-separated target paths from CLI (--gremlin-targets).
        cli_exclude: Glob patterns from CLI (--gremlin-exclude, repeatable).
        cli_workers: Worker count from CLI (--gremlin-workers), already resolved to int.
        cli_cache: Cache flag from CLI (--gremlin-cache).
        cli_report: Report format list from CLI (--gremlin-report).
        cli_batch_size: Batch size from CLI (--gremlin-batch-size).
        cli_max_pardons_pct: Max pardoned % from CLI (--gremlin-max-pardons-pct).
        cli_max_pardons: Max absolute pardon count from CLI (--max-pardons).

    Returns:
        GremlinConfig with CLI values overriding file config where provided.
    """
    operators: list[str] | None = None
    if cli_operators and cli_operators.strip():
        operators = [op.strip() for op in cli_operators.split(',')]
    elif file_config.operators is not None:
        operators = file_config.operators

    paths: list[str] | None = None
    if cli_targets and cli_targets.strip():
        paths = [p.strip() for p in cli_targets.split(',')]
    elif file_config.paths is not None:
        paths = file_config.paths

    exclude: list[str] | None = cli_exclude if cli_exclude is not None else file_config.exclude

    workers: int | None = cli_workers if cli_workers is not None else _resolve_workers(file_config.workers)
    cache: bool | None = cli_cache if cli_cache is not None else file_config.cache
    report: list[str] | None = cli_report if cli_report is not None else file_config.report
    batch_size: int | None = cli_batch_size if cli_batch_size is not None else file_config.batch_size
    max_pardons_pct: float | None = (
        cli_max_pardons_pct if cli_max_pardons_pct is not None else file_config.max_pardons_pct
    )
    max_pardons: int | None = cli_max_pardons if cli_max_pardons is not None else file_config.max_pardons

    return GremlinConfig(
        operators=operators,
        paths=paths,
        exclude=exclude,
        workers=workers,
        cache=cache,
        report=report,
        batch_size=batch_size,
        max_pardons_pct=max_pardons_pct,
        max_pardons=max_pardons,
    )

GremlinSession

The GremlinSession dataclass maintains state throughout a mutation testing run.

GremlinSession dataclass

Python
GremlinSession(enabled=False, operators=list(), report_formats=(lambda: ['console'])(), gremlins=list(), results=list(), source_files=dict(), test_files=list(), target_paths=list(), instrumented_dir=None, coverage_collector=None, test_selector=None, prioritized_selector=None, test_node_ids=dict(), total_tests=0, cache_enabled=False, cache=None, source_hashes=dict(), test_hashes=dict(), cache_hits=0, cache_misses=0, parallel_enabled=False, parallel_workers=None, batch_enabled=False, batch_size=10, xdist_item_ids=None, xdist_active=False, xdist_workers=None, coverage_mode=PRIVATE, private_coverage=None, gremlins_tmpdir=None, exclude_patterns=list(), strict_pardons=False, audit_pardons=False, max_pardons_pct=None, max_pardons=None, no_coverage_filter=False, test_name_to_node_ids=dict(), explain_gremlin_id=None, preserved_addopts='')

Session state for mutation testing.

Attributes:

Name Type Description
enabled bool

Whether mutation testing is enabled.

operators list[GremlinOperator]

List of operators to use for mutation.

report_formats list[str]

List of report formats (console, html, json).

gremlins list[Gremlin]

All gremlins found in the source code.

results list[GremlinResult]

Results from testing each gremlin.

source_files dict[str, str]

Mapping of file paths to their source code.

test_files list[Path]

List of test file paths that were collected.

instrumented_dir Path | None

Temporary directory containing instrumented source files.

coverage_collector CoverageCollector | None

Collects coverage data per-test.

test_selector TestSelector | None

Selects tests based on coverage data.

prioritized_selector PrioritizedSelector | None

Selects tests ordered by specificity (most specific first).

test_node_ids dict[str, str]

Maps test names to their pytest node IDs.

total_tests int

Total number of tests collected.

cache_enabled bool

Whether incremental caching is enabled.

cache IncrementalCache | None

The incremental cache instance (if caching is enabled).

source_hashes dict[str, str]

Content hashes for source files.

test_hashes dict[str, str]

Content hashes for test files.

cache_hits int

Number of cache hits in this session.

cache_misses int

Number of cache misses in this session.

parallel_enabled bool

Whether parallel execution is enabled.

parallel_workers int | None

Number of parallel workers (None = CPU count).

batch_enabled bool

Whether batch execution mode is enabled.

batch_size int

Number of gremlins per batch in batch mode.

xdist_item_ids list[str] | None

Test node IDs captured from the first xdist worker after collection finishes. None until the hook fires; [] if the worker collected nothing.

coverage_mode CoverageMode

Whether to reuse pytest-cov's coverage (PIGGYBACK) or manage an inline coverage instance (PRIVATE).

private_coverage Coverage | None

The inline coverage.Coverage instance used in PRIVATE mode. None in PIGGYBACK mode or before session start.

gremlins_tmpdir str | None

Path (as a string) to the shared temporary directory where xdist workers write their per-worker coverage data files in PRIVATE mode. None when xdist is not active.

exclude_patterns list[str]

Glob patterns from [tool.pytest-gremlins] exclude used to skip matching files during source discovery.

test_name_to_node_ids dict[str, list[str]]

Reverse index mapping bare function names to their full pytest node IDs. Built at session setup for O(1) lookup when resolving coverage contexts.

preserved_addopts str

The project's pytest addopts with pytest-cov flags stripped (see :func:_addopts_without_cov), threaded into the subprocess runs as -o addopts=<...> so collection-affecting options such as --import-mode=importlib survive (issue #424). '' clears all addopts.

Session Attributes

Attribute Type Description
enabled bool Whether mutation testing is active
operators list[GremlinOperator] Active mutation operators
report_formats list[str] Output formats (console/html/json)
gremlins list[Gremlin] All discovered gremlins
results list[GremlinResult] Test results for each gremlin
source_files dict[str, str] Map of file paths to source code
test_files list[Path] Collected test file paths
target_paths list[Path] Source paths to mutate
instrumented_dir Path \| None Temp directory with instrumented code
coverage_collector CoverageCollector \| None Coverage data collector
test_selector TestSelector \| None Coverage-based test selector
prioritized_selector PrioritizedSelector \| None Priority-ordered selector
test_node_ids dict[str, str] Map of test names to pytest node IDs
total_tests int Total number of collected tests
cache_enabled bool Whether caching is active
cache IncrementalCache \| None The cache instance
source_hashes dict[str, str] Content hashes for source files
test_hashes dict[str, str] Content hashes for test files
cache_hits int Number of cache hits
cache_misses int Number of cache misses
parallel_enabled bool Whether parallel mode is active
parallel_workers int \| None Number of workers (None = auto)
batch_enabled bool Whether batch mode is active
batch_size int Gremlins per batch
xdist_item_ids list[str] \| None Test IDs captured from xdist workers
xdist_active bool Whether xdist is active
xdist_workers int \| None Number of xdist workers
coverage_mode CoverageMode PIGGYBACK (reuse pytest-cov) or PRIVATE
private_coverage coverage.Coverage \| None Inline coverage instance (PRIVATE mode)
gremlins_tmpdir str \| None Shared temp dir for xdist worker coverage data
exclude_patterns list[str] Glob patterns to skip during source discovery
strict_pardons bool Treat pardoned gremlins as CI failures (exit non-zero if any exist)
audit_pardons bool Whether to audit pardon pragma usage
max_pardons_pct float \| None Maximum percentage of pardoned gremlins
max_pardons int \| None Maximum absolute pardon count

pytest Hooks

The plugin implements these pytest hooks:

pytest_addoption

Adds command-line options for mutation testing configuration.

Python
def pytest_addoption(parser: pytest.Parser) -> None:
    """Add command-line options for pytest-gremlins."""

pytest_configure

Initializes the gremlin session based on command-line options and pyproject.toml. When xdist is available, also registers pytest_configure_node and pytest_xdist_node_collection_finished hooks.

Python
def pytest_configure(config: pytest.Config) -> None:
    """Configure pytest-gremlins based on command-line options."""

pytest_sessionstart

Sets up inline coverage collection in PRIVATE mode (when --cov is not active).

Python
def pytest_sessionstart(session: pytest.Session) -> None:
    """Start inline coverage collection if needed."""

pytest_runtestloop

Hookimpl wrapper that saves and stops coverage data after the test loop completes.

Python
def pytest_runtestloop(session: pytest.Session) -> Generator[None, None, None]:
    """Wrap the test loop to capture coverage data."""

pytest_collection_finish

After test collection completes, discovers source files and generates gremlins.

Python
def pytest_collection_finish(session: pytest.Session) -> None:
    """After test collection, discover source files and generate gremlins."""

pytest_configure_node (xdist only)

Passes gremlin temp directory path to xdist workers via node.workerinput.

Python
def pytest_configure_node(node: _XdistWorkerNode) -> None:
    """Pass gremlins config to xdist worker nodes."""

pytest_xdist_node_collection_finished (xdist only)

Captures test node IDs from the first xdist worker's collection.

Python
def pytest_xdist_node_collection_finished(node: object, ids: list[str]) -> None:
    """Capture collected test IDs from xdist workers."""

pytest_sessionfinish

After all tests run, executes mutation testing against each gremlin.

Python
def pytest_sessionfinish(session: pytest.Session, exitstatus: int) -> None:
    """After all tests run, execute mutation testing."""

pytest_terminal_summary

Adds mutation testing results to pytest's terminal output.

Python
def pytest_terminal_summary(
    terminalreporter: pytest.TerminalReporter,
    exitstatus: int,
    config: pytest.Config,
) -> None:
    """Add mutation testing results to terminal output."""

pytest_unconfigure

Cleans up temporary files and closes resources.

Python
def pytest_unconfigure(config: pytest.Config) -> None:
    """Clean up after pytest-gremlins."""

Environment Variables

Variable Description
ACTIVE_GREMLIN Set by plugin to indicate which gremlin is active during test execution
PYTEST_GREMLINS_SOURCES_FILE Path to JSON file containing instrumented source code

Internal Functions

These functions are internal to the plugin but documented for understanding the implementation.

Source Discovery

Python
def _discover_source_files(
    session: pytest.Session,
    gremlin_session: GremlinSession,
) -> dict[str, str]:
    """Discover Python source files to mutate."""

Test Selection

Python
def _select_tests_for_gremlin_prioritized(
    gremlin: Gremlin,
    gremlin_session: GremlinSession,
) -> list[str]:
    """Select tests for a gremlin, ordered by specificity."""

Mutation Testing Execution

Python
def _run_mutation_testing(
    session: pytest.Session,
    gremlin_session: GremlinSession,
) -> list[GremlinResult]:
    """Run mutation testing for all gremlins (sequential mode)."""

def _run_parallel_mutation_testing(
    session: pytest.Session,
    gremlin_session: GremlinSession,
) -> list[GremlinResult]:
    """Run mutation testing in parallel across multiple workers."""

def _run_batch_mutation_testing(
    session: pytest.Session,
    gremlin_session: GremlinSession,
) -> list[GremlinResult]:
    """Run mutation testing using batch execution."""