Skip to content

Reports

pytest-gremlins generates reports in multiple formats to help you understand mutation testing results and take action on surviving gremlins.

Report Formats Overview

Format Use Case Output
console Quick feedback, local development Terminal output
html Detailed analysis, code review coverage/gremlins/index.html
json CI integration, custom tooling coverage/gremlins/gremlins.json

Console Report (Default)

The console report provides a quick summary directly in your terminal.

Enabling Console Report

Console is the default format:

Bash
pytest --gremlins

Or explicitly:

Bash
pytest --gremlins --gremlin-report=console

Example Output

Text Only
================== pytest-gremlins mutation report ==================

Zapped: 142 gremlins (85%)
Survived: 18 gremlins (11%)
Timeout: 5 gremlins (3%)
Error: 2 gremlins (1%)

Top surviving gremlins:
  src/auth.py:42    >= to >     (boundary not tested)
  src/utils.py:17   + to -      (arithmetic not verified)
  src/api.py:88     True to False (return value unchecked)

Run with --gremlin-report=html for detailed report.
=====================================================================

Timeout, Error, and Pardoned lines only appear when their count is greater than zero.

Console Output Sections

Summary Section:

Field Description
Zapped Number and percentage of gremlins caught by tests
Survived Number and percentage of gremlins that escaped tests
Timeout Number and percentage of gremlins that caused test timeouts (shown when > 0)
Error Number and percentage of gremlins that caused errors (shown when > 0)
Pardoned Number of gremlins pardoned via inline pragma, excluded from score (shown when > 0)

Top Surviving Gremlins:

Shows the most important surviving gremlins to fix. Each line contains:

  • Location - File path and line number
  • Mutation - What changed (e.g., >= -> >)
  • Operator - Which operator created this gremlin

Cache Statistics (when caching enabled):

Text Only
Cache: 85 hits, 15 misses (85% hit rate)

When to Use Console Report

  • During local development for quick feedback
  • In CI logs for basic pass/fail information
  • When you need immediate visibility without file output

HTML Report

The HTML report provides a detailed, visual representation of mutation testing results.

Enabling HTML Report

Bash
pytest --gremlins --gremlin-report=html

Output Location

By default, the HTML report is written to:

Text Only
coverage/gremlins/index.html

The location is shown in the console output:

Text Only
HTML report written to: /path/to/project/coverage/gremlins/index.html

Report Contents

The HTML report includes:

Summary Dashboard:

  • Total gremlins tested
  • Zapped count (with percentage)
  • Survived count (with percentage)
  • Timeout count (with percentage)
  • Error count (with percentage)
  • Overall mutation score

Results Table:

Column Description
File Source file path
Line Line number in source
Operator Operator that created the gremlin
Description Human-readable mutation description
Status zapped, survived, timeout, or error

Status Color Coding:

Status Color Meaning
zapped Green Test caught the mutation
survived Red Test missed the mutation
timeout Orange Mutation caused test timeout
error Purple Mutation caused an error

Example HTML Report Structure

HTML
<!DOCTYPE html>
<html>
<head>
    <title>pytest-gremlins Mutation Report</title>
    <!-- Embedded CSS for standalone viewing -->
</head>
<body>
    <div class="container">
        <h1>pytest-gremlins Mutation Report</h1>

        <!-- Summary cards -->
        <div class="summary">
            <div class="stat-card">
                <div class="stat-value">160</div>
                <div class="stat-label">Total Gremlins</div>
            </div>
            <div class="stat-card stat-zapped">
                <div class="stat-value">142</div>
                <div class="stat-label">Zapped</div>
            </div>
            <div class="stat-card stat-survived">
                <div class="stat-value">18</div>
                <div class="stat-label">Survived</div>
            </div>
            <div class="stat-card">
                <div class="stat-value">89%</div>
                <div class="stat-label">Mutation Score</div>
            </div>
        </div>

        <!-- Results table -->
        <table>
            <thead>
                <tr>
                    <th>File</th>
                    <th>Line</th>
                    <th>Operator</th>
                    <th>Description</th>
                    <th>Status</th>
                </tr>
            </thead>
            <tbody>
                <tr>
                    <td>src/auth.py</td>
                    <td>42</td>
                    <td>comparison</td>
                    <td>>= -> ></td>
                    <td class="status-survived">survived</td>
                </tr>
                <!-- More rows... -->
            </tbody>
        </table>
    </div>
</body>
</html>

Viewing the HTML Report

The HTML report is self-contained with embedded CSS. Open it directly in any web browser:

Bash
# macOS
open coverage/gremlins/index.html

# Linux
xdg-open coverage/gremlins/index.html

# Windows
start coverage/gremlins/index.html

Trend Chart

When you run mutation testing more than once with --gremlin-report=html, the HTML report includes a line chart showing your mutation score over time. The chart renders after two or more runs.

Behind the scenes, append_history_entry writes a timestamped score record to a history file alongside the HTML report each time you generate one. On the next run, load_history reads previous entries and passes them to the chart renderer. If the history file is missing or corrupted, the report still generates -- you just won't see the chart until the next run rebuilds the file.

This is useful for tracking whether your mutation score is trending up or down as your project evolves, without needing an external dashboard.

Accessibility

The HTML report meets WCAG 2.1 AA standards:

  • Contrast ratios pass for all status colors against both light and dark backgrounds
  • Keyboard navigation works for expanding file sections and navigating the results table
  • Expand-all handles overflow gracefully so the page remains usable with many files

When to Use HTML Report

  • Detailed analysis of mutation testing results
  • Code review discussions
  • Sharing results with team members
  • Tracking mutation score trends across runs

JSON Report

The JSON report provides machine-readable output for CI/CD integration and custom tooling.

Enabling JSON Report

Bash
pytest --gremlins --gremlin-report=json

Output Location

The JSON report is written to:

Text Only
coverage/gremlins/gremlins.json

The directory is created automatically if it does not exist.

JSON Schema

JSON
{
  "summary": {
    "total": 160,
    "zapped": 142,
    "survived": 18,
    "timeout": 0,
    "error": 0,
    "pardoned": 0,
    "percentage": 88.75
  },
  "files": {
    "src/auth.py": {
      "total": 80,
      "zapped": 72,
      "survived": 8,
      "percentage": 90.0
    },
    "src/utils.py": {
      "total": 80,
      "zapped": 70,
      "survived": 10,
      "percentage": 87.5
    }
  },
  "results": [
    {
      "gremlin_id": "g001",
      "file_path": "src/auth.py",
      "line_number": 42,
      "status": "survived",
      "operator": "comparison",
      "description": ">= -> >"
    },
    {
      "gremlin_id": "g002",
      "file_path": "src/utils.py",
      "line_number": 17,
      "status": "zapped",
      "operator": "arithmetic",
      "description": "+ -> -",
      "killing_test": "test_utils.py::test_calculate"
    }
  ]
}

Field Reference

Summary Object:

Field Type Description
total integer Total number of gremlins tested
zapped integer Number of gremlins caught by tests
survived integer Number of gremlins that escaped
timeout integer Number of gremlins that caused timeouts
error integer Number of gremlins that caused errors
pardoned integer Number of gremlins pardoned via inline pragma
percentage float Mutation score percentage (0-100)

Files Object:

A mapping of file paths to per-file statistics:

Field Type Description
total integer Total gremlins in this file
zapped integer Gremlins caught in this file
survived integer Gremlins that escaped in this file
percentage float Mutation score for this file (0-100)

Results Array (Gremlin Objects):

Field Type Description
gremlin_id string Unique identifier for this gremlin
file_path string Source file path
line_number integer Line number in source
status string One of: zapped, survived, timeout, error, pardoned
operator string Operator that created this gremlin
description string Human-readable mutation description
killing_test string Test that caught the mutation (only present if zapped)

Processing JSON Reports

Using jq to extract information:

Bash
# Get mutation score
jq '.summary.percentage' coverage/gremlins/gremlins.json

# List surviving gremlins
jq '.results[] | select(.status == "survived") | "\(.file_path):\(.line_number) - \(.description)"' coverage/gremlins/gremlins.json

# Count gremlins by operator
jq '.results | group_by(.operator) | map({operator: .[0].operator, count: length})' coverage/gremlins/gremlins.json

# Get per-file breakdown
jq '.files | to_entries[] | "\(.key): \(.value.percentage)%"' coverage/gremlins/gremlins.json

Python script example:

Python
import json

with open('coverage/gremlins/gremlins.json') as f:
    report = json.load(f)

print(f"Mutation Score: {report['summary']['percentage']:.1f}%")

survivors = [g for g in report['results'] if g['status'] == 'survived']
print(f"\nSurviving gremlins ({len(survivors)}):")
for g in survivors:
    print(f"  {g['file_path']}:{g['line_number']} - {g['description']}")

When to Use JSON Report

  • CI/CD pipeline integration
  • Custom reporting tools
  • Historical trend analysis
  • Automated quality gates

Multiple Report Formats

Generate multiple formats in a single run using either syntax:

Comma-separated (v1.5.1+):

Bash
pytest --gremlins --gremlin-report=json,html

Repeated flags:

Bash
pytest --gremlins --gremlin-report=json --gremlin-report=html

pyproject.toml:

TOML
[tool.pytest-gremlins]
# List syntax
report = ["json", "html"]

# Or comma-separated string
report = "json,html"

Console output always renders regardless of which formats you specify -- you do not need to include console in the list.

Duplicate formats are deduplicated automatically, and trailing commas are ignored.

Interpreting Results

Understanding Mutation Score

The mutation score is calculated as:

Text Only
score = (zapped + timeout) / (total - pardoned) * 100

Pardoned gremlins are excluded from the denominator because they represent accepted exceptions (equivalent mutants, untestable paths, or out-of-scope code). This means pardoning a gremlin does not artificially inflate your score -- it removes the gremlin from the scoring pool entirely.

Timeouts count as "zapped" because the mutation was detected (it caused the test to hang).

Score Guidelines:

Score Interpretation Action
< 50% Poor coverage Focus on adding basic tests
50-70% Below average Add tests for surviving gremlins
70-85% Good Target specific gaps
85-95% Very good Diminishing returns territory
> 95% Excellent May have equivalent mutants

Prioritizing Survivors

Not all surviving gremlins are equally important. Prioritize by:

  1. Security-critical code - Authentication, authorization, validation
  2. Business-critical code - Payments, data integrity
  3. Operator type - boolean and comparison often catch real bugs
  4. Code complexity - More complex functions need better coverage

Analyzing Patterns

Look for patterns in surviving gremlins:

Pattern: Many boundary survivors

Text Only
src/validation.py:12   >= -> >   (comparison)
src/validation.py:15   <= -> <   (comparison)
src/validation.py:23   >= -> >   (comparison)

Action: Add boundary value tests for validation functions.

Pattern: Many return survivors

Text Only
src/service.py:45   return x -> return None   (return)
src/service.py:67   return x -> return None   (return)

Action: Tests are not asserting on return values.

Pattern: Boolean logic survivors

Text Only
src/auth.py:30   and -> or   (boolean)

Action: Test all condition combinations in authorization.

CI Integration Examples

GitHub Actions with Score Threshold

YAML
name: Mutation Testing

on: [push, pull_request]

jobs:
  mutation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install uv
        uses: astral-sh/setup-uv@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: uv sync

      - name: Run mutation testing
        run: |
          uv run pytest --gremlins --gremlin-report=json

      - name: Check mutation score
        run: |
          SCORE=$(jq '.summary.percentage' coverage/gremlins/gremlins.json)
          echo "Mutation score: $SCORE%"
          if (( $(echo "$SCORE < 80" | bc -l) )); then
            echo "::error::Mutation score $SCORE% is below threshold 80%"
            exit 1
          fi

      - name: Upload mutation report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: mutation-report
          path: coverage/gremlins/

GitLab CI with Artifacts

YAML
mutation_testing:
  stage: test
  script:
    - pip install uv && uv sync
    - uv run pytest --gremlins --gremlin-report=console,html,json
    - |
      SCORE=$(jq '.summary.percentage' coverage/gremlins/gremlins.json)
      echo "Mutation score: $SCORE%"
      if (( $(echo "$SCORE < 80" | bc -l) )); then
        echo "Mutation score below threshold"
        exit 1
      fi
  artifacts:
    when: always
    paths:
      - coverage/gremlins/
    reports:
      junit: coverage/gremlins/gremlins.json

Jenkins Pipeline

Groovy
pipeline {
    agent any

    stages {
        stage('Mutation Testing') {
            steps {
                sh 'pip install uv && uv sync'
                sh 'uv run pytest --gremlins --gremlin-report=json,html'

                script {
                    def report = readJSON file: 'coverage/gremlins/gremlins.json'
                    def score = report.summary.percentage

                    echo "Mutation Score: ${score}%"

                    if (score < 80) {
                        error "Mutation score ${score}% is below threshold 80%"
                    }
                }
            }
            post {
                always {
                    archiveArtifacts artifacts: 'coverage/gremlins/**'
                    publishHTML([
                        allowMissing: false,
                        alwaysLinkToLastBuild: true,
                        keepAll: true,
                        reportDir: 'coverage/gremlins',
                        reportFiles: 'index.html',
                        reportName: 'Mutation Testing Report'
                    ])
                }
            }
        }
    }
}

Troubleshooting Reports

No Report Generated

If no report is generated:

  1. Ensure --gremlins flag is present
  2. Check that source files were found
  3. Verify tests exist and pass normally

Empty Report

If the report shows zero gremlins:

  1. Check --gremlin-targets points to source files
  2. Verify files contain mutable code
  3. Check exclude patterns are not too broad

HTML Report Not Rendering

If HTML report looks broken:

  1. Open in a modern browser
  2. Check file is not truncated
  3. Ensure no special characters in file paths

JSON Parse Errors

If JSON report fails to parse:

  1. Check for incomplete writes (disk full)
  2. Verify encoding (should be UTF-8)
  3. Look for control characters in source paths

Exporting to External Services

pytest-gremlins can export mutation testing results to external code quality platforms.

Stryker Dashboard Export

The Stryker Dashboard is a free service for hosting mutation testing reports. pytest-gremlins exports results in the standardized mutation-testing-report-schema format.

Using StrykerExporter

Python
from pytest_gremlins.reporting import MutationScore, StrykerExporter
from pathlib import Path

# After running mutation testing, get your score
score: MutationScore = ...  # from test execution

# Create exporter
exporter = StrykerExporter()

# Write full report (for detailed dashboard display)
exporter.write_report(score, Path('mutation.json'))

# Or generate simple score-only format (for badge display)
simple_json = exporter.to_score_only_json(score)
Path('mutation-score.json').write_text(simple_json)

Uploading to Stryker Dashboard

  1. Enable repository on dashboard.stryker-mutator.io
  2. Get your API key from the dashboard settings
  3. Upload report via HTTP PUT:
Bash
curl -X PUT \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: $STRYKER_DASHBOARD_API_KEY" \
  --data-binary @mutation.json \
  "https://dashboard.stryker-mutator.io/api/reports/github.com/$OWNER/$REPO/$BRANCH"

GitHub Actions for Stryker Dashboard

YAML
name: Mutation Testing

on: [push, pull_request]

jobs:
  mutation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install uv
        uses: astral-sh/setup-uv@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: uv sync

      - name: Run mutation testing
        run: uv run pytest --gremlins --gremlin-report=json

      - name: Convert to Stryker format
        run: |
          python -c "
          from pytest_gremlins.reporting import MutationScore, StrykerExporter
          import json
          from pathlib import Path

          # Load pytest-gremlins output
          data = json.loads(Path('coverage/gremlins/gremlins.json').read_text())

          # Note: This requires creating MutationScore from the JSON data
          # In practice, you would save the Stryker format during test execution
          "

      - name: Upload to Stryker Dashboard
        if: github.ref == 'refs/heads/main'
        run: |
          curl -X PUT \
            -H "Content-Type: application/json" \
            -H "X-Api-Key: ${{ secrets.STRYKER_DASHBOARD_API_KEY }}" \
            --data-binary @mutation.json \
            "https://dashboard.stryker-mutator.io/api/reports/github.com/${{ github.repository }}/${{ github.ref_name }}"

Mutation Score Badge

After uploading to Stryker Dashboard, add this badge to your README:

Markdown
[![Mutation Score](https://img.shields.io/endpoint?style=flat&url=https%3A%2F%2Fbadge-api.stryker-mutator.io%2Fgithub.com%2FOWNER%2FREPO%2Fmain)](https://dashboard.stryker-mutator.io/reports/github.com/OWNER/REPO/main)

SonarQube Export

SonarQube can import surviving mutants as external issues.

Using SonarQubeExporter

Python
from pytest_gremlins.reporting import MutationScore, SonarQubeExporter
from pathlib import Path

# After running mutation testing
score: MutationScore = ...

# Create exporter (optionally specify project root for path normalization)
exporter = SonarQubeExporter(
    project_root='/path/to/project',  # paths will be relative to this
    severity='MAJOR',  # BLOCKER, CRITICAL, MAJOR, MINOR, INFO
    effort_minutes=10,  # estimated time to fix each issue
)

# Write report
exporter.write_report(score, Path('mutation-sonar.json'))

SonarQube Import

Add the report path to your SonarQube analysis:

Bash
sonar-scanner \
  -Dsonar.externalIssuesReportPaths=mutation-sonar.json \
  -Dsonar.projectKey=my-project

What Gets Imported

Only survived mutants are imported as issues:

Field Value
Engine ID pytest-gremlins
Rule ID mutant-survived-{operator}
Severity MAJOR (configurable)
Type CODE_SMELL
Effort 10 minutes (configurable)

GitHub Actions for SonarQube

YAML
name: Mutation Testing + SonarQube

on: [push, pull_request]

jobs:
  mutation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # SonarQube needs full history

      - name: Install uv
        uses: astral-sh/setup-uv@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: uv sync

      - name: Run mutation testing
        run: uv run pytest --gremlins --gremlin-report=json

      - name: Convert to SonarQube format
        run: |
          python -c "
          from pytest_gremlins.reporting import MutationScore, SonarQubeExporter
          import json
          from pathlib import Path

          # Note: Full implementation would create MutationScore from results
          # exporter = SonarQubeExporter(project_root='.')
          # exporter.write_report(score, Path('mutation-sonar.json'))
          "

      - name: SonarQube Scan
        uses: sonarsource/sonarqube-scan-action@master
        env:
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
          SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}
        with:
          args: >
            -Dsonar.externalIssuesReportPaths=mutation-sonar.json

      - name: SonarQube Quality Gate
        uses: sonarsource/sonarqube-quality-gate-action@master
        timeout-minutes: 5
        env:
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}

Export Format Reference

Stryker Dashboard Format (mutation-testing-report-schema)

JSON
{
  "schemaVersion": "1.0",
  "thresholds": {
    "high": 80,
    "low": 60
  },
  "files": {
    "src/auth.py": {
      "language": "python",
      "mutants": [
        {
          "id": "g001",
          "mutatorName": "comparison",
          "location": {
            "start": {"line": 42, "column": 8},
            "end": {"line": 42, "column": 14}
          },
          "status": "Killed",
          "killedBy": ["test_auth.py::test_login"],
          "description": ">= to >",
          "duration": 45
        }
      ]
    }
  },
  "framework": {
    "name": "pytest-gremlins",
    "version": "1.0.0"
  }
}

Status values:

  • Killed - Test caught the mutation (gremlin zapped)
  • Survived - Mutation not detected (gremlin survived)
  • Timeout - Test timed out
  • RuntimeError - Mutation caused an error

SonarQube Generic Issue Format

JSON
{
  "issues": [
    {
      "engineId": "pytest-gremlins",
      "ruleId": "mutant-survived-comparison",
      "severity": "MAJOR",
      "type": "CODE_SMELL",
      "effortMinutes": 10,
      "primaryLocation": {
        "filePath": "src/auth.py",
        "textRange": {
          "startLine": 42
        },
        "message": "Mutant survived: >= to >"
      }
    }
  ]
}

Combining Multiple Exports

Generate multiple formats in your CI workflow:

```yaml - name: Run mutation testing with all exports run: | pytest --gremlins --gremlin-report=json,html

Text Only
# Generate Stryker format
python scripts/export_stryker.py coverage/gremlins/gremlins.json mutation.json

# Generate SonarQube format
python scripts/export_sonarqube.py coverage/gremlins/gremlins.json mutation-sonar.json

  • name: Upload to Stryker Dashboard run: curl -X PUT ... @mutation.json

  • name: SonarQube Scan run: sonar-scanner -Dsonar.externalIssuesReportPaths=mutation-sonar.json