Running Tests¶
Complete guide for running mcp-scan tests.
Quick Commands¶
# All tests
make test
# Tests with verbose
make test-verbose
# Tests with coverage
make test-coverage
# Golden tests (snapshots)
make test-golden
# Detection tests against DVMCP
make test-detection
# Benchmarks
make bench
Unit Tests¶
Run all tests¶
Run tests for a specific package¶
# Parser
go test ./internal/parser/...
# Pattern engine
go test ./internal/pattern/...
# Taint analysis
go test ./internal/taint/...
# Scanner
go test ./pkg/scanner/...
# ML classifier
go test ./internal/ml/...
Run a specific test¶
# By name
go test ./internal/pattern/... -run TestPatternEngine_PromptInjection
# With verbose
go test -v ./internal/pattern/... -run TestPatternEngine_PromptInjection
# With detailed failure output
go test -v ./internal/pattern/... -run TestPatternEngine_PromptInjection -failfast
Run tests matching a pattern¶
# All SQL tests
go test ./... -run ".*SQL.*"
# All classifier tests
go test ./... -run ".*Classifier.*"
# All taint tests
go test ./... -run ".*Taint.*"
Tests with Coverage¶
Generate coverage report¶
# Run with coverage
go test -coverprofile=coverage.out ./...
# View summary in terminal
go tool cover -func=coverage.out
# Generate interactive HTML
go tool cover -html=coverage.out -o coverage.html
open coverage.html # macOS
Coverage by package¶
# Parser coverage
go test -coverprofile=parser.out ./internal/parser/...
go tool cover -func=parser.out
# Pattern engine coverage
go test -coverprofile=pattern.out ./internal/pattern/...
go tool cover -func=pattern.out
Minimum required coverage¶
The project requires minimum coverage of 80%:
# Verify minimum coverage
go test -coverprofile=coverage.out ./...
COVERAGE=$(go tool cover -func=coverage.out | grep total | awk '{print $3}' | tr -d '%')
if (( $(echo "$COVERAGE < 80" | bc -l) )); then
echo "FAIL: Coverage $COVERAGE% < 80%"
exit 1
fi
Integration Tests¶
Run integration tests¶
# Integration tests (not skipped)
go test -v ./pkg/scanner/... -run Integration
# With extended timeout (for large projects)
go test -v -timeout 5m ./pkg/scanner/... -run Integration
Skip integration tests in fast CI¶
Complete E2E tests¶
# Build binary
make build
# Run against fixture
./bin/mcp-scan scan testdata/fixtures/class_a/ --mode fast --output json
# Verify output
./bin/mcp-scan scan testdata/fixtures/class_a/ --mode fast --output json | jq '.findings | length'
Golden Tests (Snapshots)¶
Golden tests compare current outputs against saved snapshots.
Run golden tests¶
Update snapshots¶
When the output format changes intentionally:
# Update all snapshots
UPDATE_GOLDEN=1 go test ./... -run Golden
# Update specific snapshot
UPDATE_GOLDEN=1 go test ./internal/reporter/... -run TestGolden_JSON
Snapshot location¶
testdata/golden/
├── json/
│ ├── class_a_output.json
│ ├── class_g_output.json
│ └── ...
├── sarif/
│ ├── class_a_output.sarif
│ └── ...
└── evidence/
└── ...
Benchmarks¶
Run all benchmarks¶
Package benchmarks¶
# ML classifier benchmarks
go test -bench=. ./internal/ml/...
# Scanner benchmarks
go test -bench=. ./pkg/scanner/...
Specific benchmark¶
# Only the classifier benchmark
go test -bench=BenchmarkRuleBasedClassifier ./internal/ml/...
# With memory
go test -bench=BenchmarkRuleBasedClassifier -benchmem ./internal/ml/...
Compare benchmarks¶
# Save baseline
go test -bench=. ./... > bench_old.txt
# After changes
go test -bench=. ./... > bench_new.txt
# Compare (requires benchstat)
go install golang.org/x/perf/cmd/benchstat@latest
benchstat bench_old.txt bench_new.txt
Important benchmarks¶
| Benchmark | Package | Description |
|---|---|---|
BenchmarkRuleBasedClassifier |
ml | Prompt classification |
BenchmarkFeatureExtraction |
ml | Feature extraction |
BenchmarkScanner_SmallProject |
scanner | Small project scan |
BenchmarkScanner_MediumProject |
scanner | Medium project scan |
BenchmarkParser_Python |
parser | Python parsing |
BenchmarkParser_TypeScript |
parser | TypeScript parsing |
Tests against DVMCP¶
Initial setup¶
# Clone DVMCP if it doesn't exist
if [ ! -d "testdata/damn-vulnerable-MCP-server" ]; then
git clone https://github.com/harishsg993010/damn-vulnerable-MCP-server \
testdata/damn-vulnerable-MCP-server
fi
Run detection tests¶
Validate detection coverage¶
Makefile Targets¶
# Basic tests
test:
go test ./...
# Tests with verbose
test-verbose:
go test -v ./...
# Tests with coverage
test-coverage:
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out -o coverage.html
# Golden tests
test-golden:
go test ./... -run Golden
# DVMCP detection tests
test-detection:
go test -v ./... -run TestDVMCP
# Benchmarks
bench:
go test -bench=. -benchmem ./...
# Fast tests (without integration)
test-fast:
go test -short ./...
# Tests with race detector
test-race:
go test -race ./...
CI/CD¶
GitHub Actions¶
# .github/workflows/test.yml
name: Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: '1.24'
- name: Run tests
run: go test -v -race -coverprofile=coverage.out ./...
- name: Check coverage
run: |
COVERAGE=$(go tool cover -func=coverage.out | grep total | awk '{print $3}' | tr -d '%')
echo "Coverage: $COVERAGE%"
if (( $(echo "$COVERAGE < 80" | bc -l) )); then
echo "Coverage below 80%"
exit 1
fi
- name: Run benchmarks
run: go test -bench=. -benchtime=1s ./...
Pre-commit hook¶
#!/bin/bash
# .git/hooks/pre-commit
# Run fast tests
go test -short ./...
if [ $? -ne 0 ]; then
echo "Tests failed. Commit aborted."
exit 1
fi
# Verify format
gofmt -l . | grep -q .
if [ $? -eq 0 ]; then
echo "Some files need gofmt. Run: gofmt -w ."
exit 1
fi
Debugging Tests¶
View detailed output¶
# Complete output of failed tests
go test -v ./... 2>&1 | tee test.log
# Filter only failures
go test -v ./... 2>&1 | grep -A 10 "FAIL"
Run a test in debug mode¶
# With delve debugger
dlv test ./internal/pattern/... -- -test.run TestPatternEngine_PromptInjection
Generate trace¶
Troubleshooting¶
Test timeout¶
Out of memory¶
Test cache¶
Flaky tests¶
# Run multiple times to detect flakiness
for i in {1..10}; do
go test ./internal/pattern/... -run TestFlaky
if [ $? -ne 0 ]; then
echo "Failed on iteration $i"
break
fi
done