bug0inc/passmark

AI-Native Test Orchestration

PassMark operates as a wrapper layer around standard Playwright tests, intercepting element resolution failures and routing them through an AI decision pipeline rather than immediately failing.

Design Trade-offs

• Determinism vs. Resilience: Sacrifices 100% reproducible test runs for higher pass rates on UI iterations, requiring teams to accept "probabilistic green builds."
• Latency for Maintenance: Adds 2-5 seconds per healing event (API roundtrip) but eliminates hours of selector updates.
• Cost Distribution: Shifts QA costs from engineering salaries (maintenance) to inference tokens (operational), favoring teams with high UI velocity.

The Breakthrough: PassMark treats DOM element identification as a retrieval-augmented generation problem rather than a static query problem. When a selector fails, it captures the full DOM context, viewport screenshot, and test intent, then prompts an LLM to suggest the corrected selector—effectively giving Playwright "common sense" about UI patterns.

Specific Technical Innovations

1. Semantic Selector Healing: Unlike traditional retry mechanisms, PassMark uses vision-capable models (GPT-4V/Claude 3) to analyze screenshots alongside DOM dumps. It doesn't just wait for an element—it understands that "the blue checkout button moved from header to sidebar" and updates the locator strategy dynamically.
2. Multi-Model Verification Consensus: Implements a "voting" system where cheaper models (Haiku, GPT-3.5) attempt verification first, escalating to premium models (Opus, GPT-4) only on disagreement. This reduces per-test costs by ~60% while maintaining high confidence intervals for assertions.
3. Intelligent Decision Caching: Stores successful healing decisions in a vector database keyed by DOM structure hashes. When similar UI patterns appear (e.g., React component rerenders with identical class names), it retrieves cached selectors without API calls, dropping latency to <100ms for recurring patterns.
4. Regression Diffing via Embeddings: Instead of pixel-perfect screenshots (brittle) or DOM text comparison (noisy), PassMark generates embeddings of page states to detect semantic regressions—catching when functionality breaks but visual appearance changes intentionally.
5. Playwright-Native Hook Injection: Uses TypeScript decorators and custom expect matchers rather than forking Playwright, allowing drop-in adoption with test.extend() patterns that preserve existing test semantics.

Latency & Throughput Metrics

Scalability Characteristics

• Cache Hit Rates: Projects with stable component libraries see 70-85% cache hits after the first week, reducing AI API calls to marginal noise.
• Cost at Scale: A 500-test suite running daily with 10% healing rate costs approximately $45-80/month in LLM tokens (using GPT-4 mini for 80% of operations), significantly undercutting visual testing SaaS pricing.
• Bottleneck: The current architecture appears single-threaded for AI decisions; parallel test suites may encounter rate limits or cold-start latency spikes with cloud AI providers.

Limitations

The multi-model verification, while reducing hallucinations, introduces non-deterministic flakiness—the very problem it solves. Teams must implement confidence thresholds (e.g., "fail if 2/3 models disagree") which adds configuration complexity. Additionally, vision-model API costs can spike 10x during DOM-heavy single-page applications where screenshots are large.

Competitive Landscape

Integration Points

• CI/CD: Native GitHub Actions support with passmark-action that caches AI decisions between runs, critical for keeping pipeline times under 10 minutes.
• AI Gateway: Supports Vercel AI SDK, OpenAI, and Anthropic out-of-box, with pluggable adapters for Azure OpenAI and local Ollama instances for air-gapped environments.
• Observability: Exports healing metrics (frequency, confidence scores, cost per test) to OpenTelemetry, allowing teams to track "test health" degradation over time.

Adoption Barriers

Current ecosystem risk is provider dependency. The "multi-model" approach requires API keys for multiple LLM providers, complicating enterprise procurement. The project needs a "bring your own model" abstraction for GPT-4-class local models (Llama 3.1 405B, Mixtral) to achieve adoption in regulated industries.