OpenMythos: Reverse-Engineering Claude's Speculated Architecture from Research Breadcrumbs

Hypothesized Core Design

OpenMythos builds a speculative recurrent transformer stack distinct from standard GPT-style autoregressive models. The architecture hypothesizes that Claude 3.5's extended reasoning capabilities stem from internal loops rather than simple scaling.

Key Abstractions

• Recurrent Depth vs. Layer Depth: Trades vertical layer stacking for horizontal iteration, potentially reducing parameter count while increasing effective computation per token.
• Stateful KV-Cache: Extends standard attention caching with compressed historical states, hypothesizing Claude's 200K context efficiency.

Design Trade-offs

The looped architecture introduces non-deterministic latency (learned halting) and complicates gradient flow through time. OpenMythos implements straight-through estimators and checkpointed backpropagation to mitigate this, though this remains computationally expensive compared to standard transformers.

The most significant contribution isn't the code itself, but the methodology of architectural archaeology—synthesizing sparse technical breadcrumbs (job postings, citation graphs, and performance benchmarks) into a testable hypothesis about proprietary frontier model internals.

Specific Technical Innovations

1. Dynamic Halting via Ponder Networks: Implements Graves' Adaptive Computation Time modified for transformer blocks, allowing the model to 'think longer' on complex tokens while maintaining throughput on simple ones. Uses a sigmoid halting probability p = σ(MLP(x)) with entropy regularization.
2. Cross-Loop Attention Routing: Unlike standard transformers where layers are feed-forward, this implements Attention(Q, K_prev, V_prev) allowing later iterations to attend to earlier computational states, creating a form of working memory distinct from the context window.
3. Constitutional Consistency Loss: Adds an auxiliary loss function enforcing that looped outputs remain consistent with Constitutional AI principles across iterations, simulating Anthropic's rumored 'values anchoring' mechanism.
4. JAX/PyTorch Duality: Maintains functionally equivalent implementations in both frameworks, allowing researchers to leverage JAX's pjit for parallelism studies and PyTorch's ecosystem for training stability.
5. Interpretability Hooks: Every loop iteration exports attention patterns and state norms, addressing the black-box criticism of proprietary models by making the 'thought process' observable.

Theoretical Computational Profile

Since OpenMythos is a speculative reconstruction without trained weights, performance analysis focuses on computational complexity and memory scaling of the hypothesized mechanisms.

Scalability Limitations

• Loop Divergence: Without careful initialization, recurrent transformers suffer from chaotic attractors where small input changes cause wildly different loop counts, making batched inference difficult.
• The Hardware Gap: Current CUDA kernels are optimized for standard feed-forward attention. The irregular memory access patterns of dynamic looping underutilize Tensor Cores, resulting in ~40% effective FLOP utilization compared to static architectures.

Benchmark Status

The repository currently implements random-initialized forward passes only. No training run has validated whether the hypothesized architecture actually converges on reasoning tasks, leaving performance claims purely theoretical.

Competitive Landscape

Integration Points

• Hugging Face Compatibility: Implements the PreTrainedModel interface, allowing drop-in replacement for model cards, though without pretrained weights.
• vLLM Integration: Experimental support for PagedAttention on looped states, though continuous batching across variable loop counts remains unimplemented.
• EleutherAI Eval Harness: Ready adapter for benchmarking the architecture once training commences, targeting HELM and MMLU for validation against Claude's reported scores.

Adoption Risks

The project's utility depends entirely on the accuracy of its architectural hypotheses. If Claude 3.5 actually relies on Mixture-of-Experts (MoE) rather than looping, OpenMythos becomes an educational curiosity rather than a functional alternative. The 301 forks suggest significant experimentation interest, but production adoption is non-existent without trained checkpoints.