DDTree-MLX: Metal-Optimized Tree-Based Speculative Decoding for Apple Silicon

Core Architecture

This framework implements tree-based speculative decoding optimized for Apple Silicon’s unified memory and Metal GPU backend:

• Built entirely on mlx with custom handwritten Metal kernels for hybrid draft model-target model execution
• Two-stage inference pipeline:
  1. Draft tree generation from a smaller, faster draft LLM
  2. Parallel verification of all tree branches on a larger target LLM
• Parameter count configurable via draft/target model selection (supports any MLX-compatible LLM)
• Native support for code-specific workload tuning via optimized logit sampling for programming language token distributions

Diagram Overview

Input Prompt → Draft Model Generates Token Tree → Metal Kernel Parallel Branch Validation → Target Model Confirms Valid Tokens → Output Stream

Key Technical Innovations

• First MLX-native tree-based speculative decoder: No prior MLX framework supported custom Metal kernels for hybrid speculative decoding prior to this release
• Code-workload tuning: Optimized token sampling heuristics tailored for code generation, which has sparser token distributions than natural language
• Direct integration with MLX’s automatic differentiation and device management stack, avoiding cross-framework overhead
• References the original Tree-Based Speculative Decoding paper, with modifications to reduce Metal command buffer overhead

Unlike Python-only speculative decoding wrappers, this project eliminates CPU-GPU data transfer bottlenecks by running all core logic on the Metal backend.

Benchmark Results & Hardware Requirements

• Inference Speed: Up to 85 tokens/sec on M2 Max 32GB for 7B parameter code models
• Hardware Requirements: Apple Silicon Mac with M1 Pro/Max or newer, 16GB+ unified RAM
• Limitations: Only supports single-GPU Apple Silicon, no distributed inference yet; draft model selection requires manual tuning for target model parity

Ecosystem & Deployment

• Deployment Options: Local command-line interface for LLM inference, importable Python package for integration into MLX workflows
• Fine-Tuning Support: Works with any base MLX LLM, including fine-tuned code models like CodeLlama-MLX
• Licensing: MIT open-source license, commercial use permitted without restriction
• Community Ecosystem: 107 GitHub stars as of June 2026, 8 forks, with active discussions on Apple Silicon LLM inference Discord servers

Adapters for popular MLX model packs are already available for CodeLlama, Mistral, and Llama 3 variants.