​
Executive Summary

ATLAS demonstrates consistent performance improvements across all evaluated metrics, achieving a 15.7% average accuracy gain with 50% token reduction while maintaining a 97% non-degradation guarantee.

15.7%

Average accuracy improvement

50%

Token reduction (4k → 2k)

97%

Non-degradation rate

31%

Completion rate gain

​
Benchmark Environment

Hardware: 4×H100 GPUs with NVLink interconnect Dataset: Arc-ATLAS-Teach-v0 (32 samples per problem, seed=42) Models: ATLAS-8B-Thinking and ATLAS-8B-Instruct

​
Core Performance Metrics

​
Teaching Effectiveness

Performance comparison between teacher-assisted and standalone student models:
MetricTeacher+StudentStudent AloneImprovementStatistical Significance
Average accuracy78.0%62.3%+15.7%1p < 0.001
Maximum improvement91.9%62.3%+29.6%p < 0.001
Completion rate~100%~69%+31%2p < 0.001
Non-degradation rate97%N/A97%3-

​
Efficiency Metrics

Resource utilization and computational efficiency gains:
MetricTeacher+StudentStudent AloneImprovement
Token usage~2,000~4,000-50%
Generation time (32 samples)1:101:21-13.6%
Teaching efficiency score0.372baselineefficiency metric
Memory footprint16GB8GB+8GB for teacher

​
Performance by Task Category

DifficultyTeacher+StudentStudent AloneDelta
Easy92%78%+14%
Medium81%64%+17%
Hard73%55%+18%
Stronger improvements on harder problems demonstrate ATLAS’s value for complex reasoning.

​
Key Findings

​
Non-Degradation Analysis

ATLAS achieves a 97% non-degradation rate, meaning only 3% of interactions result in worse performance:
  • Primary cause: Normalization issues in response parsing (2.1%)
  • Secondary cause: Over-specification in teaching (0.9%)
  • Target rate: ≥99% (ongoing optimization)
  • Mitigation: Improved prompt templates and parsing logic

​
Efficiency Gains

The 50% token reduction comes from:
  1. Diagnostic efficiency (20% reduction): Targeted probing identifies exact capability gaps
  2. Teaching precision (25% reduction): Focused guidance eliminates unnecessary exploration
  3. Response coherence (5% reduction): Better-structured outputs require fewer tokens

​
Scalability Profile

Performance across different scales:
ScaleGPUsBatch SizeThroughputLatency
Development1×T412 req/min30s
Production4×A100816 req/min3.75s
Enterprise8×H1003264 req/min0.94s

​
Learning Metrics

​
Teaching Effectiveness Score (TES)

TES = (accuracy_gain * completion_rate) / (teaching_tokens / 1000)
Model PairTES ScoreInterpretation
ATLAS-8B + GPT-40.42Excellent
ATLAS-8B + Claude-30.39Excellent
ATLAS-8B + Llama-70B0.36Very Good
ATLAS-8B + Qwen-4B0.31Good

​
Learning Rate Analysis

Performance improvement over teaching iterations: 
# Measured learning rate per interaction
iteration_1: +8.2%  # Initial diagnostic and teaching
iteration_2: +4.1%  # Refined guidance
iteration_3: +2.3%  # Fine-tuning
iteration_4: +1.1%  # Diminishing returns

​
Hardware Requirements

  • GPU: 16GB VRAM (RTX 4080, A5000)
  • RAM: 32GB system memory
  • Storage: 100GB for models and cache
  • Use case: Development and testing
  • Throughput: 1-2 requests/minute
  • GPU: 8×H100 80GB with NVSwitch
  • RAM: 256GB+ system memory
  • Storage: 2TB NVMe RAID
  • Use case: High-volume production
  • Throughput: 50+ requests/minute

​
Statistical Validation

All results are statistically significant with p < 0.001 using:
  • Test: Paired t-test for accuracy improvements
  • Sample size: 32 generations per problem
  • Seed: 42 for reproducibility
  • Cross-validation: 5-fold validation on held-out test set
  • Confidence intervals: 95% CI reported for all metrics

​
Next Steps

Evaluation Methodology

Understand our testing protocol

Reproduction Guide

Reproduce these results

Deploy ATLAS

Start using ATLAS

Footnotes

  1. Average across all Arc-ATLAS-Teach-v0 evaluation tasks ↩
  2. Percentage of tasks completed within token limits ↩
  3. Percentage of interactions that maintain or improve performance ↩