Performance Scores

Overall

40
Rank #44 of 45 — Top 2%

SWE-bench

32
Rank #44 of 45 — Top 2%

LiveCodeBench

42
Rank #44 of 45 — Top 2%

HumanEval

62
Rank #44 of 45 — Top 2%

BigCodeBench

26
Rank #44 of 45 — Top 2%

Strengths & Weaknesses

Strengths

  • Multimodal
  • Fast

Weaknesses

  • Basic coding
  • Niche provider

Compare with Similar-Priced Models

ModelOverall ScoreInput $/M
Reka Flash 40 $0.200
Claude 3.5 Haiku 52 $0.800
Claude 3 Haiku 45 $0.250
GPT-4o mini 58 $0.150
GPT-3.5 Turbo 40 $0.500
OpenAI o1-mini 70 $1.10