How does CCRL test and rate chess engines?

CCRL (Computer Chess Rating Lists) is a volunteer-run engine rating group formed in 2006. It publishes Elo-style rating lists built from games played by its testers, and it explains its setup so readers can interpret those ratings correctly: testing uses benchmark hardware (Intel i7-4770k), Stockfish 10 to calibrate time controls, generic opening books limited to 12 moves, and ponder off. Ratings are only saved for engines with 100+ games, and early ratings may fluctuate, so the site itself recommends consulting other rating lists for broader context.

What CCRL is

CCRL describes itself as a group of computer chess enthusiasts who enjoy engine testing, formed in 2006 by Graham Banks, Ray Banks, Sarah Bird, Kirill Kryukov, and Charles Smith. Currently active testers listed on the site include Graham Banks, Gabor Szots, Shaun Brewer, Basti, Andres Valverde, and Mattia Giambirtone.

Two points matter for reading its lists:

• Testers follow general testing guidelines, but each tester chooses what and how to test — some use systematic methods, others prefer tournaments.
• CCRL states it does not judge engine originality. Inclusion or exclusion does not imply endorsement or rejection, and viewers are expected to form their own opinions.

How testing works

The methodology page gives the concrete conditions behind the numbers:

Because time controls are calibrated against a fixed reference (Stockfish 10) on fixed benchmark hardware, the lists are internally comparable — but they are not a claim about how engines perform on your own machine or at different time controls.

How ratings are produced and read

• Rating lists and statistics are split across categories: 40/15, Blitz, Chess960, 40/2, Archive, Chess324, and DFRC. Each list is updated independently, with links located below the Best Versions lists.
• Elo ratings are saved for engines with 100+ games. Engines below that threshold may appear with unstable or provisional numbers.
• Early ratings may fluctuate. Treat a newly listed engine's first rating as provisional rather than settled.
• LOS (Likelihood of Superiority) shows the probability that an engine outperforms the one ranked directly below it. It is a pairwise comparison statistic, not a general confidence score for the whole list.
• Lists are sortable by column, and clicking an engine name reveals its opponents, homepage links, and comparison tools.
• All games are downloadable by engine, month, or ECO code.
• The openings report shows ECO statistics, draw rates, and White win percentages.

What the numbers do and don't tell you

A CCRL rating answers: how did this engine perform against this pool of opponents, under these calibrated conditions, on this benchmark hardware? It does not answer how the engine will perform on your hardware, at your time control, or with your opening book — which is why CCRL recommends consulting other rating lists for broader context.

If you want to go deeper than the summary tables, the downloadable games and the openings report let you check the underlying evidence: which opponents an engine faced, how often games were drawn, and how results break down by ECO code.

Joining CCRL

If you are interested in becoming a tester, the site's instruction is to sign up on the CCRL forum and contact a member. For security, CCRL only accepts testers known to the community or vouched for by trusted members.