EG 2020-2021

Yesterday the award for the EG 2020-2021 (sic!) tourney was published. Once again I was speechless. How is such a delay possible?! But let's switch to the studies.

The judge was Árpád Rusz and he selected two studies for 1st and 2nd prize that caught my attention, because both contain long maneuvering phases with only seven pieces. To say it straight away: these are not 2-click studies and I have no reason to assume that these studies are mined.

I am only interested in the pure academic question: How difficult is it to find these studies with my DTZ program? A description of the program with some examples can be found here. If you haven't read this article, you probably won't understand the details here.

Before we answer the question, a small addendum to the article.

My program is not public. But the concept is public and therefore it is easily possible to recreate it. And even non-programmers have a way to achieve the results. Árpád Rusz is also the author of SEE, the Syzygy Endgame Explorer. This software offers DTZ value search, though not very convenient in the version I have, since only the search for positions with a fixed DTZ value is offered. I assume that Rusz can improve this with a fingersnip. And since SEE is - according to Rusz - just a Stockfish engine with an additional command "see", I think that the source code of SEE has to be public anyway because of Stockfish's license (GPLv3). So if you are in contact with Rusz, just ask.

Personally I've done a few experiments with SEE, but they were either small or unsuccessful. Maybe it's just me and my hardware (see below).

The question "How difficult is it to find these studies with my DTZ program?" was easy to answer and no report would be required. But this text had to be written in this or another form anyway, namely for my friend and software tester Hagen, who will be visiting me for extensive discussions next week.

Let's start with the simple example.

These numbers indicate already that the position will be found by the program. But it is still unclear, whether is it just possible or also easy.

I setup the program as follows. Generate all pawn endgames with

• White having an f-, g- and h-pawn
• Black having two pawns somewhere on the f-, g- and h-file
• no passed pawns
• and White to move.

8,910,126 positions were generated and their DTZ values were determined. These are the counts for the highest DTZ values.

• DTZ 23: 282
• DTZ 25: 123
• DTZ 27: 63
• DTZ 29: 5
• DTZ 31: 1

Since White has the move, we are looking for the position after 1.g3 Kd1, which has a DTZ value of 29. So, it turns out that this position would be a pretty easy find.

(It took 36 minutes on a 12 year old notebook without SSD showing a Youtube video at the same time.

With EGTBs on a SSD the 23 minutes for fetching the DTZ values would dwindle to a few minutes and certainly would drop under the 12 minutes for position generating which however would melt down itself on a modern computer.)

I let the program output the 1000 longest game fragments as PGN (similar to the 2-click downloads at https://syzygy-tables.info/). The unique longest fragment was this one.

And what about the 994 other fragments? Well, some of them are definitily interesting. As with the pawn structure of the longest fragment, the pawn structures repeat quite often and only the start positions of the kings vary. Some of these pawn structures have already appeared in the endgame literature, but others could still be used ... somewhere.

The first prize is a much more difficult example and is unattainable for my hardware.

1. In the first example there were four different pieces (white and black kings and pawns). In the current position all seven pieces are different. This results in quite different sizes of EGTBs. The two files needed for the endgame KPPPvKPP require less than 2.2 GB diskspace, while the two files required for KRNPvsKRP need 54+46=100 GB space! But in fact much more space is needed.

2. Note that the "no passed pawns" condition above excluded White pawns on the seventh and Black pawns on the second rank. In other words: despite the five pawns above, no promotion was possible in any position! But here we have a White pawn on c7. A pure DTZ value approach would not care about the next move, but the common EGTB access libraries are written for chess engines and provide also moves for the position. Therefore, all pairs of files for the four possible promotions of the white pawn are needed too. The good news is that they are not all as big as the two files for the position before the promotion, but 60 GB for the promotion to Q is still quite a lot. All in all a dedicated hard drive is needed.

3. The number of positions to be created also increases dramatically. It is no longer possible to set up reasonable restrictions and then to deal with all positions in one program run. Therefore, many runs with different restrictions are necessary. Assume we fix the position of both pawns for each run (resulting in 1128 runs when symmetry is already  applied). Then there are roughly about 60 squares left for every other piece, which means about 778 million positions for each run. Well, many of them are illegal, but even 500 million positions for every run would be quite a lot.

So, further restrictions are needed and it is clear that most people's hardware suffices only to search for small parts of all possibilities.

While nos. 1 and 2 can be solved by an extra harddisc for EGTBs (as many people have been using for years), no. 3 needs well-tailored restrictions which guarantee the computation of results within an acceptable time span.

My simple DTZ program and every similar functionality allows to replace composing by mining. This danger also exists for material constellations such as KRNPvKRP, which can only be examined stepwise. With extra storage space everyone can generate thousands of examples every day. Gonzalez and Kopylov have shown that there are more than enough positions suitable for tourney participation even in the small set of 2-click examples. The next months will show how judges act. If they continue to value the product and ignoring the way these products were obtained, tourneys could change in an unfavourable direction.