Thanks again, I've not downloaded anything too big yet so I'll keep an eye out for the RTI2 tables. BTW, may I ask how the progress bars at https://www.freerainbowtables.com/en/tableprogress/
work? I assume the top one is table progress, but what's with the light/dark blue on the bottom one?
We should probably document this somewhere. I had to check the code to actually confirm it.
Yes the top line is the actual completion based on the perfected chains vs the required chains. Note that perfection with these larger tables (until we get our I/O subsystem upgraded) is a 26-36 hour process so the updates are far from real time. Perfecting has to take all the raw data for a table 1TB+ and remove all chains that have duplicate end points. You can look at the data growth for the last 24 hours and that's the raw size of new parts that have been validated and assimilated awaiting perfection. Right now the frequency of perfection is driven by how long it takes and how we are doing on free disk space.
The bottom line in the dark blue indicates WUs returned and the light blue on the bottom indicates WUs already processed by the work generator and in the queue to be sent to hosts and/or work that is already being processed by hosts. In general we fill the GPU queues because of the rate at which the WUs are turned around and the CPU only queue (right now that's hybrid2 because the GPU piece isn't complete) is filled in a more traditional sense as needed.
We know how many chains in total we need to achieve the desired success rate (chains in total.) We have formulas for the expected unique chains which allows us to form a reasonable guess about the number of extra chains we have to generate to create a perfect table. In theory once the bottom line is fully dark blue and perfecting is completed then the top bar should be fully dark blue. Sometimes we need to add additional chains to reach the target chains to meet the target success rate.
So the most important data for how close we really are to completion: chains completed, chains left, and those on the top line are also the least frequently updated.
Also, right now the time remaining estimates are based on the aggregate speed in billions of links/s and so it doesn't factor in that ntlm is faster than md5 or that hybrid2 is CPU only right now.