It is currently 30 Jul 2010, 12:01

All times are UTC + 1 hour [ DST ]




 Page 1 of 1 [ 15 posts ] 
Author Message
 Post subject: The Math behind Rainbow Tables
PostPosted: 18 Aug 2009, 19:11 
Perfect Table

Joined: 02 Apr 2008, 15:10
Posts: 833
Location: Romania
As the title says, I'm trying to compile all the known formulas related to rainbow tables.

key_space = (charset_length*(charset_length^max_password_length-1)/(charset_length-1))
= charset_length^1 + charset_length^2 + ... + charset_length^max_password_length
work_factor = chain_length * chain_count / key_space
bruteforce_point = key_space / (chain_length * (chain_length + 1) / 2 * number_of_tables)
perfect_size = non-perfect_size / ((work_factor_per_table / 2) + 1)
expected_unique_chains *about* = total_chain_count / (work_factor / 2 + 1)
success_rate_per_non-perfect_table = 1 - (1 - expected_unique_chains / key_space) ^ chain_length
seconds_required_to_generate_a_rt = key_space * work_factor / step_speed
success_rate_perfect_rainbow_table = 1 - (1 - no_of_unique_chains / key_space) ^ chain_length

Quote:
P = success rate of a rainbow table
N = number of possible keys
L = length of chain
Co = number of chains
Cmax = max number of unique chains
C = number of unique chains
P = 1 - (1 - C(1) / N) * (1 - C(2) / N) * ... * (1 - C(L) / N)

C(1) = Co
C(2) = N * (1 - e ^ (-C(1) / N))
C(3) = N * (1 - e ^ (-C(2) / N))
C(4) = N * (1 - e ^ (-C(3) / N))
C(5) = N * (1 - e ^ (-C(4) / N))
C(6) = N * (1 - e ^ (-C(4) / N))
...
C(L) = N * (1 - e ^ (-C(L - 1) / N))

Cmax(1) = N
Cmax(2) = N * (1 - e ^ (-Cmax(1) / N))
Cmax(3) = N * (1 - e ^ (-Cmax(2) / N))
...
Cmax(L) = N * (1 - e ^ (-Cmax(L - 1) / N))


----------
key_space - the total number of possible passwords for a given charset_length and max_password_length.
charset_length - character set length
for example :
a-z(loweralpha) = A-Z(upperalpha) = 26 = card{a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z}
0-9(numeric) = 10
perfect_size - perfect rainbow table size
non-perfect_size - "original" rainbow table size
chain - hash chains
success_rate_per_non-perfect_table - Success probability of one rainbow table

average step_speed of a quad core = 3 million links per second (3*10^6 links/sec)
average step_speed of the Free Rainbow Tables Super Computer = 1 bil links/second (10^9 links/sec)

work_factor -
Sc00bz wrote:
work_factor is chosen depending on table accuracy wanted. These work factors are found by guess and check and work for most key spaces and chain lengths (small chain lengths like 100 will be off by a little).

http://www.freerainbowtables.com/phpBB3/viewtopic.php?p=10969&f=2#p10969
99.9% success probability :
4 Tables =>work_factor per non-perfect_table = 12.78300
5 Tables =>work_factor per non-perfect_table = 4.48860
5 Tables =>work_factor per perfect_table = 1.9945

bruteforce_point >=10000 ;
bruteforce_point -
Sc00bz wrote:
Sc00bz wrote:
One way to measure how good a table is, is to do "key space / (chain length * (chain length + 1) / 2 * number of tables)" this number should be at least 10000.

I just picked a number that isn't really low. The higher the number the better as long as the size of the tables are manageable. It's basically so that someone doesn't pick a chain length of 100 or 10,000,000. You should also look at the time it takes to generate the end points when cracking a hash and make sure it is not too high. Also a nice little trick for indexing is to make the number of chains just under 2^n (where n is an integer). This way the tables are slightly more space efficient.


Offline
 Profile  
 
 Post subject: Re: The Math behind Rainbow Tables
PostPosted: 21 Aug 2009, 12:49 
Guesser

Joined: 12 Nov 2007, 03:36
Posts: 38
Sweet. I sometime have to go trolling to find a formula that I saw. Nice idea to consolidate them.


Offline
 Profile  
 
 Post subject: Re: The Math behind Rainbow Tables
PostPosted: 22 Aug 2009, 01:12 
Developer

Joined: 03 Dec 2007, 11:37
Posts: 725
I finally got off my ass and made it look all nice. Well I tried to make the text easier to read but picking 21 colors that look different and work on this background is hard :D.

chainLen = chainLenRCrack - 1
keySpace = sum i = minPwLen to maxPwLen of charSetLen ^ i = (charSetLen ^ (maxPwLen + 1) - charSetLen ^ minPwLen) / (charSetLen - 1)
tableWorkFactor = chainLen * chainCount / keySpace
totalWorkFactor = numTables * tableWorkFactor
preWork = chainLen * (chainLen + 1) / 2
bruteForcePointkeySpace / (preWork * numTables)
expectedUniqueChains = euc(chainLen) ≈ chainCount / (tableWorkFactor / 2 + 1)
euc(1) = chainCount
euc(i) = keySpace * (1 - e ^ (-euc(i - 1) / keySpace))
tableGenerationTime = keySpace * tableWorkFactor / stepSpeed = chainCount * chainLen / stepSpeed
perfectTableSuccessRate ≈ 1 - (1 - uniqueChains / keySpace) ^ chainLen ≈ 1 - (1 - expectedUniqueChains / keySpace) ^ chainLen
nonperfectTableSuccessRate ≈ 1 - product i = 1 to chainLen of 1 - euc(i) / keySpace
totalSuccessRate = 1 - product i = 1 to numTables of 1 - tableSuccessRate(i) ≈ 1 - (1 - tableSuccessRate) ^ numTables


Attachments:
rtequations.png
rtequations.png [ 30.09 KiB | Viewed 1689 times ]

_________________
http://www.tobtu.com/
Offline
 Profile  
 
 Post subject: Re: The Math behind Rainbow Tables
PostPosted: 22 Aug 2009, 03:34 
Perfect Table

Joined: 02 Apr 2008, 15:10
Posts: 833
Location: Romania
WOOOOOOOOHOOOOOOOOOO

love the colors :D
isn't it this way ?

LE: and could you give us some detailed explanation on how do you define work_factor, is it a something that you discover, how ? Why do those constants work...I'm very curious ! Could you come up with a mathematical demonstration why they work and how ? (you can license it as your own work.)
Do you also have some statistics made for rainbow tables (anything) ? charts, spreadsheets, tests and failures, maybe they would come handy to someone.


Offline
 Profile  
 
 Post subject: Re: The Math behind Rainbow Tables
PostPosted: 29 Aug 2009, 04:18 
Developer

Joined: 03 Dec 2007, 11:37
Posts: 725
_haxxor_ wrote:
isn't it this way ?

They are both equivalent. I was aiming for no redundant parentheses because it starts to look messy, but I just noticed that the first set of parentheses in totalSuccessRate equation are redundant.

_haxxor_ wrote:
LE: and could you give us some detailed explanation on how do you define work_factor, is it a something that you discover, how ? Why do those constants work...I'm very curious ! Could you come up with a mathematical demonstration why they work and how ? (you can license it as your own work.)
Do you also have some statistics made for rainbow tables (anything) ? charts, spreadsheets, tests and failures, maybe they would come handy to someone.

Work factor is how many times more work is needed to create a table than hashing every password in the key space. Chain length * chain count gets you the total number of links you have to do. Then you divide by how many passwords are in the key space. I originally did this just to see how many times you would need to use the tables before they became more efficient than brute forcing, but I noticed that tables with the same work factor had the same success rate. Then awhile later I noticed that tables with a work factor of 13.0 (99.90% success rate with 4 tables) when they were perfected they were 1/7.5 the size. Then I tried several different work factors and they all became about 1/(workFactor/2+1) the size if they were perfect. Yes I did come up with work factor and the name but I later found out that someone else did too and named it the same thing, but I don't think they noticed the relationship between the work factor and success rate.

Awesome I thought there was a nice formula for this. I can't decide which way I like it so here's both:
tableWorkFactor ≈ 2 * chainLen * (1 - (1 - perfectTableSuccessRate) ^ (1 / chainLen)) / (2 - chainLen * (1 - (1 - perfectTableSuccessRate) ^ (1 / chainLen)))
1 / tableWorkFactor ≈ 1 / (chainLen * (1 - (1 - perfectTableSuccessRate) ^ (1 / chainLen))) - 1 / 2

Edit: Apparently I can't count parentheses, missing 2 ending parentheses on the first formula and 1 ending parenthesis on the last formula. At least the picture is correct. Only took 4.5 months to notice this. Ohh I messed up once and copy and pasted it in two other places :D.


Attachments:
rtequations2.png
rtequations2.png [ 7.7 KiB | Viewed 1624 times ]


Last edited by Sc00bz on 17 Jan 2010, 04:06, edited 1 time in total.
_________________
http://www.tobtu.com/
Offline
 Profile  
 
 Post subject: Re: The Math behind Rainbow Tables
PostPosted: 02 Oct 2009, 00:50 
Developer

Joined: 03 Dec 2007, 11:37
Posts: 725
Bad news I just ran into a paper by Gildas Avoine, Pascal Junod, and Philippe Oechslin from I think 2008. Which has this formula:
maxUniqueChains ≈ 2 * keySpace / (chainLen + 1)

and when you use these formulas to find maxUniqueChains:
expectedUniqueChainschainCount / (tableWorkFactor / 2 + 1)
tableWorkFactor = chainLen * chainCount / keySpace

you get:
maxUniqueChains ≈ 2 * keySpace / (chainLen + 2)

So I'm off by one. Hmmm maybe they consider chain length to be the chainLenRCrack then the formula is the same. I guess some day soon I'll need to read and understand their proof.
chainLen = chainLenRCrack - 1
maxUniqueChains ≈ 2 * keySpace / (chainLenRCrack - 1 + 2)



_________________
http://www.tobtu.com/
Offline
 Profile  
 
 Post subject: Re: The Math behind Rainbow Tables
PostPosted: 10 Nov 2009, 03:47 
Developer

Joined: 03 Dec 2007, 11:37
Posts: 725
RT size = 16 bytes * chainCount
RTIv1 size = 8 bytes * chainCount + 11 bytes * ceiling(keySpace / (2 ^ 16))



_________________
http://www.tobtu.com/
Offline
 Profile  
 
 Post subject: Re: The Math behind Rainbow Tables
PostPosted: 10 Nov 2009, 04:37 
Developer

Joined: 15 Jul 2009, 22:38
Posts: 363
Sc00bz wrote:
RT size = 16 bytes * chainCount


Are the 16 bytes the 2 64bit values of the start point and end point of the chain?


Offline
 Profile  
 
 Post subject: Re: The Math behind Rainbow Tables
PostPosted: 10 Nov 2009, 08:04 
Developer

Joined: 03 Dec 2007, 11:37
Posts: 725
quel wrote:
Sc00bz wrote:
RT size = 16 bytes * chainCount


Are the 16 bytes the 2 64bit values of the start point and end point of the chain?

Yes



_________________
http://www.tobtu.com/
Offline
 Profile  
 
 Post subject: Re: The Math behind Rainbow Tables
PostPosted: 11 Nov 2009, 12:23 
Perfect Table

Joined: 02 Apr 2008, 15:10
Posts: 833
Location: Romania
Sc00bz wrote:
RT size = 16 bytes * chainCount
RTIv1 size = 8 bytes * chainCount + 11 bytes * ceiling(keySpace / (2 ^ 16))

(2 ^ 16) - what constant is this ?

How about RTIv2, could you give us a formula ? :D

LE: is it linked to the upper limit of chainLen ? in that case, that constant will change according to chainLen.


Offline
 Profile  
 
 Post subject: Re: The Math behind Rainbow Tables
PostPosted: 11 Nov 2009, 14:34 
Developer

Joined: 03 Dec 2007, 11:37
Posts: 725
16 is the number of bits stored for the end point. With RTIv1 you were required to index all but the last 16 bits of the end points.

With RTIv2 you can set that to whatever you want as long as it's 32 or less.
endpointBits <= 32
indexRowSize = 1, 2, 3, or 4 bytes this depends on what the maximum number of chains per index which is not predetermined.

You can find the minimum indexRowSize by doing:
numIndexes = ceiling(keySpace / (2 ^ endpointBits))
expectedChainsPerIndex = chainCount / numIndexes
minIndexRowSize = ceiling(Log2(expectedChainsPerIndex) / 8) this should be 1 or you are doing something wrong

For minimum table size expectedChainsPerIndex needs to be between (indexRowSize * 8 / 2) and (indexRowSize * 8):
if indexRowSize is 4, expectedChainsPerIndex needs to be between 16 and 32
if indexRowSize is 3, expectedChainsPerIndex needs to be between 12 and 24
if indexRowSize is 2, expectedChainsPerIndex needs to be between 8 and 16
if indexRowSize is 1, expectedChainsPerIndex needs to be between 4 and 8

Removing ceiling from expectedChainsPerIndex gives us this:
log2((indexRowSize * 8 / 2) * keySpace / chainCount) < endpointBits < log2((indexRowSize * 8) * keySpace / chainCount)
endpointBits = ceiling(log2((indexRowSize * 8 / 2) * keySpace / chainCount))
or
endpointBits = floor(log2((indexRowSize * 8) * keySpace / chainCount))

You can pretty much plan on indexRowSize being 1 because the chances that there are over 255 chains per index when the expected is between 4 and 8 are probably really low.

RTIv2 size = ceiling((checkpointBits + startpointBits + endpointBits) / 8) * chainCount + indexRowSize * ceiling(keySpace / (2 ^ endpointBits))



_________________
http://www.tobtu.com/
Offline
 Profile  
 
 Post subject: Re: The Math behind Rainbow Tables
PostPosted: 11 Nov 2009, 14:59 
Developer

Joined: 03 Dec 2007, 11:37
Posts: 725
The others are per table these are for the whole table set:
RT size = 16 bytes * totalChainCount
RTIv1 size = 8 bytes * totalChainCount + 11 bytes * numTables * ceiling(keySpace / (2 ^ 16))
RTIv2 size = ceiling((checkpointBits + startpointBits + endpointBits) / 8) * totalChainCount + indexRowSize * numTables * ceiling(keySpace / (2 ^ endpointBits))



_________________
http://www.tobtu.com/
Offline
 Profile  
 
 Post subject: Re: The Math behind Rainbow Tables
PostPosted: 18 Jan 2010, 15:56 
Developer

Joined: 03 Dec 2007, 11:37
Posts: 725
Rearranged previous formula to get in terms of perfectTableSuccessRate:
perfectTableSuccessRate ≈ 1 - (1 - tableWorkFactor / (chainLen * (1 + tableWorkFactor / 2))) ^ chainLen



_________________
http://www.tobtu.com/
Offline
 Profile  
 
 Post subject: Re: The Math behind Rainbow Tables
PostPosted: 01 Mar 2010, 13:09 
Guesser

Joined: 16 Jun 2009, 22:12
Posts: 37
Correct me if I am wrong :

Quote:
keySpace = (charSetLen ^ (maxPwLen + 1) - charSetLen ^ minPwLen) / (charSetLen - 1)


So, for lm-frt-cp437-850 :
- charSetLen = 113
- maxPwLen = 14
- minPwLen = 1

(113^ (14+ 1) - 113^1) / (113 - 1) = 55841699800868609103953425662 unique passwords in those tables ?
source : http://bit.ly/dhrNqP


Offline
 Profile  
 
 Post subject: Re: The Math behind Rainbow Tables
PostPosted: 01 Mar 2010, 13:43 
Perfect Table

Joined: 02 Apr 2008, 15:10
Posts: 833
Location: Romania
pyr wrote:
Correct me if I am wrong :

Quote:
keySpace = (charSetLen ^ (maxPwLen + 1) - charSetLen ^ minPwLen) / (charSetLen - 1)


So, for lm-frt-cp437-850 :
- charSetLen = 113
- maxPwLen = 14
- minPwLen = 1

(113^ (14+ 1) - 113^1) / (113 - 1) = 55841699800868609103953425662 unique passwords in those tables ?
source : http://bit.ly/dhrNqP

No.
The lm is cut in 2 so maxPwLen = 7.
http://en.wikipedia.org/wiki/LM_hash#Algorithm
(113 ^ (7 + 1) - 113 ^ 1) / (113 - 1) = 237361088652359 = 2^47.75 (log2 keyspace = 47.75)
http://bit.ly/dB1Dar


Offline
 Profile  
 
Display posts from previous:  Sort by  
 Page 1 of 1 [ 15 posts ] 

All times are UTC + 1 hour [ DST ]


Who is online

Users browsing this forum: No registered users and 0 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to: