Dice are fair



  • There were several threads claiming that the game loaded the dice to keep the player from winning. As "evidence" the players were including roll counts for each possible result. As a software Engineer I found these claims offensive. As an engineer I knew that the numbers were skewed based on the small sample size. (1 game, probably less than 100 rolls)

    To test this theory; I started tracking every die roll. After 1000 rolls, I tabulated the results. (1065 rolls) I then used my own die roll program and rolled it the same number of occurrences, 5 times.

    For the record, writing a die roll method is painfully easy. In Java it is:

    public static int rollDie(int faces)
    {
        double roll = Math.random();
        return (int)(roll * faces + 1);
    }
    

    Where "faces" is the number of faces on your die, in the case of Catan this is, of course, 6. For other games, it could be whatever is needed. To simulate a coin flip, for example, you would pass 2 to the method. (the +1 is necessary because the roll * faces will return a number from 0 to (faces -1) in the case of Catan 0-5. The +1 shifts that result to 1-6.)

    As you can see from the attached charts any set of 1000 rolls can have wide variations from the normal. (I did this in a nicely formatted spread sheet that I could not, of course, export here) After 5000 rolls, however, those variations have been smoothed to near 0. (a positive number means that result is under represented - didn't come up as much as expected, a negative means that result is over represented - came up more than expected)

    The inevitable conclusion, after comparing the observed results with the java results, is that the dice are not loaded. Any prolonged period were the roll that you were hoping for didn't come up is within the statistical norm. It may be frustrating, but it isn't because the game is cheating.

    Normal  
    

    Result % Count
    2 2.78% 29.58
    3 5.56% 59.17
    4 8.33% 88.75
    5 11.11% 118.33
    6 13.89% 147.92
    7 16.67% 177.5
    8 13.89% 147.92
    9 11.11% 118.33
    10 8.33% 88.75
    11 5.56% 59.17
    12 2.78% 29.58

    Observed
    Actual Delta
    % Count % Count
    2.44% 26 0.34% 3.58
    5.82% 62 -0.27% -2.83
    6.57% 70 1.76% 18.75
    12.86% 137 -1.75% -18.67
    13.33% 142 0.56% 5.92
    15.40% 164 1.27% 13.5
    15.49% 165 -1.60% -17.08
    11.27% 120 -0.16% -1.67
    7.98% 85 0.35% 3.75
    6.67% 71 -1.11% -11.83
    2.16% 23 0.62% 6.58

    Java Set 1
    Actual 1 Delta
    % Count % Count
    2.35% 25 0.43% 4.58
    5.63% 60 -0.08% -0.83
    8.64% 92 -0.31% -3.25
    10.89% 116 0.22% 2.33
    14.08% 150 -0.20% -2.08
    18.40% 196 -1.74% -18.5
    11.74% 125 2.15% 22.92
    11.46% 122 -0.34% -3.67
    8.36% 89 -0.02% -0.25
    6.20% 66 -0.64% -6.83
    2.25% 24 0.52% 5.58

    Java Set 2
    Actual 1 Delta
    % Count % Count
    1.78% 19 0.99% 10.58
    6.76% 72 -1.21% -12.83
    8.92% 95 -0.59% -6.25
    13.15% 140 -2.03% -21.67
    12.96% 138 0.93% 9.92
    15.49% 165 1.17% 12.5
    13.33% 142 0.56% 5.92
    11.83% 126 -0.72% -7.67
    7.89% 84 0.45% 4.75
    4.23% 45 1.33% 14.17
    3.66% 39 -0.88% -9.42

    Java Set 3
    Actual 1 Delta
    % Count % Count
    2.54% 27 0.24% 2.58
    4.88% 52 0.67% 7.17
    7.23% 77 1.10% 11.75
    10.70% 114 0.41% 4.33
    14.84% 158 -0.95% -10.08
    17.46% 186 -0.80% -8.5
    14.84% 158 -0.95% -10.08
    10.42% 111 0.69% 7.33
    8.92% 95 -0.59% -6.25
    6.10% 65 -0.55% -5.83
    2.07% 22 0.71% 7.58

    Java Set 4
    Actual 1 Delta
    % Count % Count
    3.19% 34 -0.41% -4.42
    5.82% 62 -0.27% -2.83
    9.86% 105 -1.53% -16.25
    11.08% 118 0.03% 0.33
    13.52% 144 0.37% 3.92
    17.65% 188 -0.99% -10.5
    12.96% 138 0.93% 9.92
    9.58% 102 1.53% 16.33
    8.36% 89 -0.02% -0.25
    4.60% 49 0.95% 10.17
    3.38% 36 -0.60% -6.42

    Java Set 5
    Actual 1 Delta
    % Count % Count
    3.57% 38 -0.79% -8.42
    4.51% 48 1.05% 11.17
    8.17% 87 0.16% 1.75
    9.11% 97 2.00% 21.33
    14.37% 153 -0.48% -5.08
    16.53% 176 0.14% 1.5
    14.74% 157 -0.85% -9.08
    11.83% 126 -0.72% -7.67
    8.26% 88 0.07% 0.75
    5.45% 58 0.11% 1.17
    3.47% 37 -0.70% -7.42

    Java Average
    Actual 1 Delta
    % Count % Count
    2.69% 28.6 0.09% 0.98
    5.52% 58.8 0.03% 0.37
    8.56% 91.2 -0.23% -2.45
    10.99% 117 0.13% 1.33
    13.95% 148.6 -0.06% -0.68
    17.11% 182.2 -0.44% -4.7
    13.52% 144 0.37% 3.92
    11.02% 117.4 0.09% 0.93
    8.36% 89 -0.02% -0.25
    5.31% 56.6 0.24% 2.57
    2.97% 31.6 -0.19% -2.02



  • What if I say that the amount of results does not matter but the order they come in?

    I'm not really sure if the AI is cheating or not but there do seem to be some irregularities based on over half of my games. Also, it seems weird that the stack traces of some errors indicate "Foo.Cheat.Bar" packages and methods. Although the Admin has said it's for testing purposes, it seems weird having it running on a public version.

    At the beginning of the game the dice favor the AI more. If the human player is not near a 4, then the 4 is rolled more. Once the player builds a settlement there, either the number does not come up as much or it doesn't matter because the AI is already ahead and benefits from it more than the human. Then all players can expand to other numbers. After that, the other numbers seem to "catch up" statistically, meaning that the statistics overall seem to be fine.
    Anyway, that's how I would hide my cheating AI if I had to do it.


  • administrators

    @CottonKit Wow! Thanks for the effort and the time you put into this! This is truly awesome! If it is okay for you i will link to this thread in the future ;)

    As far as the ongoing "debate" regarding those log entries goes:

    We have cheat codes activated for our own game accounts on the public server to be able to test on them in live mode as well during maintenance periods. The stack traces show them in some logs, but they can actually only be used by accounts that have a corresponding flag on the server: this applies only to our admin/testing accounts.



  • Go ahead and reuse this as you want. I am still tracking die rolls. I am at 1700 rolls right now. The 4 of the 5 numbers that were + or - 1%, or more, from their expected occurrence have moved closer to the expected occurrence percentage.
    4 1.76% underrepresented (6.57% observed, 8.33% expected)
    5 1.75% overrepresented (12.86% observed, 11.11% expected)
    7 1.27% underrepresented (15.40% observed, 16.67% expected)
    8 1.60% overrepresented (15.49% observed, 13.89% expected; 8 actually occurred more times than 7)
    11 1.11% overrepresented (6.67% observed, 5.56% expected)

    This may seem odd for 1000 rolls; until you look at the Java rolls, which also exhibit the same behavior. I posted the code, so you know there is no loading of the dice. The underlying dependency is the random number generator. Depending on how truly random it is, will influence the true randomness of the die roll. After 5000 (actually 1065 x 5) die rolls the Java version had dropped the deviance from the expected to less than .5%.

    I will update the results when I reach 2000 Catan die rolls. What I have already observed is that the overall deviation from the expected occurrence percentage has dropped for 4 of the 5 outliers.



  • I finished my second set of 1000 rolls. This set the 5 and 9 were + or - 1% from the expected results. Over the entire 2000 rolls, all 5 of the outliers from the first set moved toward the expected occurrence frequency. Only the 5 remains outside the + or - 1% occurrence.
    Total Normal Occurrence Actual Prob Delta
    2 33 27.08 3.38% -0.61%
    3 45 54.17 4.62% 0.94%
    4 84 81.25 8.62% -0.28%
    5 121 108.33 12.41% -1.30%
    6 145 135.42 14.87% -0.98%
    7 157 162.5 0.16 0.56%
    8 134 135.42 13.74% 0.15%
    9 97 108.33 9.95% 1.16%
    10 79 81.25 8.10% 0.23%
    11 59 54.17 6.05% -0.50%
    12 21 27.08 2.15% 0.62%

    Total   Normal Occurrence   Actual Prob Delta
    

    2 59 56.67 2.89% -0.11%
    3 107 113.33 5.25% 0.31%
    4 154 170 7.55% 0.78%
    5 258 226.67 12.65% -1.54%
    6 287 283.33 14.07% -0.18%
    7 321 340 15.74% 0.93%
    8 299 283.33 14.66% -0.77%
    9 217 226.67 10.64% 0.47%
    10 164 170 8.04% 0.29%
    11 130 113.33 6.37% -0.82%
    12 44 56.67 2.16% 0.62%



  • Maybe you can do another one where you play vs AI.

    The first 20 rolls in the game:

    1. How many resources does every AI get
    2. How many resources does human player get
    3. What are the numbers and their stats.

    Edit: You recorded 1000 dice rolls? From Catan Universe? Using AI?
    I assume that one Catan game in CU takes about 70 rolls and lasts 45 minutes. (Provided that you can complete one with the current roadbuilding critical/showstopper bug)
    (1000/70 games) *45/60 = 10.7 hours. It's basically a full day of playing the game.
    Let's use some best case scenarios: 80 rolls in 30 minutes. Usually there is some trading done by the players so that many rolls would not be done so fast IMO. This gives 6.25 hours. Still quite a long time.
    Not counting the downtime of finding new game etc.



  • @Stroom I compared the first 1000 die rolls versus what the die roll algorithm would produce over 5, 1000 roll sets. I then checked to see if there would be any significant difference. I found that the Cataan rolls matched up with what the Java program produced. Ideally that should be accompanied by a statistical analysis. The main problem is that 1000 die rolls is not a large enough sample size. I think 10,000 die rolls would ideally be necessary to validate whether something is producing an ideal distribution. 10,000 die rolls is really easy to produce in my Java program, not so much for Cataan. Your suggestion of looking at the first 20 die rolls, while interesting, would take 50 games to get 1,000 die rolls and 500 games to get 10,000 die rolls. I have been tracking die rolls with the goal of getting to 5,000. I just crossed the 4,000 roll mark. What I have observed so far is that the 5 occurs more than it is supposed to, and 7 occurs less. That is not to say that the die are loaded. That statement cannot be made without a statistical analysis of the data. I will do that once I finish collecting 5,000 rolls. All rolls were collected in normal games with 1 human player (me) and 2 or 3 AI players (depending if the game was Base, C&K, or Seafarers)



  • I basically agree that the dice rolls overall are more or less normalized. Over 5000 rolls you will probably not have a perfectly normal distribution, but that does not matter.

    What people think is happening that the AI is preferred by the dice rolls. It really seems like the games do not start off well for the player, the AI usually gets the resources for expansion sooner. This does not come out in your research because these two things are not related. You do not have enough metadata to say if the dice are fair when playing vs AI.

    If you want to cheat, you can cheat in a way to make the statistics seem OK. If the AI has locations 4,5,6 in one game and 8,9,10 in another one then overall, if the dice do favor the AI, it does not show up in the overall statistics.

    What I would like to see is the overall resources gathered by all the players like it was done in PlayCatan. And not in the end screen but dynamically.

    Also one way to make it conclusive if the AI is favored or not would be to see all the info about each player settlements, the resource cards they have and the die rolls in the order they come. Or the source code of the server, AI and the dice.



  • Update on this...

    Some people found out that the dice in Android Catan app (created by the same guys who are currently creating this game) are off even after 25k rolls. Seems like even numbers are preferred for some reason.
    https://www.reddit.com/r/Catan/comments/51hs1i/something_is_wrong_with_my_android_catan_app_dice/
    So you should really look at the functionality of your dice. It might be that in isolated cases (rolling 2 dice is the only thing you do) the dice work well but if you do it live (multiple rolls happening at the same time, very fast etc etc), it will start producing the same numbers. There are a lot of ways how your language-built-in RNG can get stuck and start giving unexpected results. Since you have so many out of sync problems when playing online (first online game for you?), it is most likely that your dice rolling is affected.

    Also, it seems weird that some topics are gone from this forum. The forum does not have a search function so you have to use Google and its' cache to see things.
    https://webcache.googleusercontent.com/search?q=cache:w7AnxO2hLMoJ:https://forum.catanuniverse.com/topic/1097/rolling-7-issues+&cd=1&hl=en&ct=clnk&gl=ee
    https://webcache.googleusercontent.com/search?q=cache:IMr64I_qX0cJ:https://forum.catanuniverse.com/topic/1146/classic-ai-cheating/4+&cd=1&hl=en&ct=clnk&gl=ee


  • administrators

    @Stroom Those threads were closed due to being inactive and being obsolet. We cleand up the forums yesterday and will continue to do so. About the other link you shared : We are very excited to see what findings the user will have with the illegal .apk he has of our game ....



  • Well, if you tested and implemented the dice properly then people would not have to resort to things like this :)


  • administrators

    @Stroom You are justifying this? For real? We always appriciate honest feedback but you can´t for real justify software piracy?!



  • Would it be better if he bought the game and then did the same thing? Yes, this is illegal and he should have bought the game himself. Piracy is bad. This part of his doings can not be justified. BUT

    I do not think it is illegal to take a (legally acquired) apk and reverse engineer it (lots of tools for it out there I'm sure). Especially if his main goal is to figure out what is wrong with the game. AND it is going to help you guys fix your game to provide better service to paying customers and earn more money. For free. (Unless the dice are intentionally unfair and publishing it will cause you to lose money. )

    If his findings help you fix the game, it would benefit you more than him buying the game so... is this specific case OK? If you have a zero-tolerance for hackers/pirates, fine.

    I did not look at this case from the point of piracy but from the point of analyzing the game to see what is wrong with it.

    What to consider as piracy?
    Knowing how to find a website which has an apk?
    Going to a website which has the apk?
    Downloading the apk (but not opening it)? (Legally I guess the line goes here, maybe not in some countries)
    Analyzing the apk to report a bug?
    Using the apk as it was intended to be used? (This is my line where piracy starts)
    Hosting a site where people can download apks? (Torrent trackers are not actually hosting the data so this is arguable)
    Personally uploading apk for others to download? (Definitely - also if you host a site too, that's doubly wrong)

    So this specific case... if he is not even going to play the game for fun... and he is helping your developers... It could be considered a hacker reporting a bug using unconventional methods.

    A lot of businesses have prizes for people who can hack into their systems and report critical bugs. You should maybe consider paying HIM if his findings reveal a bug with your dice :P


  • administrators

    @Stroom You are free to discuss your personal sense of justice in the off topic section of our forums with other users. From a legal point of view (which we prefer) it is clear that optaining and/ or providing a non authorized copy of one of our products is illegal.


Log in to reply