The Lockbox Data Collection Project

[Yokai]

Noupoi: What is the new set of "Validated Frequency" stats in the upper right of the spreadsheet? You've now got two sets of drop rates. Even more confusing is that the "validated" drop rate for oranges is exactly the same for T2 and T3. Is this a new work in progress?

On a different angle, your original "Frequency" and "Drop Rate" columns for T2 seem to be moving the T2 drop rate for oranges much closer to 2% than the 3% it appeared to be yesterday. PSA for everyone: if the true drop rate for T2 oranges is only 2%, then the "best value" for your keys spent becomes T3 lockboxes. Just saying. I'm about to go update my page in the sig block link below accordingly.

Noupoi, on a different angle, would you be willing to improve the data collection form to minimize the ability to enter impossible result sets, then wipe the data and start over? Anything we can do to improve sampling accuracy will be beneficial to all.

[jetah]

Noupoi: What is the new set of "Validated Frequency" stats in the upper right of the spreadsheet? You've now got two sets of drop rates. Even more confusing is that the "validated" drop rate for oranges is exactly the same for T2 and T3. Is this a new work in progress?

On a different angle, your original "Frequency" and "Drop Rate" columns for T2 seem to be moving the T2 drop rate for oranges much closer to 2% than the 3% it appeared to be yesterday. PSA for everyone: if the true drop rate for T2 oranges is only 2%, then the "best value" for your keys spent becomes T3 lockboxes. Just saying. I'm about to go update my page in the sig block link below accordingly.

Noupoi, on a different angle, would you be willing to improve the data collection form to minimize the ability to enter impossible result sets, then wipe the data and start over? Anything we can do to improve sampling accuracy will be beneficial to all.

I think the idea was that you could open multiple boxes and input it. ie you could open 2 T4 boxes and just have 1 entry for 8 items.

[Noupoi]

Noupoi: What is the new set of "Validated Frequency" stats in the upper right of the spreadsheet? You've now got two sets of drop rates. Even more confusing is that the "validated" drop rate for oranges is exactly the same for T2 and T3. Is this a new work in progress?

On a different angle, your original "Frequency" and "Drop Rate" columns for T2 seem to be moving the T2 drop rate for oranges much closer to 2% than the 3% it appeared to be yesterday. PSA for everyone: if the true drop rate for T2 oranges is only 2%, then the "best value" for your keys spent becomes T3 lockboxes. Just saying. I'm about to go update my page in the sig block link below accordingly.

Noupoi, on a different angle, would you be willing to improve the data collection form to minimize the ability to enter impossible result sets, then wipe the data and start over? Anything we can do to improve sampling accuracy will be beneficial to all.

Validated frequency only includes entries that pass the basic validation which checks if the number of items in that entry is a multiple of the number of items you could get from a lockbox of that tier- this means that impossible data is ignored. The idea was that you could input the results for multiple lockboxes at a time, but also to make it difficult for trolls to enter large amounts of false data at once.

I'm open for suggestions on how to improve the data entry form. If it doesn't actually involve changing the questions, but how they're inputted, there's no need to wipe the data. We could just redo/add a results page for data entered after a certain time - so not including data before a certain point, and move over to that when we have enough data for that to become reliable.

Edit: I've also given edit access to the people who've asked me for it. Hopefully you'll be seeing changes they make soon!

[Yokai]

Validated frequency only includes entries that pass the basic validation which checks if the number of items in that entry is a multiple of the number of items you could get from a lockbox of that tier- this means that impossible data is ignored. The idea was that you could input the results for multiple lockboxes at a time, but also to make it difficult for trolls to enter large amounts of false data at once.

I'm open for suggestions on how to improve the data entry form. If it doesn't actually involve changing the questions, but how they're inputted, there's no need to wipe the data. We could just redo/add a results page for data entered after a certain time - so not including data before a certain point, and move over to that when we have enough data for that to become reliable.

Okay, then the validated data is showing an interesting trend/pattern:



At first I was surprised at how closely these validated results matched the non-validated results in nearly every respect except for the steep drop in probability for oranges from T3 boxes. But then I looked more closely at the pure drop rates (not the cumulative probability). See the pattern? Here's what I see, translated to verbal narrative:

T2 has a very low drop rate for blues and higher, with nearly 75% of drops being green.
T3 keeps THE SAME LOW drop rate for orange as a T2, but greatly pushes up the drop rate for blues and purples.
T4 slightly ups the drop rate for blue, keeps purple the same as T3, and finally gives you a 250% improvement in drop rate for orange.

Honestly, this pattern makes sense for for the value proposition among the three tiers from the standpoint of drop rates alone. I could see a designer/dev picking numbers like these because they "look good and make sense, and will encourage players to hold out for T4s if they really want an orange, but to settle for T3 if what they're really looking for are purples and blues. We'll give them so many greens at T2 that they'll at least be tempted to move up to T3".

But in this case, look at the whacky, non-linear value propostion when you view the cumulative probability per X number of keys spent!

T2 has the best cumulative odds for getting an orange
But T3 has the best cumulative odds for getting a purple
T4 is still on the bottom of the heap, though.

[TomahawkJackson]

Edit: I've also given edit access to the people who've asked me for it. Hopefully you'll be seeing changes they make soon!

I'm hard at it already...

CHANGELOG
Moved Validated Results to it's own tab, where they will grow their own charts, based on the Validated data.
Corrected a raw data entry where the user input EGO 'one hundred and twenty', exactly how it says not to do it on the form. Joker. It's now 120.
Removed a raw data entry where the user input EGO 7410.
Removed several repeated raw data entries where the user said they got 10 of everything from a T2 box. 7 times in a row.

TODO
Make charts replicating the raw charts on the first tab, using the Validated data.
Create additional charts showing the likelihood of receiving multiple items based on the number of items received/the number of Lockboxes submitted (i.e. the data actually shows that the users got 686 Green Items from 241 T2 Lockboxes. You are likely to get TWO green items from a T2 box. That should be represented by a 200% chance to get a Green Item.)

[Armanewb]

Whoever changed the title of the Expected Items per 100 Keystones to lockboxes, I've changed the title back. It's not per 100 lockboxes, as they cost different amounts of keystones and thus it would not be a comparable measure.

[Stickasylum]

Whoever changed the title of the Expected Items per 100 Keystones to lockboxes, I've changed the title back. It's not per 100 lockboxes, as they cost different amounts of keystones and thus it would not be a comparable measure.

It also shouldn't be a percentage, since it is a count! If you reformat it, you'll probably need to multiply by 100 to keep the number the same.

Okay, first off, I apologize for the prickly tone. Second, I apologize for being wrong, lol. I read your most recent example in a rushed state and misinterpreted something.

No worries! Glad to know I'm not crazy

[Thunderclap]

Sledge is on live stream now. When someone asked the question stating about the different tiers he said it is supposed to be random and that you would have to open "tens of thousands" of boxes to prove that another tier is better than 4. So the offical answer is there is nothing wrong. (he's on his twitch channel if you wonder 4/12/13 10 ish to 11 ish starting. stream if he records it has no sound other than him to begin with

[Yokai]

Sledge is on live stream now. When someone asked the question stating about the different tiers he said it is supposed to be random and that you would have to open "tens of thousands" of boxes to prove that another tier is better than 4. So the offical answer is there is nothing wrong. (he's on his twitch channel if you wonder 4/12/13 10 ish to 11 ish starting. stream if he records it has no sound other than him to begin with

Not everyone understands the notion of cumulative probability very well. T4 certainly has better inherent drop rates per box. We can already see that from the observed results so far in the data collection--validated or not.

The problem is whether the relative pricing for each box supports Sledge's claim based on the notion of cumulative probability. So far the observed results indicate that other tiers are better than T4 when you look at things from the notion of cumulative probability. Put simply: for any given amount of keys, you get more T2 or T3 "item rolls" than T4 "item rolls", and the cumulative probability for those increased rolls can actually work out to a BETTER chance at scoring an orange or purple.

Unless Trion balanced the three tier prices and inherent drop rates with a specific eye towards cumulative probability, to ensure that T4 also offers the BEST cumulative probability, and T3 offers the NEXT BEST cumulative probability, then Sledge might well be wrong in his statement.

Finally, the "you'd need to open tens of thousands of boxes to prove" is a well intended and innocent statement, but also simply wrong. If we had accurate reporting to the spreadsheet collection form, actually all we'd need is 377 data points per teir to feel pretty damn confident.

[Stickasylum]

Well, to be fair, as the table gets more data (and as the data gets validated), the orange drop rates seem to be getting closer and closer to the cutoffs where the efficiencies are equivalent. From the expected drops per keycode perspective, the T3 drop rate needs to be more than half the T4 rate for T3 to better, and the T2 needs to be more than a quarter. Right now, the orange drop rate for T3 is awfully close to this cutoff, so its looking like we would need a lot more data to know either way (and the effect would likely be very small). Similarly, to show T3 is better from the cumulative probability perspective, you would need to test the hypothesis that p_3 > 1-(1-p_4)^(1/2). Again, the current numbers are awfully close to this cutoff!

It's even entirely possible that the real drop rates have a 1:2 ratio, giving T3 and T4 equivalent efficiencies for orange drops. If this is the case, however, T3 is still clearly better than T4 as it clearly has better blue and purple efficiency (and you run less risk of accidentally maxing your keycodes and wasting incoming keycode drops). The only downside to T3 would be the greater scrip cost, but scrip has never been a problem for me, even buying T2s!