Iron Chef Optimization Challenge in the Playground LIX

[DeAnno]

The concern I, and I think Amphetryon, have is that a build could theoretically do very well in UoSI, perhaps scoring just below excellent while missing a class feature or two, and have zero competitive advantage over a build that just barely avoided poor, making use of a couple of class features and perhaps not even completing the SI.

I'm not concerned about a high granularity in the way the subscores are presented, but I would be extremely alarmed if the criteria used to rank the builds was simply an averaging of those very granular subscores. This is especially a concern with UoSI set up in such a way that it will generally converge the scores to Average.

[Piggy Knowles]

From the FAQ:

So, what do these new rules mean for judging? Basically, there's a couple of big changes.
No point scores: Instead, I'll be asking judges to organise the entries much as we do in the summary tables - instead of giving your "best scoring" entry a 19.75, instead, just mention it as "First" (or "top", or any synonym thereof). Scoring in the specific categories will be a word instead of a number; I'm using an Excellent/Average/Poor guideline for now, but I'm more than willing to go to an Excellent/Above Average/Average/Below Average/Poor scale instead. Note that this score is more for the improvement of the chefs than anything else - a judge's top build could very well score three Averages and a single Excellent, and still place above an entry that had three Excellents and a Poor.
More/less responsabilities on judges: No need for a complicated formula - just rank your builds in your own preference, and give a short explanation why you scored each category as you did, and maybe an overall commentary.
More fluid judging guidelines: Guidelines? Who needs 'em. Same as always, you choose what you're judging on. So long as you judge everyone by the same standards, I wo't stand in your way.

Emphasis mine.

So unless I'm mistaken, the Excellent/Average/Poor is mostly for the chef's edification, correct? It seems like the commentary is secondary, and that "scoring" is primarily by the judge ordering builds as he or she likes (as per the second bolded line).

[Darrin]

Correct me if I'm wrong, but it sounds something like an instant runoff voting system? But with so few judges compared to the entries, I'm not sure how it will work.

Given the small number of judges compared to the entries, I'm not sure how a ranked or preferential system would work with, say, only two judges. If both judges select a different First Place, there's no other ballot to break ties.

Hmm. If the lack of judges is a problem (and it has been before), maybe move to a Borda Count where each contestant ranks every entry except his own, and then add up the points?

If the root problem is the number of disputes, then there may be another possible solution: Keep the 20-point format, but get rid of half/quarter/fractional points, and the judges post *only* the scores. No discussion or explanation is provided by the judges on why a particular score is low. Although that might have the exact opposite effect, as contestants slam the chairman with disputes but they can only guess why a particular judge scored what they did. Or rather, maybe this is the same as "There Is No Dispute": you get your score, and no matter how unfair it is, you can't change it.

[Deadline]

If the root problem is the number of disputes, then there may be another possible solution: Keep the 20-point format, but get rid of half/quarter/fractional points, and the judges post *only* the scores. No discussion or explanation is provided by the judges on why a particular score is low. Although that might have the exact opposite effect, as contestants slam the chairman with disputes but they can only guess why a particular judge scored what they did. Or rather, maybe this is the same as "There Is No Dispute": you get your score, and no matter how unfair it is, you can't change it.

These are not ideas I would like. We've had judges score without providing information, and it increased disputes. Also, scores without any commentary is not something I would want on a personal note.

And given how easy it is to miss something, eliminating disputes seems like a bad idea.

Due to a lack of time, I'm going to take a pass on judging. There are some great builds though.

[sideswipe]

i would judge but i believe it would be unfair as i don't have the same level of optimisation experience that most of the other judges have. or the high level play experience. so as much as i want to i don't think i would give a truly fair outlook on the builds. would that be an issue?

[DeAnno]

i would judge but i believe it would be unfair as i don't have the same level of optimisation experience that most of the other judges have. or the high level play experience. so as much as i want to i don't think i would give a truly fair outlook on the builds. would that be an issue?

It might be helpful for you to give the builds a detailed look over and see if you form some concrete opinions. After looking over all the builds, you might be a lot more confident in which you like than you thought you would be (and if you don't feel too confident, you could still refrain from judging.) Either way, looking over all the builds would expose you to a lot of common and uncommon optimization tricks and might help you feel better about judging or participating in a later contest.

[sideswipe]

It might be helpful for you to give the builds a detailed look over and see if you form some concrete opinions. After looking over all the builds, you might be a lot more confident in which you like than you thought you would be (and if you don't feel too confident, you could still refrain from judging.) Either way, looking over all the builds would expose you to a lot of common and uncommon optimization tricks and might help you feel better about judging or participating in a later contest.

i participated last round and did reasonably well. but i am only very familiar with my own set of likes and optimisation tricks, so yeh i will pay attention to the builds in the next couple rounds and put myself up for judging then. thanks.

[Thurbane]

All a bit quiet around here right now - any word on judging?

[OMG PONIES]

my primary concern is that there aren't enough of an ability to differencate between different builds, two builds getting the same score on the limited gradient even through one is just about to drop to the one below, while the other is top of the chart while not growing higher

That reads to me as leaving a whole bunch of ties being likely. Is that your intent, and how you read the new rules for scoring as intended?

Please keep in mind, there aren't really "scores" to be gotten here, so there aren't really going to be ties. If we assume two builds both receive "Average" rankings in all four categories, the judge is still providing a ranking of the builds in their order of opinion from first to last place. Therefore, a build that is theoretically "almost excellent" in every way could still be ranked higher than a build that is "almost poor" in every way. I read the new rules as intended to break the illusion of judging as an objective measurement and instead reinforce that they are simply a judge's [hopefully educated] opinion.

The concern I, and I think Amphetryon, have is that a build could theoretically do very well in UoSI, perhaps scoring just below excellent while missing a class feature or two, and have zero competitive advantage over a build that just barely avoided poor, making use of a couple of class features and perhaps not even completing the SI.

Again, please see above. Additionally, please remember that Use of the Secret Ingredient is not the only category judges are considering in creating our rankings. A build's competitive advantage may lie in its Originality, Power, or Elegance.

I'm not concerned about a high granularity in the way the subscores are presented, but I would be extremely alarmed if the criteria used to rank the builds was simply an averaging of those very granular subscores. This is especially a concern with UoSI set up in such a way that it will generally converge the scores to Average.

Again, a ranking of "Average" in a particular category may not have much bearing on one's final placement. Piggy's hit it on the head:

So unless I'm mistaken, the Excellent/Average/Poor is mostly for the chef's edification, correct? It seems like the commentary is secondary, and that "scoring" is primarily by the judge ordering builds as he or she likes (as per the second bolded line).

Each judge will provide a ranked list of entries, so even if the entire field is ranked "Average" in Use of the Secret Ingredient, there will still be separation between the builds' rank.

Correct me if I'm wrong, but it sounds something like an instant runoff voting system? But with so few judges compared to the entries, I'm not sure how it will work.

Given the small number of judges compared to the entries, I'm not sure how a ranked or preferential system would work with, say, only two judges. If both judges select a different First Place, there's no other ballot to break ties.

An instant runoff system is exactly what we've got here. With two judges voting differently, any ties would traditionally be resolved by the Chairman. After all, it has happened in prior rounds with our numerical scoring system.

Hmm. If the lack of judges is a problem (and it has been before), maybe move to a Borda Count where each contestant ranks every entry except his own, and then add up the points?

I like, but I think we've got the opposite problem there as we do with the new instant runoff. Instead of having too few judges to break ties, etc, we'd have to wait for too many folks to respond if we were looking for every entry. Also, this would dissuade folks who didn't enter from sharing their opinions. Perhaps we could try an open Borda Count, where everyone is allowed to contribute rankings--contestants and non-contestants alike. Also, everyone is still allowed to vote on all the entries, even their own. Even if you give yourself points by ranking yourself number one, we'd be considering the points you receive from all the other rankers as well.

If the root problem is the number of disputes, then there may be another possible solution: Keep the 20-point format, but get rid of half/quarter/fractional points, and the judges post *only* the scores. No discussion or explanation is provided by the judges on why a particular score is low. Although that might have the exact opposite effect, as contestants slam the chairman with disputes but they can only guess why a particular judge scored what they did. Or rather, maybe this is the same as "There Is No Dispute": you get your score, and no matter how unfair it is, you can't change it.

The first option--scores without commentary--has happened before and, as you predicted, caused the opposite result. The second option--disallowing disputes--was actually how the competition started, but it devolved into madness in one particular thread where people started voicing grievances in-thread, leading to the creation of our dispute system.

All a bit quiet around here right now - any word on judging?

It's still happening . To be more specific, my comments on Originality are done for all 13 builds. To give you an idea, I've ranked 4 builds as Poor in this category, 6 as Average, and only 3 as Excellent. I'm not sure what the spread will look like on Power, but Elegance should be fairly simple as long as everything's rules-legal and doesn't ruffle too many feathers. Use of the Secret Ingredient will be tricky, as the Ingredient seems to be pulling itself in different directions at the same time.

[dysprosium]

I get the idea of how this round is going to be scored. The judge puts the builds in order and each category has their rating excellent, average, poor. So even if two builds are excellent across the board one is still ranked higher by the judge.

But what happens with multiple judges? That is where I can't get my head around.

Build A is excellent across the board by both Judge 1 and Judge 2.
Build B is excellent across the board by both Judge 1 and Judge 2.

Judge 1 scores them A followed by B.
Judge 2 scores them B followed by A.
This would come out to a tie right?

Even if Judge 2 explained how Build B really fit all of his criteria and was absolutely brilliant. Judge 1 believed that Build A was only slightly better than Build B. Or am I thinking too much into this? I guess until we have more than one judge for this round this question is moot?

I do like the idea of the judges making commentary. This is how we all learn to build better. Speaking from personal experience, I've learned immensely from reading what judges have given not only me but from other entrants as well. It has made me a better builder and a better judge.

I don't like the idea of the builders/fans vote. I think it takes away from what judges do. To put another spin on what Ponies had mentioned, we would have to wait for all of the builders to vote. Some people who enter are not as vocal or post on the thread as much as some of us do. Also what of those with multiple entries? I entered two builds this round, would that mean I would get two votes?

I had another thought but I don't know if I can really voice it without causing too much of an unncessary headache.

[Darrin]

But what happens with multiple judges? That is where I can't get my head around.

Build A is excellent across the board by both Judge 1 and Judge 2.
Build B is excellent across the board by both Judge 1 and Judge 2.

Judge 1 scores them A followed by B.
Judge 2 scores them B followed by A.
This would come out to a tie right?

Even if Judge 2 explained how Build B really fit all of his criteria and was absolutely brilliant. Judge 1 believed that Build A was only slightly better than Build B. Or am I thinking too much into this? I guess until we have more than one judge for this round this question is moot?

To avoid ties, we need at least three judges. Actually, there's still a chance with an even-number of judges for a tie, but the more judges you have, the less likely that will happen. Ideally, you want an odd number of judges.

In an instant runoff voting system, when you don't have a clear winner, you look at all the other ballots cast to break ties. My concern is that in most instant runoffs, there's a large number of ballots cast compared to the candidates, so the odds of there being a tie are practically nil. But for Iron Chef, the number of judges compared to the number of entries tends to be very, very low. If there's no clear consensus on how the winners break down, we either have to let the ties stand (has happened previously) or ask the chairman to break the ties (I'm not sure if this has been done before).

Example:

Let's say Judge #1 ranks Hufflepuff as First Place and Judge #2 ranks Slytherin as First Place. Tie. So long as we have a third judge, we can resolve it. If Judge #3 ranks either Hufflepuff or Slytherin as First Place, tie is broken and we have a clear First/Second result. If Judge #3 has Ravenclaw at First Place, then it's a little more complicated, but we can look at how all the judges ranked the rest of the candidates and weight them accordingly. If Judge #1 has Hufflepuff at First, Judge #2 has Hufflepuff at Second, and Judge #3 has Hufflepuff at Third, and Judge #1/#3 both ranked Slytherin dead last, then we have something of a consensus.

While I'm generally in favor of instant runoff systems (this is how the Academy Awards and Hugo Awards are determined), I remain concerned how it will shake out with a small number of judges. We'll know more when we get some scores posted.

I don't like the idea of the builders/fans vote. I think it takes away from what judges do. To put another spin on what Ponies had mentioned, we would have to wait for all of the builders to vote. Some people who enter are not as vocal or post on the thread as much as some of us do.

Well, we have a judging deadline for a reason. If a contestant didn't have time to submit scores, and the chairman has enough scores submitted to declare a winner, then I don't see a problem there. My thinking was that if you're a contestant, you're very likely to be active in the thread, and thus much more likely to submit scores. But I have problems with this idea as well... the non-contestant judges provide invaluable feedback, and I wouldn't want to lose that. I'm not sure how to weigh contestant/non-contestant scores so it would be fair. Having both would be awesome, but from a practical standpoint I'm not sure how it would work.

Also what of those with multiple entries? I entered two builds this round, would that mean I would get two votes?

I hadn't thought of that. I'm not sure how to resolve that.

[dysprosium]

I guess in my example I should not have stopped at two. Your explanation makes sense and I can see that working.

I agree with you that the non contestant judges with appropriate commentary is the invaluable part. I just didn't want us to go the way of any of those TV shows with fan voting (since they almost always get it wrong . . . ).

[OMG PONIES]

We'll know more when we get some scores posted.

Aye, there be wisdom here. I'll focus on my scores for now and leave comments on the system itself until afterward. Please keep in mind, everyone, this is an experiment. If it doesn't work, we'll try something else. After all, we're not crazy. Well, most of us... *twitch*

[Amphetryon]

Aye, there be wisdom here. I'll focus on my scores for now and leave comments on the system itself until afterward. Please keep in mind, everyone, this is an experiment. If it doesn't work, we'll try something else. After all, we're not crazy. Well, most of us... *twitch*

This is true. My mother had me tested.

[KrimsonNekros]

Hey! What's wrong with crazy?

[Kuulvheysoon]

Come on guys - everybody knows that "sane" is just another synonym for boring.

[Kreuz]

I found myself without internet these weeks so I couldn't submit my idea...

I wanted to combine the Bounding Assault Maneuver + Greater Mobility + Cheetah's Sprint + Robilar's Gambit + Scorpion's Grasp + Air Walk + Girallon's Blessing + Stand Still + stuff that increases size to move around the battlefield, provoke AoOs, answers the AoO with unarmed strikes, grapple, pick up to 4 enemies and drag them around the battlefield, then go up to 200 feet, drop them on top of the last guy, finish that round attacking the last guy, preventing him from moving away with Stand Still, and the next round having him take 80d6 of damage, and each of the falling enemies taking 20d6 of damage, and maybe AoOs from entering the area via dropping down. I was thinking about how to fit Thicket of Blades in there, and I lost my connection... well, I'll just have to wait for the next challenge.

[KrimsonNekros]

Come on guys - everybody knows that "sane" is just another synonym for boring.

Finally someone else gets it.

[Piggy Knowles]

How are the judges doing? Is this style of judging any easier/harder than the standard? Any judges other than the illustrious Mr. Ponies still in it?

[dysprosium]

It may be that Ponies is our "guinea pig" judge. Others may be willing to judge but waiting to see what Ponies posts and what reaction the thread has.

Pure speculation of course.

[KrimsonNekros]

Pure speculation of course.

Aren't there rules clearly posted in the opening post that forbid speculation? Sir I may need to call the chair upon you!

[dysprosium]

Oops! You got me.

[Vaz]

Give him the chair! Give him the chair!

[Kuulvheysoon]

Give him the chair! Give him the chair!

Gott say, I love that .gif. Nicely played, good sir.

[Sian]

How is the judging going?

[Ikeren]

All the judges are done. They're just afraid to post

[DeAnno]

It would be a little funny after all this discussion about ties in the New system, there was only one judge anyway

[sakuuya]

It would be a little funny after all this discussion about ties in the New system, there was only one judge anyway

And then there were ties anyway.

[OMG PONIES]

It may be that Ponies is our "guinea pig" judge. Others may be willing to judge but waiting to see what Ponies posts and what reaction the thread has.

Ah yes, a guinea pig--on account of my girth, squeaky whine, and inability to do anything except urinate on a pile of woodchips. Nicely crafted insult, sirrah.

How are the judges doing? Is this style of judging any easier/harder than the standard? Any judges other than the illustrious Mr. Ponies still in it?

I'm illustrious now? Sweet. I'm actually finding this round and style of judging both make judging harder. Since this Ingredient isn't one that confines itself to one particular playstyle, there's a wide spectrum of opinion on what makes a "good" Fleet Runner. Also, the new style of judging is much more based on feel and opinion, disallowing me the opportunity to use a carefully created rubric and basic arithmetic to arrive at a score.

I found myself without internet these weeks so I couldn't submit my idea...

I wanted to combine the Bounding Assault Maneuver + Greater Mobility + Cheetah's Sprint + Robilar's Gambit + Scorpion's Grasp + Air Walk + Girallon's Blessing + Stand Still + stuff that increases size to move around the battlefield, provoke AoOs, answers the AoO with unarmed strikes, grapple, pick up to 4 enemies and drag them around the battlefield, then go up to 200 feet, drop them on top of the last guy, finish that round attacking the last guy, preventing him from moving away with Stand Still, and the next round having him take 80d6 of damage, and each of the falling enemies taking 20d6 of damage, and maybe AoOs from entering the area via dropping down. I was thinking about how to fit Thicket of Blades in there, and I lost my connection... well, I'll just have to wait for the next challenge.

This right here would have been something fun to see! I was tinkering with a similar strategy (provoke all the AoOs and use them to fuel a way to respond), but I didn't even get it as fleshed out as you have.

All the judges are done. They're just afraid to post

If only. I've got Originality done as well as most of my Elegance comments, but Power and Use of the Secret Ingredient will take me more time this round. I'm looking at the looming deadline and wondering if our Chairman may consider a one-week extension on judging? Pretty please?

[dysprosium]

Ah yes, a guinea pig--on account of my girth, squeaky whine, and inability to do anything except urinate on a pile of woodchips. Nicely crafted insult, sirrah.

It was so nicely crafted I didn't even see it as an insult . . .

I'm illustrious now? Sweet. I'm actually finding this round and style of judging both make judging harder. Since this Ingredient isn't one that confines itself to one particular playstyle, there's a wide spectrum of opinion on what makes a "good" Fleet Runner. Also, the new style of judging is much more based on feel and opinion, disallowing me the opportunity to use a carefully created rubric and basic arithmetic to arrive at a score.

Yours was a carefully created rubric that had been honed over time to become a standard among other judges. Even with using arithmetic to come to a score, you have been known for giving good commentary--which is something that we have come to enjoy/wish for.

If I were judging, personally I would not change how I did it and instead of putting numbers I would put the excellent/average/poor ratings. Could you do something similar Ponies?