Contest Reorganization

Let us discuss in this thread ideas to improve the way contests are run. I would like to turn this contest into a very open process that does not rely on just one or two people.

So, your ideas are welcome here. Let's talk about them and figure out how to make this thing work well.

i don't get it. the contests are already an open process. people can already submit contest idea's, dispute if they disagree with something, etc.. without "one or two people" in charge of a given contest, how would anybody agree on anything?

I am just looking for ways to make it so the contest is run smoothly and those one or two people don't feel the weight of the world on their shoulders. A tight nit contest group by be in order for all I know where that group of say 5-6 people would vote on final decisions. I'm not sure what the answer is, but I know the current system isn't working too well.

How about the winners of the last 6 contests vote on the final decisions.
Or the SU's + Admin's

Either way there would be plenty of people to share out the task because there will always be someone online somewhere.

zombie quit griping.

the main problem seems to be with judging the solutions .. the last 2 contests saw the participants wait a few months before the results were out

perhaps, the judging process can be improved by having the judge scripts and test data prepared *before* the actual start of the contest, but only the contest admin will have access to the test data. this also makes it easier to field questions regarding judging.

another possibility to help streamline the process is to look into writing an automated judge script. take a look at sites like topcoder.com or acm.uva.es . they both employ different methods of automated judging... perhaps someone could be kind enough to write one in php and add it to the repository of php scripts available on this site.

Dont quote me on this but i was under the impression that there was a judging script for the last contest and some before that. Also the contestants should be patient and understand that things take time and the judges are not online 24/7, they have other priorities.

i didn't say a word.. i swear. ;)

but it is evident that this hard for anyone.. (even matt can see that.. )

Since building the test data seems to be taking most of the time, I simply suggest that every participant submits his own test data, which will be included into the final set.

This way, you can at the same time work on improving your program and also submit tests where your program performs well (assuming others will perform poorly).

I participated to such a contest, and it was very interesting (the winner won the 2 parts of the contest).

JC

Or have the test data generated randomly... Just work on a contest that doesn't need test data or that can use random test data. For this one, you could make something like this:



Then make sure that PlaceRandomRoom() makes some random coordinates, and a random shape room. Then make sure that some constraints are followed (only X% of the map is rooms, no rooms touching each other, etc) and you're done.

Perhaps for certain types of problems, random could be enough, but in my experience, random never gives difficult tests.

The best tests are always designed by humans, and as they take a lot of time, and since participants use their own tests, it's better to use these tests for the final judging.

Automating the final run should not take more than one week.

For contests where there's no obvious way to generate testcases (like corridors) I think it's a good idea to let the participants submit testcases for the contest. Participants could submit their testcases along with their entry (i.e., testcases wouldn't be public until after the contest), and I'd think you'd want to set a limit (say no more than one or two) on the number of testcases per participant. The admins could still generate some testcases, perhaps with some random-generation script that's provided a priori, that would be used in addition to the participant's.

You could imagine breaking the contest into two parts: (1) who comes up with the "hardest" testcase, and (2) who comes up with the "best" script. I think this is usually a bad idea because it will limit the diversity of testcases provided. (It would've been a bad idea for corridors, I think.) It might work well if the testcase-space was big but tightly-constrained, say if every testcase was a permutation of the integers 1..N.

Of course, this idea wouldn't apply to contests like blackjack where you're simulating something based on randomly events.

I also agree that all of the judging scripts should be setup before the contest begins. I'm not sure if this has happened in the past or not, and I realize that it's a lot of work for someone, but I really think it's the best way to set things up for a smooth contest.

about the judging script, i made sure it was available _before_ the contest started.. i think that is very important, because it takes the load of contestans, and let them only deal with what is important (ie. wrinting the best script for the contest ).

about the test cases, i like the idea where contestants would submit the test cases, but i think that the contest judge(s) (matt, zack, me, whoever) whould decide what test cases would be used in the final scoring.

like, each participant would be able (not obligated) to submit up to 10 test cases, and possible even a small text describing why he/she thinks his test cases are good.

then, when everyone submits, the judge(s) would decide what submited test cases (if any) to use from each contestant (or if he is not sattisfied with them, only then would he create his own)..

there are two different kind of judging scripts here ...

1. a script that a participant can use to test his code
- input: test data
- output: result

2. a script used to rank and run multiple scripts against each other
- input: many contest entries
- output: ranked results of each entry, ...


i would suggest that both 2 types of scripts should have been prepared before the competition to ensure smooth judging. in the last contest, i would hypothesize that the 2nd type of judging script was created much later only by zack

Your hypothesis is correct.