Suggestion for how to prevent reverse-engineering of challenge problems.
Taken from the thread on CHEFPAR and reverse-engineering instance parameters. In my opinion it is better if the competition is to find the program that works best on a random problem instance, rather than being a competition to reverse-engineer the hidden (or visible) instances. I think that would make CodeChef a more attractive environment to compete in (it is already very good, by the way - I hope this suggestion might help make it a bit better).
Definitions: by “feedback-instances” I mean those which contribute to your provisional rank, and for which you get back a score (four of these in the CHEFPAR contest, one for each type). By “hidden instances” I mean those for which you don’t get a score back during the contest (sixteen of these in the CHEFPAR contest, four for each type).
As far as I can make out, there are two competing needs:
(i) It is nice if people can be confident during the competition that their program will work (not abort, time-out etc) on hidden instances, so they need some kind of feedback during the competition that it works on hidden instances. It’s also nice if people feel able to try out lots of solutions over the 10 days.
(ii) It is good (in my opinion, anyway) if you can’t use the information from the result code (AC, TLE etc) to reverse-engineer parameters of the hidden instances. It’s also good if you can’t use the result information (score, time, memory usage) from the feedback-instances to reverse-engineer their parameters.
I think you can satisfy both of these needs by the following modifications to the rules:
(i) You are still allowed to submit a lot of answers (say 200), but in only (say) 20 of these will you get result code feedback from the hidden instances. 200 should be plenty to try out lots of ideas, and 20 should be plenty to make sure your program (that you already know runs successfully on feedback-instances) doesn’t crash on hidden instances. The user would get to choose which (up to 20) submissions count for the extra result code information. (The displayed time and memory usage would be taken from the feedback-instances, so you can’t get extra information about the hidden instances that way.)
(ii) feedback-instances are excluded entirely from the final score (after competition ends). In the case of the April 2018 challenge, that would mean you would be judged on 16 hidden instances rather than 20 = 4 + 16 instances. The reason for this is that it is very easy to leak information back from the feedback-instances, much easier than from the hidden instances. For example in CHEFPAR, during the competition you got back a 16-digit final score: you could encode information in some of these digits, so you get back much more than 1 bit of information per submission.
I think these changes would keep most of the necessary feedback so that users know their code is working and preserve the fun of seeing a live scoreboard, but would mostly prevent the problem being “hacked”.