Faster convergence for experts: Compared to the previous rating system, Elo-MMR awards a much higher increase to users who do well in their first CodeChef rounds.
I can’t call myself an expert, but I was not a complete beginner when I participated in my first CodeChef contest. Still my performance was far from optimal (rank 302 in division 3), largely because I wasn’t familiar with the UI of the CodeChef platform during that contest. Now I wonder, how much did this blunder affect my curent Elo-MMR score? Does Elo-MMR quickly adapt or does it assign a high weight to the first contests, which sticks longer?
Also even complete beginners may improve really fast after only a few contests thanks to learning new tricks and algorithms (there are a lot of low hanging fruits). Does the new rating system now penalize beginners, making the results of their first contests more important?
The weight of the first contests is not increased. Instead, the weight of the provisional starting rating is decreased. Glicko and TrueSkill do essentially the same thing.
The weight of the first contests is not increased. Instead, the weight of the provisional starting rating is decreased.
Isn’t this basically the same thing? What you are saying sounds like the new system essentially replaces the initial provisional rating with the performance of the first contests. Making the first contests very important, considering that the system is also “Robust: players will never lose too much rating for a single bad day” (“players will never gain too much rating for a single good day”), which may indicate that the system adapts slowly after the initial bootstrap.
If the system converges very fast for new accounts but then adapts very slowly, then this may provide a strong incentive for accounts rerolling (to get a more favorable starting position).
Could you please calculate Elo-MMR scores of CodeChef users after discarding the results of their first contest and share the results on drive? I would like to confirm whether the impact of the first contest still persists many contests later or not.
BTW, looks like your account may be another example of an expert (you were candidate master on codeforces in 2012), who happened to have bad performance in their first CodeChef contest.
It’s not the same thing: a user who has done a lot of contests would have almost zero weight on either the provisional rating or the first few contests. For such a user, feature #8 is completely irrelevant.
If you’re worried about robustness, the system does have a bit of a memory. A player who is consistently overperforming (or underperforming), or who performs inconsistently, would be allowed larger updates, whereas a very consistent player’s rating would be less sensitive. It’s basically a gentler version of the “volatility” parameter in the older systems, but carefully designed to retain incentive-volatility.
Did you consider that the system may grant too high of a “newbie” bonus?
look at this user:
#24 Div4 and #5 in Div4. 2 contests total and his MMR is 2166. I can easily score better than him but I cannot reach 2166 MMR. Reaching almost 6* with only Div 4 contests can’t be correct.
To be fair, this user wouldn’t normally have been placed into Div 4: their Elo-MMR rating after the first contest was already too high.
The #4 placer is a better example of a user who got a huge boost on their first contest, to 1966. Is that too high? Well, keep in mind that Topcoder rated Petr at 1866 after his first round, which was half as long as a CodeChef round (and I think they use a smaller \mu of 1200 instead of 1500, though Elo-MMR has the 3\sigma penalty from feature #7 instead…)
Now, I don’t have enough experience with Div 4 contests to judge how fair this is. The rating systems are not aware of problem difficulty; they just assume that the problems are sufficiently calibrated to accurately rank all the contestants. If the user had done well in Div 2 or 1 instead, I imagine you’d be less surprised.
One approach we could take is to cap performance scores from Div 4 contests to something like 2000. I don’t know if that’s necessary, but it would be interesting to see if these users actually turn out to be overrated when they later go on to compete in Div 1-2.
@admin want to say something??.. Now everyone will make a new account and get a 5 or 6 star rating with ease. All that matters is the heighest rating, which we will write in our CV. Soon CC rating won’t be considered for evaluating someone’s skill.
Alt accounts are clearly being created to gain a higher rating…old accounts don’t gain much for the same rank…
Newbie gains are not necessarily newbie gains.A rating system that has such a loophole makes it a deterrent for people who don’t resort to such tactics. I think “fairness/robustness to cheating” is the most desired variable.
We have introduced Provisional Ratings, which signify that a new user’s ratings aren’t very indicative of their actual rating for the first few contests that they participate in.
In particular, until a user participates in at least 5 rated contests, their rating will be accompanied by a question mark, indicating that it is a Provisional Rating.
Thanks, that’s the move in the right direction. Could you please also ensure that the users with provisional ratings don’t get star badges assigned to them in this forum and in other places until they finish their 5th rated contest?
By this what changes should be done in Provisional
Rating Means In Provisional
Rating rating they get is high (if they solve more problems) +ve delta as compare to Normal Rating
I’m quite confused and disappointed with what’s been happening with the rating in Codechef. Yesterday after the STARTER50 my new rating on my profile was changed to 1913 from 1882. Today it changed to 1893 without participating in any contest. Even though ranked 11 globally my overall rating only changed 11 on profile! But I just checked the file ‘changes_after_starter50’ from the drive link. It shows the current elo_rating is 1802 and the previous elo_rating is 1836. I don’t get the math here. This whole transition is a mess.
After launching Elo-MMR, we got feedback from a lot of users about the sudden increase in ratings of new users, particularly in their first couple of contests. This was supposed to be a feature, but it had been a bit too much. So now, we have introduced a change, in which the weight of the first and second contests of a new user is changed to 60% and 80% of their old values, respectively. This leads to a slower increase in ratings of new users.
Along with Provisional Ratings, we hope this leads to a more robust rating eco-system.
This change has been applied for all users and their Elo-MMR rating recalculated from the beginning of time, and hence most users will see a change in their display ratings.
A comparison between the previous and current ratings (display ratings):
31 users have an increase of more than 50
~11700 users have an increase of between 0 and 50
~465000 users have no change
160 users have a decrease of more than 100 (70 of whom have been caught in cheating)
~4000 users have a decrease of between 50 and 100
~32000 users have a decrease of between 0 and 50.
You can find the entire list of users with their rating changes here.
