Back to the lab

What is this figure a measure of?

How well we're doing on the Netflix Prize.

In particular, it's our root mean squared error (RMSE) on a set of test data. This is what it sounds like: the square root of the sum of the square of our error on each test instance, where "error" means the difference between what we predicted and the user's actual rating.

count=0
sum=0
for all test cases x
    err = (predicted_x - actual_x)
    sum += (err*err)
    count = count + 1
rmse = sqrt(sum/count)

It's a sort of measure of how many stars off your predictions are, on average. Only slightly more complicated.

Netflix says that their algorithm, when they dumb it down, gets a score of about .9514 on the test set. You qualify for a yearly progress prize if you can improve on this by 1% (.9419), and you win the million if you can improve by 10% (.8563). The current leaders are at .8965, though the slope of progress over time has seriously leveled off after the first month and a half.

Also, I hear that someone did an experiment where he made up new ratings for all of the movies already rated in his account, and used the new set as "predictions" and the old set as "actual" and computed an RMSE of .7 something. So there's probably a pretty high lowerbound on how well you can do simply based on human idiosyncracy.

The pseudocode algorithm above looks better on the entry's archive page than in the pop-up comments, where it appears that "pre" tags center their contents. Which is retarded.

Thanks for signing in, . Now you can comment. (sign out)

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

URL:
Remember me? YesNo

Comments: