Posted by Devin Hayes — 1 Apr 2010
I woke up to an unusual amount of IM's and emails that hotness was all borkedy and screwed up, people's numbers dipped dramatically, hotness was only showing 4 or 5 items in a theme with 500 photos, etc. Whhhaaat? I slurped down my coffee, put on my Viking helmet and went to work. Here's what I found:
For those that don't know or don't care, hotness is like a robot (but far less shiny). It's an automated process that, albeit somewhat lacking by design, requires no human intervention. We don't touch it. We'd like to, but any time it does something weird, we get a flood of emails and messages. Like today, for instance.
What apparently happened was that instead of displaying xx% of the top rated items in a theme, it showed 1-3%. How? Nothing changed. So, I started running through the code line by line running each query (eg: how many photos are in a theme, how many votes per item, etc) I came across and then something weird happened. I said, "Give me all the items in theme x and order it by their ranking from highest to lowest". What it gave me was a list randomly ordered, so instead of 100, 99, 98 it was 73, 4, 97. Huh? I rubbed my eyes, took another slurp of coffee and ran it again. Same results. Rubbed my eyes, took another slurp of coffee, wash, rinse, repeat 2 more times. What appears to have happened is that the index on the ranking (remember the "ranking from highest to lowest" thing) broke. So, instead of getting the top xx%, it was returning a random array of xx%. Since it knows what qualifies as hot (anything above average), it only took what qualified out of the randomly ordered slice. Which turns out to be, not very much.
There ya' go. We weren't monkeying with your mojo or tinkering with the tubes. The nature of anything and everything man-made is that it breaks. It happens. We slap some spackle on it and it's good as new. Until it breaks again.