Back end changes

I rolled out a new version of News Sniffer last night. A lot of work went into rejigging the censored comment detection to make sure we don’t make mistakes. I also wrote a system to confirm censored comments by html scraping, which gives us a way to double check censorship.

This also allowed me to check the entire backlog of censored comments. There were a number of comments that we thought were censored but were not and those are now fixed. We also found a number of comments that we didn’t know had been censored.

Apologies for this. My understanding of the BBC HYS RSS feeds was flawed (to be frank, largely due to some brain-deadness on the part of the BBC forum software).

So you might notice that some existing censored comments disappeared but other ones appeared for the first time. I’ll be running the confirmation script regularly to monitor the new checking system (though not as regularly as the others as it’s a bit intensive).

Front end changes

The “Recommended Comments” page now lists the latest recommended comments, not the highest recommended as before. Comments in order of highest recommendations can now be found on the “Top Recommended Comments” page.

The thread listing pages now displays the number of published comments along with the number of censored. This helps give an idea of how busy a thread is (you might expect busier threads to be more censored).

And lastly, the thread display page now includes the BBC HYS description, which gives a bit of background to the thread and links to any news articles that might have prompted the it.

Thanks to datamining.typepad.com and currybet.net for the feedback that led to some of these improvements.