Decaflon Thread: Writing An Effective Spam Filter

Writing An Effective Spam Filter

corenominal — Tue, 06 May 2008 20:39:02

@Ozone42: A quote from the Wikipedia article you linked to:

Today, Sisyphean can be used as an adjective meaning that an activity is unending and/or repetitive. It could also be used to refer to tasks that are pointless and unrewarding.

I agree that this could become an unending activity, but I am not sure about the "pointless and unrewarding" part. Personally I find this type of task thoroughly interesting and rewarding. I have really enjoyed coding up my spamsnake and I am looking forward to continued tinkering :)

Also, I wonder myself about some parts of the points system [discussed in the linked article.] Having read the entire post and the comments, I am pretty sure that some of the flags/rules used are compound, so while they may appear odd on their own, they probably work well in conjunction with other rules.

Writing An Effective Spam Filter

Ozone42 — Tue, 06 May 2008 19:34:59

Sisyphus comes to mind

The point system you linked to isn't bad, but I wonder about the -10 points for "Interesting." Perhaps if that was the majority of the comment made.

I think the most important thing to keep in mind is not make it hard on the real people commenting. If you catch 90% of the spam and it's still easy to leave a comment that doesn't get flagged, I think you've succeeded. If you catch 100% of the spam over a week, but have 2 false positives, I think that's a failure. Then again, it really depends on the level of traffic and spam we're talking about here.

Writing An Effective Spam Filter

corenominal — Tue, 06 May 2008 19:07:07

This is my first Chawlk note. I only registered as a new Chawlk user this morning, and to be honest, I was not sure that I would be overly interested in the site/service; however, there seems to be a good mix of users and content on the site and it occurred to me that the Chawk notes service might be a good place to post Dear Lazy Web type posts?! So here goes...

Dear Lazy Web

I am currently in the process of writing a new spam filter for the user comments system on website [crunchbang.org]. I am following the same principles as described by Mr Snook. Do you have any experience of writing/creating similar, and if so, do you have any tips for effectively separating the spam from the ham?

Best Regards,

Philip