Dear Won H. Yoon
I own a web development company and for years we have used Hot Scripts and I have recommended Hot Scripts to dozens of people as the best script resource on the web.
My high opinion of Hot Scripts was completely shattered last night!
Let me explain.
Over a year ago we created a Perl CGI script called the L10 Hit Counter (resource 18082 ). We built it for ourselves originally because we could not find a suitable traffic metrics system for real in-depth analysis. Even the three different for-pay services we tried in the past were not adequate.
The script took almost 400 hours of development time to get to version 3.15, which we would normally charge a client $50,000 to develop. We released it to the public for free. Posting it on Hot Scripts and similar sites.
We got a great response, we even have a company that is negotiating to license a private branded version for 6 figures. As of this date we have enjoyed very good visitor ratings on the major script resources:
CGI Resource Index: 9.80 out of 10 (111 votes)
Script Search: 4.62 out of 5 (94 votes)
Hot Scripts: 4.49 out of 5 (139 votes)
However, despite this success, we learned from our community that the script had some technical issues on high volume sites, mostly revolving around the use of text files for data storage.
This year we looked to solves these problems. We decided to completely reprogram the back-end to use a database (MySQL to start and others to follow) and that PHP would make for easier installs then Perl. All reports are viewed in a Flash thin client and we kept that the exact same. Actually it was a bit of an experiment; given that the features are the exact same would a Perl CGI or PHP/MySQL version be more popular, and/or easier to install.
This project took about 120 hours to complete and we released the LevelTen Hit Counter PHP on 12/1/03, and of course posted it to Hot Scripts (resource 28819).
Despite the features and reports being the exact same as the CGI version we saw a difference in visitor rankings. We got several 5s but also a few 1s. Our only conclusion is that while the script works on our servers and several other webmasters had it running on theirs, some people must be having technical problems. So we set out to discover what they might be.
We created a new forum for people to report problems and we posted an e-mail address in the instructions for people to report problems. However, the actions of last night and our experience with our other scripts lead us to believe that the “Poor” votes are not honest opinions by people who have installed the script.
In the first 12 days our script was listed Hot Scripts, our script received 9 votes all were either a 5 or a 1. This is approximately 1 vote per 100 downloads basically the same ratio we have seen for years with our other scripts. However, last night Hot Scripts reports 12 new votes, all “poor”. What is particularly disturbing is that the script has only been downloaded 2 times during that period. Furthermore, no new support questions have been generated in the forum or via e-mail.
So one of two things happened:
A. 12 people shared two downloads and after having technical problems, instead of asking for support, all bucked the 1 vote to 100 download ratio and decided to rate hundreds of hours worth of work as “poor”.
or
B. Someone has grossly and maliciously spammed this ranking in a successful ploy to make sure no one ever downloads it again. (If I wanted to create a really long post, this is where I would liken it to “Ranking Terrorism” where a cowardly individual hides under the cloak of anonymity to undermine a well-intentioned community. But I won’t get that political.)
But wait there is more. On the same night in question another script linked with the LevelTen Hit Counter, LevelTen Formmail received 6 votes, again all “poor”. What is interesting is that in the 5 previous days since it has been posted on Hot Scripts it had not received one vote. This is a simple Formmail script, we have it installed on dozens of sites - again no request for help, just 6 votes as “poor”.
Now it gets really interesting. Another resource we have posted that is listed as a related script is the “@-Manager”. This is a fairly advanced form to email processor we created years ago. In 2.5 years of being on Hot Scripts this resource has only received 10 votes for an average of 4.5+. Last night it mysteriously received 7 new votes, all “poor”
There is only one conclusion; there is not mechanism to stop malicious spamming of rankings on Hot Scripts.
What is even more discouraging is that I have come to rely on those rankings as an important part of our business to evaluate others work. Now I know they have no integrity and can be rendered meaningless by anyone who wished to do so.
I do realize that the task of maintaining the integrity of the ranks is very difficult. However, Hot Scripts has such potential; large database of scripts, very good traffic, a solid support community. However, without honestly evaluations of the resources the site becomes pointless.
I make a few recommendations:
1. Require a 50 to 100 word comment/review with each rating submission. This will help reduce bogus ratings, but mostly, it will help script authors improve their work. How can we fix what is wrong if we don’t have proper feedback, and if someone is going to trash our scripts in public they should at least tell us why.
What would be really helpful is that if someone has an issue with a script and the author can fix it, the poor rating is removed.
2. Encourage more members to write reviews and promote the general rankings unfiltered use at your own risk evaluations. You could even create a point system to reward members for reviews.
3. Implement algorithms to identify abnormal ranking activity. E.g. if a script gets an unusually high number of rankings in ratio to its hits or if a series of rankings are submitted that significantly differ from the existing ones, ect.
4. Use the Hot Scripts community as reviewers with checks to ensure the integrity of the reviewers. Could be similar to Open Directory, which is not perfect but is far from the free-for-all that Hot Scripts appears to be now.