[webmaster@atmanager.com]

Dear Won H. Yoon

I own a web development company and for years we have used Hot Scripts and I have recommended Hot Scripts to dozens of people as the best script resource on the web.

My high opinion of Hot Scripts was completely shattered last night!

Let me explain.

Over a year ago we created a Perl CGI script called the L10 Hit Counter (resource 18082 ). We built it for ourselves originally because we could not find a suitable traffic metrics system for real in-depth analysis. Even the three different for-pay services we tried in the past were not adequate.

The script took almost 400 hours of development time to get to version 3.15, which we would normally charge a client $50,000 to develop. We released it to the public for free. Posting it on Hot Scripts and similar sites.

We got a great response, we even have a company that is negotiating to license a private branded version for 6 figures. As of this date we have enjoyed very good visitor ratings on the major script resources:

CGI Resource Index: 9.80 out of 10 (111 votes)
Script Search: 4.62 out of 5 (94 votes)
Hot Scripts: 4.49 out of 5 (139 votes)

However, despite this success, we learned from our community that the script had some technical issues on high volume sites, mostly revolving around the use of text files for data storage.

This year we looked to solves these problems. We decided to completely reprogram the back-end to use a database (MySQL to start and others to follow) and that PHP would make for easier installs then Perl. All reports are viewed in a Flash thin client and we kept that the exact same. Actually it was a bit of an experiment; given that the features are the exact same would a Perl CGI or PHP/MySQL version be more popular, and/or easier to install.

This project took about 120 hours to complete and we released the LevelTen Hit Counter PHP on 12/1/03, and of course posted it to Hot Scripts (resource 28819).

Despite the features and reports being the exact same as the CGI version we saw a difference in visitor rankings. We got several 5s but also a few 1s. Our only conclusion is that while the script works on our servers and several other webmasters had it running on theirs, some people must be having technical problems. So we set out to discover what they might be.

We created a new forum for people to report problems and we posted an e-mail address in the instructions for people to report problems. However, the actions of last night and our experience with our other scripts lead us to believe that the “Poor” votes are not honest opinions by people who have installed the script.

In the first 12 days our script was listed Hot Scripts, our script received 9 votes all were either a 5 or a 1. This is approximately 1 vote per 100 downloads basically the same ratio we have seen for years with our other scripts. However, last night Hot Scripts reports 12 new votes, all “poor”. What is particularly disturbing is that the script has only been downloaded 2 times during that period. Furthermore, no new support questions have been generated in the forum or via e-mail.

So one of two things happened:

A. 12 people shared two downloads and after having technical problems, instead of asking for support, all bucked the 1 vote to 100 download ratio and decided to rate hundreds of hours worth of work as “poor”.

or

B. Someone has grossly and maliciously spammed this ranking in a successful ploy to make sure no one ever downloads it again. (If I wanted to create a really long post, this is where I would liken it to “Ranking Terrorism” where a cowardly individual hides under the cloak of anonymity to undermine a well-intentioned community. But I won’t get that political.)

But wait there is more. On the same night in question another script linked with the LevelTen Hit Counter, LevelTen Formmail received 6 votes, again all “poor”. What is interesting is that in the 5 previous days since it has been posted on Hot Scripts it had not received one vote. This is a simple Formmail script, we have it installed on dozens of sites - again no request for help, just 6 votes as “poor”.

Now it gets really interesting. Another resource we have posted that is listed as a related script is the “@-Manager”. This is a fairly advanced form to email processor we created years ago. In 2.5 years of being on Hot Scripts this resource has only received 10 votes for an average of 4.5+. Last night it mysteriously received 7 new votes, all “poor”

There is only one conclusion; there is not mechanism to stop malicious spamming of rankings on Hot Scripts.

What is even more discouraging is that I have come to rely on those rankings as an important part of our business to evaluate others work. Now I know they have no integrity and can be rendered meaningless by anyone who wished to do so.

I do realize that the task of maintaining the integrity of the ranks is very difficult. However, Hot Scripts has such potential; large database of scripts, very good traffic, a solid support community. However, without honestly evaluations of the resources the site becomes pointless.

I make a few recommendations:
1. Require a 50 to 100 word comment/review with each rating submission. This will help reduce bogus ratings, but mostly, it will help script authors improve their work. How can we fix what is wrong if we don’t have proper feedback, and if someone is going to trash our scripts in public they should at least tell us why.

What would be really helpful is that if someone has an issue with a script and the author can fix it, the poor rating is removed.

2. Encourage more members to write reviews and promote the general rankings unfiltered use at your own risk evaluations. You could even create a point system to reward members for reviews.

3. Implement algorithms to identify abnormal ranking activity. E.g. if a script gets an unusually high number of rankings in ratio to its hits or if a series of rankings are submitted that significantly differ from the existing ones, ect.

4. Use the Hot Scripts community as reviewers with checks to ensure the integrity of the reviewers. Could be similar to Open Directory, which is not perfect but is far from the free-for-all that Hot Scripts appears to be now.

[Skeleton Man]

Here here !! I couldn't agree more with all of what webmaster@atmanager.com said..

[YourPHPPro]

I definately agree and could not have said it better myself...

[Keylar]

I totally agree, I posted about this kind of thing to another thread, I don't tend to use Hotscripts myself much but one day started getting these weird anonymous emails from someone saying they were a competitor and that I had better stop giving their application bad ratings or they were going to drive mine down in retaliation. Sure enough they did, since the bad ratings must have been coming from someone else and I couldn't do anything about it. It's very frustrating, I have the lowest rating on here of any site of this kind. Certainly makes me less likely to use it as well, knowing there is nothing preventing this kind of activity.

[Techmonkey]

Agreed.

What it comes down to is this: anonymous review systems don't work. Every review should be from a registered member. This isn't new to Hotscripts. I have seen other large review sites that are inherently flawed because they allow anonymous reviews.

[Andrew]

I posted this in another thread before I found this one:

The rating of my script (http://www.hotscripts.com/Detailed/14024.html) has been steadily going down for a while now. It used to be 4.70, now it's 4.03. I've seen it go down without the total number of votes going up. I've also seen it go down after having somebody rate it as excellent; the next day the vote total was up one, and the rating was down.

Obviously there is something wrong here, but I've contacted Hotscripts twice (actually three times now), and have gotten no response. This is bad for my business, and I can't even get anyone to tell me why this is happening.

I saw the thread starter mention how many poor ratings, etc. he had gotten. Is there a way to find this information? Because I couldn't find it.

[nakulgoyal]

100% agreed with you. I have seen this happen with some of my resources at Hostcripts as well.

[Andrew]

It's upsetting that no one from hotscripts will respond. I had planned at one time to do a sponsored listing and possibly a banner ad, but I don't think I'll be doing that now.

[webmaster@atmanager.com]

Voting Integrity Investigation part 2.

Well I have learned more about Hot Scripts rating mechanism and the reality gets worse.

To “Right the Ship” as it were, I added an external voting form to our site and encouraged users of our script to vote for it. I was hoping for a few votes each day. I was completely shocked, our users were submitting over 30 votes a day, with an average rating of 4.76.

For 4 days our ranking steadily climbed. Then our ranking stopped increasing, at 4.39. We were still getting votes at over 30 a day, but no change in the ranking. Then it slowly started to fall.

So I decided to keep track of all votes submitted from our form. As I suspected, the 5 votes our users were submitting were not being used in the average. I guess the large volume of votes caused an anti-spamming mechanism to be triggered.

However, a simple inspection of the remote IPs of those submitting via the external form would reveal the votes did not come from a single source (or even a few sources). The remote IPs prove that this could not be spamming but legitimate votes from a global community.

So now not only has someone destroyed our listing, but our user base is being deigned the right to voice their opinion by Hot Scripts themselves.

[Andrew]

It seemed like my rating started going down after I added the external voting form. I wonder if there's a connection..