468 posts
  • Envato Staff
  • Envato Developer
  • Australia
  • Bought between 1 and 9 items
  • Has been a member for 2-3 years
  • Exclusive Author
justinfrench Dev says

Hi folks, if you haven’t encountered me before, I’m Justin French, Product Owner of the Marketplaces. As I’ve previously mentioned in other threads, the rating system is the next major pain point we’d like to focus on in improving the Marketplaces this year. There’s certainly no shortage of feedback and ideas!

Like many recent projects, we’re taking this potentially epic piece of work and splitting it into smaller pieces we can confidently ship really soon, and some more we will look at later.

Show a breakdown of the average

This is an incredibly popular and long-running request! We’re redesigning the sidebar of an item page to accomodate a little more detail, rather than just the overall average rating — buyers will see the number of people who have rated the item at each step (5 stars, 4 stars, etc), the total number of ratings received and the precise numerical value of the average (e.g. “4.6” or “3.9”). This work is mostly complete and will ship soon!

Rounding to half stars

Also requested, we’ll start rounding to the nearest half-star rather than whole star. Currently an average rating of “4.6” is rounded to “5”, “4.3” is rounded to “4”. By rounding to the nearest half star, an average “4.6” or “4.3” will be rounded to “4.5” instead. Some items will go “up” a half star, and others “down”, but overall, the graphical representation will be closer to the real numerical average. This work is mostly complete and will ship soon.

Encouraging more ratings

We want to encourage and remind buyers to provide a rating for any of their unrated purchases. We’ll start with an email follow-up a week after purchase and measure what effect this has on the number of ratings per item. This is also mostly complete and will ship soon.

What are we rating, exactly?

We don’t explicitly state anywhere what a buyer should be thinking about when they’re rating an item. Is it the technical quality? Aesthetics? The author’s support? Flexibility & configuration options? The author’s willingness to help with questions or customisation? The purchasing experience? The installation process? Value for money?

To help us better understand what’s already taking place, we’re going to run a survey, interview a range of authors and buyers, and start collecting the primary reason (selected from a short list of predefined options) from buyers as they rate each item. There’s no plans to display this data on the site (not even to the author), but it will help us better understand how the rating system is currently being used, and to plan our approach to some of the remaining feature requests.

Speaking of feature remaining requests…

Who rated this, and why?

An often-requested feature is buyers providing a comment when they rate an item, displayed alongside their username — some propose this should be required for all ratings, others just for one and five star ratings. There’s a lot of hidden features we might need here too (moderation & dispute resolution, marking reviews as unhelpful, etc).

The underlying goals of these requests are:

  • improve transparency for everyone
  • provide authors with insights to help improve their items
  • provide relevant ratings to buyers to help their purchasing decisions

These goals are important to us, but we’re not starting on the build right now — the data we’re collecting and the research we’re working on will help us design the details of a system that best meets the community’s needs, as well as ours.

We’ll update you with what we’ve learned (and our next steps) in a month or two.

Average averages?

We’re also working with a statistician to explore some alternatives to the existing method of averaging we use (including bayesian and median, which have been brought up a few times by the community). These are only experiments to review what happens to the data at this point. If it looks like we’re going to make a change in the future, we’ll keep you in the loop.

Ratings are forever?

Right now, a rating is forever — we calculate the average using every available rating since the item was first uploaded, which means it’s a mixture of historical and current ratings. We’re working with a statistician to explore the impact of ignoring older ratings. If an item has greatly improved over time, this might lead to a rating which is more reflective of what you’re buying today. Again, if it looks like we’re going to make a change in the future, we’ll discuss what we’ve learned.

Stay tuned!

I wanted to share this plan before we deploy the first batch of improvements so that you aren’t left wondering “is that it?”. There’s plenty more to be done, and this is the first step!

Edit: Made a small clarification to “What are we rating, exactly?” 4pm AEST (about 5hrs after the post).

1541 posts
  • Sold between 10 000 and 50 000 dollars
  • Author had a File in an Envato Bundle
  • Most Wanted Bounty Winner
  • Russian Federation
  • Has been a member for 2-3 years
  • Exclusive Author
  • Referred between 10 and 49 users
  • Bought between 10 and 49 items
  • Grew a moustache for the Envato Movember competition
AndrVlad says

Sounds great! Thank you for the information.

Best Regards, Andrey

356 posts
  • Exclusive Author
  • Has been a member for 2-3 years
  • Malaysia
  • Referred between 1 and 9 users
  • Sold between 10 000 and 50 000 dollars
billyf says

Sounds really great! My only wish now is to have it implemented as soon as possible :)

Thank you.

250 posts
  • Exclusive Author
  • Has been a member for 2-3 years
  • Beta Tester
  • Sold between 10 000 and 50 000 dollars
  • Bought between 10 and 49 items
  • Referred between 1 and 9 users
sightsignal says

Good stuff! Thanks for the thorough insights.

1399 posts
  • Author had a File in a Mini Bundle
  • Has been a member for 4-5 years
  • Sold between 1 000 and 5 000 dollars
  • Exclusive Author
  • Bought between 50 and 99 items
  • Referred between 10 and 49 users
  • United States
WebSmacker says

This will be nice! I have a couple 4 star items that may actually be 4.5 stars :)

983 posts It's in the pipeline!
  • Elite Author
  • Has been a member for 5-6 years
  • Bought between 100 and 499 items
  • Referred between 200 and 499 users
  • Exclusive Author
  • Envato Studio (Microlancer) Beta Tester
  • Author had a Free File of the Month
+2 more
purethemes says

This is very exciting news and makes me very happy that finally something is happening here! Good job Justin and whole team!

369 posts
  • Envato Studio (Microlancer) Beta Tester
  • Exclusive Author
  • Has been a member for 3-4 years
  • Sold between 50 000 and 100 000 dollars
  • Most Wanted Bounty Winner
  • Bought between 50 and 99 items
  • Referred between 100 and 199 users
saptarang says

Great News. Can’t wait to see it active :)

1729 posts
  • Envato Studio (Microlancer) Beta Tester
  • Sold between 10 000 and 50 000 dollars
  • Exclusive Author
  • Has been a member for 3-4 years
  • Referred between 1 and 9 users
  • Bought between 1 and 9 items
gbs says

Great to know about that. I’ve been waiting for this for years :). Cheers to the team!

490 posts Code is Poetry
  • Elite Author
  • Sold between 100 000 and 250 000 dollars
  • Has been a member for 2-3 years
  • Europe
  • Exclusive Author
  • Referred between 50 and 99 users
  • Bought between 1 and 9 items
egemenerd says

Great News ! I’m looking forward to see the result ;)

119 posts
  • Exclusive Author
  • Sold between 100 and 1 000 dollars
  • Bought between 1 and 9 items
  • Has been a member for 1-2 years
  • Haiti
Rameau says

Aaaah, going to f5 all day long on my “Downloads” page to see the “first batch” :D . Awesome.

by
by
by
by
by
by