509 posts
  • Has collected 1+ items on Envato Market
  • Member of the Envato Team
  • Located in Australia
  • Has been part of the Envato Community for over 2 years
+1 more
justinfrench Envato team says

Hi folks, if you haven’t encountered me before, I’m Justin French, Product Owner of the Marketplaces. As I’ve previously mentioned in other threads, the rating system is the next major pain point we’d like to focus on in improving the Marketplaces this year. There’s certainly no shortage of feedback and ideas!

Like many recent projects, we’re taking this potentially epic piece of work and splitting it into smaller pieces we can confidently ship really soon, and some more we will look at later.

Show a breakdown of the average

This is an incredibly popular and long-running request! We’re redesigning the sidebar of an item page to accomodate a little more detail, rather than just the overall average rating — buyers will see the number of people who have rated the item at each step (5 stars, 4 stars, etc), the total number of ratings received and the precise numerical value of the average (e.g. “4.6” or “3.9”). This work is mostly complete and will ship soon!

Rounding to half stars

Also requested, we’ll start rounding to the nearest half-star rather than whole star. Currently an average rating of “4.6” is rounded to “5”, “4.3” is rounded to “4”. By rounding to the nearest half star, an average “4.6” or “4.3” will be rounded to “4.5” instead. Some items will go “up” a half star, and others “down”, but overall, the graphical representation will be closer to the real numerical average. This work is mostly complete and will ship soon.

Encouraging more ratings

We want to encourage and remind buyers to provide a rating for any of their unrated purchases. We’ll start with an email follow-up a week after purchase and measure what effect this has on the number of ratings per item. This is also mostly complete and will ship soon.

What are we rating, exactly?

We don’t explicitly state anywhere what a buyer should be thinking about when they’re rating an item. Is it the technical quality? Aesthetics? The author’s support? Flexibility & configuration options? The author’s willingness to help with questions or customisation? The purchasing experience? The installation process? Value for money?

To help us better understand what’s already taking place, we’re going to run a survey, interview a range of authors and buyers, and start collecting the primary reason (selected from a short list of predefined options) from buyers as they rate each item. There’s no plans to display this data on the site (not even to the author), but it will help us better understand how the rating system is currently being used, and to plan our approach to some of the remaining feature requests.

Speaking of feature remaining requests…

Who rated this, and why?

An often-requested feature is buyers providing a comment when they rate an item, displayed alongside their username — some propose this should be required for all ratings, others just for one and five star ratings. There’s a lot of hidden features we might need here too (moderation & dispute resolution, marking reviews as unhelpful, etc).

The underlying goals of these requests are:

  • improve transparency for everyone
  • provide authors with insights to help improve their items
  • provide relevant ratings to buyers to help their purchasing decisions

These goals are important to us, but we’re not starting on the build right now — the data we’re collecting and the research we’re working on will help us design the details of a system that best meets the community’s needs, as well as ours.

We’ll update you with what we’ve learned (and our next steps) in a month or two.

Average averages?

We’re also working with a statistician to explore some alternatives to the existing method of averaging we use (including bayesian and median, which have been brought up a few times by the community). These are only experiments to review what happens to the data at this point. If it looks like we’re going to make a change in the future, we’ll keep you in the loop.

Ratings are forever?

Right now, a rating is forever — we calculate the average using every available rating since the item was first uploaded, which means it’s a mixture of historical and current ratings. We’re working with a statistician to explore the impact of ignoring older ratings. If an item has greatly improved over time, this might lead to a rating which is more reflective of what you’re buying today. Again, if it looks like we’re going to make a change in the future, we’ll discuss what we’ve learned.

Stay tuned!

I wanted to share this plan before we deploy the first batch of improvements so that you aren’t left wondering “is that it?”. There’s plenty more to be done, and this is the first step!

Edit: Made a small clarification to “What are we rating, exactly?” 4pm AEST (about 5hrs after the post).

1565 posts
  • Sells items exclusively on Envato Market
  • Has sold $10,000+ on Envato Market
  • Had an item featured in an Envato Bundle
  • Made it to the Authors' Hall of Fame
+6 more
AndrVlad says

Sounds great! Thank you for the information.

Best Regards, Andrey

357 posts
  • Has been part of the Envato Community for over 2 years
  • Has referred 1+ members
  • Has sold $40,000+ on Envato Market
  • Sells items exclusively on Envato Market
+1 more
billyf says

Sounds really great! My only wish now is to have it implemented as soon as possible :)

Thank you.

282 posts
  • Sells items exclusively on Envato Market
  • Has been part of the Envato Community for over 2 years
  • Has been a beta tester for an Envato feature
  • Has sold $10,000+ on Envato Market
+2 more
sightsignal says

Good stuff! Thanks for the thorough insights.

1460 posts
  • Has been part of the Envato Community for over 4 years
  • Had an item featured in an Envato Pack
  • Has sold $1,000+ on Envato Market
  • Located in United States
+3 more
WebSmacker says

This will be nice! I have a couple 4 star items that may actually be 4.5 stars :)

1012 posts It's in the pipeline!
  • Has sold $500,000+ on Envato Market
  • Elite Author: Sold more than $75,000 on Envato Market
  • Has been part of the Envato Community for over 6 years
  • Sells items exclusively on Envato Market
+7 more
purethemes says

This is very exciting news and makes me very happy that finally something is happening here! Good job Justin and whole team!

372 posts
  • Has referred 100+ members
  • Has sold $40,000+ on Envato Market
  • Has been a beta tester for an Envato feature
  • Has collected 50+ items on Envato Market
+3 more
saptarang says

Great News. Can’t wait to see it active :)

1739 posts
  • Has referred 1+ members
  • Has sold $10,000+ on Envato Market
  • Has been a beta tester for an Envato feature
  • Has collected 1+ items on Envato Market
+2 more
gbs says

Great to know about that. I’ve been waiting for this for years :). Cheers to the team!

506 posts Code is Poetry
  • Elite Author: Sold more than $75,000 on Envato Market
  • Has sold $125,000+ on Envato Market
  • Has referred 50+ members
  • Has collected 1+ items on Envato Market
+3 more
egemenerd says

Great News ! I’m looking forward to see the result ;)

121 posts
  • Has sold $100+ on Envato Market
  • Has collected 1+ items on Envato Market
  • Sells items exclusively on Envato Market
  • Has been part of the Envato Community for over 1 year
+1 more
Rameau says

Aaaah, going to f5 all day long on my “Downloads” page to see the “first batch” :D . Awesome.

by
by
by
by
by
by