RT Audience Score Rigged Again

RT has been a good metric for professional reviews for a long time now, even back during the prequel days, so I was very surprised that it's now this "thing," like average people all of a sudden discovered it in the last 3 years.

Anyway, whenever you have these type of geeky franchises with huge swaths of rabid fans you get people review bombing them to artificially lower the score. It became such a problem on Steam that they had to change their user-review structure, because they realized that people were creating an account just to give a low rating to a single game and then no further account activity. Now people discovered that RT is a thing, and a similar situation is happening. I'm always inclined to trust critic reviews over user reviews personally, partially because user reviews are dishonest for franchises, but also because my tastes tend to run more with people who have studied film and analyse it for a living, but I still have to take it with a grain of salt for certain genre things where I know it's simply outside the tastes of many. Case in point, one of the critics I learned the most from about movies was Roger Ebert, I usually learned a lot about what makes films work just by reading his reviews, but he absolutely hated horror films and with some exceptions always gave them a bad rating, which was why it always made me turn my head when he gave one a good review.

Anyway, I don't even think most people understand how RT works. Like, I see people saying "screw Rotten Tomatoes" and it's like...what? That's like looking at the sports score page and saying "screw the guy who printed this." It's just a measurement. And I don't think people even understand what Rotten Tomatoes is a measurement of, the way they talk about it. It's not an average. They do provide average ratings, it's in the smaller print under the big number, I think Solo is something like a 6.5/10. Metacritic measures the reviews as an average, and I actually prefer this method myself. But the "Tomatometer" is essentially a ratio, it's the percentage of how many people rated it "fresh", which is a 6.0/10 or higher, essentially is it worth seeing, would you recommend it. So it's a percentage of how many people recommend the film. It doesn't matter if you rate it 9.5/10 or 6.5/10, it still would be "fresh", it's a binary yes/no measurement. This form of measurement has a lot of advantages over the average, in that everyone has different tastes and it's problematic if people are rating it as a 6.5/10 or 9.5/10, but the bottom line is that both those ratings are basically saying they liked it, one liked it a bit and the other liked it a lot, but bottom line is that they both agree you should see the film. That's why TLJ's 90% fresh rating isn't rating the film 9/10 on average, I think the average is something like 8/10, but 90% of reviewers gave the film a recommendation (6/10 or higher).

I don't know if they still do this but they also use to have an optional weighted score for "top reviewers", in other words if you have some small blog that is somehow counted as being part of the media that might only get half the weight of something like The Guardian or Rolling Stone.

RT is really only designed to measure how the film performed in the media, it was never intended to be useful for measuring general audience, and the user review score has always been an afterthought that no one ever paid attention to, I think it might even be a newish feature because until it got controversial as late I sort of forgot it existed. That's why it was so open to exploitation and manipulation, as it was just a minor bonus feature the site had just cuz, the user score wasn't meant to be taken very seriously because it wasn't the point of the site and so they didn't make it all that secure.

 

I used to trust critics but if they think Wonder Woman, Black Panther, The Avengers and The Last Jedi are great movies, then I don’t know what they consider a great movie?

These are all popcorn blockbuster movies that appeal to a demo of fans and everybody has an opinion of them which is cool with me. But they are not ‘great’ movies, meaning they aren’t on the level of The Godfather. Lawrence of Arabia, or even The Original Star Wars.

Again, I’m not demeaning anyone’s opinion of The Last Jedi, as I LOVED The Force Awakens, but I’m not going to say it’s a great movie. Loving a movie is all our personal tastes as I loved Ace Ventura and have seen it hundreds of times, but again it’s not a great movie.

Somewhere the critics began to blur the lines of a fun/entertaining and now saying it’s great. Wonder Woman got those accolades from a lot of reviewers. It’s an entertaining blockbuster movie, but great?

 

I'm not sure if you are meaning this with regards to critical score metrics.

But for the record, Wonder Woman wasn't off the charts in that regard. Metacritic and RT measured the exact same average rating, which is 7.6/10

I think I'd agree with that, it was an alright movie but nothing too special IMO, that rating is positive but not great. Again, it gets a 92% Tomatometer rating, but keep in mind what that actually is and as it relates to the average score, it just means almost all critics liked the movie, only 8% disliked it, it's not an average and doesn't mean 9.2/10.

 

a good critic judges a movie based on a fair set of criteria. yes, there are movies we just consider "great" on a wide band of cinematic achievement, and then there are movies that are "great" within their genre (like a blockbuster), or for their demographic (like special interest films), or for their market (like indie films).

a good critic also judges a film based on the film's implied intent. is it a horror movie made to scare you? was it scary?
is it a comedy? did you laugh? and even then, what kind of comedy it is? is it crass, is it sweet, is it romantic, is it slapstick?

you can't compare Young Frankenstein to Citizen Kane on many many levels. but both are great films in terms of their achievements within their criteria.

 

Well, this seems like the right place for this.
A week or so ago, I decided to make a chart which kind of shows my thinking regarding the "problem" (read: context) of online self-elected reviews, such as RT's audience scores.

The thing is, these are somewhat similar to "customer surveys" hanging out at the door of a business.
There's some logic that has to be applied into them.

One of those pieces of logic is that whatever negative reviews that you hear, you have to keep in mind that it represents roughly (on average) about 4% of the total negative experience audience because most people don't bother telling you anything at all.
So for TLJ, that 53% dislike (in some fashion) population would represent 4% of the total negative opinion demographic under this industry standard consumer market model.

Another is that a grade is a grade; not a head-count.
If you want to know how many people enjoyed something, then a rating (which is a form of KPI) is not going to tell you anything.
You'll have to pull that data through inference if you aren't the original data holder.

A while ago, someone elsewhere posted a ream of data for TLJ scattered across a large volume of diverse online self-elect review rating services.
When you wiped out the scores and counted heads, you had approximately around 850,000 reviewers combined (as a median of possible total sums).

Now, if 53% (disfavor percent on RT) of 850,000 reviewers is 4% of the total population who didn't like it (which is a subset of the total general audience population), then you're looking at approximately a bit over 11 million people who didn't like it in some way.

The natural next question is how many people watched the film.
The only real way to get that (from where we sit) is to take the dollar amount and slice it up into ticket sales by the average ticket prices for the films.
That has problems because of repeat viewers, naturally, however, the MPAA can help us inform our gauge setting because they report that the average audience member sees an average of 5 films in a year.

The demographic slicing is further broken out, via graphs, but we can infer the bins differently, to 3% seeing a film they love an average of 3.5 times, 5% of the general audience seeing a film they love 2.5 times, and 92% of the general audience seeing a film once and moving on with their life.

If you take a million tickets, then, and follow this in 'per 1 million' fashion, then you end up with a factor modulation of about 1.15 to 1.
That means you take your ticket number and filter it by a factor of 1.15.
So if you had a ticket volume of 100 million, that you would be (under this model) translated as approximately 87 million individual customers.

Now, to get the total reviewing population I employed a bit of an axiom, which means I need to also supply a standard error margin (which I do).
The axiom is that, on the average, we'll assume that the reviewing audience volume will scale per film as TLJ did within the Star Wars saga.
Since IMDB has a good and easily accessible rating volume for every film, I could check what percent of the total 850,000 reviewing customers IMDB accounted for.
It's about 44% and change.

So, now I can divide other titles by that percent to discern the total reviewing audience, assuming the same relative range of customer review interest across Star Wars titles in online review sites.

OK, so now we have our total audience and our total review group.
To dicern the "like" rating, I just went with RT's audience review score. I figure this is a "worst case scenario" condition because RT tends to have the loudest voice and the lowest ratings in some interesting cases (AOTC, TLJ, to name a couple examples) compared to other services on average.

So then I flipped that around and treated that as real (it's not, but hey; let's play pretend here and error on the side against favor for good measure).
That means I treated the TLJ 47% audience score as 53% of my calculated reviewing audience (850,000) as disliking the film out of a total general population of around 129 million viewers, which then converts to an implied total calculated disfavor audience of 11.2 million out of 129 million general population.

Now, further, I know there will be a bunch said about the fact that all of these services didn't exist for the original films, but that's fine.
We're basically looking at how they're painted today when we run them through this model; which is to say, practically deified as holy relics.
I'm sure scoring, if possible, would have been very different right after their releases, but that achievement is literally impossible without asking the universe to click the CTRL+Z on its metaphysical keyboard and send us back in time by undoing all physical interactions that have ever happened anywhere in the universe since then.

So, we'll just take the rose-colored-view regarding the original and carry them as what they are: highly favored currently (which they also were really well favored then as well, but probably just not always this high upon release...things drift a bit over time when they catch fire in culture).

One further note here is that for the average and standard deviation, I weighted things to offset the over bearing nostalgia-tint we'll get out of the original trilogy.
The weighting is around 10% with the remainder split equally between the prequel and sequel trilogies. This shifts the average for the set from an unweighted value of 94% to a weighted average of 93%. It's a small difference, but it should be done for good measure.

Aside from the average standard deviation and general average, I also supplied a past three film rolling average and that rolling average's average standard deviation over that rolling period.

Again, you'll see error bars; these are standard error margins widened for correction to 5% (due to how many axioms we have in this thing).
The difference here is, for example, TLJ's low range on the standard error margin would be about 89%, where the widened 5% margin would allow for down to 87%.
I prefer to represent such sampling as this with wider margins of error than with narrower, so I'm fine with this.

To run a check for sanity, I ran a T-test against the Rotten Tomatoes Audience Ratings and the Ratings here at a standard alpha setting of 0.05.
The resulting test was 0.034877, which is a fairly decent result.
It could be better (in physics I wouldn't be happy), but it's not bad (my friend in pharmaceuticals in the medical industry would be delighted with the result).
For population demographic work, it's acceptable as a result.
I also ran a correlation coefficient for good measure, and that result was 0.904484, which means that we can say they are well positively matched proportionately, which is good.

So, with all of that out of the way, here's the results.



To no one's surprise, this basically translates to: People like Star Wars.
Fans might complain, but people in general really like Star Wars.
Even at its worst, Star Wars is sitting just fine; regardless how much any one of us fans cringed watching any one film or not.

Cheers,
Jayson

 

What they did to Lando? Possibly.

 

@Jayson , your stats math is reminding me why I never went into physics but well done.

I think this is the bottom line that people forget, and again I have to remind them of the context of the franchise in general. After the Special Editions and prequels, we all came back and saw the next one. Anyone that hated TLJ will eventually go see Solo, even if it's not on opening day. The people complaining the loudest are people who obviously care quite a bit, and for every person complaining there's another ten people buying a ticket and then moving on with their life after its over. Especially in the context of what reviewers and the small amount of people who have seen it are saying, the consensus is that it's not terrible, and if people convinced themselves to see Attack of the Clones more than once (me), then I think most fans will not have a huge problem with the film.

The franchise will face a very big challenge in a few years when they run out of steam milking the OT, but not for another 5 years or so. The franchise has never survived without the OT as a brace--every film so far has been a prequel or a sequel to the OT, and with a Kenobi spinoff, Episode IX, and two more Han Solo films down the pipeline, they got the next few years covered. The question of how much people will care for an "iffy" SW film that has nothing to do with the OT is really hard to predict right now. And once profits start showing any concerning decline, they have a lot of tricks up their sleeve still. "Think you don't care about Star Wars any more? Well too bad we're making the Darth Vader stand-alone film next." I'm assuming that the Benioff and Weiss series in development is the KOTOR, so they will have the fan-base eating out of their hand soon enough, and if it's successful they can build a shared universe just in that time period; it's like a franchise within a franchise, with the possibility for connected Sagas and endless spinoffs. Disney knows what they are doing better than some people think.

 

.....and once again @Jayson rides in on his white horse of DATA (I'm a DATA guy) and puts this whole thing into perspective......with real data.....and standard deviations, and correlation coefficients (ahhhh the good ole days)

Bottom line....people like Star Wars. The data set is pretty powerful.

 

Lando has no character or depth to him whatsoever. He seized to be interesting the second the credits rolled on empire. So let's not have rose colored glasses

 

2 things are clear.

1 - The RT % method of scoring a film does not lend itself well to comparison. Black Panther has a higher RT score than The Shawshank Redemption. Let that sink in for a moment..

2 - All those saying the audience score wasn't accurate for TLJ might be right. It certainly divided fans. But seeing the Solo audience, the talk of vocal minority or vocal angry fans has merit.

 

I'm going to do this backwards a bit here...

For the following, I'm going to define a fanbase as those who habitually purchase franchise media products between major product releases because it involves the franchise brand.

I think most fanbases assume their worth is a bit more than it is when it comes to massive franchises.

Fanbases are important, but more in regards to a form of advertisement and topical conversation.
Ironically, the most damning thing that anyone who's mad at a franchise can actually do is not complain; no, it's be silent and stop consuming.
That has more damage by a long shot because the thing with getting in the news over contention, when it comes to music and films/TV, is that it tends to cause someone to get curious and check it out if they hadn't already done so, or they might check it out again and try to see what they missed.
In other words, it creates a consumer discussion, and the most important thing about a consumer discussion is that it revolves around consuming the product.

So fanbases are of value, but then there's the general audience and the general audience FAR out-numbers the fanbase.
There are scores more, what I call, pop-culture-proxy-fans (these are people that buy a bumper sticker or shirt because it's cool or cute, but only interact with the brand when a major product hits) who show up because the pop-culture event itself is fun.

There's far more of these, and combined with just regular people in general, there's far more non-fanbase people at a film than there are fanbase audience members, and so it's not really possible for major franchises to be gutted by a fanbase.
What is possible is for a fanbase to annoy the ever-loving-piss out of a good artist and make them quit and walk away, as happened with George Lucas, for example.

The angry fanbase didn't put a dent in the monetary success of the PT, but they absolutely put a dent in the creator's interest in making something that he felt he would have to listen to hate from yet again; even if it wasn't the largest portion of the noise (there's also something to be said about Lucas' softness - he's pretty sensitive to critique, so that's also a factor; other artists don't really care).

But for business? Yeah, no.
I'm right there with you - fanbases are that nice extra, but nowhere near the actual monetary meat and potatoes of a mammoth beast this size.
Now, if we're talking small cult franchises like Firefly or back in the day's Evil Dead; then the fanbase matters a hell of a lot, but Star Wars is very much far beyond that - it has theme parks...plural.

Hehe. Thank you both!

I love data crunching...which is weird because I was an absolute bohemian art nut (I mean...full on; drawing, film, music, drama; that was my gig) all growing up (and sucked at math - I once passed math by performing Monty Python and making the teacher laugh too much); it wasn't until I was in the military that I got into data analysis, but, there you have it. Strange life.

Luckily I get to do this as a job, so YAY!
And if that's not enough, I also do it for fun.

@S_Foils, if you want, since you like data - check this one out (click spoiler...warning, I tangent off topic...hence the spoiler).

Spoiler: DATA DORK TANGENT!

Check this value out.
0.000039.
Take any star radius (which are usually in magnitude of our Sun, so convert that to km first) and divide it by that value (and if you want to make it easy on yourself, whip up google and run a quick conversion from km to AU since most things in star systems are in AU ranges).
What you will have just returned will be the approximate distance from that star where you will likely find an astrosphere (like our heliosphere, or heliopause...like the whole solar system's "atmosphere" - ish, if you will). You'll be within stellar ear shot most of the time with that one simple trick (about 1.2-ish mark, but close enough for ball parking).
If you want to see more about that, message me and I'll shoot you the white paper that we're currently using from the first study (prepping to approach a re-do now that the first study showed value in fully committing some resources on this).

Here's another one that I derived out of music using semitones because I wanted a "unified theory of scales"...so to speak rather than all the wandering 'if it's this, then this, else if it's this, then the degrees are, but if it's, then the degrees are....blah' No...there's got to be a mathematical foundation below that garbage, and yep; there is, and it's way the hell easier. It's 2,2,1,2,2 and 2,1
You just slide around where that 2 and 1 are mostly and that's pretty much it. Then you can borrow from values and slide things around a bit, but it's pretty damn simple and you don't need to care what anything's named - just the principles of how to move a set of notes around to build a scale and then you're on your merry way. Here's a chart of what that looks like.


And at work, recently, I did this - which was TOTALLY FUN!
https://thecantina.starwarsnewsnet....value-estimation-calculation-nerd-time.54714/
Business Analytics Reporting Value Estimation Calculation - Nerd Time!

Awesome to see another data lover!
*high five!*

Have you done any recent data work??

Cheers,
Jayson

 

The sad thing is, the more people like SW, the less I do. I don't know what it is; SW has always been popular, but something has changed after the acquisition. SW has never quite felt so populist. People say it's great that the audience is broadening, I feel that something's lost. The identity. Lucas never tried to please everyone. He was creating something (I'm talking movies, byproducts are other thing) mostly for himself. Now it's a... production line.

Okay, I'm going off-topic here.

 

I agree with the first paragraph which is why I find RT's critic rating has some merit in gauging noteworthy releases. Maybe I'm a film elitist too but, as in many other things, a more informed opinion often provides a wiser analysis.

Of course if someone's movie enjoyment is based on a different set of criteria than pro critics it would make sense to discount critical opinion!

Audience reviews such as this example (complete with rallying cry) abound on RT.

I'm no pro critic but am curious if expectations may have also played a part in the positive critical reception of Hollywood blockbusters mentioned in this thread such as Deadpool, Wonder Woman, and Black Panther?

Some of these bubble-gum genre film (unlike some of their source material) don't have much history of challenging audiences intellectually or marrying action with substantial philosophical or existential themes.

Is it plausible that when films arrive that break new ground within a genre or add nuanced layers to the standard recipe they may be received more warmly by critics or at least afforded some respect for the ambition of the filmmakers?

Of course the other side of the coin is the danger of a “sophomore slump” when expectations are raised. Then there's the exhaustive list of incredible films that were poorly received by critics or viewers upon their initial release!

 

i hate to be cynical, but yes, anything that breaks ground and doesn't just flat out suck is likely to get glowing reviews.

none of this is wrong or bad. these are all just opinions in the end. one hopes a critic has the objectivity and knowledge to understanding and respect the medium and judge appropriately. that's not always the case, but i'd say it's generally true. everyone has biases. we find balance in relating to the people whose biases match our own.

i haven't seen Deadpool or Black Panther, but i did see Wonder Woman. i liked it okay. it sucked less that most of the other DC films. XD
but it was also a female superhero movie with a female director, so it made a big impression.
that's fair enough. still not a "great" film in my estimation, but i can appreciate it for what it is and what it means to the industry.

because a film does not have to entertain or make money or be greatly loved to have significant impact.

 

Oh wow. The lunacy. Yeeeesh.

Yes. Absolutely. Deadpool, Wonder Woman, and Black Panther are fun, great films. But they don't have much depth. You nail it when you say they "don't have much history of challenging audiences intellectually or marrying action with substantial philosophical or existential themes." Very true.

Last Jedi did just that. It raised challenging questions on what makes someone "great." It raised questions on what it means to be spiritual, and what religion is. Is it an all-powerful Force that makes you powerful because you believe/have the power. Or is it the natural world that binds us all together, and there is power in that, and power in sharing that? These are just two of the ideas the film played with. Critics loved it. However, a portion of the general audience, and a very vocal portion of Star Wars fans...did not.

I believe that this has much to do with them simply either not understanding the movie and the themes it raised, not connecting to these themes, or outright rejecting them. Further, it also had much to do with the audience being allergic to the changes made to the beloved characters (as if people don't change...yet another theme of this film), and big changes to the set expectations of the audience. Further, many came into the theater with their "head canon" that they obsessed and preened over for years, ready to see the precious stories they created in their head play out on the big screen. And when it didn't, but instead went off in a completely different direction, they became enraged because it wasn't the "right" story. Despite its populist themes, Last Jedi wasn't a populist movie. It wasn't for everyone. It proved too challenging for many who had set expectations they stubbornly (ignorantly?) refused to budge from.

 

Solo has ~5259 votes on Rotten Tomatoes...if you remove all the negative ones. And that's less than 3 days away from Thursday night previews.
Incredibles 2 has 4707 total votes and that's slightly less than a month away.
Jurassic World 2 has 16501 total and that's a month away.
Ant-Man 2 has 10847 total votes and that's a month and a half away.
Deadpool 2 had 15090 total votes days before its opening.



It's the YouTube channels during the past week sending their audiences over to vote on Solo.

The bad reactions were all over the planet to The Last Jedi.



The big drop-off of box office and disc sales (U.K.) shows it too.





https://thecantina.starwarsnewsnet....nancial-performance.53377/page-58#post-502313

Rian Johnson said years ago that his goal in life is to make divisive movies:

(Younger) Rian Johnson: "I would be worried if everybody across the board was like, 'Yeah, it was a good movie.' It's much more exciting to me when you get, you know, a group of people who are like coming up to you and really really excited about it and then there are other people who walk out just I mean literally saying, 'It was the worst movie I've ever seen.' (gestures with hands) Having those two extremes to me is, you know, the mark of the type of movie that I want to make so..."

 

If Solo makes only just more than half the money/tickets that TLJ did, that's in reality a lot of people that didn't actually want to see the movie. If TFA represents peak Star Wars, then it's possible that only a bit more than one-third of that audience shows up for Solo.