|
| T O P I C | |
|
Limeys review Black & White, gamers scratch collective heads.
February 20th 2001, 20:25 CET by Zachack Recently, British gaming website PC Zone managed to <a href="http://www.pczone.co.uk/guest/php3/openframe.php3?page=http://www.pczone.co.uk/guest/newreviews/printrev.php3?id=33679">review</a> Lionhead Games' (or more specifically, Molyneux Games') Black & White, a game scheduled to come out right before the end of March (in the United States, at least). What makes this review curious is that, theoretically, Black & White is still in the testing stage. While the bug hunt is supposedly almost finished, this still raises the question of "What in the hell are they reviewing, and how can I get it?". Granted, many game reviewers base their reviews off of gold master copies, allowing them to review the game before it hits stands, thereby (hopefully) increasing sales. But unless I missed something, B&W hasn't reached that point (I assume that for a game of B&W's hype the "we went gold" news will be fairly hard to miss). So what are they reviewing? Here are some questions to ponder and blather on about: 1. Assuming that PC Zone has a final version, what do you think about exclusive reviews? Also, how and why are they handed out? Bribes? Blackmail? Just being a really nice guy? Inquiring minds want to know! 2. If it isn't a final version, how unethical is this? Is it unethical at all? Since Lionhead hasn't attacked the review, should we assume that it is valid? 3. Now that you're talking, tell us about any bad experiences you may have had with the press, or vice versa. C'mon! I need gossip to live! And don't forget to play name-dropping games! P.S. Some German site apparently has also reviewed B&W. Since I can't read German, I am going to ignore it. However, feel free to make inflammatory remarks about the Germans if you are feeling grumpy. - Zachack |
| C O M M E N T S |
|
Home »
Topic: Limeys review Black & White, gamers scratch collective heads.
|«« - Previous Page - Next Page - »»| |
|
This was a news item on LionGames.com waaaaaaaaaaaaaay back on Jan. 10th: Here is another LionGames exlusive for you folks. We've received following quote, taken from the PC Zone UK online-forums, by mail just a few minutes ago. Have a read: "Unbelievably, we've been told that it's ready, and we've got the world exclusive review in issue 100 (on-sale Feb 9th). I haven't seen it yet, but I can't wait" We've contacted Lionheads' PR Lady to confirm this information. Here is what we got for you: Q: Can you confirm that Black & White is done, or is this referring to a press/review version? Cathy Campos: They are referring to a press version. |
|
Yeah, I'm new here.. greetings to all of the peeps on the planetcrap boards.. I've read your posts for a couple of months now... LURKER NO MORE! Err.. yeah. Anyway. Hi. Back to the regularly scheduled programming. Speaking of B&W.. the latest word on the release looks like March 30th (at the latest).. All praise Jamie Durant. Waiting for this game is torture.. back to more Populous then, I suppose. |
|
Been a while since I posted here. Do love the new (retro) look though ;) 1. About the reviewcopies : I don't know how it's being handled in the US or UK, but what I CAN tell you is how it's done in the Benelux (and believe me, that's not a standard). Basically, reviewcopies are being sent out by large distributors or the companies themselves. Wether you get a reviewcopy or not is pure guessing. Sometimes you're the first one to receive a copy, sometimes you just don't get one. What the distributors use to see who they'll send out a copy to is a complete mistery. Having the biggest amount of visitors or pageviews does not immediately mean that you'll receive the biggest amount of reviewcopies. It seems to me that they care more about the company that's behind the site, rather than how much traffic the site gets (example : The Gaming Shrimp is owned by PlanetInternet - a big ISP - and they get ALOT of reviewcopies, while Gamespot.be gets a fair amount of copies, but still less than the shrimp). Also, magazines get copies easier than websites. It seems that over here in the Benelux, the distributors still don't see the light that magazines have a smaller amount of readers than websites, and that therefor the impact of a review is smaller (or maybe that's just what they want -> less viewers means more people that will buy the game when it's crap ;p) When a review is posted, one should send a link to the distributor to proove they've checked the game out (and personally I think it's to see wether you gave it a positive review or not, which probably has an effect on the amount of reviewcopies you'll receive in the future -> we at Fragland defenitely notice it since we're getting really small amounts - up to 0 - from certain distributors since we've posted some pretty harsh reviews that broke down some games but fuck them, we don't care and won't be bribed !) Personally I must say that my best experiences are with companies that send out reviewcopies themselves (Ubi Soft, Microsoft, ...). You get them on time and don't have to keep asking for them Companies like Activision and EA work with distributors in Belgium, and the distribution of reviewcopies plain sucks (you can even ask the guys from Gamespot over here, they'll say the same) 2. Reviewing a beta is not per se bad, but you have to mention that it's a beta, and I would rather call it a preview then. About the PS : A beta has been distributed over here aswell. Guess they're all reviewing the beta version (which Fragland doesn't do because we think it doesn't give an accurate view of what the game will be like) Speed Fragland.net |
|
It is my understanding that B&W was feature-complete back in November, and the release date was pushed back to March only for bugfixing and many square centimeters of polish. It wouldn't surprise me if some journos have been playing the game since then, and only now can start talking about it. |
|
However, it's not particularly useful to consumers because that publication is not reviewing the version you're going to buy at the store. A lot can be fixed or screwed up in the last couple of weeks of development. I've been luckly enough not to be working for a publcation and see what a difference a month can make. Lets take a look at midtown madness 2 one month before release. 1: Lake in San Francisco that you should be able to drive though has no bottom, shame since this lake features promitantly in one of the races and going though it is the fastest route. 2: All Crash course races are possible to win on pro mode. 3: All races on pro are extremly easy Month latter 1: Lake has been fixed, you can now enjoy running the race around this lake with out the frustration caused by avoiding the bottom less pit. 2: One of the San Francisco crash course races has had it's time greatly reduced on pro. The race is next to impossible to finish even with a perfict run. 3: Most pro races are still really easy If I had to do a review based on the beta and ignored the ovious bug 1 and got word from the developers that the races would be made harder on pro the game would have gotten a rather high score. With the final some of the pro races have goten easyer not harder out side of that one race that I just can not win even tho I'd done it perfictly 20 times now. I find is unexcusable that the harder setting should give me no chalange and play more like the easy settings exepted for one race. |
|
Madre de dios! Es el planeta de la Mierda! What the hell does that mean? |
|
Mother of God! It's the planet of the crap! |
|
I cannot wait for B&W, im all hyped up for it. But I have this fear that its going to some grand version of the Sims. Fantastic reviews, fantastic hype, fantastic sales but after 2 weeks it pains me to play the game again. Someone smack me around and tell my why Im wrong, please!? |
|
"A founding member of the PC Zone team now works at lionhead. Being it's 100 issue (longest PC gaming magazine ever in the world) there were some niftyness. It was the cover story and hell, they got the exclusive review. uhh wrong. CGW(Computer Gaming World), just had their 200th issue anniversery. Been going since 1981. I guess only time will tell if they had the permission of Lionehead to do this review. If they didn't I'm sure we'll be seeing some angered responses from Lionhead for reviewing what must be still a beta product. |
|
Being it's 100 issue (longest PC gaming magazine ever in the world) there were some niftyness. Did they really say they've been around longer then any PC game magazine in the world? Hell, even the magazine I work for (Computer Games Magazine, formerly Computer Games Strategy Plus, formerly Strategy Plus) has been around longer. We just celebrated our 10 year anniversary and are on our 126th issue. Not that 100 issues isn't something to be proud of, mind you... |
|
I love the Curse of Monkey Island reference there, Morn. I think my favorite quote from that is actually "Nice Fluffy Towel." I ripped that sound out of the game and made an mp3 out of it. It just repeats and repeats on winamp.... well... actually it doesn't. Back to topic, from the impression of the review, i'm even more psyched to get this game. Even if it was before final testing and all that, do you think that Lionhead could break the game after having it practically done? I don't really think so. |
|
Even if it was before final testing and all that, do you think that Lionhead could break the game after having it practically done? I don't really think so. Not really, but I do have to wonder what wasn't really done and thus not talked about in the 'review '. Is the Gesture system really as good as they say it is, or was there a bug in this that Lionhead assured them would be fixed. Can you really create unlimited wonders or is there a limit that just wasn't working in their copy, did they just assume this is the way it would be in the retail verion. What other bugs are there that where not talked about, will some of these not be fixed or be worse in the final. I'm not about to pass up on the game because of the questions I have about the review, but I'm not getting more hopfull because of this (which would be what lionhead would have wanted). On that last note I want to know one thing. If the review trashed the game how many people think lionhead would be all over PCZone for running a reivew on a early copy? |
|
Pc Zone tends to be a tad enthusiastic about some games and get 'exclusives' eg, all the reviews of Mechwarrior 4 in November.. when the game only came Uk Retail in February (pissingly annoying for a me..... bought a sidewinder 2 JUST for that) That aside, theres very little in the review that hasnt been said in previews, looks at, most anticipated game of 20XX features that have been running since Peter Molyneux said 'Lionhead studios is working on this.....'. Its a bit like the saturation surrounding Tribes 2, I mean, does anyone HONESTLY expect to find any neat cool nifty fun swiftly groovy features when they get to play it for themselves? I dont, and Im sick to death of the 923412424352432.4 (One was a corrupt jpg) that have been smeared around on various websites. To me B&W sounds very like Dungeon Keeper blended back with Populous, you can play nice and expand and only do war when needed. Or you can be, like Im almost sure Mr0n is going to be, an utterly evil shit and torture people just to see what happens. The first quest of rescuing the lost boy for his sister... dropping the sister in front of him (or on him) killing him, boy sees it and dies.... Well fuck a duck, that sounds like a level of freeform that superceeds Deus Ex's mark and multiple solutions means that the game should be different for everybody. I wonder tho, how much of its going to rely on the personality of the player and how much on logic and how much on 'now what happens if I'.... Would love to see the logic heuristics going on in the background that can cope with the user just offing and doing something whacked.... Oh like say gathering up a mound of rocks and building them into a pyramid and stranding one of your peeps up on top of it .... then smiting the area with some overly powerful spell whilst the rest of your peeps look on... Oh yes ... fucked up behaviour is going to be good. Ds *smites Cleghorn with Black&White cow of Kung Fu +7* |
|
A Belgian magazine called PC Gameplay (www.pcgameplay.com DUH) also reviewed B&W, after printing the Lionhead diary for 3 years. They even got an interview with Peter Molyneux and had him explain the whole game. Something else: this is going on for years, dudes! Remember beta-reviews? Think about publishers making exclusive deals with certain mags!...and other stuff If I remember correctly we've had this discussion before. Zachak, I personally don't understand why you're starting to whine about this issue NOW, while it's firstly already discussed, and secondly it's been going on for SOOOOOO long. Some German site apparently has also reviewed B&W. Since I can't read German, I am going to ignore it. However, feel free to make inflammatory remarks about the Germans if you are feeling grumpy. Do I sense some aversion against Germans here? LOL! I don't mind, I'm Dutch, but it seems you're not taking them seriously because you dont understand the language...? |
|
#30: You'd have to be Belgian to call your site Game Shrimp (www.spelletjesgarnaal.be) :-P |
|
Check out http://www.boards.ie/bulletin/Forum6/HTML/001062.html This sheds some light on what people think. Wasn't the entire story that 1) Some guy at pc-zone reckoned black&white would be all icing and no cake 2) Lionhead invite him down to prove him wrong 3) He arrives down, is impressed and then (this is the dodgy, perhaps untrue step/link) 4) Wasn't this the same magazine that provided the exclusive "Review" and published it against lionhead's wishes? |
|
Bollix link didn't work... ../Forum6/HTML/001062.html |
|
Wow, no registration, less Andy, I might start posting here again............. probably not. |
|
Zachack, NEVER mess with Germans !!! |
|
#37 Steve Bauman talked about "Strategy Plus" Magazine Steve, I can remember going to GenCon many moons ago and these guys from Stategy Plus had a booth promoting their new magazine. That was in like 1994 or so. |
|
#24 -- Steve wrote: "Also their methodology for comparing reviews is flawed, as there is no baseline. Who's to say PC Gamer's apparent "soft" high review scores aren't more representative of the general public's tastes? I recommended they use their own site's ratings for that comparison, as I'm guessing they know their reviews aren't influenced by advertising, but they said that would look too much like self-promotion." I agree with this. They should have found games that were reviewed in all three mags, then compared them, and after doing this 100 times you'd have a pretty good idea of how soft or hard a mag is on reviews. My opinion is that an average game should score a 50%, in other words, the middle rating should represent an utterly average game, and better-than-average games scale up from there, and vice versa. Instead, it seems that a game that is merely average scores around a 70%. Kinda of reminds me of how investment houses and banks that rate stocks rarely use a "sell" rating -- just heard yesterday that of the 5000 most recent ratings by all the top rating banks, on 47 were "sell." I am encouraged when I see a high profile game like Timeline get a 10% rating, as it did in the most recent Computer Games. ;-) Scott |
|
Sorry guys, off topic... and I'm sure this has already been said before... but is there a version of CrapSpy that supports PC5? Or is Morn forcing us to us his interface (Which is great BTW Morn! ; ) -- Mav |
|
Steve, I can remember going to GenCon many moons ago and these guys from Stategy Plus had a booth promoting their new magazine. That was in like 1994 or so. That was probably us, though we weren't exactly "new" then, though there had been some editorial changes around that time. The magazine was launched in 1990. |
|
I agree with this. They should have found games that were reviewed in all three mags, then compared them, and after doing this 100 times you'd have a pretty good idea of how soft or hard a mag is on reviews. But without some baseline, they're still just relative to each other. If you went to one of those game ranking sites that averages ratings for like hundreds of editorial sites, and compared them to that average, you might be able to make a case for one being "soft" or "hard." My opinion is that an average game should score a 50%, in other words, the middle rating should represent an utterly average game, and better-than-average games scale up from there, and vice versa. Instead, it seems that a game that is merely average scores around a 70%. Absolutely, but you won't get that with a 100 point system because it's tied to grades. A "C" grade (or average) is 70%. So most 100 point rating systems are really 60-100, as anything below 60 is just varying levels of awful. We average just over 3 stars for all of our reviews (out of 5), which is about right as it's generally considered an average or mediocre rating. |
|
I wouldn't really trust UK mags, but at the time PC Zone made that review Lionhead announced that they had only a few bug left to fix (11 if I'm right), so that version was pretty close to completion already. I am very disappointed though, cause that review was VERY poorly written. A review usually tells you "every detail" about the game, but this one really was a preview with a score. I don't think you can really review a game after just trying out the game. 1. Assuming that PC Zone has a final version, what do you think about exclusive reviews? Also, how and why are they handed out? Bribes? Blackmail? Just being a really nice guy? Inquiring minds want to know! They had play it at Lionhead Studios. More about that coming up :) 2. If it isn't a final version, how unethical is this? Is it unethical at all? Since Lionhead hasn't attacked the review, should we assume that it is valid? The game was almost ready, so that doesn't make it unethical, but like I said you can review a game just by trying it out. I don't think they really had time to try out the multiplayer mode (which is cool BTW. Can't say more, because of the NDA). And it IS valid, cause Cathy Campos confirmed it. 3. Now that you're talking, tell us about any bad experiences you may have had with the press, or vice versa. C'mon! I need gossip to live! And don't forget to play name-dropping games! Don't get offended, but in my opinion the UK mags are a little bit ridicilous. All they want is EXCLUSIVE HAND-ON PREVIEWS (notice bold, capital letters :) Why do they write often almost 10 page long previews of games they haven't even tried or maybe played for 10 minutes? |
|
Scott Miller said: My opinion is that an average game should score a 50%, in other words, the middle rating should represent an utterly average game, and better-than-average games scale up from there, and vice versa. Instead, it seems that a game that is merely average scores around a 70%. I'll have to disagree here. While this may be a better system, the fact is that on a 100-point scale, people are used to seeing 70% as an average grade ('C' grade in school), and they bring that expectation to any 100-point scale. Hell, if I didn't read the fine print, and saw a 50% rating, my initial thought wouldn't be "average," it would be "is it THAT bad?" Of course, a reader should always look at the fine print, and I personally would, but I don't know that most people do. I think in a case like this you should stick to people's expectations or use a different (5-star, maybe) scale. -brennan |
|
Steve: "We average just over 3 stars for all of our reviews (out of 5), which is about right as it's generally considered an average or mediocre rating." I don't expect the average review to be 50% in any magazine, because many bad games are not worth reviewing, which skews the average rating in a magazine upward. Brennen: "I'll have to disagree here. While this may be a better system, the fact is that on a 100-point scale, people are used to seeing 70% as an average grade ('C' grade in school), and they bring that expectation to any 100-point scale. Hell, if I didn't read the fine print, and saw a 50% rating, my initial thought wouldn't be "average," it would be "is it THAT bad?" Yup, because we incorrectly view 70% as average on a percentage scale thanks to the grade-school system, I prefer using a 5-star system, which is functionally equal to a 0-10 system (as long as you don't use half points, like 5.5 -- not to say that half points are bad, but anything more than 20 gradations and the ratings system becomes too subjective, I think). Scott |
|
#49 by MaverickUK: Sorry guys, off topic... and I'm sure this has already been said before... but is there a version of CrapSpy that supports PC5? Craig is going to update CrapSpy's PC script soon -- if you have Auto-Update enabled, it will notify you once the new script is available. - Morn |
|
Sorry, Steve, but your argument about methodology doesn't hold up. You can only analyse game review scores in the aggregate in comparison to the aggregate for another publication. There is no other common point of reference, no absolute zero, no velocity of light. To compare the aggregate ratings to some other source, such as AVault, as you suggest, assumes that those AVault ratings are the absolutely correct, pristine ratings, the ones that a game should have received (in the absence of advertising influence, or whatever) ... something that could not be taken by a reader on face value and is otherwise impossible to prove. In these kinds of statistical analysis, you are stuck with "greater-than/lesser-than" operations... there cannot by definition be any objective baseline. It's like Chomsky's analysis of pages devoted to foreign country news by the New York Times... he can't effectively establish how many pages a country *should* have got... so he compares East Timor coverage to, say, Israel or El Salvador, and draw the inherent conclusions about editorial bias the column-inch counts reveal. The methodology was not "flawed", thank you very much. It is the only methodology available. PS to Scott Miller: in fact, we compared *105* reviews in Computer Games, to 80 in PC Gamer to 68 in CGW (all the reviews run in our five month survey period). So I'm afraid you lost me there. |
|
By the way, Steve, I also disagree with your insistence on a need for a disclaimer on media analysis pieces outside one's own medium. That's simply not common practice in any media industry today. That's because my site's relation to your magazine is not an NBC-ABC relationship, as you suggest, but a New York Times-NBC relationship (to take on airs for a sec.) Plenty of newspaper sports sections criticized Monday Night Football for bringing in Dennis Miller without prefacing their remarks with a disclaimer that they both take ads each time they did. Sure we both accept ads, and we can safely assume our readers are smart enough to know that, but there's no real comparison between ad bases. Ergo, any disclaimer would be lip service. BruceR |
|
Oh, one last thing. It would also be statistically invalid to compare a magazine's 5-star ratings to a website's 5-star ratings with the idea of correlating to advertising expense, for the simple reason that the majority of games reviewed by a magazine are advertised in that magazine, and the vast majority of games reviewed by a website aren't. You could certainly compare websites' ratings in the aggregate to magazines' in the aggregate, and draw conclusions about the two media as a whole, but that can't tell you anything about an individual title in either medium. But as I said to you in email the other day, it would certainly be an interesting article for someone to write someday. BruceR |
|
#26, Yeah, and sometimes I wonder how gamers.com blew through $11 million in six months, too, while we've managed to last five years. But it doesn't keep me up at nights... |
|
Sorry, Steve, but your argument about methodology doesn't hold up. You can only analyse game review scores in the aggregate in comparison to the aggregate for another publication. But here's the problem Steve is talking about, the readers for Computer Games, PC Gamer, and CGW are very different. And the reivew scores will take that difference in taste into account. A reader for Computer Games may like a game more then the readers of PC Gamer or CGW. You can not judge editorial covarage with out making a survey of the readers and their responce to the covarage. Take the two local newspapapers in fortwayne, both are owned by the same company, both have very different takes on the issuse. One tends to be more conservative the other more liberal. I can not judge the editoral covarage of the newspapers by simply compareing them to each other, rather I have to compare what the average reader of that paper thinks of the covarage. |
|
You can only analyse game review scores in the aggregate in comparison to the aggregate for another publication. There is no other common point of reference, no absolute zero, no velocity of light. Then why do the comparison at all? The article did not note the problems you've noted here. Unless the point of the analysis was to raise the question that there could be bias, something which could not be proved without actual evidence, i.e. a smoking gun, a memo, an e-mail, someone stepping forward and speaking on the record. And if that was the reason for doing the article was to merely raise the possibility, knowing full well no conclusion could be drawn, I question the motivation for writing it in the first place. Editorially speaking, Adrenaline Vault can't be a media watchdog when it's in direct competition with the publication's it's watching over. A media watchdog site, to work, would have to focus solely on watching the media; as soon as it crosses over to covering the news in addition to the media, it loses its ability to appear unbiased. Sure we both accept ads, and we can safely assume our readers are smart enough to know that, but there's no real comparison between ad bases. That's only because websites aren't receiving that many ads today; it was a different story a year ago. the majority of games reviewed by a magazine are advertised in that magazine, and the vast majority of games reviewed by a website aren't. That's irrelevant, because I suspect Avault is calling on these companies looking for their advertising dollars. If you're going to imply there could be some sort of correlation between review scores and advertising, might you consider there could be an attempt to win favor with high ratings in order to secure ads? I assume websites aren't getting game ads for a lack of trying, unless of course Adrenaline Vault refuses game company ads. By the way, I did a semi-random sampling of Adrenaline Vault reviews that may have been used in your study and found the average rating was 3.8/5, with the majority of ratings being 3.5 and 4, which would put it about equal with PC Gamer. So would you say Avault's reviews are "soft?" |
|
None, I absolutely agree media has biases, and for reasons not limited to either advertising or readership. But what's at question here is whether it was appropriate to see if game magazines show *any* evidence of bias in their review scores. Just to clarify for those who haven't read the piece linked above (thx, Apache, btw), here are our big two statistically based observations that Steve is saying here I shouldn't have made: *Overall, of the three game magazines in North American distribution, two (Steve's own CG and CGW) give out an average game review rating of a C. (3.5 stars). One (PC Gamer) gives a B to the same average game. *Of the three magazines, CGW gives far more F's to games: a third of their "marks" are F's, as opposed to less than a fifth for the other two mags. I didn't say any of these approaches to review scores was better or more just, just that they were different. (Personally, I'm going to keep reading all three.) And Steve: obviously Steven Brill doesn't feel his role as self-proclaimed world media watchdog is jeopardized by Brill's Content taking advertisements. Neither do I. And again, your rule against media outlets commenting on each other's content is so far from the general practice in every other industry, that I can't help wondering if it's not due in part to this just being the first time it's ever been done to your magazine. My main background is newspapers, but every other major media outlet I've worked for has these kinds of media analyses made about them all the time. (I'm sure it's not the first time, by the way, but I'll tell you a lot of readers have written me to thank me for reaffirming or challenging preconceptions they had made on a purely subjective basis about you and your competitors, so I feel safe in assuming there hasn't been a lot of this statistical analysis done heretofore.) The motivation of the article was simple: to help readers make a more informed choice at the newsstand. Period. If you wanted to read the review first, statistically speaking your best choice was PC Gamer. If you wanted to read reviews of every single last new game, your best choice was CG. And if you like your reviews vicious, your best choice in our five-month period was CGW. Those are just statistical facts. Again, I'm not saying one way of writing about the industry is better than the other, just that differences exist. AVault has its own way (with reviews written by far more intelligent reviewers than me, thank God... I just do news). And yes, I think an article comparing our average review scores to, say, Gamespot, would be very informative, too. I'm just respectfully suggesting a comparison to any website's review scores, composed as they are in a completely different environment (fewer advertisers, higher pressure to publish faster, whatever) wouldn't have increased the reader's understanding of our magazine statistics as much as you make out. BruceR |
|
Steven Brill doesn't feel his role as self-proclaimed world media watchdog is jeopardized by Brill's Content taking advertisements. I said nothing above about advertising conflict in my most recent message; I specifically referred to editorial competition. Here's what I said: Editorially speaking, Adrenaline Vault can't be a media watchdog when it's in direct competition with the publication's it's watching over. A media watchdog site, to work, would have to focus solely on watching the media; as soon as it crosses over to covering the news in addition to the media, it loses its ability to appear unbiased. A watchdog cannot cover anything other then what it's watching over. Brill's Content covers the MEDIA, and how the media covers the news, not the news itself. If it does cover the news (which it has gravitated toward, I believe), it loses its credibility as a watchdog because it's now in direct competition editorially speaking and its motives become suspect in the eyes of readers--are you pointing out flaws in the coverage, drawing correlations and conclusions, to make you look better and to draw negative attention to the other publication or are you doing it for some greater good? As soon as someone asks that question, the article is pretty much rendered useless. For example, do you think it would be fair for me to write an article about websites that write glowing previews, get box quotes, and then somehow magically reverse their uniformly positive preview opinions and produce negative reviews? For example, I could say, "Adrenaline Vault had this to say about Daikatana in a preview: 'If anyone can make a killer game, it's John Romero. With such an awesome storyline, exciting characters, and varied gameplay environments, Daikatana should throw the whole 3D action genre into another dimension.' Their review, though, painted an entirely different picture of the game. ..the final product does not manifest the qualities that a talented team with a singular goal and proper guidance can produce. ... I could not ignore its substantial discordant harmonies. Most damning is the crude programming, the inconsistent map design and the jagged artificial intelligence. ... Although fun and proficient at times, it also comes across as a sometimes lusterless simulation of better titles.'" Now what would be my motivation for producing this article? You could make a valid argument that I'm doing it for our readers, but in reality people might perceive that it was designed solely to embarass websites and make our own publication look like a pillar of integrity by comparison, because WE ARE COMPETING EDITORIALLY. here are our big two statistically based observations that Steve is saying here I shouldn't have made: That's not what I'm saying. Those are all measurable numbers, but you tried to draw correlation between those numbers and advertising, and there is no statistical basis to reach that conclusion. You said in the article, "We also tell you who you should look to for game reviews, and if those reviews are being influenced by the advertisements that pay for them." That's a perfectly valid question to ask, but how were you planning on drawing that conclusion? What statistical evidence COULD you produce that would show influence? You said, "One wonders, though, if the influence of ad purchasing isn't being more subtly felt? And if not, why is PC Gamer the industry darling right now? The answer may lie in the reviews." How could you prove that one way or another? What evidence, using ratings and counting ad pages, could show "subtle" influences? "One wonders if this more positive view of the industry isn't having some effect on the magazine's ability to get and keep its advertisers." Again, how could you prove this with your data? So again, what was the motivation for creating the article in the first place if you had no way of proving, one way or another, the basic questions the article was written to address? |
|
Also, the article seems to assume readers know a lot about a magazine's internal operations. It essentially gives the (incorrect) appearance that magazines, like websites, rely solely on advertising for their revenue. This ignores the significant role newsstand sales and subscriptions play in a magazine's finances, and they are the two areas where editorial quality, reputation and integrity have the greatest effect. They're also areas where letting advertising influence ratings would ultimately cause you the most problems; readers would, in theory, leave in droves if you no longer represented their opinions editorially. Also, in dicussing revenues you really have to give a minor mention of the costs of producing a magazine so people don't get the impression we're sitting on millions of dollars in profits every month from all of that ad revenue. Printing, postage, and duplicating hundreds of thousands of CD-ROMs every month costs a lot of money, hundreds of thousands of dollars in revenue every month. So we'd better have hundreds of thousands of dollars in revenue every month, otherwise we're history. A couple of sentences mentioning these issues would give a more accurate representation of a magazine then the article currently does. |
|
Fair enough on all points, but a couple rejoinders. I personally would have no problem with an article in any media outlet about any other outlet who previewed a game and got it wrong. Others may, I don't know. Again, though, in any other medium that kind of treatment is de rigeur. Who is supposed to call the media on its mistakes, if not other media? The government? All the quotes you make are correct. But you neglect the answers that were given, specifically our statement that there's *no* evidence that individual reviews are being influenced by advertising revenue. I can't answer the question without asking it, surely. And I believe I'm allowed under any pretense of objectivity to still suggest a possible interpretation of two facts we were able to prove: that PC Gamer is more generally upbeat, statistically speaking, in its reviews of about the industry, and it also has the broadest ad base. Yes, you're right: the exact meaning and interrelation of those facts is open to interpretation: but this sub-thread started in a critique of our methodology, not a search for the reasons for PC Gamer's current success. Why do you think they attract more advertisers than you do, Steve? What we promised we delivered. We told readers which magazine to look to for the fastest, hardest, fairest, most upbeat, and most comprehensive reviews. And we stated definitively that there was *no* evidence any magazine (including PC Gamer and CG) was allowing its reviewers to be influenced by the number of ads, or lack thereof, for the same game. Any other inferences that the reader draws from the facts presented are their own to make, frankly. |
|
None, I absolutely agree media has biases, and for reasons not limited to either advertising or readership. But what's at question here is whether it was appropriate to see if game magazines show *any* evidence of bias in their review scores. Then exactly what is the point of linking that *any* bias to ad revenue. The article makes NO meation that the 'soft' reivews found in PCGamer could be because the readers tend to be a bit 'softer'. Rather the article trys to lead the reader to belive that this is to get and keep more advertisers. The article also attempts to get us to belive games from companies like Simon and Schuster and SSI are trashed in both PC Gamer and CGW, yet makes no attempt to explane why those companies may not wish to advertise in that mag. It also tries to get us to belive PC Gamer is ignoring games produced by companies not advertising with them. Yet says nothing that these games may not be reviewed or get lower scores because the readers of that mag do not like the types of games those companies are producing. |
|
Again, though, in any other medium that kind of treatment is de rigeur. Who is supposed to call the media on its mistakes, if not other media? The government? Media watchdogs that do not directly compete editorially, whose sole purpose IS to watch the media and not produce the same news stories the publications they're watching are producing. Do you have examples of direct competitors covering each other's coverage of an event, assuming something about it wasn't particularly newsworthy? While you might see a story in Time about Newsweek being forced to pull a story, would Time run an article about Newsweek's problems attracting advertisers? But you neglect the answers that were given, specifically our statement that there's *no* evidence that individual reviews are being influenced by advertising revenue. I can't answer the question without asking it, surely. But you didn't answer my questions. Based on the data you collected and its methodology, how DID you plan on answering the questions asked in the article? What basis do you have for saying there is NO evidence? What would you have expected to see to prove some correlation? All 90+ ratings in PC Gamer for advertised products? A cursory glance at the publication would show that wasn't true. [quote[Why do you think they attract more advertisers than you do, Steve? Because they have higher circulation, it's as simple as that. And we stated definitively that there was *no* evidence any magazine (including PC Gamer and CG) was allowing its reviewers to be influenced by the number of ads, or lack thereof, for the same game. Any other inferences that the reader draws from the facts presented are their own to make, frankly. But that still doesn't answer my basic question: using your research, what evidence COULD you have possibly presented? You asked a series of questions--worthwhile questions, no doubt--but as far as I can tell, you had no way to actually answer them with the information you obtained. If that's the case, why does the article even exist? |
|
And we stated definitively that there was *no* evidence any magazine (including PC Gamer and CG) was allowing its reviewers to be influenced by the number of ads, or lack thereof, for the same game. Any other inferences that the reader draws from the facts presented are their own to make, frankly. It may say there is no evidence, but it's not leading me in that direction with lines like these One wonders if this more positive view of the industry isn't having some effect on the magazine's ability to get and keep its advertisers. Of PC Gamer's 80 reviews in the period we're looking at, all but 11 (86 percent) were for products whose publisher was advertising in the magazine in the same period. CGW was the same, with 79 percent of its reviews being about games put out by its advertisers. PC Gamer is certainly suspect: products by advertisers got a median score of 78, as compared to a 67 average for non-advertisers. CGW is much the same: companies that don't or can't afford to advertise, like Simon and Schuster and SSI, get lower reviews for their games. In particular Timeline (Eidos), Gunman Chronicles (Sierra) and Tomb Raider Chronicles (Eidos) all spent vast sums of money with little apparent effect on reviewers' perceptions. The whole article up till the end of the 5th page lead in a totaly different direction then the findings show. And that last quote seams to clame that those three games should have scored higher based on the ads placed (this is in a section stating that there is no connection). |
|
I have to question your motives and the entire article Mr. Rolston when I see this paragraph as the lead in for the story as the link on the main page of the site... "Computer Game Print Mags: The State of the Industry While some online gaming sites struggle to survive, their print counterparts appear to thrive. Bruce Rolston takes an in-depth look at three top contenders, analyzing a number of factors including review content, length…and influences." You never make anything clear in the article and just put up a whole lot of questions even AFTER you've been given good answers by the interviewees or your own skewed comparison process fails to show anything. It's a clearly inflammatory piece to my eye. Isn't it true that these types of articles are mandated by your employment at Adrenaline Vault. If I'm not mistaken, the last time AVault advertised for help, a prerequisite was the need to write a feature article once a month. Could it be you couldn't come up with anything else and once this gave you no satisfaction, you decided to make it more questioning and play to the audience you figured you'd have? You also haven't really addressed the fact that AVault reviews average out excessively high overall showing a much more decided bias in favor of not offending any advertisers. Could it be that the real story is elsewhere and you're just trying to deflect the attention to the 'net's favorite whipping boys, the print mags? --Dave |
|
Come on, Steve. Let's leave aside the circularity of the circulation/ad base reasoning you posit and get to your latter point. It would have been very easy to statistically establish a stronger correlation, if one had in fact existed. You have lots of games that are only advertised in one or two magazines, and reviewed in all three, for instance. If the aggregate of review scores of games that were advertised was significantly different from the same game reviews where they weren't advertised that would be clear evidence of likely publisher influence on publications. That wasn't the case. Alternately, if there was a clear trend line in the comparison of average review scores with money expended on advertising, that would be evidence of an industry-wide bias in favour of the big spenders. That was tested, and also found not to be the case. But seriously, do you really think I was the first person who ever wondered about the connection? Don't you think every devoted game magazine reader would like to know, and hasn't wondered, and probably had a theory of their own on, which one of you goes easiest on the game companies? I did. Mine was wrong. But I did. Are people not even allowed to *question* your uncorruptibility now? And I ask again, who is this pristine untouchable source who would be the only person you'd deign to allow to analyse your magazines' coverage? You're not going to write about each other this way, for all the reasons you cite. So if a magazine were to behave inappropriately, who calls them on it? (And I gotta tell ya, given the mail I've been getting, I think my theory that people prefer to know more about what they're buying at the newsstand is generally holding water here.) And None, I said we found no evidence that individual review scores are being altered by publisher influence. But the facts we found *do* show that 2 of the 3 mags both give a lower score on average to a non-advertised game, and rarely review games that aren't advertised... but that correlation is not strong enough that we could safely discount the hypothesis that companies that can't afford to advertise generally make lower quality games, for instance. The rules of logic tell us one cannot prove a negative: i.e. that advertising definitely had no influence whatsoever, anymore than I can prove that aliens haven't ever landed on Earth. The best we can say in the game mags' defense based on statistical reasoning is that there's no strongly *positive* correlation between ad dollars and "game grades." That's what we found and that's what we said. |
|
Oh, and to Dave Long... Considering no one has yet to challenge a significant fact in the article, I feel justified still taking exception to your allegation of "skew." If what you're trying to accuse me (and by extension my current employer) of is producing articles that people actually want to read, well, I'm guilty of that, I guess. If you really knew every single last one of those facts about the game mag industry before we reported them, then I apologize for wasting your time. But I don't believe that's true. |
|
<i>Thinking...</i> |
|
#34: Mother of God! It's the planet of the crap! More like Mother of God! It's the planet of the shit! Crap would be more like basura, perhaps... |
|
that would be clear evidence of likely publisher influence on publications. "Clear" evidence of "likely" publisher influence? Kind of says it all right there. "Clear" and "likely" generally don't go together when talking about evidence. Clear evidence leads to actual proof. Actual proof in this case would probably be an advertiser or editor (or former editor) speaking on the record about such a relationship, or an internal memo or e-mail that promised editorial coverage if publisher's advertised. That would be clear evidence of direct influence, and we could skip this whole "likely" business. And that would be a story. But seriously, do you really think I was the first person who ever wondered about the connection? Of course not, but again, was there any way you could prove actual publisher influence using your methodology? Proving "likely" influence... c'mon, you have to do better then that. There are tons of connections reporters wonder about, but if they can't make those connections with actual evidence the stories get dropped until evidence is presented. Anyone publishing stories drawing "likely" connections in, say, a murder case would run the risk of a libel suit. (Of course you can dance around libel using specific language but that doesn't make the article less sleazy because it raises the issue and does damage the reputations of those involved.) Are people not even allowed to *question* your uncorruptibility now? Nope, never said. I wish MORE people would ask those questions, but I'd prefer them not to be publications that covered computer games for a living since it's in their interest to make the competition look bad. I was interviewed by Brill's Content a few years ago, and would have no problems with their scrutiny, regardless of the outcome, because they do not cover videogames. They will not benefit in any way from saying good or bad things about our publication. Avault, on the other hand, stands to gain traffic and respect in the eyes of gamers, your audience and our audience, by drawing these sorts of conclusions. (And I gotta tell ya, given the mail I've been getting, I think my theory that people prefer to know more about what they're buying at the newsstand is generally holding water here.) Well duh. If any magazine ran an article reviewing websites or our competitors we'd get the same feedback. But most people probably don't spend time thinking about issues like motivations and ethics and conflicts-of-interest; that doesn't mean we shouldn't be concerned with those issues. The best we can say in the game mags' defense based on statistical reasoning is that there's no strongly *positive* correlation between ad dollars and "game grades." That's what we found and that's what we said. I'll ask this one more time, was it even possible to find a correlation between ad dollars and ratings? What if PC Gamer rated every advertised game at 95%? What does that really mean? Because there's no baseline, no "right" review, can you only conclude their reviews are "likely" influenced by advertising? What does logic tell you about "likely" influence? Aren't they just as "likely" to have no influence? Would you want to be convicted on the "likeliness" of being guilty? Or to ask you more directly, would you want your publication's reputation tarnished because in studying arbitrary ratings, a single writer interpreted those numbers to mean you're "likely" influenced by advertising dollars? |
|
And None, I said we found no evidence that individual review scores are being altered by publisher influence. So their not influenced But the facts we found *do* show that 2 of the 3 mags both give a lower score on average to a non-advertised game, and rarely review games that aren't advertised... So they are influenced but that correlation is not strong enough that we could safely discount the hypothesis that companies that can't afford to advertise generally make lower quality games, for instance. SO they aren't influenced The rules of logic tell us one cannot prove a negative: i.e. that advertising definitely had no influence whatsoever, anymore than I can prove that aliens haven't ever landed on Earth. So they could be influenced The best we can say in the game mags' defense based on statistical reasoning is that there's no strongly *positive* correlation between ad dollars and "game grades." That's what we found and that's what we said. So they're not influenced. The whole article ends up brining the same things to my mind as that post did. I'm not far more confused about the reviews I formatly trusted (based on the fact that the games I did buy and play all got scores I agreed with). All your telling me is that since we could prove there was a connection there very well might be a connection between review scores and ad dollors, yet nothing in the article or that you've posted shows any connection between the two above and beyond what one would espect from reader or quality differeces. |
|
Let me correct that last section All your telling me is that since we couldn't prove there was a connection there very well might be a connection between review scores and ad dollors, yet nothing in the article or that you've posted shows any connection between the two above and beyond what one would espect from reader or quality differeces. Must use preview |
|
I'm glad to learn that I'm not the only one who thought that the Avault article was instantly biased from the beginning, all the while presenting evidence contrary to what they seem to be subtly suggesting. Personally, I find most Avault articles seem to serve Avault's own interests, such as protesting the CPL's innocence ("All the News That's Fit?", interestingly following an article obviously influenced by (though not specifically refering to) the whole PlanetCrap 3DR screenshot fiasco), attempting to add more validity to the CPL ("The Killer App Of The 19th Century"), reducing the connection between violence and gaming (news reporting which is quick to point out even the most strenously related faults with studies which don't agree with Avault's view of no such link, and articles such as "The Skeptic" presented with no similar articles arguing critics views), etc. Now, this bias that I detect is probably because A) I don't read most of Avault's articles anymore, as I got sick of this bias I was percieving, and B) the articles I do read tend to read are the ones that get my attention, and as I am tend to look at Avault with an attitude of "huh, I wonder how they are going to push the CPL's cause today", I tend to notice ones with blurbs such as the one following "The Killer App Of The 19th Century". This bias I perceive is undoubtly due to the enthusiasm of the Avault staff for the subjects I have mentioned, and I'm sure that it has nothing to due with pressure from New World upon the editorial staff, I'm sure. Really. But can you imagine if another website published the previous opinion, backed up by a review of lines of text or some-such non-resolution-specific statistic which "proves that avault is editorially unbiased, and any percieved bias (which there appears to be some of, doesn't there, hint hint) is probably due to the similarity of opinions between the editrorial staff and New World. Really. Although it does look a tad fishy if you ask me. Not that there is any proof."? The problem with the article on avault is that it seems to insinuate that there is back-room dealmaking going on at the print magazines (which Avault, the staunch defender of the public interest takes no part in. Please ignore the fact that we cover game news and reviews, and also please ignore the advertising from game companies surrounding this article. They are unrelated.), but can not be proved. This appearance of insinuation is being pointed at by most of the posts here, but Bruce seems to be ignoring the suggestion of it. The discussion on this topic (which I personally find much more interesting that a magazine reviewing a press copy of Black & White, so I don't really mind contributing to the slide into off-topic land) seems (to me at least) to be going something like this: Poster: The article is inherently skewed as it is done by a game site and takes about its competion. It doesn't include anything about Avault's score averages between advertisers and non-advertisers. And it insinuates that the mags accept bribes even though it presents no evidence. Bruce: If same-media didn't engage in a fit of bickering over who was more impartial we would have Communism! I could point to several examples... I, er, just don't have time right now. And we are a website. The article was about print mags. A comparison would be incrediably poor taste. And it wasn't insinuating anything. THE EVIDENCE PROVES THAT WE COULDN'T PROVE ANYTHING. Poster: Yes, but could you please explain why you say stuff like <followed by quotes that insinuate the print mags are taking bribes... both in and out of context>? And please give me just one example of a media that reports on its competition, please. And how do you not compete for our audience and advertisers? Bruce: Don't you understand? THE EVIDENCE PROVES THAT WE COULDN'T PROVE ANYTHING!!!! HOW DENSE ARE YOU? Poster: Yes, but the insinuation, what about that? Bruce: THE EVIDENCE!!!!! etc. Of course, this summary rather villifies Bruce, but it is in the interest of exciting discussion so I feel it fits into the Planet Crap way. The summary also leaves out extensive name dropping of Brill's Content and who held what "real" jobs in the news industry, etc. So what I'm basically trying to say is, if you feel this post worthy of a response Bruce, please answer at least this question: "Do you intend the article to insinuate that the print mags accept bribes, and how do you feel this general suggestion of the bribes contributes to the article's overall attempt at an unbiased review of the subject? Also, why do you feel that including Avault in the 'test of purity', as it were, is not a valid suggestion?" Thank you, Nick Tamm (my first post on Planetcrap) |
| C O M M E N T S |
|
Home »
Topic: Limeys review Black & White, gamers scratch collective heads.
|«« - Previous Page - Next Page - »»| |
| P O S T A C O M M E N T |
|
|
| C R A P T A G S | ||||||||
|
|
| There are currently 0 people browsing this site. [Details] |
Powered by blah 0.9.1-dev •
PlanetCrap is © 1997-2035 Hendrik "Morn" Mans
|