How Microsoft allowed critically flawed console to reach production

[moycon]

Taken from IGN:

 

Xbox 360 RRoD Failures Finally Explained

Microsoft bungle cost company a billion.

by Gerry Block

 

June 11, 2008 - The Xbox 360's proclivity for suffering the infamous three red lights / Red Rings of Death hardware failure is legend these days. Despite Microsoft's initial claims that hardware failures were a slim 3% through much of the console's first year of availability, the gaming community knew otherwise, with real-world failure rates estimated to be in the 40-80% range for consoles manufactured in the first production runs.

 

After a great deal of pressure, Microsoft tacitly recognized the RRoD epidemic, and announced extended warranty support that would cost the company more than one-billion dollars. Microsoft refused, however, to discuss exactly what had caused so many consoles to fail, noting only that the problem stemmed from a "design issue."

 

The gaming public, however, was well aware of the truth of the situation after a number of peeps in the modding community opened their broken consoles and found the 360's GPU had run too hot, resulting in warped motherboards, unseated chips, and disconnected heat-sinks. Such evidence was only given greater credential when redesigned Falcon Xbox 360s arrived with a newly beefed up heatsink on the GPU.

 

Though RRoD failures are now generally a thing of the past after several chipset redesigns on Microsoft's part, the question has long remained as to how Microsoft allowed critically flawed Xbox 360 consoles to reach production, much less become a billion-dollar epidemic. Today, light seems to have finally been shed on the issue, as eetimes.com reports from the Design Automation Conference.

 

Speaking at the show, Bryan Lewis, research vice president and chief analyst at Gartner, revealed that Microsoft, bucking industry norms in what was hoped to be a cost-saving decision, chose not to seek the input of a traditional ASIC (application-specific integrated circuit) vendor in designing the architecture of the 360's GPU. Relying purely upon in-house processor-architecture design expertise (or lack thereof), Microsoft went directly to Taiwan Semiconductor Manufacturing Co to put the Xbox 360's GPU chips into production.

 

The result was the mass production of ungainly processors that ran quite inefficiently, generating far more waste heat than expected. According to Lewis, had Microsoft brought an ASIC vendor into the design process, a far more elegant chip architecture would likely have been developed, leading to much cooler-running GPUs for the Xbox 360. Bryan Lewis noted that Microsoft has since developed a new version of the Xbox 360 GPU with the support of an established ASIC vendor, widely thought to be ATI / AMD

Edited June 12, 2008 by moycon

[Atari5200]

I can't take any article seriously that uses "peeps" as an actual word to represent a group of people.

[cimerians]

Overall, I feel they forced this machine out the door before Sony and Nintendo can get theirs out.

 

Its a clunker but Xboxlive is great and it seems developers can make great games on it.

[MCHufnagel]

Overall, I feel they forced this machine out the door before Sony and Nintendo can get theirs out.

 

Its a clunker but Xboxlive is great and it seems developers can make great games on it.

 

I agree with you on both points.

 

Game-wise, the Xbox 360 might be my all-time favorite console. I rank it up there with the Genesis and Dreamcast.

[Reaperman]

I'm not sure I'll ever be able to recognize a 360 without problems, so I'm staying away until I'm *really* sure my system will make it for at least 10-20 years.

It's a shame since I jumped on their last console fairly early and still play it more than the other last gens. That 360 controller is about the best it can be too. Nearly perfect, it's easily one of my top 5 favorite stock controllers.

 

 

Game-wise, the Xbox 360 might be my all-time favorite console. I rank it up there with the Genesis and Dreamcast.

really? I'm having a bit of trouble getting excited about that too. It's got forza, and other racing games, but so does everybody else and most of the genres I love are missing on 360. Really 360 seems to thrive on graphical updates to previous xbox titles, but too few of my favorites seem to have made the cut.

Edited June 12, 2008 by Reaperman

[Lost Monkey]

I love the 360 but I cringe every time I hear it make a strange sound... I just can't get over this apprehension about the quality of it...

 

Plenty of nice games though - can't compain about that...

 

I like to hang onto my consoles forever and I find I am having a hard enough time keeping myself with a working Playstation, Duo and DC, I don't want to add another system to the repair table too soon..

[Gregory DG]

Overall, I feel they forced this machine out the door before Sony and Nintendo can get theirs out.

Exactly. They wanted to beat Sony to market no matter the cost. History will answer if this was a smart move or not.

[TwinChargers]

I'm not sure I'll ever be able to recognize a 360 without problems, so I'm staying away until I'm *really* sure my system will make it for at least 10-20 years.

 

That is the biggest scare that will probably keep me from ever owning one. Has there ever been a console that has had a higher failure rate? That and I fear that in 10 years these games might not have the same playability factor as games that are currently a decade old, due to the fact that online play is key for this gen.

[Crazy Climber]

I have owned 2, Both systems worked great.

[Reaperman]

I've just conducted some "highly scientific" research about the failure rate and have just concluded that it's around 10%.

 

Searching ebay game systems for

"360 "as is"" yielded 323 hits where

"360 -"as is"" yielded 2699 results.

It's better than I thought it would be, but that's around 10%, without factoring in the number microsoft has fixed, search problems, etc. Really those are huge factors, but for 10 seconds of my time, the numbers are still high enough to be alarming.

 

the reason I expected worse is that of the four people I know with 360's, two are on their second system. One of the survivors has this neat little cooler with a/v outputs on it. I think if I went 360, that'd be on my shopping list.

 

I kind of miss game systems without fans and optical drives. They seem to last pretty much forever.

Edited June 12, 2008 by Reaperman

[Atari5200]

what surprises me, and I have to be skeptical, is that according to the article, heat seems to be the problem correct?? Yet no system that I'm aware of has ever caught fire. Doesn't that seem odd that it's hot enough to melt motherboards and components, but never caught dust, or anything inside on fire?

[moycon]

I can't take any article seriously that uses "peeps" as an actual word to represent a group of people.

 

It's a writer from a videogame site...Not CBS news.

 

And what exactly are you skeptical aboot? Members of this very site have opened 360s after the RRoD and posted their findings on this forum. Do you suppose maybe MS planted people on AA (and all over the net on various video game forums) to start the viscous rumor that their units ran too hot so we wouldn't discover the REAL problem which would be??? What?? What do you suppose they are hiding? Maybe aliens got into some of the units?

 

I'm all for skepticism, but what exactly would the motivation be here? LOL

 

If the units were running hot enough to catch fire, MS wouldn't offer 3 years extended warrantees, they'd be recalling the 360.

[Lost Monkey]

If the units were running hot enough to catch fire, MS wouldn't offer 3 years extended warrantees, they'd be recalling the 360.

 

And paying out some huge settlements...

 

They don't need to catch fire to melt... they were just slow cookin'.

[retrogeek]

I'm not sure I'll ever be able to recognize a 360 without problems, so I'm staying away until I'm *really* sure my system will make it for at least 10-20 years.

 

That is the biggest scare that will probably keep me from ever owning one. Has there ever been a console that has had a higher failure rate? That and I fear that in 10 years these games might not have the same playability factor as games that are currently a decade old, due to the fact that online play is key for this gen.

 

 

I believe the first run of Playstations had a very high malfunction rate based on overheating issues. I actually waited a few years before buying one in fear of the system burning up!

[Atari5200]

i'm not saying it WASN'T heat, I'm just saying I think it's something more, heat + something else. I understand things melt and stuff, I've fried many electrical things in my life and every time a component blows or melts, it at least smells and may even smoke a little. As far as I know, nobody has reported that have they?

[hex65000]

i'm not saying it WASN'T heat, I'm just saying I think it's something more, heat + something else. I understand things melt and stuff, I've fried many electrical things in my life and every time a component blows or melts, it at least smells and may even smoke a little. As far as I know, nobody has reported that have they?

You don't need it to blow up for it to be a failure. This is really true with today's surface mount technology. You get enough expansions and contractions on a ball grid part with bad thermal management and you can introduce an intermittent solder joint or outright failure. Not to mention, a microprocessor is not a voltage regulator. If it overheats, it'll more likely not perform as expected. (IE: an enable line doesn't go low enough, a bit doesn't flip in time, you get data corruption and it's downhill from there.)

 

Not to mention, if the part get stressed badly enough through heat, it may start exhibiting that behavior even when operating within its temperature spec. Then you're hosed because it's not a bad solder joint, the whole part has lost its mind and the only solution is to replace it.

 

Side story:

At my old job they did hybrids (tiny surface mount electronics with bare ICs in really small packages) They would do wild stuff like wirebond onto a chip die to minimize the footprint. They were doing a device that was being used at extremey hot hostile temperatures and they were getting failures because the epoxy they were using was expanding at a different rate than the board it was glued onto.

 

While MS isn't using ceramic boards and industrial-rated parts, the concept remains: Heat + tiny electronics = stress * bad thing

 

As a guess on the 360 architecture, there is likely some kind of POST / sanity check to make sure everybody is happy. If not, throw an err. How elaborate they get depends on how quickly/accurately they can detect an error and RROD the machine.

 

So really the problem is : Heat, bad design, band-aids on a bad design, and stressed parts.

 

Hex.

[ Don't ask... the results will be 'undefined' ... ]

[harveymush]

I don't know if I believe that the RROF has been resolved yet as the article claims, my second 360 died on sunday which was a replacement for my first 360 which had the same fault, and this one was only 5months old. Its now waiting to be picked up to take it away to be repaired, but I think its more luck than anything with the 360, I am really starting to think that every 360 carrys the same risk because of that stupid X clamp design that holds the heatsinks on, once that fails then the heatsink moves and the cpu/gpu fires.

 

Its is a real shame, after watching many youtube videos of people fixing the 360 with just screws and getting rid of the X clamp altogther its a wonder why Microsoft doesn't do the same, cause its really put me off buying any future system Mircosoft releases.

 

The system itself is great of all the consoles I have ever owed the 360 is by far the best, it has great games and the on line side of the things is second to none, just a pity of the reliablity.

 

Rob

[+remowilliams]

This is really true with today's surface mount technology. You get enough expansions and contractions on a ball grid part with bad thermal management and you can introduce an intermittent solder joint or outright failure.

And that's exactly what's happened to tons of 360s due to the stupid xbrace design that eventually over time kills the BGA connections. Then presto - RROD.

 

I've seen this myself first hand, and repaired one myself as well.

[hex65000]

This is really true with today's surface mount technology. You get enough expansions and contractions on a ball grid part with bad thermal management and you can introduce an intermittent solder joint or outright failure.

And that's exactly what's happened to tons of 360s due to the stupid xbrace design that eventually over time kills the BGA connections. Then presto - RROD.

I've seen this myself first hand, and repaired one myself as well.

If you have the opportunity and can take close up photos of the damage, I'd love to see them.

 

I'm not a big fan of BGA technology. I know it works when done well, but it always seemed like something that could get screwed up and an absolute bear to detect and fix through inspection. You can only catch these kinds of flaws through test. If its designed like most things, the design engineers never leave enough hooks in the circuit for you to do 100% coverage.

 

Hex.

[ Wonders if these units will ever get fixed right... ]

[Lost Monkey]

This is really true with today's surface mount technology. You get enough expansions and contractions on a ball grid part with bad thermal management and you can introduce an intermittent solder joint or outright failure.

And that's exactly what's happened to tons of 360s due to the stupid xbrace design that eventually over time kills the BGA connections. Then presto - RROD.

 

I've seen this myself first hand, and repaired one myself as well.

 

okay - so from someone who knows the 360's innards about as well as he knows Bea Arthur's - does this mean we can expect the same failure on the newer boards too? (w/ the Falcon[?])

[Hyper_Eye]

This is really true with today's surface mount technology. You get enough expansions and contractions on a ball grid part with bad thermal management and you can introduce an intermittent solder joint or outright failure.

And that's exactly what's happened to tons of 360s due to the stupid xbrace design that eventually over time kills the BGA connections. Then presto - RROD.

 

I've seen this myself first hand, and repaired one myself as well.

 

okay - so from someone who knows the 360's innards about as well as he knows Bea Arthur's - does this mean we can expect the same failure on the newer boards too? (w/ the Falcon[?])

 

No. While the x-braces are the defect heat was still the thing that killed the system. It is the combination of heat and the x-braces that created the issue that led to RROD. The new systems generate much less heat and as such will not suffer the same failure rate. Also, the heatsinks were redesigned and now have heatpipes. This redistributes the heat to different areas from where the heat was before. So, even if the same heat were being generated, it is not all in the same area it used to be.

 

I am not saying that RROD is eliminated. I am not saying it isn't either. At the very least it should be significantly reduced. I would not be surprised if the failure rate of the newest production units are finally down to industry standards. Time will tell.