Stephen’s Personal Blog

A story that didn’t make the news was the meltdown of the VA’s IT infrastructure on 8/31 for one day this year. This article from Computerworld is an illuminating look at Enterprise level Information Technology. The principles interviewed have blamed the problem on employees failing to follow procedure. I’ve heard this used before in the IT department of every organization, all two, that I worked for. It’s a drum beat that the process owners beat relentlessly.

It’s also completely impossible for one simple reason. People aren’t computers. There is no way to tell a person to follow a set of rules every single time and expect them to do it. Computers are good at doing this. That’s why we use them, but don’t confuse computers with people.

This article competely misses two points. One is that this is going to happen again without a doubt. The second point is poor design of their infrastructure.

The VA should plan for IT mistakes just like any other disaster scenario. They seemed to have planned for natural disasters and network problems quite well. No forethought is given to getting the data back in spec. That’s simply not supposed to happen. Paper forms specifically made for resyncing data should be prepared and sent to replace any old forms that existed prior the technology “upgrade”. Plans for workarounds should be made for getting vital information from medical instrumentation. There is no excuse for test results or machine readings to be unavailable.

When the computers failed people did the natural thing. They reverted to the prior process which was paper based. It took two weeks of data entry to get one day of paper records in the computer. This is the real disaster. Where is the plan for the computers being down for 1 day, 1 week, 1 month? Apperantly, there isn’t one. Zero downtime and 100 percent uptime are impossible. Anyone who sells you this is lying. Get over and plan for it.

Why did the VA IT infrastructure fail? We’ve all seen the internet weather several storms. It slows down and some systems fail, but the whole Internet keeps going. If this system can be brought down by one port misconfiguration it has not been properly designed. Sites on the Internet are exposed to large scale intentional attacks every day and survive. The architectural descriptions of high volume sites like Slashdot are very interesting. Where is this attention to detail? Where is this level of skill? Where is this fault tolerance?

This instance happened to the Veterans Administration. It is no less applicable to any system that puts people lives in the hands of technology. At the medical malpractice suit your family brings, do you want Bob the 23 yr old computer geek to be your cause of death?. You can’t blame him. He’s not a computer.

The VA’s computer systems meltdown: What happened and why
Computerworld
Dian Schaffhauser

The articles title is overstated to say the least. It’s pretty good and makes a point.

The Vista Death Watch
Microsoft has extended the life of Windows XP because Vista has simply not shown any life in the market. We have to begin to ask ourselves if we are really looking at Windows Me/2007, destined to be a disdained flop. By all estimates the number of Vista installations hovers around the number of Macs in use.

How did this happen? And what’s going to happen next? Does Microsoft have a Plan B? A number of possibilities come to mind, and these things must be considered by the company itself.

This completely blows by those that have not dealt with bureaucracy of Dilbert’s company. For the rest, enjoy..

We have lost control of the apparatus
I know, I know, they stare at our work for eight, ten, or twelve hours a day. So you would think that we would set the standard for how computers ought to be. But the Good Old Days when most of users had never seen a computer before work have gone. Some of our users, fresh out of school, have already been using computers for ten years!

I’m not kidding. Our users are being exposed to applications we don’t control. And it messes things up. You see, the users get exposed to other ways of doing things, ways that are more convenient for users, ways that make them more productive, and they incorrectly think we ought to do things that way for them.

I’ve been working on selecting the components for an always on file server that will, hopefully, be run by solar power. If you keep up with this topic, you’ll know that it’s something like ten times cheaper to pick low power components than build large sources of power. Last night, I made my choice and place the order for the motherboard and power supply.

The most important aspect is the motherboard. It’s the single largest consumer of electricity. This component determines the power saving features like Wake on Lan and Sleep modes. All the devices plug into this one. It will stay on as long as one hard drive or network connection remains open.

A lot of PATA and SATA connectors with at least on expansion card are needed to be a file server. Many USB ports are nice, but not necessary. USB can be chained and any USB device probably means power wasted in an external brick. The minimum I settled on is 4 PATA hard drives, to support the existing set. And some SATA connectors, again, to support an existing pair. The expansion card will probably be used to provide an additional set of hard drive connections. Though, the way my onboard NICs fail it might serve as a backup NIC.

The goal was the lowest power usage that can support a lot of connected devices. There are three types of motherboards that can do this. The first is made of Via components. For years, they’ve been making motherboards that can fit easily into a high school biology textbook. I built a computer for Grandma off a 5000 board 2 years ago and am quite satisfied with it. Intel began making boards in this space a few years ago. Their performance is quite a bit better than Via, but the power needs are 5-15 Watts more too. Lately, AMD has begun making boards. These are by far the most efficient. The AMD Geode uses 5 Watts.

The Intel boards were out, because both AMD and Via scored better on power and processing power isn’t something I care about. The AMD boards were very tempting, but they didn’t support all the hard drive necessary connections. AMD and Intel have a limited selection of boards compared to Via.

I chose the lowest power Via board I could find, the Via CN10000EG. It’s related to the better performing LN10000. Idle is <10 Watts and Load is ~16 Watts. The LN was just a 1-2 Watts more. It supports 4 PATA, 2 SATA, 2 PCI cards (with riser), built in NIC, 4 USB, and six more USB internally. Of course, video and audio are built in.

The board and power supply should come in this week. Or not. Thanksgiving is Thursday.