David Auerbach at Slate clears up the clutter of reporting on the Healthcare.gov website mess.
Last weekend, some anonymous “specialist” told the New York Times that “5 million lines of software code may need to be rewritten” in order to fix the mess that is healthcare.gov. (The good news, according to the source, is that the project has a total of “500 million lines of software code,” so only 1 percent has to be rewritten. So the code’s 99 percent good—or something.)
I don’t mean to jump back on my hobbyhorse of complaining about lack of knowledge in tech journalism, but printing a claim like that is egregious.
Why? Well, here’s a line of C++ code:
The close curly brace signals the end of a block of code. It could be put on the same line as the previous, more substantive line, but for the sake of cleanliness, programmers tend to put it on a line of its own. When it comes to coding in HTML, Perl, and AJAX, different programmers have different styles. Some will split code up into many lines; others will compress it into a handful of lines. I’ve seen nearly identical segments of code written in 10 lines or in 50.
Here’s another line of C++ code.
// TODO: make sure this code doesn’t crash!
That’s a comment. It doesn’t do anything—those two slashes at the beginning tell the compiler (which converts code into actual computer instruction) to ignore the line. It’s there to explain things to people reading the source code, or in this case to remind the programmer to fix whatever lies immediately below. I’ve written cryptic bits of code that required more lines of comments than lines of actual code, simply to explain what on earth was going on.
So not all lines of code are created equal. As a programmer, I had weeks where I produced 1,000 lines of code. I had weeks where I produced 20. Usually the latter weeks were more grueling, because any 20 lines requiring that much time and effort are going to be a) important, b) complicated, and c) bug-prone. The 1,000 lines were far more likely to be simple stuff that I could code by rote. I even had weeks where I removed 2,000 lines of code by removing redundancies between similar blocks of code. Those were the best weeks of all, because less code means fewer bugs.
Programmers who do user interface code—which is responsible for the visuals and input components of software—tend to produce far more code than other programmers, because user interface code requires a lot of boilerplate. I knew programmers who wrote 10,000 (good) lines of user interface code in a week. Many of them were copied and slightly modified from other projects or example code.
Consequently, it’s rather silly to say, as the Times article does, that “a large bank’s computer system is typically” 100 million lines of code. Investment banks have far more complex code than commercial banks—they need more in order to do all their clever, sneaky trading. Assuming the Times is referring to commercial banks, there is such variety among implementations and coding standards that speaking of an “average” amount of code is meaningless. Bank code written in FORTRAN will be far longer than bank code written in Python. Does it make a difference? Not really.
But while the numbers in the Times article don’t tell us much about the healthcare.gov codebase itself, they do tell us something about the “specialist” sources that inform the article. The sources are not programmers, because programmers would not speak in terms of lines of code with no further context. We hear that “disarray has distinguished the project” in part because government “officials modified hardware and software requirements for the exchange seven times.” The officials probably modified them 70 times—requirements for any software project are constantly in flux, and it’s expected that project managers and software engineers will adapt. Modifications alone do not signal a project in disarray.
On top of the fact that writing code for anything is complicated and error-prone, there’s the simple fact that working for the federal government’s IT system is complicated, as this fellow at World of Pie explains.
I’m writing this post as a rant. I am tired of hearing people who have never worked in Federal IT try and come forward with ideas about what was wrong about the way Healthcare.gov was developed. I have one statement for all of you who think you could have done better.
You would have ALL failed miserably.
Federal IT is broken. Hell, all of Federal contracting is broken from what I’ve seen, but I want to focus on the IT side for now.
Before I get started, a quick reminder of my background. My first Federal project was back in the late 90s as the tech lead for the Secretary of the Air Force’s correspondence tracking system. Over the years, I have worked on a multitude of projects and managed many more while I was the Director of Technology Solutions for Washington Consulting . I’ve responded to many proposals and run Federal IT projects through the wide variety of hurdles that they face.
I can tell you right now, I am impressed that Healthcare.gov even boots up.
Let’s forget the obvious facts surrounding a system that is interfacing with over 36 states, a multitude of insurance carriers, and several federal agencies that think they have the best IT shop in the land. I want to talk about the over-arching process.
In Federal IT, typically nobody with both knowledge and authority owns all the components of a system. Even on smaller efforts, one contractor owns the data center, another runs the database infrastructure, one is developing the actual system, and a fourth contractor is in charge of making sure all the rules are followed.
Also take into consideration that we’re dealing with a website that deals with a very complicated industry — insurance — in the first place. Getting a quote for car insurance online isn’t as easy as they would have you believe. Try it and see.
Finally, in the effort to make a system that is open and accountable to the people and Congress, the federal government has set up so many checks and balances that it breeds red tape like minks in heat.
Try writing the code for that.
PS: It’s nostalgic to see FORTRAN mentioned. That was the computer language I learned when I was a freshman in high school in 1967.
HT to CLW.