Sometimes it’s best to leave old software systems alone.
Last night at the pub, a friend and colleague of mine was telling me of a recent experience he had at a company he was doing some IT work for. I think the lesson learned is a very important one, and thus I wish to share it. But first I’ll describe the situation he encountered.
In the mid-1990s, the company in question built their IT operations around systems from Sun. They wrote much of their in-house code using C++, and used Oracle for their database needs. On the front-end, they used PCs running a mix of Windows NT 3.51, FreeBSD 2.x, and even OS/2, depending on the department. While that is not a unique setup by any means, what is somewhat unique is that they essentially continued to use those same systems up until 2006.
One of the main reasons why they didn’t switch is because their software systems worked just fine, even if the UIs were somewhat archaic. Their software was mature and well-understood by the company’s employees. They even got extremely lucky in the first place, as the developers who initially designed and implemented their software systems did so in a way that allowed for the systems to easily scale as the need arose over time.
The hardware proved to be the main instigator of change. After a decade, many of the front-end PCs they were using started to exhibit a variety of physical problems. Some had been replaced earlier, but eventually it was decided to replace them all with newer systems. However, to the best of my friend’s knowledge, the back-end Sun systems were working just fine.
However, at the same time they decided to also replace the back-end systems. A variety of consultants were apparently called in to appraise the situation. For whatever reason, it was eventually decided that the new back-end systems would be built around Windows Server 2003 and SQL Server 2005. The new back-end software was to be built upon .NET, while Web-based client-side apps would be developed and used. My friend wasn’t sure exactly when this effort started, but he believed it was in early 2006.
By the end of 2006, the consultants and developers deemed the new system ready to go. Over the course of the December 2006 holidays, the new systems were rolled out. It turned out to be a pretty major disaster. The first problem they ran into was a complete lack of performance. As they moved into the first weeks of 2007, their back-end systems just wouldn’t scale. As an emergency fix, they ended up throwing more hardware at the problem, which did ease the burden on the existing servers somewhat. But it was in no means a permanent solution.
The front-end software systems proved to be an even bigger disaster. Many of the AJAX-based applications used Internet Explorer-specific functionality. But the IT managers of some of the front-end networks would not allow IE to be used, for security reasons. They only allowed for Firefox to be used. So the Web-based front-end software needed significant modifications right away, as well.
What was perhaps the worst failure involved the in-house users and their productivity. Large portions of the old system were built around a curses-based UI. Although it apparently wasn’t very pretty, it did allow for a great deal of user productivity. One of the main complaints about the new Web-based software was that the keyboard support was quite poor, requiring the user to select input fields using the mouse, and at times even having to scroll the page to input or manipulate certain data. With the earlier system, the nagivation could rapidly be performed using just the keyboard. Some of the more experienced users were apparently so efficient with the older system that their productivity was reduced to 25% of what it was before the switch.
My friend and his colleagues were called in to try to remedy the situation as best they could. The company was not willing to invest in a completely new system, but instead insisted that the old system be brought to a usable, if not optimal, state. The work is still on-going, as we speak, five months later.
There are many lessons to be learned here. The one my friend emphasized the most was that it’s often a good idea to leave existing systems as they are. What they had worked, for the most part. The problems they experienced with their front-end hardware could have been easily dealt with by buying new PC systems. But the decision to replace their working server hardware, and to rewrite their existing (and well-functioning) back-end software, were obviously terrible ones. The use of AJAX and Web-based software for their front-end systems was also a poor idea.
I think some of the other major lessons are as follows:
- Use mature, well-tested, effective software (eg. Solaris, Oracle, FreeBSD).
- Avoid immature fad “technologies” like AJAX.
- Traditional applications offer more flexibility than Web-based applications.
- Always give much consideration to back-end scalability.
- Sometimes a text-based interface is far more efficient than a GUI.
- Get user feedback on software early and often.
- Maintain a reasonable level of heterogeneity, when it comes to software, hardware and vendors.
Hopefully we can all learn from these lessons made obvious by this situation. Although given my years of experience, somehow I think that won’t be the case.
July 28th, 2007 at 10:59 pm
A few addditional lessons learned might be:
– Collect client requirements in advance, including constraints like “no IE on the desktop”.
– Don’t change to many things at once. Incremental change is better.
– Curses works. period. stop.
– Beware outside consultants.
July 28th, 2007 at 11:03 pm
Some good lessons, but some you didn’t parse that well:
“Use mature, well-tested, effective software (eg. Solaris, Oracle, FreeBSD)”
There are plenty of very large scale systems running on Windows and .Net. Don’t blame the implementation on the platform.
“Avoid immature fad “technologies” like AJAX.”
Well, kinda. Google and Microsoft seem to be doing okay with AJAX, and they probably have a few more users than the object system. Again - don’t blame the platform for the implementation.
“Traditional applications offer more flexibility than Web-based applications.”
No argument here at all.
“Always give much consideration to back-end scalability.”
Also agreed.
“Sometimes a text-based interface is far more efficient than a GUI.”
Yes. And sometimes it isn’t. Here’s that “platform vs. implementation” thing again - the lesson here is that GUI’s should always have keyboard shortcuts and primary mission critical systems should go through a LOT of iterative testing with real users. (Not simplified use-case testing; real world “day in the life of” style testing)
“Get user feedback on software early and often.”
Exactly.
“Maintain a reasonable level of heterogeneity, when it comes to software, hardware and vendors.”
Nah. There are other reasons to do that, but that doesn’t seem to be the issue from what we’ve seen here.
July 28th, 2007 at 11:58 pm
I think the core problem with Windows/.net on the back end is that you get Window/.net developers and consultants, who, by and large, are idiots. The Unix guys tend to be smarter, and so, platform issues aside, tend to implement better systems. Windows isn’t quite as good a server OS as Unix, but it’s mostly a people issue.
With regards to AJAX vs. traditional UIs: AJAX is nice because anyone on the Internet can use it. AJAX is a pain in the ass to develop with, and is much more restrictive in what you can do. Goole and Microsoft use AJAX where they have no choice. A normal application is almost universally faster to develop, and, if commonly used, gives better user productivity.
July 29th, 2007 at 12:07 am
AJAX is not an immature technology: calling it such is a sign of an immature programmer. I’ve been programming with it in a cross-platform, cross-browser environment for 8 years. The fact that it only worked with IE and not with Firefox isn’t the fault of the technology, it’s the fault of the idiots who built their app to work on only one browser.
Further, the lack of keyboard support is also not the fault of the web environment - keyboard support can be built into a web based application using AJAX, but whoever built that one didn’t bother to think about it or build it in, which they damned well could have if they’d wanted to.
The lesson to be learned from the story is not “web applications and AJAX are lame”, but rather, “when it works, it’s a bit dumb to fix it, and when you replace something that works, make sure the replacement works too before you roll it out.”
July 29th, 2007 at 12:27 am
I don’t know where AJAX not being able to use keyboard shortcuts came from, all the GOOG AJAX does keyboard just fine (and quite well).
I do however disagree on requiring IE: Just because it comes with the world’s most installed OS doesn’t mean it’s what everybody’s going to use. If it works on standards, it’ll work on IE, Firefox, Opera, and Safari (iPhone), and probably the blackberry too.
But yeah, overall, I don’t see why they had to upgrade. People get stuck in their ways and while this can be a problems at time, if they’re stuck in very EFFICIENT ways then there’s no problem at all.
July 29th, 2007 at 12:49 am
Ajax is not immature, but I wouldn’t say it has completely matured yet either.
There are times when Ajax is the right key for the job.
However, on this occation, there was nothing wrong with the key. It was the key-makers that were intellectually-challenged, and couldn’t see that the door didn’t even require a key.
Many developers (especially MS drones), believe the .NET and Silverlight are the “Silver-bullets” of software development. They believe they can accomplish ANYTHING with it. The fact is that you still need brains to get any task completed. Visual Studio only gets a programmer to stage 1, and then he/she needs to complete everything else.
I do agree that smart *NIX people are easier to come by… than smart Windows people.
I do believe this is mainly due to a *NIX person being regularly required to completely understand their actions taken to get something working.
A person using windows would click-click-click, and then windows would do the rest.
But, back to the point.
This is not about whether *NIX people are smarter.
This is not about whether Ajax was the wrong tool for the job.
All in all, the developers screwed up and made a mess.
Actually, this story reminds me of when my country’s Traffic Department went and developed a new system for the registration of vehicle licenses (and other stuff).
It was almost a complete mess as well! The system barely worked for the first few months, since the developers didn’t “expect so many connections into the main servers”.
July 29th, 2007 at 12:59 am
[…] I shared the experience a friend had with one of the companies he works with. It involved a failed transition from what was […]
July 29th, 2007 at 1:20 am
Very cool post. I’d like to submit “5 Tips For Redesign” in response.
1. Don’t fight the client’s minimum browser requirements. The more people that can use the application, the better.
2. Don’t bash or cheer any technology unless you’ve got a business case to do so.
3. Forget Flash, AJAX, Java, etc. unless there’s a good business case to use it - like measurably enhanced process clarity for the end user, resulting in faster task execution.
4. Set VERY CLEARLY defined and measurable business goals. An example might be to “allow the average user to complete x task in y amount of time.” Define, Measure, Analyze.
5. You’re not “requirements-gathering”, you’re “requirements-excavating”. Get your client to talk. They’re probably not developers, so they don’t know what to ask. You’re not in the client’s business, so you don’t know what to ask either. Converse, explore, share… be there to help, and do your best to unearth requirements that the client doesn’t even know they have.
July 29th, 2007 at 1:56 am
ahem.
I worked for a (very) large company that had that exact problem on the “front end”. I’d bet it was the same system.(OTP)
As an aspiring AJAX developer, it was obvious the problems were caused by bad quality checks in the software design, or total incompetence by whom ever decided to buy a broken system.
July 29th, 2007 at 1:58 am
[…] Old Software= Good! Well Sometimes. Posted July 29, 2007 I’m reading this guy’s article about how sometimes it’s best to leave old software systems alone. […]
July 29th, 2007 at 7:08 am
Interesting story! I must say that the conclusions drawn seem a bit arbitrary and it’s not really clear that there is evidence that “Traditional applications offer more flexibility than Web-based applications” or “Maintain a reasonable level of heterogeneity”. Maybe more like “hire developers who know how to write *real* web apps” and “Don’t use Microsoft”
July 29th, 2007 at 8:02 am
There are smart developers and there are so-so developers. There’s no evidence that showed Windows developers are stupid and *NIX developers are smarter.
You know that most universities teach students *NIX for as long as 4 years right?
At the end of the day, a developer is a developer whether he/she developed in Windows or *NIX.
July 29th, 2007 at 3:54 pm
I wouldn’t blame the consultants, they were just doing their job. I would blame the company, or more specifically the employee at the company who hired the consultants, for replacing the backend that worked just fine. There are a ton of legacy systems out there still being used in Windows env with terminal sessions, and asking if things really needed to be upgraded should have been the very first question. Sure upgrade the PC’s, but keep the back end as is unless there was some compelling reason to change it.
July 29th, 2007 at 5:26 pm
Seems like another case of sales people *cough* Microsoft *cough* influencing the process in cahoots with the mgmt.
We recently went through a similar nightmare. Needless to say, as a Mac / Unix shop, the CTO didn’t last long after rolling out a disaster in the form of Windows Server / ASP.
*Shudders*
July 30th, 2007 at 10:53 am
Pointing out the obvious; why was such a major change done as a big bang event?
Yes, there’s always resistance to any change. However, the lack of keyboard navigation suggest that the end users weren’t given the chance to try prototypes and give feedback, and the browser issue suggests that no-one checked that the changes being made would function in the end-user system.
Also, it’s rare to require an immediate move of *everyone* from one platform to another; it should be possible to migrate data, let the customer verify that the new system works, then migrate data again (to pick up the remaining work done while the customer verified the new system), and finally migrate the customer from old system to new. Again, why wasn’t this done?
July 30th, 2007 at 3:44 pm
I have been working in the IT industry for nearly 20 years.
I have seen a lot of good and bad projects.
This is a perfect example of Change for the sake of change.
There are far too many I.T. folks out there that have more balls then brains. I can see the concern about very old back end equipment. The concern should be what will happen of this old equipment fails and cannot be fix or replaced. The fact of the matter, it was working and therefore there should have been no reason to rush a change like this. In my experience the greed of the consulting company that makes such costly recommendations can never be underestimated. Think about who has the most to gain and who has the most to loose. If this was done entirely inhouse then the motivation for the recommendation needs to be strongly considered. The most responsible person here was the manager in charge of IT. He did not perform due diligence when considering this change. There are an huge number of “Paper only” MCSE’s in the world. They think they have all the answers. There is more then one way to skin a cat as they say and that is where experience and prudence comes into play.
Its an expensive lesson for everyone. I hope the Manager that gave the recommendation has learned from this and he is now a better man from the experience.
September 24th, 2007 at 8:15 am
Back in 1998 I was working for a large German company. In their IT centre they had the motto: “never touch a working system”. And they knew this for good reason: they had a mainframe that was doing its work since 1970. I guess it works still.
September 24th, 2007 at 9:47 am
the project manager(s) should be blamed first, it sounds like a bunch of IT cowboys getting all hard over the technology and forgetting about the real point of the software function - to help people do their jobs.