Complexity makes for exponential maintenance costs?!

The title of this article isn’t a fact, just a thought. However the thought comes from the experience of putting a very complex application into production and watching what happens. So long as there is stuff to do on the application, e.g. further releases adding more functionality, the maintenance costs are minimal, because you can plan the people who do the maintenance to do spend say 20% of their week on maintenance tasks while spending the rest of their time on building the new functionality.

But what happens when there is no more new functionality to build? What happens if suddenly there is no budget for new functionality? Well, you normally scale down the team, reducing head count to the minimum required to keep the knowledge about the application. You will probably put them part-time on other projects.

But now it gets hard. If the application is very complex or uses many differing technologies, it might not be possible to efficiently (in terms of reducing knowledge loss) reduce the size of the team below a specific size. If you application uses say a BPEL Engine, an Oracle Database, a Java Application Server, has a web front end and connects to several external business partners requiring knowledge of complex business rules, you probably already have the requirement to keep more than five people for maintenance. Mostly, this doesn’t cause a problem, because those people can happily staff other projects. But if the application is so complicated that they are certain to forget parts of the system if they start working elsewhere, then you face a problem – knowledge loss is guaranteed if these people are not kept on the project for 100% of their time. Saying that, it’s not even that simple, because the people might start forgetting parts of the system if they are not actively using / updateing / maintaining them. Just because you are on a project doesn’t mean you are not forgetting how it works.

What can you do if you have such a complex project? First of all, from the start of development, ensure it is well documented, has good unit and regression tests. These will help reduce maintenance costs because it helps remind developers of the pitfalls in the system and checks that any changes they make don’t introduce further bugs. But we know that we should do this on all projects anyway right? What else can you do? Well, the example I am thinking of used to have 50 people working on it. That was cut down to around 10, of which 5 were developers. If none of the developers are happy to work on anything else for more than 50% of the time because they would lose knowledge of the system, if you include server costs, an application manager and a few other comparatively small costs, you soon come to a budget of a million dollars a year just to keep your application in production. That cost is not related to number of bugs or number of technology updates (e.g. upgrading the application server in line with company policy to keep up with supported versions). That cost is just to keep the application running and have a safety net, in case a bug crops up (although it does of course include the cost to fix such bugs). Oh, and that price assumes there are other projects for your staff to go to – which might not be the case, if for example there is no budget, or if other projects are only interested in resources who are at least 75% available… If these problems apply, you suddenly need to double your maintenance budget because you have to keep these people on board 100% of their time!

Other options… Well if you compare this phase of a project to an earlier one where you have your staff doing 20% maintenance and 80% development. In this early project phase, maintenance time is optimum because the people are still working with the system and have all its nuances at the forefront of their minds. In the later phase (call it the maintenance phase where no new functionality is delivered), maintenance at 20% might actually cost the equivalent of 30% in the early phase, since the people are idling (working elsewhere) for 50% of their time and forgetting how the system works. So of their 50% availability, they only have 20% available (one day a week) in which to potentially build new functionality, should there be such requirements. Compare that to the earlier phase, a developer has 80% of their time to devote to new functionality (4 days a week) – they are four times as productive. But they are there for 100% of the time, so its costing you double compared to the later phase. In total, that makes them TWICE as efficient in terms of what is built per dollar!

Another problem with having staff working for only 50% on maintenance tasks is that they will get bored and want to move on. Bad morale is never a good thing.

Another option is to just ramp down regardless of knowledge loss and save money up front. That saved money can be put in a pot and be used to pay for any ramp up time needed if and when bugs or requirements for new functionality come your way (although it is unlikely that it will remain in that pot, since other more active projects are likely to use it). As a techy I would shy away from this approach, because experience has shown that ramping up a project at a later time almost guarantees that the newly built stuff will be badly integrated making the entire thing less maintainable, ensuring that maintenance costs in the future will be increased. Still it’s your project, your choice. Since software decisions are often based on political decisions these days, you probably don’t care if the project fails during a later maintenance stage, because it was successful while you were there (after all, you saved loads of money on maintenance back then!), and the person who took over after you left, must have screwed up because it now costs more to maintain or upgrade than when you were there. That sounds like good cause for another promotion for you! Just start praying now, that there are no bugs or change requests between the start of the maintenance phase where you slash the head count and the time you are no longer responsible for the project…

Side Note: The last paragraph reminds me of a project from some years back, where a project manager totally disregarded software quality and maintainability in order to get the first phase of the project (in fact entire platform) into production as fast as possible. He turned the whole project into a high profile one and span it all in his favour so that he delivered a great new platform on time and under budget guaranteeing his promotion into a role well above the previous one. By the time anyone realised that the platform was very expensive to maintain he was well out of sight and the person who took over spent a large proportion of their time defending the platform and trying to improve it. But from that place far above where seagulls fly, it was the new guy who looked bad, not the one who delivered a high profile project successfully… (note the reference to why directors are like seagulls, because they go around with their heads in the clouds crapping on the people below them).

Anyhow… it appears from the discussion above that there are three options when it comes to deciding how to put a very complex project into a maintenance phase.

  1. Do it traditionally, which is inefficient in terms of developers time
  2. Cut all staff to a bear minimum, and pray you are long gone before the crap hits the fan
  3. Don’t do it at all! Keep developing, and justify it by showing that development is twice as efficient in doing so…

OK, well that last option does depend upon there actually being something to do. But most enterprise applications that I have worked on always went into the maintenance phase before they were completely finished. And of course more seriously, any software should always show a return on its investment – you can’t just keep developing to the point that you are streaming live TV through your application because there was nothing else left to do! But I guess the important point is this: delay your maintenance phase until the ABSOLUTE last possible moment. If you intend on putting a project into a maintenance phase and then developing some other functionality in a year or twos time, it will be far more successful if you replan, keep your team running at 100% and bring the budgets for the new functionality forwards.