Why is software development in good quality, in time and in budget so difficult?
Actually it should be easy to develop high quality software. Software engineering technology, -methodology and -tools have matured tremendously within the last twenty years. There have been so many bad examples to learn from, and companies already invested – and still invest a not insignificant percentage – of their software engineering budget in software quality. So why do software engineering projects miss deadlines and exceed budgets?
Navigation tools for software engineering are needed
Long story short: Large scale software engineering is like finding a new passage to India in the fifteenth century. There are so many obstacles and influencing factors that – if you don’t have appropriate navigation tools – you can easily end-up discovering a new continent. Although this is certainly true, so far this blog post is not really of any help. So let’s dig a little deeper to unravel the mysteries of software engineering projects.
Large scale software engineering projects behave a bit like modernizing a house, in which the heating or electrical installations must be replaced. Only when the construction manager’s men have opened up the walls, he can tell reliably which pipes, wires or cables have to be reinstalled, how long the repair will take and how high the cost will be. In software development projects, management, project managers, quality managers or developers face similar difficulties.
Software engineering calls for detailed knowledge of the system
All you want to know is: What are the problems, how long does the development take and how much will it cost? And, like the renovation of a building, the managers work with many estimates and past experience. It is no coincidence here that many construction projects exceed the initially calculated budget.
Therefore, to ensure the success of the software engineering project or the operation of the application landscape, all stakeholders urgently need detailed knowledge of the system:
- The CIO has to convince the departments that they invest money for the modernization and sustainability of applications. This can be achieved by making the technical risks strikingly visible even for non-techies.
- The Development Manager must have an overview on the activities of his distributed development teams and the tools to verify that external suppliers deliver the agreed quality.
- Also the Quality Manager needs genuine knowledge about the state of the software, he needs to see the points at which the software is too complex and where the critical development hotspots are located in order to test the software accordingly.
- Developers would be happy to have proof, why it is necessary to invest in refactoring of certain modules or software objects.
Most times, in order to get an overview, project managers use methods and tools, which mainly include code analysis, like taking into account the number of code lines or the complexity of an algorithm.
Pure code analysis doesn’t indicate much
The great weakness of this approach is: The analysis of the source code only allows a selective view. For example, a quality manager can only take a look at the number of dependencies within a program part, not across the whole application. Although he gets an idea of the complexity within the module, whether this complexity is justified or developers have simply done a sloppy programming job, he cannot see.
Furthermore, a high level of complexity does not mean that action is needed. To decide whether actions are required or not, further information and data sources such as the configuration management system are necessary. Thus the pure code analysis itself doesn’t indicate much. It has to be put in larger perspective of the developed application that includes all available software engineering data sources.
Software maps in software engineering
Additionally, the information from existing analysis tools are simply not “user-friendly”, especially not “management friendly”. In many cases it is incredibly long Excel spreadsheets that need to be combed through with difficulty. To remedy these shortcomings and not end-up in a faraway land, companies such as Adidas, the German stock exchange or Daimler rely on seereneTM which brings together all the data from the source code, run-time analysis and the existing repositories to a meaningful data base – and then analyze it using data mining methods.
Already the combination of metrics for complexity and the location of the latest code changes, lead to valuable guidance. This shows in which program parts bugs are very likely to find and extensive testing is required. To uncover more complex interconnections, algorithms for pattern recognition can be applied. With these, developers can identify hidden dependencies, such as change in module A leads to a required modification of module B.
Results like this can be displayed in a software map. The form of a map allows a complete overview of the system. Blue, slim houses on the map represent unproblematic program parts. Red towers with floor area indicate a danger of collapse. A red building represents program parts of large, monolithic structures that bring a high degree of complexity and have been frequently changed.
A collection of buildings in one district represents a module. This allows to recognize risks and obstacles very early before they become high rise buildings. A project manager can see at a glance how much effort is necessary to repair the module, or if it is worthwhile to demolish and rebuild this area completely by refactoring it. Knowing all the hotspots, Quality managers can direct testers on the critical points and thus make sure that the project is not getting off course.