Software engineering as a discipline is still in a lower stage of evolution than the traditional, older engineering sciences. Compared to software development, automotive engineering, for example, has been working at the highest process level since at least the 1980s with the 'lean manufacturing' approach introduced by Toyota. Why? In addition to the actual tools and conveyor belts and everything that is needed to screw and weld a car together from the individual components, you should also use tools and platforms that allow you to observe how the overall process works from start to finish. This also makes it easy to optimize this overall process – a must-have in automotive manufacturing. Without this, automotive manufacturers would have no chance in the market. Unfortunately, we have not yet reached this point in the software development world. Or have we?
The next type of tools and instruments are the so-called build automation systems. Today, these are referred to as CI/CD systems (Continuous Integration/Continuous Delivery systems). These take source code and automatically convert it into executable components. These are the robots that go into the warehouse racks, take out the components and then put them on assembly lines, and the robotic arms that assemble the moving car from them. It is common for CI/CD systems to also run tests of the programmed code, written by the developers, which are used to verify that the source code actually does what it is supposed to do without errors. If the tests fail, digital alarm bells are rung and the developers must make improvements. So, in the spirit of the robot analogy, the assembly line would stop if the production line determines that the passenger door opens on its own when the right turn signal is activated – 'Oops!' – In the software world, the most prominent tool at this point is 'Jenkins'.
Then came the era of static code analysis tools and their integration with CI/CD systems. Such systems act as automated inspection devices. Source code, when converted into executable system components, is examined for known security vulnerabilities, potential bugs, consistent programming style, maintainability, performance, etc. Analogous to the automotive factory, all components on the assembly lines are inspected and defective parts identified. A well-known example of this type of tool is 'Sonarqube'.
The last category that has been added is the so-called issue tracking systems, where the stacks of work orders are located. There, there are tools that allow for improved coordination of work orders. Using the manufacturing analogy, software developers accept a work order, go into the warehouse, and get the components. Afterwards, they put the modified components back on the shelf. In reality, they get the code from Git. They modify the code according the work order. Then they check the amended code back into Git. One tool for this issue tracking that has now established itself as the top dog is 'Jira'.
Everyone involved in the software development process is well equipped with tools that he needs and must hold in his hands in order to do his own work efficiently. The names that have been mentioned here are purely exemplary. There are several hundred tools in this ecosystem – hammers, screwdrivers, special pliers and so on.
But what has not yet emerged in the software development area is a way to see the process as a whole–from start to finish. When you have this transparency, you can start performing optimization tasks that will improve the development organization more and more. However, until now there was no term for this category. As the person in charge of a large software development organization, if I wanted to keep track of everything, I wouldn't even know what search terms to type into Google to find help.
But there is a glimmer of hope. From the university environment, methodological approaches, and platforms, are emerging that can be applied on a large scale to real major industrial software development organizations. These platforms analyze what data traces the above tools leave behind. For example, when you run a CI/CD system, data traces are created. Down in the metaphorical basement of code there is then a lot of data gold. These new platforms leverage this data gold to then reconstruct the real-life development process from the data traces, with all its flourishes and inefficiencies.
This gives complete transparency about how well the software factory is actually performing–what is really happening on the factory floor. This is not just something that a person puts in a report and says, 'Yes, the situation is okay,' but the data traces are also fundamental. These data traces are used to derive KPIs (Key Performance Indicators), with which one has indicators at the strategic-operational level that go beyond mere code observation. All dimensions of the software development can be supported with objective numbers, thus a person responsible for software development can grasp and observe the situation quantified. Finally, a kind of Digital Boardroom is created where all these important KPIs are stored. The person in charge can see every day in real time where there are problems on their factory floor and where he/she should perhaps intervene in the development organization – analogous to the plant manager.
However, this top-down approach is no longer the culture in most software development, but is supported by collaborative work. The team as a whole works and decides, especially in the agile world. But transparency is needed nonetheless so that everyone involved in the software factory can see objective figures on the digital boardroom: "Here we are. If we rush through the release projects with such haste, we'll be piling up technical debt all along the way. In other words, you should plan for that in the next release cycle - a technical debt reduction like that." That's the kind of decision you can consciously make when you can see how the KPIs are changing.
This is the only way to get a good grip on software development. A term is now also crystallizing for this type of platform: Digital Engineering. They combine techniques of software analytics and
software process mining. As the highest stage of software evolution so far, digital engineering platforms can help the software factory achieve outright success. Soon now, the conviction will have taken hold that it is not enough to equip developers or stakeholders with hammers, screwdrivers, conveyor belts, or robotic arms. This alone is not enough for excellent software development; one should act as in mechanical engineering, namely with the help of data-based process optimization. As such, digital engineering will become a must-have for an organization that does software development.
Authors: Dr. Johannes Bohnet, Dr. Uta Morgenstern
Translation: Brandon Lewis
This article originally appeared in German. You can read it here.
