You are responsible for the IT organization and must assure that the agile delivery model is better than the former waterfall-style.
You must assess the performance of your organization and lead it to highest operational excellence.
The IT organization in your enterprise has transformed from hierarchical structures and waterfall projects to a flat hierarchy of self-empowered teams that deliver with agile methodology.
Or the IT organization is in the middle of this transformation process.
After the switch from “culture of control” to “culture of trust” you can only hope “helplessly” that the self-organized swarm intelligence is aligned with your organization-wide goals.
Project management expertise and KPIs from waterfall-times primarily rely on plan versus as-is comparison. Hence, they are useless in agile contexts because of the conceptional lack of upfront planning. So how can you assess progress and team productivity?
Agile KPIs were invented to help individual teams to improve; but a comparison of KPIs across teams doesn’t make sense by design (e.g. Story Points). How can the entire organization reach highest excellence if there is no objective way to see if there is a laggard team that needs help?
An internal team benchmark would enable you to identify best performers and to enforce their best practices across the entire organization.
KPIs that work for both waterfall and agile settings would enable you to assess and quantify your transformation success.
KPIs as fact-based proof of superiority of agile over waterfall would help you to convince your customers (the business side) to more quickly switch to the agile model – even if they are reluctant due to missing upfront planning and not knowing in advance what to get and how much it costs.