CIOs from two major European banks believe adopting Agile development and running DevOps teams have been revelatory ways of working they would not go back from, but warned that it created a new cultural problem of management and measurement for large enterprises.

CIO at ING Benelux, Johan Kestens, and BNP Paribas Fortis CIO Paul Thysens, both added that measuring software - the quality of the packages they are paying for, their overall effect on productivity and innovation, and the blurred metrics of licences and software audits - were emerging worries for CIOs but that software quality measurement was a way of helping move forward in vendor relationships by creating a clarity and transparency which could improve collaboration between buyers and suppliers.

Speaking at the CAST European CIO Forum, BNP Paribas Fortis CIO Thysens said that during the organisation's first experiment with Agile, the methodology was implemented with no boundaries or parameters - and needless to say it failed horribly.

"We then started doing Agile in a very structured way with clear definitions and short sprints effectively with a lot of measurement," he said. "I could see the way they were working they did not want to go back and work differently. They were very creative in DevOps and created new ways of testing - it all came from the team and they were measuring themselves.

"Agile creates a very positive mindset and they are now challenging themselves. It is part of a big programme within the bank which is moving towards digital, and not just the IT department."

Getting closer to the customer

Thysens said that Agile and DevOps shifted the developers closer to the customer by working in more diverse teams. "There was cross-fertilisation from those autonomous teams," he said.

"Teams were able to find again the pride of developing things of quality. They get the notion of who the customer is. With Waterfall they deliver, but now they are delivering a product or service to the customer and that customer is no longer just one place before you in a long line of delivery."

Kestens, CIO at ING Belgium, said that previously software development had much in common with the construction industry and a car production line. If you were going to make a million cars, Kestens explained, by the time 5,000 had been created you had made most of the mistakes and optimised the process, but by the time the 500,000th had rolled off the production line it was usually time to start building a better car.

With his goal and remit of enabling 2,500 people to make a worthwhile contribution to the financial objectives of the bank, his major challenge was of planning and then getting the work done.

"We chose Agile as a way to plan, change and deploy," Kestens said. "Modern software tools allow us to abandon the Waterfall way of doing things, like assembling cars on a line, and going back to a team approach more like the way a surgical team operates. A whole team that works on the patient; everyone is highly skilled, knows what to do, and it's a satisfying way to work.

"Agile is a way which allows great engineers to be great.

"Using the SAFe framework [Scaled Agile Framework] links the work with the DevOps team to the business strategy. We are empowering the team for rapid delivery but we still need to learn and master this at ING. It's like learning a new craft; the first few chairs I create might not be very good but eventually I hope to become a master."

Agile measurement and software measurement

The CAST European CIO Forum in Brussels was hosted by the former CIO of Dutch bank ABN Amro Geert Ensing, who said that he had become very worried about the cost and quality of IT in his role in financial services, querying the current processes of contracting with vendors, how CIOs can measure innovation and the drivers of innovation, and how they can certify software in a meaningful way to improve their relationship between vendors and CIOs - and ultimately give some power back to the customer.

The views struck a chord with Ensing's financial sector CIO peers, with BNP Paribas Fortis CIO Thysens saying two of his biggest challenges were evolving the middle layer of his team to deal with DevOps, as well as measuring a new method of productivity and tools.

"With Agile you have to fundamentally change how you deal with middle management," he said, with the role moving from one of controllers to becoming coaches and trainers.

"The project manager becomes a servant leader. There is a clear definition of what is expected of them in terms of results. But the PM has to support the team and help them grow. We are asking our middle management things they were not used to doing, but with more than 1,000 people I cannot fire them and hire 1,200 new people - so I need to upgrade the people I already have."

Thysens also said that software measurement and transparency from vendors about what they are supplying was critical to maintaining a strong relationship with CIOs.

"I hate the word outsourcing because I like the word sourcing," he said. "We talk about what are we going to develop and maintain, with whom and in what way. Every partner in that chain is a partner so transparency is a must.

"I hope that vendors don't see that as unwanted; it's the best way to get to something which works. Software quality measurement is helping us move forwards in that collaboration because we create clarity and transparency. We can agree on what criteria something is delivered to, and get rid of some of the subjectivity in the creation of these tools."

Transparency in vendor relations

Kestens agreed, saying that measuring the tools used at ING could help the bank increase productivity and add value in the long run, but warned that measurement in itself was just a guide to help an organisation achieve its business goals.

He said: "Every way of measuring adds value as long as you understand the limitations. Measurement gives you insight, like a compass which tells you not where north is but where Magnetic North is. It doesn't matter though because it is accurate enough for people to sail everywhere.

"Measurement of software is a problem that will never be solved. But with the right tools and appreciating the limitations you can understand true value, and we can understand what is real quality and what is real productivity and inspire people to relative progress."

Methods of measurement were two-pronged for Kestens, with inspiring his own team also supported by the relationship-building it can foster which key software suppliers and for IT with its own internal business partners.

"If something is late, the business blames IT - but the IT department says the supplier was delayed. They are probably right but exaggerating a bit, but it leads to a lack of trust. By giving them these tools, it build confidence because they can be transparent and show that they are correct."