Chief Technology Officer Sharon Cooper is making the BMJ – previously known as the British Medical Journal – entirely open source and has moved the 175-year-old organisation to a DevOps environment to reduce costs and provide greater agility in the digital age.

While the BMJ is best known for its publishing services, the institution has expanded in recent years to offer 20 new digital products such as its Med School Selector online tool. Since joining in 2013, CTO Cooper has led a transformation of IT systems and development processes, enabling BMJ to start embedding analytics in its products to support delivery of new functionality.

But embracing open source, Agile development and DevOps has seen the BMJ drastically improve its release cycle while improving performance and lowering cost under Cooper – who was recognised recently in the CIO 100.

'Everything we do is open source'

In order to build its wide range of products - more than 80 in total - the organisation relies heavily on bespoke development and open source technologies. This offers BMJ greater agility, as well as the cost benefits, said Cooper.

"Open source has always been really important to us and probably why we have a reasonably big in-house team to manage all of that open source technology and to tie it all together. And I think it will become increasingly important," she said.

"Also we can take something out and putting something else in quite quickly, so it is not just open source but also open standards. I talk to the business a lot about what we need to build our Lego bricks. At the moment we have Duplo - these massive great big chunks and we need to get down to those tiny little microservices. You can build much more refined structures if you are working with much smaller components.

BMJ currently uses a number of open source software including Java, Drupal, Wordpress, and is starting to bring a lot more open source technologies into its backend systems.

"Everything we do is open source, and if it isn't open source now, it will be very soon," she says, adding that the firm now plans to migrate all remaining licensed software to open source platforms, including a 'costly' Oracle database.

"We are removing Oracle and replacing it with Postgres - in terms of the products we create that is probably our only big piece licensed piece of software we have," she said.

"A lot of it is for performance and it is also cost. It is a huge amount and we don't really use much of it. What we do with databases is relatively straightforward. We don't need a lot of the complexity, and we have tools which use very little functionality.

She added: "It is also about speed of access and the ability to do things faster."

Agile and DevOps

Another key part of BMJ's modernisation efforts has been an overhaul of its development practices.

"Our market is being disrupted enormously, but in the world of journals it takes a very long time to change," Cooper said. "I needed a team that could work much faster and more Agile. So we embarked on an agile transformation process."

Cooper said that when she arrived at the firm two years ago a release would only be done every four or five months, and would take a week to complete. "There were usually rollbacks, there were huge bugs and masses of technical debt," she added.

She soon began to adopt Agile development practices across the IT team. The process took around 18 months to complete. "The first thing we did was removed all the partitions between the desks, we whiteboard the walls, created a really nice working environment and allowed the team to make a lot of the decisions.

"We said the outcome is that you need to be able to produce engineering-ready code at the end of every fortnight, and the business decides if it wants to deploy it, but we are ready to release."

The results were impressive, Cooper said, and the company began to move to a DevOps setup, removing the workload on its ops staff to focus on other projects.

"In 2012 when I first joined I did one release a month, a year later we were doing 12 releases, a year after that we were doing 35 releases," she said. "This month our ops team isn’t doing releases any more, the developers are doing it themselves, so we have moved to a DevOps environment very quickly.

She added: "The operations team isn’t just doing a load of stuff on other applications to really change the way all of this back end infrastructure is architected. That is the next big challenge - having sorted out the working practices and having created a much smoother flow of work through, we actually need to dig into some of our legacy hardware and try and make changes there."

Embedding analytical thinking

Embedding analytical in the organisation has also been a key element in Cooper and the BMJ improving its products and customer engagement.

"We have started some of these journeys and we can absolutely tie increases in revenue and sales to making changes in functionality," said Cooper. "We would have never been able to do that two years ago."

For example, BMJ is now using analytics to drive decisions around commissioning of content, well as A/B product development.

"A lot of our content has to be updated very frequently because medical practices change, drug doses change, but we don't have to change everything immediately, so analytics enables us to look at whether we need to do that," she said.

"Then, in the last six months, we have started to do a huge amount of A/B testing to look at whether we need to have certain product functionality, and, if nobody is using it, why don't we just remove it? We are also reducing registration processes, re-architecting authentication processes. So really looking at some of the more complex parts of our business."

Investing in analytics

The analytics plans have involved investing in a number of tools. Among others, this includes Google Analytics and AppDynamics, which is used to monitor user behaviour as well as a back office monitoring tool, Cooper said.

"We are using that a lot for performance and to be able to look at not just what an application is doing, but how we are using that to see what people are doing in the application," she said.

"We also use Tableaux for looking at a lot of our back office and customer data when it is not in the live products. But I think we will have a wholesale look at our BI in the next 12 to 18 months."

Going forward, Cooper said the company plans to use a wider variety of analytics, including Big Data tools such as Hadoop. However before embarking on a big data project, the main challenge is deciding what data the company needs to analyse.

"What we need to work out is how big our Big Data is," she said. "Right now we are not in such a mess, but what we have got is so fragmented and we are just trying to work out what it is we need to track, what is the important data, what do we need to measure, because we have a lot of very industry specific data models that come with being an academic publisher."

"So I would say yes we are looking at using Hadoop or similar tools, but we are just not there yet."

Skills challenge

With a complex range of products and technologies, Cooper said that one of the main challenges encountered by BMJ is finding staff with necessary skills. Specifically, a proficiency in a range of programming languages.

"What is hard to find is people who can speak more than one language, as we still have quite a mixed portfolio," she said. "We have predominantly Java and Javascript, but we also have - mainly because of acquisition - bizarrely we still have a fairly big chunk of work done in Delphi, which is not very easy to find expertise.

"We have C# and .NET, but we have also moved into Node, Angular, and some of the guys are experimenting in Ruby and Python, so some of the challenge is just finding people who can speak more than one language.

"The core principle is that every single person has to be able to work on every single product. Whereas a team of people worked one product, and one product only. Now everyone can pick up every product and fix things and develop it."