The government is not measuring whether its IT strategy is working, in spite of pouring millions of pounds of taxpayer money into it.

That is according to the National Audit Office, which said the Cabinet Office had "not yet developed a system for measuring the extent to which the strategy is resulting in sustained change". Whitehall needed much clearer timescales and baseline measurements.

Additionally, the government has only been "informally" managing resources to deliver the strategy, the NAO said. "Without a clear resource plan, gaps may start to hinder progress," it said.

Additionally, a shortage of IT skills in the public sector remained a "serious challenge", and was hurting the management of technology. Some 140 people, double the existing number, would need to work on the strategy.

The Cabinet Office must do "more work" to fully engage departments with the strategy, the NAO stated, urging it to move ahead with a "simplified and shared ICT infrastructure".

Nevertheless, in its six month review of progress, the NAO commended "early progress" in the strategy, as the government strives to end lengthy and complex projects – in which it has a long history of failure.

By bringing together a broader set of IT management skills, establishing a panel of departmental chief information officers, watching departments' project decisions and improving programme management, the current government had done much better than any of its predecessors, the NAO said.

It has taken "a pragmatic and collaborative approach" so far, and has "largely met the first round of deadlines for taking action", the NAO said, with key delivery areas identified.

Amyas Morse, head of the NAO, said the changes needed to be successful because IT "is going to play an increasingly important role in changing how government works and how services are provided".

In spite of the early success, he said, "new ways of working are as dependent on developing the skills of people in the public sector as they are on changes to technology and processes; the big challenge is to ensure that the strategy delivers value in each of these areas."