Although there are some large companies who have become known for a culture of innovation (such as 3M), it is generally recognised that small companies are usually better at innovation than large ones. In the case of enterprise software that holds true, with most of the innovation occurring in small angel- or VC-backed companies, with larger companies waiting to see whether a market takes off before eventually buying one of the innovative firms and using that as a vehicle for entering a promising market. There is an analogy with the pharmaceutical industry, where despite the industry giants having vast R&D budgets, many of the most promising drugs come from smaller bio-tech firms, whose products are either licensed by or bought up entirely by the industry giants.

It is interesting to consider why this state of affairs persists. On the face of it, a software company with large R&D resources is in a much better position to develop products, and they already have the marketing muscle and sales channels to exploit the products that their labs produce. Yet time and again new markets are developed by innovative products from small software firms, who do this despite the obvious disadvantage of having far fewer resources at their disposal.

One reason for this may be that in a large company there are (rightly) more processes in place, which have the effect of meaning that several layers of people need to be convinced of the merit of an idea before significant resources are devoted to it. If a software engineer has a bright idea for a new product then not only do they have to convince their supervisor, but the corporate marketing department and assorted executives, who may or may not buy into the bright idea, or think it is a priority for the company. The more people that look at the project, the more objections will spring up: "we tried something like that years ago and it failed"; "is there a real market for that?"; "if it is such a good idea then why hasn't someone else already built this?" and so on. It takes great determination to wade through this morass of obstacles and objections.

One reason why smaller firms innovate more is that they simply have less filters like this. Writing software does not have high barriers to entry, so someone with a bright idea, determination and at least enough capital to live on for a few months can actually produce at least a prototype, and possibly catch the eye of a potential buyer. Of course most of these bright-eyed individuals will fail, but the truth is that in reality human beings are not very good at picking out the great ideas from the dross anyway. I have met a very experienced and successful venture capitalist who turned down Google (who were entering an existing market with several well established brands, some from large companies). Even very wily venture capital firms, whose job it is to spot promising people and ideas, routinely turn down opportunities to invest in what turn out to be excellent companies.

Large software companies are no better at this murky art, and because by their nature they are more bureaucratic, they are likely to be less agile and good at selecting the best ideas.

Most software companies that turn out to be innovative and successful did not look like world-beaters in their early days, and there are a hundred reasons why a new idea might not succeed. The more stages of process and filters that we apply to weed out the good from the bad, the more likely we are to miss some hidden gems. There are plenty of such cases where people end up presenting a promising idea within a large company, having it rejected, and then leave to prove their bosses wrong; Siebel was a well known case in point.

In the enterprise software space few innovations come out of the industry giants, who more and more sit back and wait to observe which ideas turn out to have real traction in the market, and then just purchase one of the promising start-ups that created that new, emerging market. They then apply their sales channels and established brands to shift far greater volumes of that software than was possible for the smaller company.

The conservative buying habits of large companies, especially over the past decade, have amplified this behaviour, making it very hard for smaller companies to build up to a significant size without the comforting presence of a large familiar brand.

The disappearance of the IPO market for enterprise software companies that existed in the 1990s has had the effect of reducing the amount of reducing the amount of venture capital available, so fewer ideas are being backed that was the case in the late 1980s and early 1990s (never mind the late 1990s gold rush). This is a pity, since although there are doubtless many dismal ideas which got backing in those headier days and wasted some investor money before dying off, at least some of these companies went on to produce genuinely innovative products which benefitted everyone. I don't believe that we are really any smarter than before at picking winners, so the best thing for all of us would be to have an environment to "let a hundred flowers blossom" (ironically this is a quote from a 1957 Mao Zedong speech, a man not noted for his capitalist views), and see which ideas really take off and deliver customer value. Difficult economic times militate against such an environment, and in the end we will all be worse off as a result.

Read Andy Hayler's previous column on innovation.