The advertising and sales industry is being digitally disruped by the development of real-time bidding (RTB) for ad space on web pages. CIO UK's Pat Brans looks at the transformation that has seen online advertising overtake all other mediums.

Don’t you love all the free services you get on Internet these days? You can hold video conferences at no cost. You can get in touch with long lost friends and share pictures at no charge. You can use sophisticated email and calendar applications without paying a pence.

It’s hard to believe some of the richest people in the world amassed virtually all their wealth offering consumers (and enterprises) so many things for free. Then again, none of these services are really free. As TV and radio have proven since we started using those inventions several decades ago, mass media is big business.

With TV or radio, the choice for funding content has always come down to five options:

1) The state owns and runs the networks.

2)    The state levies taxes to subsidise the media and exercise influence over content without going so far as to own the networks.

3)    The general public donates money to fund media and content.

4) The users of the media pay for a subscription, which goes towards funding the network.

5)    Advertisers pay to push content designed to convince the users of the media to take some action, such as buying a product.

In most countries you see a little of each of these five models, with advertisement playing a dominant role in western countries, as most people in free market economies prefer to be brainwashed by private companies than by their government. Besides, most people swear they are not swayed by the ads, so in their minds, the media really is free.

Funding the Internet

The models for funding internet are the same as those for TV and radio, but with internet, advertisement becomes much more efficient for two reasons. First hyper targeting is possible. Because the medium is interactive by nature, you can know much more about each user.

The second reason advertisement is more efficient over internet is that, when users view a message and then click through to make a purchase, communications agencies can measure the effectiveness of their campaigns. Whereas before advertisers were plagued by the problem of not knowing how well they were spending their money, now they can know with great precision.

This brings to mind the following famous, but unattributed, quote that sums up the problem that has plagued marketing professionals for decades: “I know I’m wasting half of my advertisement budget. I just don’t know which half.”

Thanks to hyper-targeted advertisement, agencies no longer have to waste so much money. Instead of broadcasting messages based on demographic information of viewers, they can tailor ads to individuals. What’s more, they can measure the return on their advertising spend.

If you don’t believe advertisement is effective, just remember that Google, the company that provided a better search engine fifteen years ago, is now a $55-billion-a-year power house; and more than 90% of their revenue comes from selling advertisement space.

In the early days, Google, and companies like Google, didn’t need to show you an ad based on your immediate activities. Or to put it more accurately, the solutions and ecosystem were not in a state that allowed advertisers to respond to what you were doing right this second, or fraction of a second. But now they are.

The advertisement industry is being transformed by the ecosystem that has developed to allow real-time bidding (RTB) for ad space on web pages. Companies who are part of the ecosystem follow guidelines and use compatible systems that exchange data and make decisions within milliseconds.

How real-time bidding works

Let’s say you want to buy airline tickets to fly to Miami, Florida this summer. Like most people, you’ll probably search online for the lowest prices first. You may look at the site of a discount ticket broker, and you’ll probably also look at the sites of some of the major airlines. After visiting a few sites, you find the best ticket for your needs and you buy it online.

While you’re shopping around, and for a few days after your purchase, you’ll get ads for car rentals in Miami. Who was looking over your shoulder? And how did they find the ads that target you specifically?

Publishers who want to sell ad space are looking over your shoulder - and they’re keeping track of you thanks to cookies, which are a fundamental part of the process. The interesting thing is, cookies were invented for a completely different reason.

Remember that the web was designed for physicists to share papers online and to be able to click on a reference to see another document. These kinds of exchanges were simple transactions that required no memory of preceding exchanges between the two systems.

Sessions weren’t necessary until people started to realise they could use HTTP and HTML as a basis for developing online applications. One problem that had to be overcome was that, because HTTP provided no way of maintaining sessions, web sites could not link the different requests to put them into context.

Cookies, files that store information on previous transaction and sent with each request, solved this problem. This first kind of cookie was called a “session cookie.” Later on, a second type of cookie, called “persistent cookies,” was invented to allow web sites to retain information about a user from one session to another.

More recently, a new type of cookie was invented to track user behaviour across web sites. These were called “third-party cookies.” The third-party cookie is what companies like Google use to track your behaviour and sell advertisement space based on what they detect.

We just answered the questions about who’s looking over your shoulder and how they do it. Now let’s look at how they find the ads that target you specifically.

Way before your search for tickets, car rental companies have given ad agencies a budget to find new clients. These agencies configure bidding engines that respond to opportunities to post an ad. The agencies can also track how successful each ad is and provide detailed reports to clients.

When you look for the plane ticket online, a cookie on your computer allows companies like Google to follow what you’re doing. The cookie indicates that you bought something, and that before making the purchase, you visited several other sites to compare prices. Google may not know car rental companies directly, but they do know about the ad agencies.

As soon as Google knows what you’re doing on the Internet it calls on an ad exchange to start an auction and invite ad agencies to bid for ad space. The ad exchange and the ad agencies all have computer systems that allow an auction occur automatically. In less than 50 milliseconds, one of the ad agencies buys the space, and a car rental ad pops up in your browser.

Needless to say, the different computer systems powering RTB are doing an awful lot of work. According to Eric Janvier, Chief Data Officer of Numsight, ”Advertisement contracts are managed in real time. You get a big flow of information - huge amounts. In a single day, you can get two to three-hundred thousand transactions. This requires a system that detects what’s happening and makes decisions - all in real time.”

The other thing that’s required is that the ad exchange predict peaks in demand like major sporting events such as the forthcoming Rugby World Cup and make sure there are enough bidders when needed. “During the last American football season,” Janvier says, “there was a huge amount of ad space sold on social media during each match. Communications agencies predict this, so they go out and round up clients beforehand.”

Hyper-targeting has been around for several years. The newest innovation is the real time aspect of RTB. To post a relevant ad, publishers have to know what the user is looking for at the very moment. Once they know what the person wants, they have to immediately go out and find an appropriate ad, so they get ad exchange to open up an auction, strike a deal, and then the publisher posts the ad.

Some ethical questions around real-time bidding

Many industry observers think real-time bidding is a win-win-win proposition. Publishers get to fill unused ad space at market prices, advertisers get to spend their money more effectively, and consumers are only pestered with messages of interest to them. That’s the idea, at least.

Janvier says, “Real-time bidding is a trend with huge economic stakes and a lot of big players behind it. But there are also big advantages to the consumer who connects to the Internet every day, and doesn’t want to be bothered with a bunch of irrelevant ads. Users are impatient and want to quickly find what they’re looking for. Targeted ads help them do so.”

David Bevan, Senior Fellow at the Institute for Business and Professional Ethics in Chicago, says, “From the perspective of selling to a fully informed consumer, targeted advertising could be considered as more ethical as it is at least trying to engage the consumer as an individual in a mutually valuable transaction. Unlike the default propaganda approach of scattershot campaigns there is a rationale which at least appears to go beyond the goal of ringing a cash register.”

The scary part is that somebody is always looking over your shoulder on the Internet; and the better computer systems get, the better they get at crossing data, and drawing accurate inferences about you personally.

According to Eric Janvier, while some of the smaller players might ignore data protection laws forbidding them from keeping a history of personal information, the larger brands are very careful not to overstep their bounds.

Companies like McDonalds and Starbucks make sure they are working with ad agencies that play by the rules - ad agencies, who themselves are careful to work with ad exchanges and publishers that also play by the rules. The big brands know that if they are ever perceived to be invading privacy, the news would spread like wildfire on the Internet. Janvier says that what’s protecting the consumer, for now, is this combination of laws and the fear of damaged reputations.