How We’re Helping Marketers Beat Bots

In the second or so it takes for a consumer to click on a link, a great drama unfolds, including a little game of cops and robbers with bots.

Bots have claimed a lot of victories in this game. They siphoned some $7.2 billion out of digital advertising last year. One particularly hard to catch type of bot is the type that appears, spams and quickly disappears. Such bots appear to browse thousands of web pages a minute.

That sounds easy enough to solve, right? Just count their requests, identify them and then block them. In practice, it’s hard to do at scale. That’s why we built a new tool to fight them using a count-min sketch in a distributed way. This tool is giving us a leg up on ephemeral bots.

Probabilistic data structure to the rescue!

What is a count-min sketch? It’s a probabilistic data structure that consumes a stream of events and counts how many of each event it saw in the stream. As the name suggests, it offers an approximate answer, not an exact one. In this case, that lack of exactness is acceptable, and in exchange it allows us to inspect every request in real time to expose short-lived bots.

One crucial feature for our purposes is that multiple sketches can be aggregated. Two sketches can be added together to see all the events of the original sketches. This feature is great for bot-fighting.

Turn’s real-time bidding pipeline is designed so that each individual bid request can come into any of several hundred bid servers. So we need a system where each server can quickly know how many times the entire cluster saw a single user. Each server uses a local sketch. Every minute or so we aggregate all of the local sketches into a global sketch. That global sketch can tell us how many times the entire cluster has seen a particular user in the previous minute. If one user was seen more times than is humanly possible, then it’s a bot.

Results

Another indication that these were bots was that “users” flagged by the sketch didn’t do much else besides make bid requests. There were no downstream clicks, actions and events in their histories. They would also appear and disappear within minutes or hours. When we rigged the sketch to only look for old bots, and stopped detecting new ones, the detection rate fell about 50% in 15 minutes. So checking every minute catches a lot of bot mayflies.

We also compared our solution to a third-party fraud detection service that we use. That service doesn’t see all bid requests and doesn’t have a short detection loop to catch those ephemeral bots. As a result, we see little overlap between the two. That means that we’re both catching different bots at the same time. This patent-pending method is designed to catch more bots before they perpetrate any more fraud.

This is all great news and we’re satisfied with the results so far. But we won’t be fully satisfied until we’ve eradicated 100% of the bots plaguing our partners. With this latest enhancement, we’re a step closer to that goal.

For more on how we help create a more transparent ad ecosystem for marketers, see this post on working with trusted programmatic partners.

About Amobee

Founded in 2005, Amobee is an advertising platform that understands how people consume content. Our goal is to optimize outcomes for advertisers and media companies, while providing a better consumer experience. Through our platform, we help customers further their audience development, optimize their cross channel performance across all TV, connected TV, and digital media, and drive new customer growth through detailed analytics and reporting. Amobee is a wholly owned subsidiary of Tremor International, a collection of brands built to unite creativity, data and technology across the open internet.

If you’re curious to learn more, watch the on-demand demo or take a deep dive into our Research & Insights section where you can find recent webinars on-demand, media plan insights & activation templates, and more data-driven content. If you’re ready to take the next step into a sustainable, consumer-first advertising future, contact us today.