Building Ad Tech at Scale Means Putting it to the Test

Turn has built a powerful ad tech platform that performs at scale to crunch tremendous amounts of data and power complex marketing tasks for some of the world’s top brands and agencies. To build and scale in a dynamic industry and ensure the quality and stability of our software, we put it through its paces in test environments that mimic our production systems and external environment at a smaller scale. This approach works well, but it does have some inherent limitations which we’re seeking to surpass and solve for through innovative technology solutions.

The number of test environments at our disposal at any given time can present challenges. Although the test environments are smaller than production, they are still too big to fit on an engineer’s personal machine. Multiple server-class machines are needed to build a test environment. This causes resource contention between engineers, teams, and projects, often wasting valuable time while engineers wait for an environment to become available or they try to share an environment with other team members. Sharing an environment is not pain-free. For example, schedules are not always aligned. While one engineer wants to proceed with testing, another engineer might want to take control of some service for debugging a problem or bring the environment down in order to deploy a new binary. Other problems include conflicting data being added to the environment, leading to incorrect or unexpected test results. In short, having another team poking and prodding around in the test environment you are using often leads to headaches and expensive delays.

There are two important characteristics of a good test environment:

1. The environment is in a known state when testing begins
2. The environment is protected from unwanted interference

But here’s the catch…It’s impossible to achieve these in a static and shared environment!

So, what’s a viable solution to counteract these headaches? A simple course is to give everyone their own test environment to use; however, this solution isn’t practical as the machines necessary to host these environments aren’t cheap, so handing them out to everyone costs a pretty penny. Also, given the sporadic nature of testing, these environments are not heavily used. Most of the time, the machines are sitting idle, wasting money. What’s needed is a solution where engineers can access a private environment on demand. When an engineer is not using the environment, the resources can be freed up for other engineers to use.

This is where Maestro comes in. Turn built Maestro as a RESTful microservice that allows users to have their own test environments, tailored to their needs and free from outside interference. The service exposes an API through which users can request a new environment, terminate an existing environment, or re-deploy new binaries to an environment. Managing the lifecycle of an environment is a complex task. Maestro conducts the show and makes orchestrating multiple resources and process, each with its unique set of quirks, strengths, and limitations, look easy!

As an example, let’s look at how a request for a new environment is processed by Maestro:

1. Parse the request and prepare a list of instances necessary to build that environment
2. Requisition the correct type and quantity of instances from OpenStack
3. Once the OpenStack instances are ready, download an appropriate Docker image to each instance
4. Start Docker containers
5. Distribute Turn binaries
6. Upgrade the databases
7. Start up necessary Hadoop nodes
8. Deploy Oozie jobs
9. Start Turn services

Because most of these steps are prone to failure, Maestro is designed to be sufficiently fault tolerant to achieve a more than 99% success rate.

After the environment is built, the engineer is free to interact with the machines in whatever way they see fit. This means they can re-deploy often, leave services offline, and debug as necessary without inconveniencing other teams. The test environment comes up in a known state every time. The only variance between environments is the code deployed to them. If something happens that places the environment into an unusable state, the engineer can tell Maestro to release the resources and create a brand new one. The fact that the environment is not shared enables engineers to test changes to critical parts of the platform without the fear of impacting another engineer’s work.

There are two contexts in which Maestro is used. First, described above, is the manual testing carried out by engineers during the course of a project. The other involves end-to-end automated testing as a part of Turn’s Continuous Delivery pipeline. The final step in that pipeline spins-up a new environment, runs the tests, collects the results and logs for archival, and finally terminates the environment and releases the resources. Maestro is a perfect tool to create disposable single-use environments for automated testing. Having an environment that always starts up in a known state and is free from unwanted interference makes our CD pipeline more stable and reliable.

Here at Turn, we constantly strive to make our platform more stable yet remain agile so that we can thrive within an ever-evolving ecosystem. Maestro allows us to conduct pre-production rehearsals and raise the overall quality of our platform another notch while simultaneously allowing us to move faster. It’s another example of the innovation enabling Turn to deliver incredible value to our clients and our partners.

About Amobee

Founded in 2005, Amobee is an advertising platform that understands how people consume content. Our goal is to optimize outcomes for advertisers and media companies, while providing a better consumer experience. Through our platform, we help customers further their audience development, optimize their cross channel performance across all TV, connected TV, and digital media, and drive new customer growth through detailed analytics and reporting. Amobee is a wholly owned subsidiary of Tremor International, a collection of brands built to unite creativity, data and technology across the open internet.

If you’re curious to learn more, watch the on-demand demo or take a deep dive into our Research & Insights section where you can find recent webinars on-demand, media plan insights & activation templates, and more data-driven content. If you’re ready to take the next step into a sustainable, consumer-first advertising future, contact us today.