TOP PUBLISHERS USE PUBNATION TO MANAGE AD QUALITY
read whyRequest A Demo
Thank you! Your submission has been received!
Something went wrong while submitting the form
AB TestingPrice Floors
June 22, 2016

A Simple A/B Test that Raised AdX Revenue by 10%

A simple price floor A/B test in AdX that resulted in a revenue increase of 10%. We’ll also show you the details of how we ran the test, so you can give it a try for yourself to see if it boosts your

Chris Cummings

In this post we are going to share a simple price floor A/B test in AdX that resulted in a revenue increase of 10%. We’ll also show you the details of how we ran the test, so you can give it a try for yourself to see if it boosts your website’s revenue.

The Setup

A few weeks ago, we wrote about a model to optimize price floors in AdX, which found that price floors may increase revenue but their impact depends heavily on the replacement rate when no bidders exceed the floor. We then published an analysis supporting the idea that many AdX impressions result inbusted auctions, which suggested that price floors may add value by putting auction pressure on higher bidders. What we haven’t done, until now, is share the results of an actual price floor experiment.

In this test, we wanted to start simple. We confined the test to all ad units on desktop traffic in the US, which comprises the majority of our traffic and an even higher percentage of our revenue. We then used the test groups from oura/b testing framework to target different price floors to different users. For one group of users, the “A” channel, we had no price floor, which was our existing setup. For the other group of users, the “B” channel, we set the price floor at $1, a floor we arrived at based on a more recent analysis using our price floor model. We created rules in AdX for each group to make reporting on the results easier. Here are what the rules looked like in AdX:

The tags we included ensured that we focused on US, desktop inventory, by channel, for all our add units.

The Results

We ran the test over a 24 hour period across more than 7 million ad requests. In AdX we used the “Price Rules” report to analyze the results. Here’s what we found:

You can see in the results that the eCPM increased by 30%--a significant gain--while the matched coverage rate dropped by 28%--an even more significant drop. If those unmatched impressions went unserved, the net impact of the test would be a 6% decline in the original revenue (100% * 130% * 72% = 94%). As we found in the model, the impact of the test hinges on the recovery rate for those unmatched impressions.

So we created a simple model (which you can download here) to run a scenario analysis on the revenue impact given the full range of recovery rates. Here’s what we found:

Unfortunately, AdX reporting doesn’t currently allow for the calculation of the actual recovery rate, so we had to make an assumption. At just a 30% recovery rate or above, the experiment had a net positive impact on revenue. We went with a relatively conservative 50-60% rate, on the idea that, on average, one of our header bidder partners would fill the impression with at least half of the revenue AdX would provide. Under this scenario, the result was an approximate 10% increase in revenue. Not bad.

Given the positive impact of the price floor across most recovery rate scenarios, we chose to implement the new floor immediately. But just to be sure the impact was sustained over time, we kept a portion of our traffic price-floor free, essentially continuing to test that the floor was in fact adding value.

Next Steps

The price floor strategy used in this test is relatively crude, making the magnitude of the results surprising to us. It also left us with a number of questions. Will the impact of this price floor be sustained over time? Is $1 the right floor? Would we achieve an even greater revenue lift if we segmented the price floor by ad unit, or device, or some other factor? As much as anything, this experiment showed us how easy it is to run price floor A/B tests. These are questions that we intend to explore in future experiments.

Have any questions or feedback, let us know in the comments below.

Follow PubNation Blog

Header Bidder Optimization: When Ads Compete, You Win

In the past two months, we've radically overhauled our programmatic advertising stack on SpanishDict.com, shifting our focus to header bidding advertisers--advertisers that submit the price they wi...

Publishers Fight Back: How The Top 50 Websites Combat Adblock

On our site, SpanishDict.com, adblock has become a major problem. We lose over 15% of our revenue each year because of adblockers--and the impact is growing larger every month. We are now embarking on