read whyRequest A Demo
Thank you! Your submission has been received!
Something went wrong while submitting the form
Ad QualityAd Speed
June 22, 2016

How to Measure the Cost of Ads

Ad ops has traditionally been focused on one metric: revenue. Revenue is the ad ops north star, the guiding light, the arbiter of decisions. It's how we evaluate, prioritize, and judge the effectiv...

Chris Cummings

Ad ops has traditionally been focused on one metric: revenue. Revenue is the ad ops north star, the guiding light, the arbiter of decisions. It's how we evaluate, prioritize, and judge the effectiveness of what we do.

Considering whether to add a new ad partner? Analyze the impact on revenue. What about trying a new ad format? Lock in the CPM. Should we introduce another ad unit to the page? Determine the effect on the bottom line.

But clearly, revenue is not enough. Without being able to measure the cost of ad changes, we may be making decisions that increase revenue in the short term but lower profitability in the long term. And just as importantly, we may be leaving significant money on the table. Here's why.

Decision Making in the Dark

Let’s imagine you’re in a team meeting and an engineer proposes adding a new 300x250 ad unit to the right sidebar of your website. A member of the ad ops team becomes excited and suggests that this could lead to significant revenue improvements. Skeptical, another member of the ad ops team says that this new unit could lower the CPMs of other units on the page. The designer on the team grumbles saying that the user experience of the site is going to take a hit. Then the product manager says that page views will decline. Another engineer on the team says that this ad unit is going to slow down the site even further.

Hearing all of these critiques--lowered CPMs, worse user experience, fewer page views, slower site--the original engineer thinks for a bit and proposes the exact opposite: let’s remove an existing 300x250 from the sidebar. The room looks shocked. We have to survive, says the business manager. The ad units we have on the site are no different from every other site out there, says the product manager. That would kill our revenue, says the ad sales team.

Puzzled, the original engineer looks around the room and asks: How can we make such an important call as to whether to add or remove an ad unit without any real measurement of the impact on our site? Does anyone have any data on how much these ads actually impact user experience, ad speed, or page views?

The room falls silent.

A Framework for Measuring the Cost of Ads

Decisions on whether to change ad setups are made constantly. Teams choose ad formats, units, partners, campaigns, even specific creatives. Clarifying the costs and benefits of these ad-related decisions can dramatically improve decision making. Conceptually, sites should be focused on long term value, as short term bumps in revenue may hide future damage to brand, user engagement, and traffic. But to understand long term benefits and costs, it’s necessary to look at early indicators of changes, including shifts in engagement, feedback, speed, and security. Here’s a 5-point framework that will help you measure the impact of changes in your ads more holistically.

1. Total revenue

When making a change, measure impact on revenue across all ad units units--even ones that did not change--in order to account for unintended gains or losses. In a recent test on our site,, we tested adding a new 160x600 sidebar unit on the right side of certain pages. The unit itself added to revenue, but we also saw a decrease in CPM across other ad units. Had we not examined the overall change in the CPM per pageview, we would have misunderstood the net impact on revenue. Be sure to look at both the isolated impact of a change and the net impact across page.

Example metrics: CPM per Pageview, Revenue per User

2. User engagement

Measure changes in key engagement metrics for your site. Every site on the internet faces a tradeoff between ads and user engagement. Too often sites are following the herd in how they approach their ad setups. A famous example of bucking the trend is Google’s ad-free homepage. They’ve likely found that engagement (searches) is higher if the page is ad free, and overall it’s better to have higher engagement overall than increased monetization from ads on the homepage alone. To assess what’s right for your site, you must measure engagement.

Example metrics: Page Views per visit, 7-day repeat visit rate, signups, searches, shares

3. User feedback

Measure what users say about the specific ads themselves. In our research on ad quality at, spanning dozens of sites and more than 1 billion monthly pageviews, we’ve found that users are particularly loathsome of ads that expand to cover the page or play audio automatically, but also share feedback quite often on ads that are slowing down the site or displaying inappropriate content. Unless there is a mechanism for feedback on the ads, it may be difficult for publishers to gather this user feedback. The most targeted way of getting specific feedback on ads is with a link placed next to the ad that invites direct feedback, but it’s also possible to read through more general user satisfaction surveys to identify changes in user’s overall opinion of the site.

Example metrics: Negative ad feedback; Change in overall user satisfaction

4. Speed

Measure the impact a change in the ad setup has on site performance. A wealth of research has established that faster sites garner more page views, more engagement, and more purchases. In research we’ve conducted at PubNation analyzing more than 101 websites, we recently found that, on average, ads more than doubled website load times. While the engineering team is busy trying to shave off milliseconds on the backend, the ads are loading huge, slow-loading files on the front-end. Many ad ops and engineering teams are flying blind to the problem. The the first step to regain control of site speed is to measure the impact of ads on your site.

Example metrics: Page load times, Speed index, Time to view page content (using the filmstrip view on, Page download size, especially on mobile.

5. Privacy and security

Measure the impact of changes in your ad set up on privacy and security. One of the top three reasons users install ad block is because of concerns over tracking. Nearly all sites have a privacy policy in place that explains that data will be collected for the purpose of serving ads. Most sites, however, don’t actually know what data is collected by third parties, how the data is being used, or even which third parties are collecting the data. Moreover, many sites struggle to ensure that third party exchanges are not serving ads that contain malware or automatic redirects. To regain the trust of users, publishers must start measuring the impact of third party ads on privacy and security.

Example metrics: # of trackers, 100% passage of malware scans, Compliance with Ad Choices and Tag

An Investment with Big Payoffs

For most publishers, the choice of what ads to run on the page is among the most important business decisions. Measuring the cost of ads requires an investment. But the reward is smarter decisions. That means fewer decisions that trade short term gains for long term losses, and more decisions that deliver long term gains for little or no loss. As a simple test of whether your site has the information to make smart decisions, ask yourself, should your team add another ad unit to the side bar? Should you remove one? How do you know? If this is hard to answer, it might be time to improve the measurement of the cost of your ads.

Follow PubNation Blog

A/B Testing Advertising: A Playbook for Publishers

Six months ago we launched an A/B testing framework for our site,, that has allowed us to make ad ops decisions smarter, faster, and with measurable improvements to revenue. Our fir...

Header Bidder Optimization: When Ads Compete, You Win

In the past two months, we've radically overhauled our programmatic advertising stack on, shifting our focus to header bidding advertisers--advertisers that submit the price they wi...