Tuesday, September 15, 2009

Arbitron’s PPM: Wobbly As She Goes?

We all knew the Arbitron diary was far from perfect. As a buyer of radio time, I was pretty excited the new Portable People Meter was coming to Atlanta. Soon we would find out what the ratings really were. However, after reviewing the PPM numbers for the better part of a year, I’ve been finding some inexplicable swings in individual station ratings from one report to the next.

Over the years, I’ve looked through countless Arbitron books in numerous markets based on the diary methodology. Overall I found the numbers very reasonable and logical. And I say that based not only on consistency but also on my knowledge of radio in those markets. Occasionally something looking fluky would be there. But, the mantra was to look at the past few books and not just the latest.

Ad agencies adhered to a policy of basing buys on a 2-book or 4-book average. Buyers probably would have told you that combining multiple books was done to nullify any unexplainable bounces, and they would have been correct. The reason that multiple books accomplished that was they doubled or quadrupled the small sample size of one survey.

In the Atlanta market, the monthly diary sample was 1,347, which came to 4,040 over the 3-month survey. Each participant kept a diary for one week. Atlanta’s PPM sample over 3 months is 1,335, about the same as one month of the diary. Nevertheless, Arbitron claimed a method to its madness; each PPM panelist would record his or her listening for 84 days per survey compared to 7 with the diary. The result would be 112,140 “days of observation” from PPM panelists compared to 28,280 from diary keepers.

Arbitron told us that we should shift our focus from sample size to days of observation. The fourfold increase in days would make the PPM methodology more reliable. Charlie Sislen of Research Director, Inc. told a group of CBS Radio/Atlanta clients that he saw no need to base buys on multiple PPM reports. Moreover, using only the latest data would factor in any changes in the market over the past 45 days; no more waiting 4 months. I was sold; my buys would be based just on the latest PPM report.

When I looked at the Persons 6+ shares for August, I saw that Star 94 (WSTR-FM) lost approximately 25% of its average quarter-hour audience in one month. And that was after trending at about the same share for the past 3 months. Star 94 had not changed anything in August, so was suddenly losing a quarter of its audience logical?

Star 94’s most direct competitor, Q100 (WWWQ-FM), had been correcting its music in recent weeks, so did Star listeners react to the change and go to back to Q100? Well, no. Q100 went from a 3.4% share to 3.6%, where it had been in June. And the third station in the CHR genre, 95-5 The Beat, went from 3.6% to 3.4%. Star’s audience apparently left the format yet Q100’s and The Beat’s listeners did not.

Since early this year, Dave-FM (WZGC) had been on an upswing. In the May PPM, the station slid from 3.2% in April to 2.7% but then bounced way up to 3.6% in June. We all thought April was a fluke. In fact, Market Manager Rick Caffey talked about Dave’s steady growth. Then, Dave-FM’s 6+ shares were 2.6 in July and 2.4 for August. Now the June share rather than the May number looks like the fluke.

When Mickey Luckoff, the highly-regarded GM of San Francisco’s KGO, complained that the PPM samples were too small, I thought he needed to see the PPM presentation that explained that days of observation were paramount. However, after reading an analysis of the New York PPM panel by a company called Harker Research, I’m edging toward Mr. Luckoff’s camp.

When Arbitron recruits diary keepers and now PPM panelists, the company attempts to closely match the market’s demographics. After the diary sample or PPM panel is in place, Arbitron determines how it compares to the market’s composition and weights accordingly. For example, if people 18-34 make up 10% of the population but only 5% of the sample, each diary keeper or PPM panelist who is 18-34 counts twice. The survey is more reliable when the percent of the sample is the same or very close to the percent of the population.

Arbitron told us that the PPM panel would be very carefully recruited to mirror the market’s demographics. This would result in more stable survey-to-survey trends than with the diary, according to PPM literature. In the New York market, based on the Harker findings, Arbitron has fallen considerably short of that promise.

Harker says the problem appears to be the panel; that participants are not consistently carrying their meter. Research Director, Inc.’s Sislen expressed concern that the reason the PPM is showing far less listening for females than males is that females have a difficult time carrying their meter in a place that picks up radio signals. As a result of these compliance problems, according to Harker, Arbitron has had to employ a dynamic weighting system to compensate for participants who come and go within the active panel.

Adding this weighting system to the demographic weighting, according to Harker Research, complicates things to the extent that swings in the ratings are inevitable. In the New York PPM report analyzed, the age and sex cells were off by an average of plus or minus 18% compared to the population versus 11% with the diaries. In other words, the PPM’s range of error was almost twice as high as the diary’s. That means some demographic cells are being weighted much more severely now with the panel than they were when Arbitron used one-week diaries.

The Harker Research study looked only at age and sex cells but not at ethnic weighting. Had it looked at ethnic weighting, contends Harker, the differences would have been even greater.

In fairness to Arbitron, the overall PPM results for Atlanta have looked reasonable. In some other markets, Washington, DC for example, most urban stations that were previously near the top of the diary ratings have taken quite a tumble. Furthermore, I’m extrapolating the New York situation into Atlanta, but I think that’s reasonable. I plan to evaluate stations for buys looking at the past few months while considering the most recent month if the market was going through major changes—format flips, personalities jumping from one station to another and the like.

The PPM does not seem like the ratings panacea that I had anticipated. I applaud Arbitron for jumping into PPM and realize that any needed changes could be costly. Stations, which pay big dollars for Arbitron compared to the pennies forked over by agencies, want better measurement but are not excitedly running to the piggybank to pay for it.

With Nielsen now in the radio fray and Arbitron talking about re-entering the TV ratings game, the next year will be an interesting one. Frankly, if Nielsen had continued to partner with Arbitron as was the case in the Houston PPM test market, I wonder whether both companies as well as the radio and agency communities would be better off today.

It’s Shout-Out Time
I want to salute Rob Stearns, who has left Star 94 after over 24 years of service. He was an account executive for 9 and a half years (starting at 94Q), Local Sales Manager for 5 years and National Sales Manager for 10 more. Rob is a solid person who contributed mightily to Star 94’s billings success over the decades. We wish him the best in his future endeavors.

Jeff Hullinger, as most of you know, was released from Cox Radio last week. At Cox, he worked long hours, starting his day as newsman on B98.5 FM during morning drive, and then walking down the hall to WSB-AM after lunch to anchor the afternoon news. Jeff is one of the most versatile broadcasters around, having also done sports on WAGA-TV’s early and late evening newscasts, both radio and TV play-by-play, and a morning drive radio show. We hope Jeff lands in the kind of top-notch position for which he is so well qualified.

Best of Atlanta Radio?
If you’re wondering what I’ve been doing the past few weekends, I’ve been looking for the right flak jacket. That’s because I’m thinking about devoting an issue of AAA to the best jocks in Atlanta radio. I want to be ready for any reaction from those not on the list.

Years back, I wrote a similar column for RadioDigest.com. A local personality called Stick, who was not on the list, emailed to inform me that I knew less about radio than anyone he knew.

Roddy Freeman

Thanks for reading. Feel free to email me at roddyfreeman@bellsouth.net.

Link to Rodney Ho’s AJC Radio & TV Blog:

1 comment:

  1. Roddy:

    Wow - terrific, well thought out post. This is one of the best, most reasoned PPM thought pieces I have seen.

    Looking forward to reading more of your stuff!