Garbage In, Garbage Out: How NOT to Design Survey Research
It’s not often an advocacy group releases quasi-research with findings antithetical to long-held public policy positions, but the National Community Pharmacists Association (NCPA) recently did just that.
The NCPA has been hostile to many of the pharmacy benefit management tools that private and public payers rely upon to maximize the value and effectiveness of prescription drug benefits, including the mail-service pharmacy option. Speaking of that option, seniors reported, by a 3-1 margin in the survey, that they found mail-service pharmacies cost less than independent drug stores.
The trade group has repeatedly disparaged the value of mail-service pharmacies, notwithstanding the fact that consumers and businesses report high satisfaction and numerous government and industry research have documented the cost savings associated with the option. One thing we know about American health care: if consumers are not satisfied or payers do not feel they are getting value with a benefit, they will demand changes. If anything, over the past decade, we have seen a dramatic growth in the mail-service pharmacy option.
The NCPA “report” purports to be a survey of 669 Medicare-eligible Americans and their attitudes toward mail-service pharmacies, but it contains some glaring flaws:
- The NCPA survey failed to ask/report on the percentage of respondents who take maintenance medications for chronic conditions and the duration for which they have been taking them. This is a fundamental omission. How do you assess attitudes when you don’t even know how many seniors have actually used the service and/or for how long?
- The NCPA survey did not report any findings included in peer-reviewed literature about the better adherence and persistence found at mail-service pharmacies.
- In numerous places, the NCPA make conclusions not supported by the data in their own survey. For example, their interpretation of data presented in figure 3 on page 9 of the NCPA report states, “Nearly half of respondents were neutral or disagreed with this statement,” referring to mail being less costly. It is simply incorrect to interpret the neutral group together with those who disagreed. As per the data, only 17.3 percent disagreed with mail-order being less expensive compared to the majority of respondents who agreed, or 51.6 percent. Data clearly indicates that those who agreed in favor of mail-order on this item were nearly three times those who disagreed.
- NCPA reports that one in eight respondents (12.3 percent) reported that they are currently required to use a mail-order pharmacy in their prescription benefit plan. This is clearly a case of inaccurate self-reported information on required mail order use as required mail order is explicitly prohibited in Medicare Part D. The author never mentions this fact.
- Interestingly, there are different sample sizes for different questions, which indicate the presence of missing information. No information was provided regarding any tests that were done to see how this missing information impacted the robustness of the results.
We are not aware what the dues levels are for independent pharmacists who are member at the NCPA. Whatever they are, one would think their membership would deserve a higher caliber of “research.”