Measure Testing and Evaluation
The AMA-convened Physician Consortium for Performance Improvement® (PCPI®) acknowledges the importance of testing performance measures for feasibility, reliability, validity and unintended consequences. To that end, the Measures Implementation and Evaluation (MIE) Advisory Committee to the PCPI Executive Committee has created the PCPI Measure Testing Protocol. It is important to the PCPI that we are able to provide confidence in the results of our performance measures.
The PCPI professional staff conducts measure testing projects to determine if PCPI measures are sound and reliable for measuring performance. Testing is an important step in the process of development, specification, public comment, membership voting and evidence to support endorsement or adoption. The project reports detail the measures tested, method, testing results and lessons learned. Testing performed for these projects may include:
- Reliability: How confident are we that the measures accurately and consistently assess the performance of physicians providing the care assessed in the measure?
- Feasibility: How confident are we that practices can accurately collect and report these measures in a sustainable fashion?
- Inter-rater reliability: The extent to which observations from two or more human observers are congruent with each other.
- Parallel forms reliability: The extent to which multiple formats or versions of a test yield the same results.
- Reliability on the form of signal to noise ratio (SNR): measure of confidence in differentiating between physicians and other providers or how confident are we that two physicians judged to be different really are different.
PCPI professional staff also worked with several PCPI member organizations to carry out an assessment of measure reliability in the form of signal to noise ratio (SNR). In the SNR analysis, a physician level registry or claims database was used for extracting the relevant physician level information. The results from this testing have been included in perioperative, melanoma and gastroenterology measure submissions for NQF endorsement.