A
fundamental assumption behind all scoring systems is
that the past and future are the same. Unfortunately,
this assumption is never really true; the world is always
changing. Because scoring systems are built using historical
data, all scoring systems lose predictive power over
time and eventually require redevelopment.
To maintain the performance of a scoring system, its
components must be validated on an ongoing basis so
that any deterioration can be identified and corrected.
PMA provides scoring system validation for all scoring
applications. We use industry standard diagnostic tools
to measure the reliability and validity of a scoring
system. Our scoring system validations cover three
crucial areas:
- Scorecard and Characteristic
Discriminant Power: Our tests measure the
power of the scoring system -- and each of the
characteristics in the scoring system -- to separate
the "good" and "bad" populations. We use various
statistical performance measures, such as the
Komolgorov-Smirnov(K-S) test, to measure
the ability of the score to separate goods and
bads and provide the expected "lift" between low
and high scoring segments.
- Scorecard and Characteristic
Stability: Economic climates and business
enviroments change. These changes can influence
population distributions and affect the performance
of a scoring system. To determine if a scoring
system can continue to be used effectively, the
stability of the target population needs to be
measured. The PMA Population Stability Index™
measures the degree of change within the target
populations, both for the overall scoring system,
and for each characteristic in the scoring system.
High index values indicate the population has
changed and may suggest the scoring system needs
recalibration or redevelopment.
- Cutoff score performance:
A scoring system can continue to rank order the
population and provide acceptable discrimant power,
yet fail to produce desired performance. Shifts
in scorecard performance by score range may demand
adjustments to score cut-offs. Our validations
include updated score performance tables for each
key portfolio segment so score cut-offs can be
adjusted.
After each successful validation,
we will certify that the scoring system is statistically
sound, that the population stability tests meet industry
standards, and provide full back-up documentation, including
revised score performance tables. If the scores do not
adequately validate, we will provide recommendations
to address any deficiencies identified.
Should your scoring system fail to validate, we are
able to assist you with necessary remedial action, which
can range from simple re-weighting of the existing characteristics
to complete score
redevelopment. In addition, if desired, we can realign
scorecards to standard performance odds.
On an on-going basis, we can provide MIS
support to track performance of your scoring systems.
|
| |
|
