View in Plain Text


Validating data for clinical and forensic use of CBTIs


Alex B. Caldwell, Ph.D.


COMMENTARY ON: Relative user ratings of MMPI-2 computer-based test interpretations
John E. Williams and Nathan C. Weed, (2004). Assessment, 11, #4, 316-329. Caldwell Report will provide you with a copy if requested.


This study set out to do a meaningful competitive comparison of the then eight publicly available Computer Based Test Interpretation services (CBTIs) for the MMPI-2. In a thorough study they answered many prior criticisms of preceding studies, none of which had provided a comprehensive appraisal of all of the available services.


The reports that were rated included protocols from inpatient, outpatient, college counseling, and prison samples. The participants submitted an answer sheet from one of their own clients to the authors, and they received either a CBTI analysis of that profile or else an analysis of a profile that was modal and gender matched for whichever of those four groups to which it corresponded (257 valid protocols were used). The participants then rated the report which they received on 10 variables (they knew it could be their case or a modal profile, but they only found out which after having sent their ratings back to the authors). These ratings were as follows:


1. Conciseness
2. Confirmation of therapist’s impressions of the client
3. Usefulness for diagnosis and/or treatment
4. Accuracy
5. Provision of new and important information
6. Presence of contradictory information
7. Organization and clarity
8. Presence of useless information
9. Omission of important information
10. Appropriateness of diagnostic considerations


The ability to compare the ratings of the actual reports versus the modal reports enabled them to demonstrate that the CBTI reports were adding a large amount of information above and beyond what could be attributed to stereotype accuracy or Meehl’s "Barnum" effect. This latter is the potential for descriptive statements to be considered highly accurate despite their lack of discrimination among individuals (e.g., "Some days are better than other days."). For clinical purposes this increment is happily reassuring as well as work-facilitating, and for forensic purposes it stands as a strong support of the expectation that the CBTI reports we use really are saying specific things that can potentially make substantial differences in the determinations to be made by the trier of fact.


The averaged ratings were then extensively analyzed by rank ordering the levels of favorability across the eight CBTI programs. The reports by Automated Assessment Associates (Strassberg & Cooper) received the highest ratings on accuracy, clinical usefulness, confirmation of opinion, and diagnostic suggestions. Strassberg and Cooper described their systems as conservative and to be used only in conjunction with other information; conservatism in this predictive context may enhance accuracy. The reports that were offered by Western Psychological Services obtained the highest ratings on being concise and free of useless information. The NCS-Pearson reports rated highest on organization and absence of contradictory statements. Williams and Weed distinguished between their report accuracy versus report style and organization variables; the WPS and NCS-Pearson reports thus topped out on the style variables although not topping out on content (except the authors grouped conciseness with content).


The reports from Caldwell Report had top ratings on the inclusion of new and important information and on not omitting important information (the latter by a wide margin over all of the other reports). I find this gratifyingly consistent with my obvious long-term intent to provide thorough reports that take as full an advantage as possible of the wealth of information that is embedded in the profiles and the other scores. Williams and Weed mentioned a prior study (Adams and Shore, 1976) in which there was a modest relationship between length of a report and its accuracy rating, with longer reports rated as less accurate overall. This makes obvious sense to me in that the more different things one says and the more specific one’s statements, the more opportunities one has to "go wrong." I would confess some gratification at being third ranked - and close to second - in overall accuracy despite going out on so many "limbs" where my specificity could easily be rated as not "on target."


Considering the use of a CBTI as a consultation, I believe that providing new and important information and not leaving out important issues is a crucial contribution of the "consultant." That one’s clinical impression is confirmed is reassuring as well as strengthening of one’s clinical interventions and forensic presentation. But calling attention to what the client may have consciously avoided confronting or unconsciously led attention away from can be a significant gain for the clinician. A psychiatrist with whom I enjoyed working many years ago (Ulrich Jacobsen, M.D.) spoke of using the MMPI either "to confirm or to alert." Not omitting important information corresponds, of course, to the alert function. I believe that having confidence that the MMPI has been thoroughly searched for overlooked or avoided issues should be strongly reassuring to the therapist or examiner.