I’ve got a great episode for you today. We are talking with Dr. Joni Mihura, all about the Rorschach Performance Assessment System or R-PAS.
Dr. Mihura is one of the co-developers or co-authors of the R-PAS, along with a few other council members from the Exner Research Council for the Rorschach Comprehensive System. After his death, she and her partners continued the development of the test and in 2011 published the R-PAS, which is now largely regarded as the most current up-to-date, and well-supported in terms of research method of administering and scoring the Rorschach.
[00:01:00] Let’s see. What else? Joni has an incredible bio. Her CV is 27 pages long and all of it is completely legit. I looked through all of it. But here are some highlights.She got her Ph.D. from Oklahoma State University. She did her internship at the Massachusetts Mental Health Center at Harvard Medical School. She generally has a specialty in psychological assessment and she is Board Certified in Assessment Psychology. Joni is currently a Professor at the University of Toledo in their department of Psychology. She is a Licensed Psychologist. Like I mentioned, she was previously on John Exner’s Research Council for the Rorschach Comprehensive System, and what else?
In 2013, she published a huge meta-analysis of the 65 Rorschach variables and produced an article that [00:02:00] resulted in something pretty remarkable. A lot of the Rorschach critics lifted their recommendation for a moratorium on the use of the Rorschach after her meta-analysis was published.
In 2018, Joni co-authored a book called Using the Rorschach Performance Assessment System with her husband Greg Meyer, and she’s currently serving as the President-elect for the Society for Personality Assessment.
So, Joni’s been doing this for a long time. She is clearly one of the experts in the Rorschach. I’m so grateful to have her on the show today to talk all about the Rorschach and the R-PAS and how it’s helpful, its clinical applications, the research behind it, and the evolution and path of the Rorschach over the years. We cover a lot of topics. I hope you will stick around to enjoy it.
[00:03:00] Before we get to the interview, I think at the time of publication of this episode, we may have one or two spots left in the Beginner Practice Mastermind and Advanced Practice Mastermind groups. If you’re interested in these group coaching experiences, like I said, there’s one for folks just getting started with their testing practices, and then there’s one for folks who have been at it for a while and are relatively successful but looking to take their testing practices to the next level.You can get more information at thetestingpsychologist.com/consulting, and on that page, you can find a little more information about both the Beginner Practice and Advanced Practice groups. You can also schedule a phone call to chat with me about membership and whether they would be a good fit.
All right, without further ado, here is my conversation with Dr. Joni Mihura.
Hey everybody. Welcome back to another episode of the Testing Psychologist Podcast. I’m Dr. Jeremy Sharp, and like you heard in the introduction, I am here with Dr. Joni Mihura. I’m so fortunate to be speaking with her. She has, I don’t know, probably more knowledge on the R-PAS than anybody else out there. Would you say that’s fair, Joni?
Dr. Mihura: Oh, I don’t know. My husband might tie with me.
Dr. Sharp: Okay. We can include him too just for a number of […]
Dr. Mihura: Okay.
Dr. Sharp: Nice. I’m so fortunate to have you here. Welcome to our podcast.
Dr. Mihura: Oh, thank you. It’s great to be here. Thanks for inviting me.
Dr. Sharp: Of course. I have to give a shout-out again to Dr. Raja David who [00:05:00] not only did a pretty amazing podcast series for me but then introduced me to all of you other folks who are fantastic. I’ve gotten so many great interviews from connecting with them. So another shout-out there.
Let’s see. I think as usual, I would love to just start and hear a little about you in your own words, what you’re up to these days, and how you got to where you are right now.
Dr. Mihura: Interesting question for me because I don’t think I’ve ever been asked that. So I will try to give a Reader’s Digest version, a short version of it. Let’s start in studying and deciding what I was going to major in. I initially wanted to be either a psychologist or a coach. I played basketball. I played [00:06:00] in a number one team. I’m 6ft tall. You can’t see me for anybody who’s out there, but I’m quite tall.
Dr. Sharp: Nice.
Dr. Mihura: And my father was like, “No, you can’t do that. You need to be something more respectable. None of that emotional stuff.” So I started off in engineering. So for a year and a half, I was in engineering, and in the summers I got paid pretty good for doing some drafting jobs for the AT&T when I used to do that by hand.
And then I finally just took off a little bit of time, decided, do I want to get my father’s approval or am I going to be waiting for that when I’m 40 or what? So I ended up going to psychology and as soon as I did, I was so happy. I just love it. It’s an immense playground of knowledge and information and things to be curious about and helping people. It’s very multi-dimensional.
So anyway, I didn’t know what I wanted to do when I started graduate school, [00:07:00] whether I even wanted to be a clinician or to go into academics, and certainly didn’t know anything about assessment. And I think most of the time students who come work with me here at the University of Toledo, if they are interested in assessment or even the Rorschach or even know about it, they’ve had some experience in some way. Either they’ve worked with somebody as an undergraduate or most typically they’ve had a master’s training or they’ve had experience and learned that they like it.
And so that’s what happened with me too. I took the class in graduate school and unbeknownst to me; my instructor had recommended me to do the assessment lab. And I said, sure. So I did like four semesters of cognitive assessment and what they used to call projective assessment that we now call performance-based assessment.
Dr. Sharp: Yes.
Dr. Mihura: And so I wasn’t [00:08:00] sure initially whether I wanted to do research in psychotherapy. To fast forward quite a bit, I am doing research with the Rorschach and John Exner of the Comprehensive System asked me to become part of his research council. And I was surprised. I was also the only woman who was ever on his council, so I was pretty glad that it was men. And then he died.
Let’s go fast forward to 2006. And as part of the council, he had always told us that we were going to continue the development of the Comprehensive System, but then he left it to his family. And after discussing with them across two years, he had been collecting norms we were wanting to put, they decided they didn’t know what to do. And they just would leave it as it was. And we could go on and develop our own system, but it would have to be completely unique which is what we did.
I never expected to be a test [00:09:00] developer. It’s led to a lot of interesting opportunities to go places to do talks. But anyway, that’s the short version of how I ended up where I am now. I think of myself actually. I did Honors Math Meets in high school. So I think of myself as a math person, but now, I guess, I’m a psychologist now.
Dr. Sharp: That’s hilarious. The places that we find ourselves, right?
Dr. Mihura: Yeah.
Dr. Sharp: Let me go back. I’m very curious. That event of getting invited on to Exner’s Research Council, how did that even come about?
Dr. Mihura: Well, I had been doing research with the Rorschach. I also won an early career award for those few people who are doing research with the Rorschach in their early career. And I hung out with all the people who were in the [00:10:00] midst of working with Rorschach. I was on the discussion list of Jack Gruber, who had a Rorschach discussion list, and in my early days, I would jump in and be the authority before I probably should have even been the authority.
And so, I don’t know, he asked me, actually, he didn’t ask me to join in person. He was sick during that time. He had leukemia and he was at the time of that meeting sick. And Philip Erdberg, who is now part of our R-PAS group from California, the sweetest guy asked me to join. I was very touched and surprised. I had no idea I was going to be asked. I was like, wow, yes, I like that. I like all the people part of the group. And in fact, eventually started dating and married one of them, Greg Meyer.
Dr. Sharp: That worked out.
Dr. Mihura: That was pretty good.
Dr. Sharp: That’s fantastic. I know you’d mentioned that [00:11:00] your husband also so steeped in all of this.
Dr. Mihura: Yeah, he’s the first author of R-PAS.
Dr. Sharp: There you go.
Dr. Mihura: Yeah.
Dr. Sharp: Well, I’m just thinking about it, I’m married to a therapist and…
Dr. Mihura: Oh, yeah, I’ve heard that in your podcast.
Dr. Sharp: Yeah. It’s a gift to be so closely connected to someone who gets what you do, right?
Dr. Mihura: Yeah. Who gets it. Yeah, who understands.
Dr. Sharp: Yeah. That’s very cool.
Dr. Mihura: Yeah, we’re hardly ever apart now. Offices are right next to each other. He was in Alaska at the University of Alaska for nine years before he came here.
Dr. Sharp: Got you.
Dr. Mihura: Working.
Dr. Sharp: Sure. Here you are. There’s so much I want to ask about, but maybe I’ll just start with what might be a basic [00:12:00] question. How did you even, I’m always curious, how do you pick your research area? How did you get into the Rorschach specifically, and why put all this time and energy into that?
Dr. Mihura: That’s a good question because in our research lab, we focus on performance-based instruments and then we call other instruments of questionnaires, and self-reports. So I think my answer would have to be my own self-report of why, because I’m not really completely sure why I was captivated by the Rorschach.
I know I like visual imagery and understanding things. When I’m stressed, I’ll create something visual to help me. And so I think in imagery. But it really was what Greg, my husband now, calls seeing what’s called personality and action. You see what people say about themselves, but then you see what they do and what they say and how they interact with you [00:13:00] on the Rorschach and if you just got test results and you tried to put them together as a person without ever meeting the person, it’d be harder to know how all this hangs together. And so you got to see a lot of things in action.
Also, I knew that there was a need for somebody in the field to both have the research interest and knowledge and also be very tenacious. The Rorschach is really difficult to do research with and to teach it and to learn it. And to also have a balance between being able to conceptualize psychologically and being able to understand the research behind it. I don’t know. I just at one point decided this is where I feel like I need to be. This place [00:14:00] needs me here. At one point in my early career, I said, where will I be best use of my tongue? I picked. I really can’t tell you. Maybe there’s other reasons that I don’t know.
Dr. Sharp: No, I think that it just speaks to circumstances and time and place. Your interest gets captured and then it just…
Dr. Mihura: I found it was a challenge. I guess I would add, because there’s definitely people who have their own ideas about what the Rorschach is without knowing the research. Every now and then I have a little semi-trauma of somebody, an editor or reviewer or something, making some conclusions about what I’m doing that are completely wrong because they have an idea about what the Rorschach is. I also like challenges so that’s [00:15:00] maybe part of it.
Dr. Sharp: That’s great. Yeah, already just from the brief time we’ve been talking, you strike me as a person who’s got a little bit of fire in your personality to rise to those challenges.
Dr. Mihura: Yeah. Where am I? Yeah, I am that way.
Dr. Sharp: That’s great. I’m glad you say that right out front because in our discussions as we were scheduling the podcast, I was pretty clear that I do not know a lot about the Rorschach. I went through graduate school at a time when I think it was fairly out of favor and had an advisor who was way down.
Dr. Mihura: Oh, it still is.
Dr. Sharp: Okay. I’m coming at this from a place where a lot of curiosity but also a history of just being down on the Rorschach. So I’m just excited to hear more.
Dr. Mihura: Yeah. And [00:16:00] also because of that, I was really glad your open-mindedness of inviting me to the podcast and your own curiosity and interest in things. So I also appreciate that.
Dr. Sharp: Absolutely. Well, I know I’m not the only one, and I know there’s a ton of discussion about performance-based measures and Rorschach in particular in our Facebook group. It’s super valuable. We got to know about it, right?
So from here, these days, are you putting much all research and development and talks and presentations? Are you doing any clinical work at all at this point?
Dr. Mihura: Yeah. I do supervise an assessment practicum. Right now, we just started last week and picked up a client. So I do some supervision. I used to have a small part-time practice and because of R-PAS, that wiped out any extra time I have. I really do like [00:17:00] seeing clients. I like things that are challenging. I like challenging cases. So yes, I have currently two pro bono cases.
When I was in graduate school, I had a three year NSF fellowship, and I was so grateful for it that I told myself I would always want to give back. And so I would always see clients who couldn’t afford therapy. So I’ve always, since day one, done at least at one or two pro bono clients that have multiple life problems that need a lot of help.
Dr. Sharp: That’s fantastic. Very cool. Well, I wonder if you’d be willing to just trace a history of the Rorschach just to start for anybody, well myself included, who may not be well versed in the history and all the iterations and developments and how we got where we are today. And I [00:18:00] know this could be an hour’s long conversation but just a basic history of where it started and how we got where we’re.
Dr. Mihura: Right. That could be a long time, but I’ll try to stay focused on not yet too tangential. Rorschach is called the Rorschach because of Hermann Rorschach’s last name. I wish everybody could see what I’m showing you right now. This is a coffee mug with his face.
Dr. Sharp: I love it.
Dr. Mihura: He was a pretty handsome guy. He looks a little bit like Brad Pitt. So anybody who’s wondering what he looks like, if you just google Hermann Rorschach, you’ll see a lot of pictures of him right next to Brad Pitt because they look really similar. It’s pretty funny.
Hermann Rorschach created inkblots but there are also many other inkblots. Some early inkblots by [00:19:00] Benét were created thinking about assessing creativity and there’s other inkblots that have been used, but just to try to keep this short, Hermann Rorschach’s inkblots have become the most popular. If you see some of the other inkblots, they don’t evoke as many images. Whatever he did, and he died a year after his test was published, he died of appendicitis. Anyway, that started almost now, 100 years ago.
Dr. Sharp: Wow.
Dr. Mihura: And so those inkblots have then after he died. Many people tried to create their own system. So there became a proliferation of systems. And they were called Rorschach Systems by people who were called Rorschach Systematizers for whatever reason. And so [00:20:00] even in other countries, there are like Crawford approaches still popular in a lot of countries. And Crawford was before the more popular one in the United States, John Exner.
Dr. Sharp: Okay.
Dr. Mihura: So anyway, I’m trying to make this short, there were five popular systems in the United States. And John Exner wanted to… People were using parts of tests and the scoring was done a little bit differently. They combined things instead of just choosing one. To make a long story short, Exner then compiled the best parts of the tests and collected norms on them, created scoring guidelines that could be scored reliably and helped people doing research that would all feed to one system. But there were still other people using other [00:21:00] approaches.
There was a time when the Rorschach was more popular. I wasn’t around then so I really can’t give you like a firsthand account of it but there became this time that it became associated with psychoanalysis. And so if you read undergraduate textbooks, you’ll see a lot of erroneous statements about the Rorschach. And one of those is that it is a psychoanalytic instrument.
I think people just knowing that in trying to move forward in our field, people became associated with a really outdated method. And so sometimes people will say the Rorschach as if they know what they’re talking about, but they just mean an outdated thing. Lots of people will say too, like, “That thing is still used? I didn’t even know that was still around.”
And so about the same [00:22:00] time, the challenges were coming in with behaviorism and CBT was coming more popular. It also came time where there were a lot of managed care companies came in and shortened the amount of money that you got for assessments. So assessments that are performance assessments that include the Rorschach but also the TAT, but even like the WISC used to be much more popular. Intelligence tests were given, especially in hospital settings, psychiatric hospital settings.
And then at about 1995, the people who referred to themselves as the Rorschach critics; James Woods, Scott Lilienfeld, Howard Garb, started writing a series of articles and doing a lot of interviews with New York Times, with a lot of places [00:23:00] challenging the Rorschach. They wrote a book called What’s Wrong with the Rorschach, popular press book. And in 2003 that was published. And so within a span of about 10 years, a lot of negative press came for the Rorschach, so much that really people couldn’t keep up with it. You couldn’t even read it all. Even if you read in the area, it’s like every time you turned around, something was being published.
Dr. Sharp: What spurred that? Do you know of why then and why…?
Dr. Mihura: I didn’t know it, I think, my own ideas, but I won’t put that in the podcast. I don’t know. In some ways, people who were thinking they wanted to be the bastion, so they wanted be like, we are about science and we’re things that are pseudoscience. Then for some people they chose the Rorschach as being an exemplar of pseudoscience, is what it’s been referred, which that’s not even an accurate use of that term, but so it came to [00:24:00] represent that.
Dr. Sharp: Yes.
Dr. Mihura: And during that time, I think of what a lot of people did was hear all these criticisms and knew the criticisms, and because they also didn’t understand what the Rorschach was, it just seemed silly, hokey, like Joni, what is that? How can you tell from images? So most people think it’s just what you see. It’s like, oh, I saw a picture of my mother or something. And it what you see as part of it, but it’s also how you describe it, cognitively complex your images are, but also it’s a measure of a thought disorder.
The one thing that the critics have always said that I think gets lost in the Rorschach is that they have always supported it as a measure of psychosis. So there are many quotes of them saying, it has strong support and it now has strong meta-analytic support with two different meta-analyses. [00:25:00] But that got washed away and people just think the Rorschach.
But for thought disorder, it’s like a speech sample. You’re using a standardized method to collect and record verbatim speech, which can then be coded for thought disorder. And there is what’s called the Thought Disorder Index out there that’s used in a lot of studies in Schizophrenia Research, Schizophrenia Bulletin but they don’t use the word Rorschach because it’s collected with administering the Rorschach but it’s just scored called the Thought Disorder Index.
Like where we are now is that it would most likely be that everything would be on a downhill turn, except I ended up doing what the Rorschach critics said had to be done before they would release what they called their moratorium on the use of the Rorschach. And they wanted a meta-analysis on every scale on the test of which there’s 65. [00:26:00] And usually meta-analysis on test variables only do one at a time. So anyway, it took me about 6.5 years, over 4,000 hours.
Dr. Sharp: Oh gosh.
Dr. Mihura: I put it all into one article. I still can’t believe I did that. It was crazy. And it got published in Psychological Bulletin to my surprise, which I never even thought I’d get published in that journal. And I felt like people opened the journal and saw the Rorschach in there and went, wait, have I got the right journal? So I was really grateful that the action editor was open enough to like look at the science instead of immediately dismiss the idea because of the Rorschach. So I was very grateful for that.
And then two years later, the critics in response to it said for the scores that were found to be valid, they lifted their cult moratorium on the Rorschach. They had still some other [00:27:00] criticisms. One of them had been about norms, but now we have new norms. The path of the Rorschach I’m mostly familiar with when I’ve been part of the game and before that I don’t exactly know, but can speculate about the reasons why.
Dr. Sharp: Yeah. Well, it gives me some historical context too. I started graduate school in 2003 and just hearing what you described, that whole period, it sounds like that was really…
Dr. Mihura: You were right in the middle of it.
Dr. Sharp: Yeah, the criticism and that all of a sudden makes a lot more sense.
Dr. Mihura: Yeah. You’re right.
Dr. Sharp: Timing-wise, when did that huge meta-analysis come out that you wrote? When did that fall in this whole thing?
Dr. Mihura: Just to back up, I’ll say I started it in 2005, literally because I wanted to be teaching my students what was there. I knew they were using it with their [00:28:00] clients, and I was just doing it for my class initially. So it was a long progress. It came out online in 2012, and then it came out in print in 2013.
Dr. Sharp: I see. Wow.
Dr. Mihura: I presented it early on in like 2008.
Dr. Sharp: My gosh, I have so many questions with that. Where do you even start? When you’re at the beginning of a project like that, how do you even figure out where to go first and how to tackle such a huge…?
Dr. Mihura: It’s a good question because I think in anybody who does meta-analytic work, your goal is to summarize the literature, but you also don’t know exactly what’s out there yet and the challenges that you’re going to encounter when you do the meta-analysis. So I always feel a little bit like, sometimes people with meta-analysis can look at the literature first, and it’s like looking at your raw data and making some decisions first, and then doing the meta-analysis.
[00:29:00] The scales on the Rorschach, it’s complicated. The scales on the Rorschachs, they’re in some ways like scales on intelligence tests and neuropsychological tests. And probably just as soon as I say that people who are in neuropsychology will think, “What, how is that possible?”What I mean by that is that you are asking the person to perform a task and based on their behavior and what you believe is operating at the time in which they respond to the task, that response process leads to your interpretation of the task. And there are many parts like Block Design. You interpret it based on mental abilities, but also there are things like how well people’s fine motor skills are, and their vision. There’s a lot of anxiety things that go into it.
A lot of the Rorschach scores have names that are like those on [00:30:00] say the WISC. We don’t care about people’s ability really to put together blocks. Some of the tasks that we have them do, that we try to make very basic, you’re trying to assess something else. And so, they have names that aren’t the names like attention concentration or working memory and the Rorschach has similar names like that.
So what you have to do is say, what do we think that this assesses in particular? And based on your description of the construct, what would you expect then to be associated with it as validity criteria? And you can either span out and say, well, this could be slightly associated, but then you get a small effect size or really targeting the construct, you would expect a larger effect.
And so initially what we had to do was come up with definitions of these scales that were a construct that everybody could say, okay, I know what that is. It’s not shading or [00:31:00] cognitive scores or achromatic color or something like that.
And then the second thing that you encounter is, first scales that are, well, I should say for tests like the MMPI and the PAI and the Rorschach are multi-scale tests. You get a lot of research studies where someone has just thrown in the kitchen sink and they’ve looked at comparison between groups like say ADHD and controls on all of the scales. Or psychosis and controls and all of the scales.
And so then from those articles you have to say, which of these are really targeting the construct of the scale? Some of these aren’t, they’re done to use the test validly to assess these groups, but you can also say, you would expect certain relationships for the test itself, for testability. So all of that had to be taken into account about how do we [00:32:00] tackle this?
That was huge. Plus you had to prove you weren’t biased in your selection. And to explain all that, you’d really need to read the methods section of the Psychological Bulletin. Sometimes, I never thought I would be able to figure out a way to make this work. And I did. And so some of it was a little bit of creativity about how do I tackle this problem because this is not a common thing to do what’s called construct validity meta-analysis.
Usually, there is criterion validity meta-analysis. You know the criterion you’re not looking for all the studies that you think would assess this broader construct. So like for attention concentration, you might expect people who are psychotic, but you’re not measuring psychosis, but you would expect that to be lower in that group. It also means having to know a lot about a lot of disorders, and a lot of different areas of research to understand what the [00:33:00] construct is. I might be making that too complicated.
Dr. Sharp: No, I think it’s good to hear about that process and just to recognize what a massive undertaking and the meticulousness that you have to employ too.
Dr. Mihura: Yes. And the systematicness. This is one of my favorite words. I had to be very systematic and detailed. I’m like, how many people have this ridiculous brain that I have that likes that stuff? So, I enjoy it.
Dr. Sharp: I’m glad somebody does. That’s good for you. We need it.
Dr. Mihura: But for any graduate students out there, undergraduate students who might be interested in this area, let me know because we need more people working in this area who like challenges like this.
Dr. Sharp: Yeah, absolutely. Well, I’ll ask you before we take off about how to get in touch with you and if there are folks who want to reach out. So we’ll [00:34:00] put that in the show notes for sure.
Dr. Mihura: Okay.
Dr. Sharp: Let’s see. Could you talk just a little bit then about the transition from, and this is likely a naive question, I’m going to ask a lot of those, but from Exner System to R-PAS, what the evolution there is and what we’re even looking at on a very functional level, like how does that change what we’re doing with the Rorschach and how we scored and all that kind of stuff?
Dr. Mihura: In general, four of us who were at the R-PAS, were on the research council for the Comprehensive System, and we already had things that we were doing with Exner that, and talking about changes of the system, that were in play that you can see in his newsletters at the time that he was planning on doing that he didn’t do before he died.
And so it’s not like there was a [00:35:00] test and then somebody developed a different one. This is more like a fast-forwarded Comprehensive System to R-PAS. But some of the things that we did that were different and were addressing challenges, one of the criticisms for years on the Rorschach is it’s challenging psychometrically because you don’t always have the same number of items per scale. You have a different number of responses. And on some of those responses, what Rorschach systems were doing were saying, for example, how many thought disorder scores did you have? But if you had a really long protocol, you might have more just by length of what’s called number of responses. So the number of responses on the test because of that had been an issue for a while. So it’s one of the first things when I joined the council that we were talking about, how to tackle this problem.
[00:36:00] In short, what we decided to do was to ask for two, maybe three responses per card. Previously, what Exner did was, he didn’t give any instructions about that. And if the person just gave one to the first card, he would ask for another one because there are 10 cards. But you also need a big enough sample of behavior to be able to have reliable results. So you need a large enough sample, but you also don’t want too large for one thing.A 50-Response Protocol takes up a clinician’s time. How do you schedule that? So just practically, you don’t know how long you schedule your administration for. And so we decided to do this two, maybe three. Well, and we discuss that a lot. And I was like, no, just two maybe three. And interestingly enough, people have one of the [00:37:00] strongest reactions of people who use the Rorschach of us changing that.
Dr. Sharp: Wonderful.
Dr. Mihura: A lot of people really liked it, but then people who really felt like there was something special about one being able to allow the patient to do whatever they wanted like what about that fifth response? What if it was really important? Maybe it could have been, but also that’d be like if that happens so rarely in a person’s life, and you only get one of something out of 50 something, maybe it’s particularly important. Personally, I don’t know for the person.
So we changed that. And what that did is it reduced the amount of variability in a lot of the scores because it’s just of the number of responses you’re reducing variability. And in the first study that was published by Dean, it was a study that was designed with Exner to look at using, [00:38:00] we modified the instructions a little bit, but an early version of the 2, maybe 3, we didn’t use to give a reminder if somebody gave one. Now we give a reminder. She did it with some criterion variables in a very severe psychotic group or very severe disturbed group and showed that using this method of administration, you had stronger effect sizes with the validity criteria than otherwise. And since that’s also been shown in some other studies.
One of the other things we did was we excluded some of the variables that didn’t have support in the meta-analysis. The Comprehensive System now is in its 5th revision, but it’s 2003 and it can never change. So it’s just going to stay that way, and it will always end with the same variables forever unless the family changed their mind at some point and decided to revise it, but it doesn’t look like that’s [00:39:00] happening.
Dr. Sharp: Sorry, could you talk about that just a little bit, this distinction between what the family is doing and has control over and the diversion that y’all took, just to make it clear for the audience?
Dr. Mihura: They have what’s called the Comprehensive System.
Dr. Sharp: Yeah. And that was Exner’s.
Dr. Mihura: It’s a little bit like the MMPI situation where now we have from the same pool of items, you have two different tests. So the MMPI-2-RF takes from the same pool of items from the MMPI-2, but we have two different tests and different scales from it and different people who are developers of the test. And even though a lot of people think like the RF has now taken over, the MMPI-2 still is used by a lot of people. And so you have two tests then. It’s a little bit different for us because the developer here died. [00:40:00] Here we still have both developers with MPPI, who are living and supporting their tests.
Dr. Sharp: Right.
Dr. Mihura: And I’m not really sure if I’m answering your questions. There maybe something I’m not realizing aren’t clear.
Dr. Sharp: No, I just wanted to make that distinction clear that when Exner died, like you said, the family, I don’t know if retained control is the right thing to do, but they sort of owned.
Dr. Mihura: So he wrote in the will to leave to the family.
Dr. Sharp: Yes. And whereas then y’all who were on the research council with him then broke off and I don’t know if evolved is the right word, but took what was going on and developed this separate way to administer and score.
Dr. Mihura: Yes. We were shocked that it wasn’t left to us. He had said he was leaving it to us and he didn’t. We tried to continue the development. [00:41:00] In some ways it’s probably better because we were able to make decisions on our own.The family wouldn’t have known what to do. That’s what they said. They don’t know what he would’ve wanted. They’re not psychologists. And so they didn’t know what to do. And they didn’t expect this either. So it was a really weird situation. And his son tried for a while to, like he was on the side of like, yeah, let’s develop it more. And the wife Doris was like, oh, I don’t know what to do. And so it was really complicated.
Dr. Sharp: My gosh. Yeah, it sounds like it.
Dr. Mihura: And so right now, if you look at the results of the test, it’s even different because we have results reported in standard scores and it’s plotted in standard scores whereas, the Comprehensive System, you have to look up every one of the normative values and compare it to the raw score. There’s not plotted in standard scores. And some people have all these scores memorized, like, oh, [00:42:00] there’s an average of eight human movements or whatever useless knowledge to have that you had to have in your brain.
So if you look at the test, it even looks different. You have standard scores. We’ve got page 1 and page 2. Page 2 is like not as much support, we’re still working on it, but you can use it. Page one is the ones that have the strong solid support.
Dr. Sharp: I see.
Dr. Mihura: There’s differences. Even if you look at the test, it looks different to test results.
Dr. Sharp: Yeah. Are there any other major differences that you can speak to without giving away too much about the test results?
Dr. Mihura: Well, like I said, different scores.
Dr. Sharp: That’s a big one.
Dr. Mihura: There’s probably things that I’m forgetting that would stand out. The norms; different norms. [00:43:00] The norms in the Comprehensive System are clearly off, and we didn’t think so for a while. Enough studies have come out to show that the scores, especially the ones for psychosis, are over-pathologizing people. They’re too healthy. It turns out Exner did not score like his manual did. So his scores look a lot healthier. He’s scoring things that have better reality testing way.
And so we have now norms that are not pathologizing in those ways. And so we recommend that if people do continue to use the Comprehensive System, they use what are called the International Norms that were published in 2007. And then that they also use the meta-analytic results to see which scores are valid. In doing those two things, you can argue that [00:44:00] you’re doing what’s known to be best in the literature. You’re still going to have maybe a wider variation of number of responses and so some errors going to come from that.
Dr. Sharp: I see. Goodness. So here you are. You’ve been working on this now for, I don’t know, 8 to 10 years, longer than that maybe. Well, for people who were maybe trained in the Comprehensive System, how do they make the leap if they want to make the leap to R-PAS. How does that happen?
Dr. Mihura: Well, we have a lot of trainings. People who can join webinars. If you look at our website, r-pas.org, you can see all our workshops. We’ve given lots of workshops. Some people feel like they can just learn it by reading the manual. I have put together some [00:45:00] recommendations for people who are making that transition but we also have two free hour and a half webinars on our website for people who have an account.
In whatever country you are in, be the level of having a license to have an account and once you have an account, which is free, we have a lot of helpful training information there that’s free.
We have free webinars for people making the transition who are doing practice. And then I’ve done a webinar for people who are teaching and supervising and making that transition. We have a thousand PowerPoint slides for instructors to choose from for free. We have two videos in there of administration. We have 10 webinars discussing through all of our scoring examples.
[00:46:00] We’ve got packets where people can practice administering with each other; one person role plays and reads the responses if they’re the person taking the test, the other person is examiner. And then you can also have a coach. It’s also made for three people to be watching and helping out. And so people can learn the administration that way. We have a really detailed 10 page checklist you can use for review before and after you do an administration. For people who are collecting remediative data, we have proficiencies that we give people who we evaluate their administrations for proficiency. Everybody collecting norms has to have that done. And then also for scoring, they have to take a test for proficiency. So we have a lot of materials designed to be helpful.Dr. Sharp: That’s amazing. We’ll definitely link to all of that in the show notes.
Dr. Mihura: Okay.
Dr. Sharp: So what about someone who, like [00:47:00] myself, has never taken class on the Rorschach, really starting from ground zero and would like to add it. How does that happen?
Dr. Mihura: So just like any new test that comes out or a revision of the test, sometimes people decide to just do it on their own, right?
Dr. Sharp: Sure.
Dr. Mihura: But we also have workshops that are for people who have no experience at all. So we have a four-day workshop that we give and people come who have no experience with the workshop whatsoever. And so we start from the beginning. And we’re also going to be having some in-person webinars training coming up. I don’t know when that will be, but sometime in the next year, we’re going to start having an in-person webinar for the people who would need to travel otherwise. We also thought about using a model that where you do the first part [00:48:00] didactic online and then come in person for the applied component.
Dr. Sharp: Great. It sounds like there are resources out there.
Dr. Mihura: Yeah. And for anybody who’s making the transition or just learning, we also now have some recommended steps that you do. They’re somewhere on our website, I can’t keep up with everything. We just came up with that recently. I had one that I gave everybody, and then we came up with one that was more formal. So if we have two people, we have a business manager and an information person that you can also email and ask them for copies of things. We’re happy to give people these steps instructions of things that you can do to make sure you become proficient.
Dr. Sharp: Yeah. I think that that’s helpful for people because there are, I’m sure, many folks who would just jump in and read the [00:49:00] manual and try to do it, but that sounds like a huge undertaking and […]
Dr. Mihura: And on a ResearchGate, there’s the first chapter of our case book through Guilford that came out in 2018. Myself and Greg are editors. There’s a free chapter for people who want to just have a short introduction to R-PAS on ResearchGate. And because it’s free from the publisher, it’s one of those you can have open. No one has to request it. You can just download it from ResearchGate for free if somebody wants just a brief introduction to it.
Dr. Sharp: Yes. Okay. I’ll also link to that. Very cool. I might be going out of order here a little bit, but I think we have to talk about the clinical utility of the Rorschach at this point, like what is it helpful for? When would we use it? What can it do?
Dr. Mihura: Yeah. So I would say that people have different [00:50:00] recommendations depending on their view their clinical, the way they conceptualize. You would think as a test developer, I might be more open and come up with a lot of different ways that can be used, but I’m a little bit more picky. I think, than some of the other people who might recommend things.
In general, if you think that somebody has either an emerging psychosis or has some problems with their thinking by either schizotypal symptoms or borderline confused thinking that may also be wrapped up in some morbidity and damage objects that they see. So for assessing any range of thought disturbance that you think that somebody might have, it’s definitely good for that.
And I would say, even though you would can say that I’m biased, it’s the only normed [00:51:00] test of psychosis- the performance test for thought disorder. And it’s also age normed because we know that kids are, they have more of what looks like it’s a thought disorder, of course developmentally their thoughts just aren’t as coherent and logical. And so you also, for younger folks, you need to have those norms.
Also for a person’s reality testing and that can also be wrapped up in maybe some schizotypal or paranoid symptoms where people are seeing the inkblots. And it’s one of the other main scores other than thought disorder, misperceptions on the Rorschach when people see images that really just don’t look like the thing and have some problems with reality testing.
Dr. Sharp: Yes.
Dr. Mihura: There’s also a woman that I’m working with now at the University of Amsterdam, that they’re working with a, it’s [00:52:00] almost you could think of it like a auditory Rorschach where they place sounds that are somewhat like white noise then people that are more likely to hear things, others misperceptions.
Dr. Sharp: Oh, that is fascinating.
Dr. Mihura: So this is more like the visual version of it.
And I would say, for someone’s ability to healthy represent themselves and other people, when you have people who will represent human representations that are damaged, that are aggressive and bleeding and really scary stuff, and so, you know that just in general, these are the kind of images whether they’re aware of it or not, they’re filtering their understandings of human interactions because they’re tending to see these things on the Rorschach.
It is also true that some people can fake these things and I won’t go into the details about how to do that, but we do have [00:53:00] some a scale now that we’re developing that has a lot of research behind it that I started a meta-analysis, but I’ve just been too busy with lots of things, but there’s a scale for detecting malingering of psychosis on the Rorschach.
And we also have comparison groups from the Center for Forensic Psychiatry and people who actually do, they’re there for guilty for reasons of insanity or incompetent to stand trial. And there’s several faking studies of people trying to present themselves to see what they do. Present themselves as psychotic or crazy however they’ve been instructed to do it.
I would also say the other thing is what you get with a person who has taken the Rorschach is you get a sample of how well they’re able to describe their experiences with the assessor. And so there’s been studies looking at it. It’s two of them, [00:54:00] I’m an author and one of them, Greg’s an author on one of them, looking at the relationship to Rorschachiana.
So basically what someone’s doing on the Rorschach is you get a sample of them being able to tell you why it looks that way and what it was about what they saw; what was on their mind and what led to the response because you’re asking them, tell me why it looks that way? What about the inkblots? So they have to say, color, shape, shading, whatever they thought it looked like. It was moving and whatever. And so people who have like Alzheimer are really flat on that. It’s just like, well, here’s that wing. And they don’t even realize a lot of times that what they’re seeing is not really the image of that. It just kind of concretely, well, that’s a wing and that’s ahead.
And so you get the ability to see how well can somebody work in a kind of therapy where they are going to have to be doing this thing, noticing what is on their mind and what led to their reaction or their thoughts, and to also work collaboratively with another person in therapy. So somebody who [00:55:00] can work in a more complex therapy where they’re working on processing their emotions, and being able to take a reflection back and think about why they thought about something or what does the psychologist need for them to know what it looks like for you so they can take some type of perspective.
And so mentalization, there’s actually really interesting several studies that have looked at the mirror neuron system in relationship to human movement responses on the Rorschach. And there’s two groups of people in Italy who’ve worked in this area who’ve done like EEG and FMRI and they’re also doing eye-tracking research. And so they’re doing a lot of the brain behavior research with the Rorschach showing that people who have, how likely is it this person is going to be able to mentalize.
And so one of the things we’re really looking for more help with research is [00:56:00] with the Autistic Spectrum Disorder, because you also get some odd thinking and some words that aren’t real words. So far the literature that I’ve reviewed, the one thing that stands out that’s the difference between the thought disorder of psychosis and of Asperger’s or Autism Spectrum Disorder, depending on when the study was done, is the logicality and thinking is more common for people who are psychotic. It’s more related to delusional kinds of thinking.
Also people use the Rorschach for lots of other reasons, but those, I would say, are the main reasons that are really supported by the literature and that I think would be really helpful for either diagnosis or doing psychotherapy with the person.
Dr. Sharp: I see. So just to recap a little bit, the [00:57:00] research support, it sounds like is pretty solid for psychosis, thought disorders, suspected disorder thinking, like you said, borderline thought processes, reality testing, so forth. And maybe, this might be the wrong term, but just gauging insight or intuition or ability to…
Dr. Mihura: Say psycho mindedness.
Dr. Sharp: Yeah.
Dr. Mihura: Social cognition as well. So understanding of yourself and other people.
Dr. Sharp: Yeah. I wanted to ask… I’m glad you brought up the autism issue. I just wanted to ask explicitly if there is some clinical utility there that’s born out there in the research so far, that is it helpful in diagnosing autism versus distinguishing between…
Dr. Mihura: One example that I gave you has, this is an [00:58:00] area of much needed research and people have asked this question many times and I have even put it out there to our users that we need some, if you have a sample of people who have taken the Rorschach, even the Comprehensive System because it can be scored by R-PAS. Like if you could share your data or do a research study yourself, we are looking actually for research in this area. There are studies out there in this area and one of the things that I just summarized as the distinction between the two is this illogicality, it’s called peculiar logic on the Rorschach. Illogicality on some other disorder measures is more likely to be a psychotic process.
Dr. Sharp: I see.
Dr. Mihura: I would also say, although there’s not research in this area, a particular [00:59:00] benefit would be to able to see if the person was able to accurately mentalize other people with the human movement responses that fit the contours of the block that are based on reality testing. But that’s just hypothetically. I don’t know of research that’s looked at that, but it’s definitely something people have expressed interest in. And it would be really great if we could have more. Like I said, there are four studies out there now that can be summarized, but because they used to use different samples in different ages like ones with children’s illness and two with adults, you are going to get different levels of thought disorder and whatever. So we need more studies out there.
Dr. Sharp: Got you.
Dr. Mihura: And for anybody listening who has any sample or knows somebody in your practice or something that could help us out, let’s share.
Dr. Sharp: Sure. Well, that might be a nice time to ask what the best way to get in touch with you is if [01:00:00] folks want to do that for any assistance.
Dr. Mihura: Yeah. Sure. I should also say there’s a forum on our website for people who are users. It’s a very active forum. We have research assistants who answer scoring questions and everything on there. So sometimes people will ask questions on that forum. It’s like a broader question, but if you want to contact us about, there’s also a research section in there, but if you want to contact me you can do so with my Gmail account. It’s jlmihura@gmail.com.
You can also contact the info person on our website. If somebody feels like, they’ll be like, oh, I’d rather not contact you, I’ll go through your help desk. That’s fine too. It’ll eventually come to us , so you might as well just contact me. Be happy to talk with anybody if they have any ideas about [01:01:00] research or ways they want to help out or questions for me about doing research.
I also am open to people who are wanting to consult for research study. If you consult about the design or questions about it, I’m happy to do that. If it became really complicated, I would want to charge for it or become an author, but initial consultations would be for free.
Dr. Sharp: Yeah. Certainly. What other resources might be helpful for folks if they want to learn more about the R-PAS, or I know we talked a lot about training, but are there any other books or websites really?
Dr. Mihura: It’s the case book I mentioned. So that was published by Guilford in 2018 and it’s also for sale on our website. I think it’s like $40. You asked what ways can the Rorschach be used? That case book is set up [01:02:00] with chapters in which either the Rorschach is helpful to be used in this kind of setting and how would you use it in this setting. So we have an emerging psychosis case in there. We have a neuropsych case with an adult, a neuropsych case with a child. We have a school case for emotional disturbance in there.
We have a violence risk case and we have a psychopathy case in there, but I will say, the Rorschach is not set up to predict these things. What was being looked at in these cases in particular was some thought disorder. We have a female psychopath in there that has some instances of thought disorder. She’s not schizophrenic but it’s more of a borderline process, but some pretty scary things like snakes coming out of people’s chests and gross, scary, odd [01:03:00] things. So, if people want to check out that book, we tried to write the chapters in a really concise way so they’re not arduous to read. They’re pretty quick reads on each one of them.
Dr. Sharp: That’s great.
Dr. Mihura: Yeah. And the resources, like I said, the forum, the discussion list online is good.
Dr. Sharp: Okay.
Dr. Mihura: If you are a member on our website and you have an account, we have a whole lot of articles that you can get by request. It’s really an automated request. We don’t have them posted there but you can send a request and you get an email with the article. And online, if you’re not a member, you can look this up, you can search for things, but you can just see the abstract.
So we’ve posted especially for people in practice who don’t have access to free pdf [01:04:00] through the university. People in other countries who they might, their academics, but their university doesn’t have really expensive packages of journals that they can download pdf. Particularly, we put that there as a resource for people so they could access these articles about our past, but also articles that are preliminary to our past that fed into our development of our pasts are there. And we’ve got it set up with a search where you can look for key terms or certain types of articles, like forensic articles. For people who work in forensic settings, we’ve got some articles there about using R-PAS and forensic settings, for example.
Dr. Sharp: Yes. Great. So any exciting projects coming up? Any research you’re doing that you’re really in right now?
Dr. Mihura: Well, right now I am creating a new [01:05:00] area of expertise for myself, which is like a lot of reading. I feel like a graduate student again and don’t know what I’m doing. And the interested, a little insecure about my knowledge, so I’m developing an interest in emerging psychosis. I have two students who are working with me now who just joined to work in this area.
I submitted a grant, federal grants, who knows how that will go, but the grant is not just about the Rorschach, it’s about assessing emergency psychosis and thought disorder. But we have several articles in the pipeline on using the Rorschach with thought disorder. Three are under review now and one is revise and resubmit. And we’re working on revisions.
We’re also working on developing a short form of assessing psychosis. So this would be [01:06:00] able to use a 4 card set to assess psychosis only for reality testing scores, which are called form quality scores and the cognitive scores which are the thought disorder scores. They are very similar to the scores that are like on the thought and language and communication, the scale for thought, language and communication in the TLC.
And so, what that means though is that the administration of the test is much easier because you don’t have to, for people who use the Rorschach, they’ll know what I’m talking about when I say you don’t have to follow up an inquiry on all of the determinants and why people saw it the way they saw it. You ask them questions to get a speech sample, but the people administering it, my goal is so that eventually it can be used in settings where people don’t have to be, they can just be like a psychometrist training. They don’t have to be a psychologist and they can just know how to administer it. We’re working also on a text to speech so [01:07:00] that could help in recording the speech.
And so my goal is that it can be used in settings where people have emerging psychosis: a college counseling center, maybe a high school counselor, where they can collect the speech and then maybe send it off to be scored by a freshman. We have 3 research assistants who do scoring for less than an actual cost of a psychologist to score it.
So that’s an exciting new area is that there’s a lot to it, a lot of steps, but we do have the first article that was published in 2018. It’s called A Thought and Perception Assessment System. And we’re also working on a dimensional scale for scoring thought disorder. That’s one of the articles that’s under review right now.
Dr. Sharp: Wow, that is exciting. I just, well, not just, I guess it’s been a few weeks [01:08:00] now, but I talked with Michelle Friedman-Yakoobian over at the Cedar Center in Massachusetts. And they do a lot of work with emerging psychosis and early onset psychosis. I wonder if y’all’s paths may cross at some point.
Dr. Mihura: Yeah. I’ve just started this. I’m just submitting some stuff to present it in a conference in Florence in the spring and so I’m looking forward to meeting people. Our paths will cross. I was in Boston for my internship.
Dr. Sharp: Oh yeah. Very cool. Small world.
Well, let’s see, we’ve covered a lot of ground, Joni. I admire how you can succinctly describe a lot of topics that could take hours, right?
Dr. Mihura: Oh, good. I’m glad because I feel like I’m talking too much.
Dr. Sharp: Oh no, this is great. I can’t imagine how to wrap all of this knowledge into a [01:09:00] small package.
Dr. Mihura: Well, I appreciate you saying that because it is one of the things I strive for as a way to not use Rorschach language and sound like you’re part of a Star Trek conference or something giving a bunch of sounds and terms that nobody knows what you’re talking about. I find ways to put things that really just make sense to people.
And I think one of the examples of that is the thought disorder scores or what’s really disordered thinking or disordered speech on the Rorschach. The names of those scores have been used for so long. I didn’t even realize what they were at first. They were called […]. What the heck does that mean? No one knows what that means. And so I have just written a manuscript now that I’m submitting for publication describing what these scores are. Disordered thinking is what they are. They’re all those scores, but they’re called a funny name.
So it started occurring to me how there’s a lot of things [01:10:00] that we’re doing that you could really put in a language we should be able to communicate it with other people rather than use these odd names of variables.
Dr. Sharp: I like that. Simplify. Be direct. Well, thank you so much for your time talking with us. I know that we could go into great detail on any number of the topics that we covered today, but hopefully, this is enough for folks. I approached it from a naive perspective just to get a sense of where things are and where they’ve been over the years with the Rorschach and what y’all are up to and how people can get more involved if they want to.
Dr. Mihura: Yeah. Thank you. I appreciate that. I really appreciate you inviting me to do this. I hope you know that whoever listened to it finds it interesting. But I appreciate the opportunity.
Dr. Sharp: Of course, I have no doubt that it will be helpful, so don’t be surprised if you get some inquiries.
Dr. Mihura: Okay. No [01:11:00] problem. That’s good.
Dr. Sharp: All right, Joni. Thanks so much.
Dr. Mihura: All right, thank you.
Dr. Sharp: Hey, y’all, thanks again for listening to the interview with Dr. Joni Mihura. There’s so much more to say on this topic, but we just barely scratched the surface. She gave a lot of resources that are contained in the show notes. The R-PAS website has a lot of great information on it, especially if you register for an account, you can get access to any number of resources that can be super helpful. So check that out.
Thanks for listening as always.
If you have not taken a moment to rate and review the podcast, I would love for you to do that. That helps get exposure in iTunes and elsewhere, and it just helps more folks discover the podcast, which is great.
I will say this in a moment of vulnerability, I have recently found that I’m getting some one-star reviews in iTunes, but none of those [01:12:00] folks are leaving a review. So if you are out there and you left a one-star review, please get in touch with me and let me know what led to that one-star review because it’s keeping me up at night folks.
All right, y’all, take care. Stay tuned. Subscribe if you haven’t subscribed. There are some great interviews coming up and I would hate for you to miss any of those.
All right, till next time.