Dr. Sharp: [00:00:00] Hello, everyone. Welcome to The Testing Psychologist podcast, the podcast where we talk all about the business and practice of psychological and neuropsychological assessment. I’m your host, Dr. Jeremy Sharp, licensed psychologist, group practice owner, and private practice coach.
This episode is brought to you by PAR. The BRIEF-2 ADHD form uses BRIEF-2 scores to predict the likelihood of ADHD. It is available on PARiConnect- PAR’s online assessment platform. Learn more at parinc.com.
Hey everyone. I am back with you today talking with a true pioneer in our field. Dr. Dave Corey is here talking with me all about the remote administration of the MMPI instrument. We talk about a number of things relevant to remote administration as well as some of Dave’s own practices in his clinical work. [00:01:00] We dip into the forensic realm a bit and how remote assessment plays into that. We just have a generally informative yet entertaining and interesting conversation. Dave is a great interviewee and made it really fun.
If you don’t know who Dr. Dave Corey is, let me tell you a little bit about him. He’s a licensed psychologist with more than 40 years of experience performing high stakes, civil forensic evaluations, and occupationally mandated evaluations of police and other public safety personnel. He’s the founding president of the American Board of Police and Public Safety Psychology, and a Fellow of the American Psychological Association Division 18. He’s board-certified in both forensic psychology and in police and public safety psychology.
Dr. Corey’s practice consists primarily of performing, testifying, teaching, and researching police officers’ suitability and fitness evaluations. He’s licensed to [00:02:00] practice in Oregon, Hawaii, Washington, California, Utah, Arizona, Wisconsin, and Ireland. With Dr. Yossef Ben-Porath, he is the co-author of the MMPI-2-RF and MMPI-3 Police Candidate Interpretive Report.
So chances are, if you work with adults at all, you have stumbled across some of Dave’s work and I’m guessing many of you know Dave’s work quite well if you work in the police and public safety realm. So I am truly honored to be talking with him and excited to bring you this conversation.
Without further ado, let’s get to my interview with Dr. Dave Corey.
Hey Dave, welcome to the podcast.
Dr. Corey: Thank you, Jeremy. Glad to be here.
Dr. Sharp: Yes. I’m so honored to be talking with you. I think a lot of people probably recognize your name. I’m just glad that we’re able to find some time and glad that you’re willing to come to chat with me for a little bit. I really appreciate it.
Dr. Corey: You’re welcome.
Dr. Sharp: I was hoping to lead with a bit of a random question, but a question that at least I’m interested in. I think others are probably interested as well. Looking over your CV, I see that you are licensed in maybe 6 or 7 states. I’m very curious. Why are you maintaining a license in so many states right now?
Dr. Corey: That’s a great question. I wish I didn’t have to, but when one is physically present in a state conducting evaluations for an agency, or even when one is conducting evaluations for an employer, police department, city, state, government [00:04:00] remotely, it’s important to check the jurisdictional requirements and the states in which I practice require that I have a license. So I’m licensed in California because I need to conduct an evaluation for a police candidate that an agency can rely on them in California, and the psychologist who conducted that evaluation must be licensed there.
Why I gained a temporary license, for example, in Wisconsin is that the requirement for Telepractice does not require you to have a regular license, but you must have a temporary license and you have to apply for that. And it has to be other specific requirements that are stipulated in the statute. So it’s really to comply with jurisdictional requirements that I think that’s really a great way to start out this [00:05:00] conversation, and that is that when one engages in telepractice, one has to think through the requirements that might be applicable to the state in which they’re working.
Dr. Sharp: Of course. Yeah, it’s not as much of a random question as it might seem. We’re talking all about remote administration and assessment using the MMPI-3 which is brand new.
Dr. Corey: Exactly.
Dr. Sharp: Yeah. Just to be as actionable as possible, how do you go about figuring out what these guidelines and stipulations are to determine if you need to get a license or a temporary license? Where is that information residing?
Dr. Corey: That’s another good question. It really varies from jurisdiction to jurisdiction. Just as one example, if you go to the Association of State & Provincial Psychology Boards, they have on their [00:06:00] website a table of requirements for each state and links to those requirements and they indicate whether or not there are any exceptions for the pandemic. Many states have waived licensure requirements for Telepractice during the pandemic, but not all have. Some have partially waived some.
As an example, Wisconsin, still requires that you obtain a temporary license to practice in that state. It’s another arduous process that in fact, they don’t even charge a fee for it. But when I read what ASPPB had published about Wisconsin’s requirements, it appeared to me that the waiver had been lifted and that one had to get a regular license.
So I pick up the phone and I call the Wisconsin board and eventually get a call back from someone because they’re also working remotely. And I was informed that [00:07:00] they had continued, the governor signed a new waiver, but there were some restrictions that had to be complied with and an application still had to be provided. So it’s some combination. I guess what I would say is to check the websites for the various state boards and then confirm that those are up to date by making a phone call.
Dr. Sharp: Fair enough. Yeah, that makes sense. So let me ask, for people who may not know you and your practice, what’s the nature of your practice that you are performing services in so many states leading to this licensure and maybe some remote assessment as well? How’s that all come about?
Dr. Corey: Sure. So our practice involves the practice of civil forensic psychology most frequently in the employment context. So it would be a disability evaluation, fitness for duty evaluation, [00:08:00] workplace threat assessment as opposed to criminal forensic evaluations, which I don’t conduct. So in addition to civil forensics, the majority of our practices are police and public safety psychology within the assessment domain as opposed to the treatment or intervention or operational consultation domains. In the assessment domain, our principle focuses on pre-employment psychological screening and fitness for duty evaluations of incumbent police and other public safety personnel.
And as I had indicated earlier, many states require you to conduct those evaluations for a psychologist to conduct those evaluations. If their opinion is to be considered by the agency as meeting the statutory requirements for the psychological evaluation, the psychologist who does those evaluations has to be licensed in that state. That’s not true for all states. Oregon, for example, can accept an [00:09:00] evaluation conducted by a psychologist not licensed in Oregon. I shouldn’t perhaps say that over the air. I’ll be getting a lot of competition, but that’s the rule in Oregon. It varies from jurisdiction to jurisdiction.
Dr. Sharp: I see. Fair enough. I won’t spend too much time on this. I know we’re really here to talk about the MMPI-3 and remote administration of that in particular, but I am curious, I know people are probably wondering there are a lot of forensic folks out there. Is that typical to be working with departments in other states or is that something that you’ve pursued or been offered outside the norm? I feel like a lot of people have local practices and to go across state lines is a little different.
Dr. Corey: I think it is unusual. Over the years, as my work has become known to other psychologists as well, [00:10:00] psychologists who are what are often called in-house, who are employed by the agency and are seeking specialists services such as fitness for duty evaluations, they will often ask me to contract with the agency to do these evaluations or to assist in pre-employment psychological screening.
So I have a contract with the Honolulu Police Department to provide psychological evaluations and work with the local psychologist who does the face-to-face evaluations and I’ll conduct evaluations that can be done remotely. And so, it’s really in response to requests from agencies that for a variety of reasons are looking to receive some specialists services outside of what can be provided [00:11:00] locally.
Dr. Sharp: Yes, that makes sense. And I have to say, if you’re going to be practicing in other states, you have to add Hawaii to that list. I mean, of all places, if you’re going to go that route, make sure that Hawaii is included.
Dr. Corey: Absolutely.
Dr. Sharp: Fantastic. Well, let’s jump into the MMPI-3. So we’re talking about remote administration of the MMPI-3. I guess I’m curious, just right off the bat how things might be different with the 3 than they were with the 2 or the 2-RF in terms of remote administration, if at all.
Dr. Corey: No differences at all, Jeremy. This is precisely the same procedure, the same platform through Pearson’s Q-global Software platform web-based. And the other procedures are precisely the same.
Dr. Sharp: I got you. [00:12:00] Okay, that’s easy. So the logistics are the same.
Dr. Corey: The logistics are the same.
Dr. Sharp: Okay. So what are we concerned about when we go to administer any personality measure remotely, but, of course, speak specifically to the MMPI-3 as much as possible. What are the things that we need to be aware of in terms of either security or administration guidelines, anything in that realm?
Dr. Corey: Great. So for psychologists that are looking for guidance in Tele-psychology, whether it’s the administration of a psychological test or a merging test administration with a clinical interview or any other elements of an assessment, referring to the guidelines for the practice of Tele-psychology, which APA published in 2013 is a great starting point. Guideline 7 [00:13:00] of that publication states in rather aspirational terms that psychologists are encouraged to consider the unique issues that may arise with test instruments and assessment approaches that are designed for in-person use are applied in a Telepractice context.
So we really begin with that admonishment which is to think through the issues. There’s nothing unique to the administration of the MMPI-3 in that context. It is with any other. There are some practical considerations simply that really rest with the administration platform that one has to think through, but think about [00:14:00] what are the core elements that protect the interpretability of a test?
Well, one is ensuring that the psychometric properties and the integrity of those psychometric properties are preserved when one moves from a paper-pencil administration to a computer-administered test. There’s significant literature that tells us that at least with respect to the MMPI instruments, whether you complete it by hand- paper-pencil or you complete it on a desktop or a tablet makes no difference. The mean scores and standard deviations associated with those means [00:15:00] are not substantially different.
There is, however, another risk. And that is that the conditions of the administration also have to be preserved if we want to be able to make inferences from a test score that are derived from validity evidence gathered under conditions that match what the test manual indicates that are required for the administration.
So for example, whether it’s the MMPI-2 or the MMPI-2-RF, or the MMPI-3, all of those manuals indicate that the test needs to be administered in a context that is suitable for test administration, that is free of distractions, reasonably comfortable where you can confirm the identity of the test taker, [00:16:00] where you can monitor their engagement with the task, and you can ensure that the test taker completed the items without coaching or other interference from third parties, and finally that you can manage and maintain test security. They don’t take copies home with them. They don’t take pictures with their smartphones of the pages of the test. So you maintain test security.
That’s the test administration condition that is stipulated in the manual. If we’re going to meet guidelines 7 of the guidelines for the practice of telepsychology, we have to think through the issues that may arise in a telepractice context that may not preserve those conditions.
And why is this so important? Because if we don’t preserve the conditions of administration, [00:17:00] then we cannot generalize from the validity of the evidence that was based upon the administration of a test under strict conditions. So we can’t enjoy the benefits of that validity evidence and therefore make inferences from test scores. When we read the items to them over a zoom meeting and they answered true or false in response, that’s not a standard test administration and there’s no evidence to indicate that those scores would be the same and plenty of reason to think they might not be.
Dr. Sharp: Sure. I think this is such a good point. A lot of us before the pandemic were administering personality measures over the computer, right? That happened a lot in our offices with some amount of proctoring. I know we’re going to [00:18:00] touch on proctoring and what that involves, but with the onset of this pandemic, all of these factors, all of a sudden controlling the test environment got a lot harder as people were either in their homes or their cars, in some cases, who knows where trying to take these measures.
Dr. Corey: That’s exactly right.
Dr. Sharp: So I’m curious how you have dealt with this, and if you’ve seen anything literature-wise, or just talking with colleagues, how folks have really handled this test security aspect over the course of the pandemic.
Dr. Corey: Sure. I recall getting a phone call from Yossef Ben-Porath, who as you know is, with Auke Tellegen, the author of the MMPI-2 restructured form, as well as the MMPI-3. And [00:19:00] as coauthor with Yossef of the MMPI-2-RF and MPI-3 Police Candidate Interpreter Report, we’ve maintained a longstanding collaboration. And in the course of that phone call, he was asking me my thoughts about making the MMPI-2-RF available for remote on-screen administration through Pearson’s Q-global web-based platform because at the time it was not available.
Dr. Sharp: I remember that. That was a new big piece of news that we could…
Dr. Corey: Yeah, it was a big piece of news. And so I expressed my concerns around test security and we engaged in exactly what guidelines seven said we needed to do. We began to think about what problems might arise.
Well, one of the things that you [00:20:00] have to consider is what happens in a worst-case scenario. Let’s talk about test security for a moment. With many test publishers, a remotely available test is provided to the test taker through a link. They’ve sent an email with the link that connects them with the test and they began taking the test.
And I know we’ll talk about this, Jeremy, that proctoring of a task, whether it’s in your office or whether it’s done remotely is an essential part of that valid test administration. You can’t assure that the conditions are proper and maintained unless you witness those conditions. So the witnessing of that, when it’s done remotely, is via teleconference.
So you connect with the test taker. You assure [00:21:00] that they are who they say they are, perhaps show a photo ID. You ask them to confirm the conditions of the room, perhaps giving you an environmental scan of the room, seeing that there’s no one else present that it’s free of distractions, they’re able to stay focused. And then you maintain a visual and auditory connection with them throughout the test-taking session so that you can document that in fact they were not coached, they were not otherwise interrupted, and that the conditions of test-taking are consistent with what would have been similar conditions that they were in your office.
So let’s say that you’ve given them that. You’ve confirmed that the test-taking conditions are suitable for test administration. You then send them the link while you’re connected with them on zoom for healthcare connection, as one example.
Dr. Sharp: Can I [00:22:00] interrupt you just for one moment?
Dr. Corey: Yes.
Dr. Sharp: I would love to ask, you said, send them the link while you’re connected with them. Is that what you advise rather than sending the link ahead of time, for example, or outside of a session when you’re connected to them?
Dr. Corey: Thank you for clarifying Jeremy. Absolutely. And the reason for that is if for some reason you’re not able to make that connection, they now have access to that test. They can complete that test without you being with them. They can take pictures of the test items with their smartphone or simply screenshot the test. And you’ve lost test security at that point.
Well, the same thing can happen after you’ve made a visual connection. You are in the teleconference and let’s say you lose it because sometimes it happens. Your internet goes out, you break connectivity and you can’t reestablish it. Now the person has the test and they’re [00:23:00] completing the tests without your proctoring. You’ve lost test security. You lost the ability to confirm that the conditions are suitable. What do you do?
Well in considering this potential complication and discussing it with Pearson, their technical staff went to work and figured out a solution. So the practical solution to that problem with Q-global is you log on to Q-global, you select that individual, you select the test that you had previously assigned to them when you sent them the link and you delete the assessment. Once you delete that assessment, they can no longer advance to the next item because items of the MMPI-3 and the MMPI-2-RF are presented item by item rather than on a page [00:24:00] of items. So they are done. And of course, the test cannot be scored, but you would have no interest in scoring in any way that hasn’t been completed.
But that’s a very thoughtful, practical solution to, although unlikely, a potential problem that could otherwise compromise test security. So it’s really all about thinking through the issues that might arise just as we’re admonished to do so in guideline 7.
Dr. Sharp: I love that. And I would expect no less, of course, from y’all knowing the history that you would think through things deliberately. And I know this pandemic has forced us all to think on our feet in many regards and really figure out how to do this the right way.
Dr. Corey: Well, when we were thinking about how the test can be administered remotely [00:25:00] without compromising either protocol, validity, interpretability, or test security, we realized quickly that we’re going to need to get some information out to the public to know how to do this properly. And we were very grateful that the editor of professional psychology research and practice had agreed to expedite a paper on that topic.
So when we published in June of this year, practical guidance on the use of the MMPI instruments in remote psychological testing, it was intended to provide some necessary guidance as opposed to, Hey folks, the MMPI is now available for administration remotely. Instead, here are some practical suggestions on how to do this in a way that preserves the [00:26:00] conditions of administration necessary for proper interpretation.
Dr. Sharp: I love them. And we’ll link to that article and all the other things that we’re mentioning in the show notes. So we’ll definitely have a nice little resource section for folks. Are there other considerations from that article that you’d like to touch on in terms of either security or just the logistics of administration aside from proctoring? I know we keep tabling that, but we’ll talk about that quite a bit. Are there other considerations that we need to be aware of?
Dr. Corey: Yossef and I made reference to what is a favorite author of mine, Atul Gawande, who is a surgeon. In 2009, he wrote The Checklist Manifesto where he’s advocating for the use of checklists for important procedures. And [00:27:00] he introduces the concept of pause points.
I love this because pause points are essentially a checklist embedded within a checklist so that if you think about it, you’re going on a trip, and there are things that you want to do before you leave for the trip, and there are things you want to do right after you leave for the trip, and there were things that you want to do during your trip. These are pause points where we can pause it at a point and think, have I done all the things that need to be done at this particular point?
One of the pause points is before you ever administer the test remotely, what do you want to do? Well, some of the things you have to consider are first, is this even appropriate for this person? Is it appropriate to be assessing this person remotely?
Think for [00:28:00] example, about a person who may be you’re administering the assessment in a forensic context or a clinical context where the person is living with an abuser and you’re administering the test in the context of an interview, perhaps an interview that you’re going to have as soon as the test is completed and you’ve looked at the scores to see what are the issues that the person’s reporting. You want to make sure that that person is in a safe place where the abuser can’t hear their responses to your clinical questions that might in fact be used in a harmful way.
So contacting the person ahead of time and discussing whether or not this is appropriate, whether they have a private place that [00:29:00] allows them to be candid and doesn’t put them in any jeopardy. Whether they can even manage the technical aspects of connecting by teleconference. I mean, there are details that can be confusing for people.
I often have connectivity issues with the person who I can see but can’t hear them. They haven’t chosen the right microphone and they don’t know how to do that. So I got to get on the cell phone with them and I have to walk them through how to do that. Not everyone can handle those instructions.
In short, Jeremy, not everyone is well-suited to conduct this testing remotely. And that’s a pause point is to think, is it appropriate for this individual? And then letting them know ahead of time, all the things they’re going to need. They’re going to need to have a computer with a camera [00:30:00] and a microphone. Now, most modern ones have that, but a lot of people have old equipment. They may not be able to manage that.
Maybe they have only a tablet. Well, the problem with the tablet is for most tablets anyway, or phones, although you could take the test, it’s technically possible to complete the MMPI-3 on a phone or tablet. Once you move to the browser on a tablet or a phone, typically the video blurs and you don’t see the person any longer, you see a blurred image of them. Well, you can’t maintain test security and confirm the integrity of the testing conditions if you can’t see them.
So we tell them that if your laptop or your desktop doesn’t have a camera, you can use your smartphone to maintain the teleconference connection, and you can use your [00:31:00] computer to take the test, but I need to see you and I need to hear you. If we can’t maintain that we can’t do it.
Those are pause points beforehand. And then there are pause points at the start of the remote psychological testing: confirming the person’s identity, reasserting what the various conditions are to be maintained throughout, making sure… This is a new one for me Jeremy, it occurred to me the other day when I was interviewing someone who had a history of some cardiac problems. I thought what would happen if this person had a heart attack or a seizure while I’m going to teleconference with them and I am entirely helpless? I have their cell phones but don’t have any other information that would be able to direct first responders to their homes.
So routinely now, [00:32:00] I ask the person for the address where they are and a phone number for emergency contact. That way, I can direct first responders to their location. The last thing I want to do is be watching someone on a teleconference who’s having a medical emergency and I can’t do anything about it.
These are things that you think about at various pause points? And so we learned over time and we include in this article, I referenced earlier various pause point considerations in that checklist.
Dr. Sharp: Let’s take a quick break to hear from our featured partner.
The BRIEF-2 ADHD form is the latest addition to the BRIEF family of assessment instruments using the power of the BRIEF-2 the gold standard rating forum for executive function. The BRIEF-2 ADHD form usesBRIEF-2 scores and classification statistics within an evidence-based approach to predict the likelihood of ADHD and to help determine the specific subtype. It can [00:33:00] also help evaluators rule in ADHD and rule out other explanations for observed behaviors.
Please note that BRIEF-2 parent and teacher form scores are required to use this form. The BRIEF-2 ADHD form is available on PARiConnect- PAR’s online assessment platform. You can learn more by visiting parinc.com\brief2_adhd.
While we’re on this topic, are there other considerations or circumstances that you would deem inappropriate for completing an assessment? You mentioned the person in the home of an abuser. I’m thinking specifically clinically, not so much tech or logistical issues with the computer and so forth, but from a clinical standpoint, are there other circumstances you can think of or examples where it just wasn’t a good idea to try to move forward.
[00:34:00] Dr. Corey: Sure. There are persons who have emotional or mental conditions where the test items can trigger emotional reactions. That can be quite distressing and potentially impairing for persons who are suicidal, with even more serious consequences. And if I have reason to believe that these tests items might trigger an adverse psychological reaction and the person is alone, I certainly don’t want to do that.So in that situation, if I were, for example, in a different state and that person had been referred to me by another professional or an attorney or I should say another psychologist or a mental health professional, or an attorney, that didn’t mean to imply that attorneys aren’t professionals, [00:35:00] then I might arrange for them to take the test remotely just as I would normally do but with their attorney or other mental health professionals in another room in case they had an adverse reaction.
Dr. Sharp: I see. That’s fair. I think it’s important to think through those concerns, it’s rare, but like you said, if something were to happen, the worst thing is to be sitting helplessly on the other side.
Dr. Corey: Absolutely.
Dr. Sharp: And I love that you referenced Atul Gawande. I really like his work and enjoyed listening to him on any other podcasts and his books and so forth. I love that.
I think we’ve put it off long enough. We’ve talked a lot about proctoring off the recording. Let’s get into that because I know that that’s really important in this process and something that you utilize really [00:36:00] effectively. So can you walk us through the whole proctoring approach and why that’s important?
Dr. Corey: Sure. Well, keep in mind that proctoring is a routine aspect of all psychological testing. You don’t send people home with an MMPI and say, hey, just put it in the mail and send it back to me.
Dr. Sharp: You don’t do that. That’s not okay, I thought.
Dr. Corey: That’s not okay. So, there’s nothing new to proctoring in a remote setting. Just think of it as exactly what you would do if they came into your office. You’re going to be available. You’re going to be able to answer questions. You want to make sure that they remain engaged with the test. You want to make sure that they’re not coached or otherwise interfered with or interrupted. You want to make sure that they’re not on the phone while they’re taking the test or that they’re [00:37:00] taking pictures of it as I indicated earlier.
These are things that you would normally do when you’re proctoring a test. We all proctor tests. This is no different. It’s just that it’s being done over a teleconference connection. So all the things you would normally do, you’re still going to do.
Now, in some jurisdictions, the psychometrician has to be registered with the board of psychology in that jurisdiction. You can’t use administrative staff to do so. So keep in mind that there are jurisdictional requirements that define who that Proctor can be within the laws of that area. Whoever is permitted to proctor should be instructed on all of the proper procedures including when using Q-global, the deletion of [00:38:00] the test if conductivity is lost and cannot be quickly restored.
The practical aspects of proctoring began before test administration: making sure that the person knows what equipment they’re going to need to have, and what’s going to take place during the evaluation. We include the conditions and expectations around remote psychological testing and our disclosure and informed consent document. I think it’s a good idea that even if you don’t intend to administer the tests remotely, it’s a standard part of your disclosure informed consent document, you’ve covered these issues should you decide to administer a test remotely.
Dr. Sharp: Can I jump in real quick because I’m guessing people might be curious? Could you talk to us a little bit about what specifically you would put in your informed consent and how much you’re spelling out this process [00:39:00] for people?
Dr. Corey: Sure, thanks. One of the things we point out, first of all, we state what the expectations are. That they would be in an environment free of distraction, that they would be taking the test alone with no other parties present unless otherwise known and approved by the psychologist, that they will not record the session either video or steal pictures without the other’s permission- that no one will record without the other party’s permission. We want to have a cell phone number or a landline number where we can contact them in case we lose connectivity. We state all of that in the disclosure of the informed consent document.
And that they understand that there are some risks. One of those risks I mentioned a moment ago, [00:40:00] there are 335 items on the MMPI-3. If they get the number 334 and lose conductivity and we can’t regain it. And I don’t know that they’re on 334, because there’s no way for me to know, I’m going to delete the assessment. They’ve now put in whatever time that it took them to nearly complete the test, and they’ll have to take it again. Those are risks that are associated with remote psychological testing.
Dr. Sharp: Of course. Thank you. Thanks for going down that path for a second.
Dr. Corey: You bet. So the Proctor then would ensure that the test taker has the necessary equipment that they know how to… let’s say that you’re using, in my case, I use zoom for healthcare. There’s a subscription price associated with that but it [00:41:00] provides protections that are otherwise not available and ensures compliance with the HIPAA security rule.
And by the way, I know that there is a current waiver during the pandemic for enforcement of good faith violations of HIPAA, but as we indicate in our article, Yossef and I, simply because there is a waiver of the enforcement of good faith violations, it is not a license just ignore the security rule associated with HIPAA. And so if you have an option that is HIPAA compliant, use it.
Dr. Sharp: Of course. Thank you for highlighting that.
Dr. Corey: So once we’re confident that the person has the information and equipment needed to be able to connect and they’re scheduled for that evaluation, [00:42:00] a proctor whether they’re in your office or whether they’re administering a test in person or whether they’re administering the test remotely, the process is the same. They need to be trained. They need to understand their role. And part of their role is to document any deviations from standard test administration and standard test conduct.
So if there is a loss of connectivity, I want them to record how long that loss of connectivity lasted. What’s the duration of that? I want to know whether there were any other interruptions, how long those interruptions were, and whether there were any behavioral observations that are notable. All of this helps me when it comes [00:43:00] the time to interpret the test. I need to know whether there was some event that makes this particular test-taking session unusual, and therefore may undermine the inferences that I can associate with those test scores.
During the in-person test administration, we’ve had fire alarms go off in our office and we’ve had people have to leave the building and leave their tests on the table come back after 45 minutes or two hours. Those are interruptions in the test-taking that are unusual. They can affect interpretability and they have to be thought about, but they can’t be thought about if you don’t know about them. And so part of the Proctor’s responsibility is to document any of those events and inform you as the psychologist.
Dr. Sharp: Yeah, that makes a lot of sense. And let me back up just to be super [00:44:00] clear. So when you’re talking about using a proctor, this is a process that happens when the client is in your office, right?
Dr. Corey: That’s right.
Dr. Sharp: I don’t know if you’re doing any, I call it direct to home, but direct to somewhere else where they’re not in your office. And I wonder if we might talk about how to proctor those situations if that’s even possible.
Dr. Corey: Sure. There are two, if you want, contexts in which remote psychological testing might take place and where the proctoring would potentially differ.
One is when you’re sending the test link to an onsite proctor- that is there onsite with the person who’s taking the test but they’re not where you are. In my work, that might involve… In civil forensic psychology, I might send the link [00:45:00] to the attorney who will then assess the person in connecting to the test in their office.
So they’re both masked and the attorney or the paralegal is connecting to the link and then connects to the test, make sure that the person is comfortably situated in a private office while they’re completing the test. And they periodically check in with the person. If they have questions, they might call me, and get those questions answered. So we’re working in tandem to proctor the test. That might also be in a police department where the background investigator or a personnel administrator has been given instructions on how to proctor the test while they’re completing it in their office.
The other [00:46:00] test administration to test proctoring is where you or your staff are personally observing the individual all by teleconference and they’re alone in their home or place of business or the location.
Dr. Sharp: So would you say those later examples are legit, maybe not recommended? I’m curious how you manage those kinds of situations if they’re ever truly necessary or how you might approach them.
Dr. Corey: Yeah, I think proctoring and either of those conditions requires training. As long as the person is properly trained and you are able to supervise that work, that is to say, you engage with them to ensure that the steps that they’re taking are the [00:47:00] approved steps and that you’re available for any deviations from those to consult, I think it’s absolutely legitimate because it’s no different than when you’re in your office, right?
When your administrative assistant administers the MMPI to a test taker and hands them the test booklet and the answer sheet, you’re not present when that’s occurring typically. But you’re available to them and you periodically may observe them to ensure that what they’re doing is proper.
Dr. Sharp: Sure. I apologize if I’m retreading, but I just want to be clear for folks, would you ever advocate or approve, I suppose, administering an MMPI remotely to someone’s home or to a business where [00:48:00] there is no proctor? it’s just you watching them over the computer and there’s nobody else involved.
Dr. Corey: Yes. That’s the second form of proctoring that I think is absolutely appropriate. As long as you can maintain visual and auditory connections throughout the test-taking session to ensure that test security is maintained and that the conditions for test administration are appropriate and maintained throughout the testing session. That’s the purpose of proctoring is to accomplish both of those goals throughout the testing session. So if you don’t have a trusted trained proctor physically on-site, you have to provide that through other teleconference means.
Dr. Sharp: Okay. Perfect.
Dr. Corey: What I never do, just to be clear to your listeners, nor would I ever advocate is [00:49:00] sending the link to a person to take the test at home without direct observation.
Dr. Sharp: Thank you for being very clear on that. I don’t do any adult assessments these days. So that’s not something luckily that I have stumbled into or a mistake that I’ve made, but I could easily see myself doing that. I think a lot of us are trying to streamline and be as efficient as we can. And that’s one of those steps that might get overlooked.
Dr. Corey: Of course. And the software makes it easy to make that mistake. It allows you to email the person the link.
Dr. Sharp: Yes.
Dr. Corey: I never do that. Even though in the software, you can put that person’s email address in, and the link will go to them, I never do that. Instead, as we indicate in our guidance, [00:50:00] send the email to yourself, put your email address, and now you have the invitation and only when you’re connected to the person, if by teleconference and you’ve confirmed the appropriateness of the environment that they’re in, that they’re alone, you’ve confirmed their identity and all the other factors that we’ve indicated need to be thought through and that we include in table 1 of our article, once you’ve done that, then you send them the link.
You can do that by chat, or text, or you can do that by email and they then connect to the link. And now you’re ready to roll because you can observe them taking the test. You can monitor in Proctor but if you send it to them ahead of time and there’s any glitch in getting connected, now you lost test security. Now you’ve got trouble. I mean, you can delete it through Q-global, [00:51:00] but if you’re using some other assessment modality, you may not be able to delete it.
Dr. Sharp: Right. That’s just a disclaimer, I suppose, to check, whatever software you might be using.
Dr. Corey: Yeah. I think we all know you don’t send an MMPI instrument or other test secure or a PAI or any other clinical assessment instrument home with the person to complete. We just wouldn’t do that. We wouldn’t have them completed under non-monitored unproctored conditions. So we don’t want to send them the link either and tell them we’re in the proctoring context.
Dr. Sharp: That’s fair. Now, what do you think about the setup of giving someone mouse control and you have it on your screen and then you turn over mouse control to them and they’re answering on your screen as an added layer of, well, at least perceived [00:52:00] security.
Dr. Corey: I’ve heard of psychologists doing that and I want to be careful about being too critical of that approach, but let me tell you why I don’t. One of the consistent features of test-taking and personality testing whether you’re doing it, whether the person is completing the test in paper-pencil format, or whether they’re completing it on a computer is that they are interacting with an inanimate object.
It’s going to be the computer terminal or the keyboard or the score sheet. I’m not as the psychologist seeing each of their responses as they produce them. That’s true though whether it’s remote psychological testing [00:53:00] or in-person paper-pencil testing.
But once I changed the administration to here’s one item that you give me your answer, and I can see it on my screen and they know I can see it on my screen because they have control over my screen, I don’t know what demand characteristic is created by that new interaction that they know that I’m seeing their individual response to each individual item. I’m concerned that that could create differences. And I think that the burden is on the test user to be able to demonstrate that means and standard deviations are the same for that administration.
Dr. Sharp: That’s a great point. That adds another layer to the process that complicates things. All of a sudden now it’s like you said, the main characteristics and people interacting with one another versus just [00:54:00] answering in a vacuum.
Dr. Corey: Yeah. I think we may have contaminated the predictor somehow and you have to be careful about it.
Dr. Sharp: That’s fair. Well, I want to spend a little bit of time, because I know this is really your area of expertise, talking about the forensic setting and how, if any what interaction there might be between remote assessment and forensic applications. And a lot of these are naive questions from someone who doesn’t work in that realm at all. So keep that in mind, but hopefully, others will take some usefulness from this.
Dr. Corey: I think, by and large, the issues are no different than what we’ve talked about already. The one consideration that I think is unique to forensic evaluations is, is this remote administration, is the modality of test administration likely to undermine [00:55:00] the reliability of the results? And if so, then it can undermine its probative value in court and may not be admissible. That’s a risk and it’s one of the risks that would need to be communicated in your informed consent and disclosure is that any testimony derived from this could be undermined by the administration modality.
I think in practical terms, there’s growing evidence that courts have treated telemedicine as equivalent to direct face-to-face medicine in terms of the findings that psychiatrists have reached when they’re testifying about patients. For example in prisons where telepsychiatry is ubiquitous.
So there’s a growing [00:56:00] body of case law that suggests that courts are not inclined to render inadmissible evidence based upon remote test administration but rather it’s one of the facts that a jury might consider and consider what weight to give to that testimony. I think the more we’re able to demonstrate as a profession, as a scientist-practitioners, that the psychometric properties of a test have not changed when it’s administered and proctored remotely, the better we’ll be able to combat those challenges. But at these early stages, being able to anticipate challenges to the admissibility of [00:57:00] evidence-based upon remote test administration is something I think every forensic examiner should contemplate.
Dr. Sharp: Of course. I think we were talking before we were planning the episode about how you might document the mode of administration in a forensic report, or I think you phrased it like, I’m totally paraphrasing, but like put it out there ahead of time so that they can’t surprise you with it.
Dr. Corey: Absolutely. I’ve read reports where I know the test was administered during the shutdown and there was no way that person was in the office. In fact, I know they weren’t because I interviewed the person later and they told me that they had tested remotely, but nowhere in the report does it say so. I think that’s a mistake.
[00:58:00] It’s important that we’re transparent. So in any report where scores from tests that were administered remotely or even if the interview was conducted remotely, I disclose what platform I used, what the essential features of the encryption were, and whether there were any unusual events, i.e, I was able to maintain connectivity throughout the entire session, I was able to confirm the person’s identity, I was able to assure that there were no interruptions.So I document all aspects of that because should there ever be a challenge, in most cases, it’s not yet [00:59:00] adjudicated. So I’m not even concerned about admissibility at that point. I’m thinking ahead of time, about what might happen if this is to become a focus of litigation. For my testimony to be admissible, I need to be able to handle any potential challenges. So I want to get ahead of that and anticipate.
When we write reports just as an aside, Jeremy, I think it’s a mistake to think about who we’re sending it to as the audience. There’re really multiple audiences when we write reports. Some of them are standing in line to potentially read our reports later. There might be a licensing board is one of the audiences. A court of law, and a jury is potential audiences. A district attorney might be another audience, or [01:00:00] a plaintiff or defense attorney might be another audience. It might be another psychologist who is an expert witness reviewing my work. These are all potential audiences.
So I anticipate those various audiences and I write to them as well. So when I’m documenting all of the aspects of the teleconference, I’m not writing that for my initial reader. They probably don’t care. I’m documenting it for a potential future member of a broader audience.
Dr. Sharp: Such a good point. I think, especially those of us who don’t live in this world much- the forensic world, I think that’s really easy to forget. Even if we’re just doing typical pediatric evaluations, there are many people who are [01:01:00] likely going to read that evaluation. So I just want to highlight that you really need to consider and think ahead of what you’re writing and who’s going to see it.
Dr. Corey: Indeed.
Dr. Sharp: Yes. I appreciate you walking through that. I’ve seen questions in our Facebook group around just discussions and message boards and so forth around the admissibility of remote assessment and forensic settings and how to handle that. I think we’re all trying to figure out what to document and the right way to do it.
So thank you.
Let me maybe start to close and just ask, we talked about a lot of different considerations. Are there other points that you want to make sure that we highlight, or touch on, and maybe as a follow-up, are there resources for folks who would like to learn more about remote administration here in these settings?
Dr. Corey: Well, I know that you’ve enclosed, or included [01:02:00] in your episode notes a number of resources. Pearson has a link that provides a broad range of documents including the link to the guidelines for the practice of telepsychology. So I think that’s a very useful one. And of course the article I referenced earlier that Yossef and I published in professional psychology research and practice is I think also useful.
One of the things that we haven’t talked about, but I think it’s just important to emphasize is how important it is to get comfortable with the technology. I mean, the last thing you want to do is… no one would think about administering a WAIS with an actual patient or examinee without ever having practiced those subtests first. You got to get comfortable with them. You got to be comfortable [01:03:00] with the instructions.
So it’s no different here. Take this test yourself remotely, send yourself a link, and go through it. Perhaps get on a teleconference with a colleague and each of you practices taking the test and deleting a test and seeing how it actually works, perhaps even engaging in a little collegial misbehavior just to be able to document some glitches.
By the way, we saw in action the importance of this delete function with a colleague who was reporting that they were administering a test to a person who was in an institutional as an adolescent. And this was the first time the adolescent had access to a computer with the internet.
Dr. Sharp: I see where this is going.
Dr. Corey: Instead of taking the [01:04:00] test, they were on MapQuest or Google maps trying to figure out a way to get out of there.
Dr. Sharp: Oh my gosh.
Dr. Corey: And he had muted himself and he had turned off his volume so that the psychologist who was proctoring the test could not communicate with him. He was paying no attention at all. That’s what we discovered, or we saw in action the importance of this delete function that Q-global has implemented because you would not want that person being able to have ongoing access to the test when they’re no longer engaged with you at all.
So I think it’s important to practice using those features of the platform. Get comfortable with the technology. If you’re proctoring two people at the same time, for example, [01:05:00] getting comfortable with breakout rooms so that you’re able to give instructions in a breakout room rather than interrupting, or I should say, the person doesn’t want to have to answer or ask and have answered a question in front of someone else. So using breakout rooms to do that might be one way to accomplish it. But the important thing is to practice and become comfortable with it so that you’re not stumbling around when you actually are administering it the first time.
Dr. Sharp: I love that. I’ve done a number of workshops on remote administration of cognitive med, the WISC, WIAT and so forth. And I spend so much time, much to the participant’s chagrin I’m sure talking about preparation, and like you have to practice. When we were learning how to do it, I likened it to learning the measures for the first time back in grad school. It’s [01:06:00] a different process and you just got to get comfortable with it before you throw yourself in front of a client.
Dr. Corey: Well, you demonstrate that very well. We met before this episode to discuss what was going to happen and how it was going to take place and how we would get through the interview. You met with me beforehand to explain all of that so that I was comfortable rather than surprised by how it all worked out then. I’m a podcast Virgin. This is my first one. And I’m happy to have done my first podcast with you, Jeremy.
Dr. Sharp: Well, I really appreciate it. Yeah, you’ve been a fantastic guest, and no indication at all that this was your first time. You nailed it. Thanks for the time and the knowledge Dave.
Dr. Corey: My pleasure.
Dr. Sharp: Okay, everybody. Thank you so much for listening to my episode with Dr. Dave Corey. I hope that you took away some nuggets [01:07:00] for remote administration of personality measures specifically the MMPI. It’s so interesting to hear from a true expert in the field about how to utilize these measures remotely, the process that Dave goes through as far as proctoring and maintaining test security, and so forth. So I hope that was helpful for you.
It’s interesting to interview Dave and not really even scratched the surface of his true area of expertise, which is using the MMPI and forensic and police contacts. So maybe that is an interview that can happen down the road. I’ll keep my fingers crossed, but for the time being suffice it to say that Dave was a great interviewee. I really enjoyed speaking with him. And I hope that you found this useful.
So if you’ve not subscribed and, or rated the podcast, that will be fantastic. You can do that pretty easily in the [01:08:00] Apple podcast app and on Spotify, you can hit follow and that will just increase the visibility of the podcast and spread the word. Alternatively, you can tell a colleague that works too, good old-fashioned word of mouth. I appreciate everyone who’s been spreading the word and we’ve seen pretty solid growth of the podcast over the last few months. So thank you as always.
All right. Hope everyone is doing well. Y’all take care and we’ll be back on Thursday with the business episode. Bye for now.
The information contained in this podcast and on The Testing Psychologists website is intended for informational and educational purposes only. Nothing in this [01:09:00] podcast or on the website is intended to be a substitute for professional, psychological, psychiatric, or medical advice, diagnosis, or treatment.
Please note that no doctor-patient relationship is formed here, and similarly, no supervisory or consultative relationship is formed between the host or guests of this podcast and the listeners of this podcast. If you need the qualified advice of any mental health practitioner or medical provider, please seek one in your area. Similarly, if you need supervision on clinical matters, please find a supervisor with expertise that fits your needs.