Many of y’all know that I have been using TherapyNotes as our practice EHR for over 10 years now. I’ve looked at others and I keep coming back to TherapyNotes because they do it all. If you’re interested in an EHR for your practice, you can get two free months of TherapyNotes by going to thetestingpsychologist.com/therapynotes and enter the code “testing”.
This podcast is brought to you in part by PAR.
The Clinical Assessment of Attention Deficit-Adult assesses symptoms of ADD and ADHD in adults. Now you can administer and score the CAT-A on PARiConnect, PAR’s online assessment platform. Visit parinc.com\cata.
Hello again, welcome back to a [00:01:00] clinical episode of The Testing Psychologist. Today, we are talking all about continuous performance tests with my guest, Dr. Chris Mulchay. Chris has joined me before to talk about child custody evaluations. He’s also the co-author of The Field Guide to High Conflict Litigation. With Jonathan Gould, he co-authored the chapter on child custody for the Oxford Handbook of Psychology and Law, as well as the APA Handbook of Forensic Psychology, but as I said, he is here today to talk about continuous performance tests and the many aspects thereof.
He also has published recent articles on evaluations of dyslexia, dyscalculia, dysgraphia, and the virtual reality ADHD continuous performance test, Nesplora. You may also see him in our Facebook group, which he helps moderate.
So like I said, we are doing a deep dive into CPTs today. CPTs come up frequently. They are quite [00:02:00] fraught in our testing world due to the research or lack thereof behind many CPTs, but the demand for the CPTs by any number of entities and belief that they provide some concrete measure of attention and executive functioning.
So lots for us to talk about today. We cover things like the history of CPTs, basic test construction, and sensitivity versus specificity, and why that’s important with CPTs. We talk about using a true hypothesis testing model in assessment and how CPTs fall into that model. And of course, we go into the pros and cons of all the major CPTs available on the market right now. So if you’re someone who has wrestled with the decision to purchase or use a CPT in your battery, this is a great episode for you.
So without further ado, let me get to this [00:03:00] conversation with Dr. Chris Mulchay.
Chris, hey, welcome back.
Dr. Chris: Thanks, Jeremy. Thanks for having me.
Dr. Sharp: I’m always glad to have you, although, it’s funny, things feel like yesterday more and more in my life and this is one of those things, but I looked back and the last time you were on was four years ago.
Dr. Chris: Four years ago?
Dr. Sharp: Yes. August, 2020.
Dr. Chris: Wow. We get to hang out sometimes when you come to Asheville and I know I need to visit you in Colorado but maybe just hanging out in between doesn’t feel like that long.
Dr. Sharp: I agree.
Dr. Chris: It’s hard to believe. Our kids were at completely different stages of their lives. It’s hard to believe it was four years ago.
[00:04:00] Dr. Sharp: I know.Dr. Chris: Plus, the listeners don’t know that we’re messaging each other frequently regarding The Testing Psychologist forum and things that come up in the group. So we have a lot more contact than just a podcast every four years.
Dr. Sharp: This is true. Yes, it doesn’t feel like it’s been that long. We have a lot of contact in between. For anyone who may not know, Chris, you’re a fellow moderator of The Testing Psychologist community on Facebook. So we get to discuss all sorts of things testing related and sometimes not testing related or that.
Dr. Chris: Primarily rule 1; if you’re in the Facebook group, rule 1 is to be kind and respectful. Jeremy and I are often talking about whether or not your comments are kind and respectful.
Dr. Sharp: Just a little glimpse into our world as moderators. We think a lot about kindness here.
Dr. Chris: That’s right.
Dr. Sharp: Harder to define [00:05:00] than you might expect. Well, I’m grateful to have you here. We’re talking about a topic that you’ve gained a little bit of notoriety for, in the group and in the world outside, and that is continuous performance tests. So I will start where I start with most guests, which is spending a little bit of time on why you are spending time on this. Why is this important? Why do you care about this, of all the topics in our testing world?
Dr. Chris: During my postdoc, I was trained on how to conduct a proper ADHD evaluation considering multiple sources of data for multiple methods to rule in and rule out ADHD and other factors that might be contributing to a client’s symptoms. After my postdoc, I moved to North Carolina and read a book called The ADHD Explosion by Steve Hinshaw and Richard [00:06:00] Scheffler.
I was reading this book and they cited research that 30% of boys in North Carolina were diagnosed with ADHD, which was shocking because some Duke researchers did an awesome epidemiological study of where I live near Asheville, North Carolina, and they found that the ADHD rate in the community was 1 to 2%.
So there’s great conflict between these two studies. And so I started a practice where I would primarily advertise and focus on ADHD evaluations using multiple methods and multiple sources of data to rule in and rule out all the different factors that it could be. It became very apparent early on in my practice that it was hard to rule in and rule out everything; [00:07:00] whether the kid was having sleep issues or had a history of trauma or anxiety or they were situational factors. I wanted an easy answer to diagnosing ADHD and CPT seemed to offer it.
Newly licensed that I was making charts and trying to figure out which CPT I could afford and which one was the best. I went with the cheapest because I didn’t have much money. And so I get this CPT and I download it on my el cheapo computer from Best Buy and it doesn’t work. The CPT completely fails multiple times when I got clients right there and I’m trying to test them.
I contact the company and they said I had to pay [00:08:00] extra to get customer support. At the time, I think it was $99 or something for customer support, but the fact that I had to pay for customer support when I just spent $2000 on a CPT made me question the sunken cost bias that I had.
I’d fallen in love with this thing because I’d spent so much money on it. It was my new Ferrari or for me, my new Toyota Land Cruiser. I spent so much money on this thing. I wanted to love it but they wouldn’t provide me support unless I paid them more money.
And then I started digging in deep into CPTs and trying to justify the expense, justify what was going on with why it wouldn’t work, and then whether or not the data was important or helpful for me. I don’t like to give up so I [00:09:00] then purchased other CPTs and went through the same process, although each company didn’t charge me for customer support. Some of them just simply didn’t provide it but I went through the process of learning and trying to figure out whether CPTs could be helpful in private practice. I learned a lot about each CPT through the process of purchasing them and trying them out.
Dr. Sharp: I love that story, not because you’ve struggled so much with the CPTs but I just have so many examples of that where everything’s good until there’s this straw that breaks the camel’s back moment, and then the floodgates are open. Like, oh my gosh, I’m going hard on this and I’m going to figure out why I’m putting up with this. And in this case, it’s led to some pretty, it sounds like valuable clinical information.
Dr. Chris: That’s right. I had this information but I didn’t have anybody to talk to about it. It was [00:10:00] essentially me in a vacuum with this information and then The Testing Psychologist was launched 7+ years ago.
Dr. Sharp: Yeah.
Dr. Chris: And then, Jeremy, you could talk all about this, the most frequent question we would get in The Testing Psychologist forum was, which CPT should I use?
Dr. Sharp: Right. I’m going to stop right there. Of course, the implied issue there is you should be using a CPT at all. We’re going to dig into that. But then, of course, it’s which one and people want to know the best and so forth. So then you stepped in.
Dr. Chris: That’s right. We frequently got that question. I was frequently responding to that question with all sorts of reactions that I’m sure people can search and find in our forum.
And [00:11:00] then maybe 4 or 5 years ago, at some point, probably I think 5 years ago, I put together a table comparing a number of the CPTs and started sharing that because it was either easier than just regurgitating my history and experience with the different tests. They were so different and they purported to do, in measure and answer different questions that I thought the table was helpful. I hoped it was helpful but I didn’t stop there. I continued to purchase CPTs in the background, tried to use them and I’ve struggled to weigh the strengths and challenges or issues with the CPTs over the years.
Dr. Sharp: I think of all the things, I’m trying to think of other issues in testing that maybe have a [00:12:00] similar fraught nature to them, but there’s a love-hate relationship with CPTs. Do we use them? Do we not? If we do, do we like them? Not really, most of the time, but there’s nothing better. There’s so much to dig into and a lot of people have a lot of conflicting feelings about CPTs, I think
Dr. Chris: They do. My hope is that we’ll have a balanced conversation today about them. I do think that it’s important to start where I like to start when I’m talking to my colleagues, staff, and mentees. I like to talk about what we should be doing with all psychological assessment, because I think that there’s a lure to CPTs that they quickly answer the question, is it ADHD, yes or no?
I think that we shouldn’t be asking that question as you alluded to already, that’s the wrong question. I think what we should be doing is following [00:13:00] essentially the scientific model and using the hypothesis testing model.
If you’re listening to this podcast and you’re thinking, well, what is that or where can I find it? The best example of how to use it in psychological assessment that I found is our good friend and friend of the program, Jordan Wright has written a few things that I’m going to mention. The first thing that Jordan has written is a book. It’s in its second edition called Conducting Psychological Assessment, and it’s published by Wiley, I think in 2001.
The book starts out on page three with Jordan’s description of the hypothesis testing model. I think it’s excellent. I encourage all psychologists to go through that. I think it’s six pages. It’s succinct. It’s beautiful. Again, Jordan’s a friend of the program. Jeremy and I both have a great affection for him and he writes [00:14:00] in such a nice, simple way that you can follow along.
So if you don’t have the book handy, step 1 of the hypothesis testing model is that we conduct an initial clinical interview. In that clinical interview, we’re gathering lots of information about the client, and most importantly, we are generating hypotheses. We’re listing all possible causes of the functional challenges the child or client is having and using our individual theories or orientations as a guide to formulate those hypotheses.
So it’s not just one question; is it ADHD or not? It’s multiple hypotheses about what could be happening that’s causing challenges for this child or this client that you’re meeting with or that you’re talking to their parents.
After you generated those multiple hypotheses, then you move on to step 2, which is selecting [00:15:00] the tests and that’s hopefully, why you’re listening today is to listen to what we have to say about these tests. We select tests based on their own internal psychometric properties.
I got to remind everybody, especially those U.S. psychologists. I know we have psychologists listening from all over the world but for those that follow APA’s ethical standards, standard 9.02 Use of Assessments, I’m going to read A and B.
So 9.02a, psychologists administer, adapt, score, interpret, or use assessment techniques, interviews, tests, or instruments in a manner and for purposes that are appropriate in light of the research on or evidence of usefulness and proper application of the techniques.
In 9.02b, which I think is absolutely applicable to [00:16:00] today’s discussion, is psychologists use assessment instruments whose validity and reliability have been established for use with members of the population tested. When such validity or reliability has not been established, psychologists describe the strengths and limitations of test results in interpretation.
I think that’s key to our discussion. We’re going to be talking about normative data, whether or not validity and reliability have been established for these tests and the strengths and limitations of the studies that have been done for these tests. That second part of 9.02b is super critical when such validity or reliability has not been established, then we describe the strengths and limitations of test [00:17:00] results and interpretation.
Dr. Sharp: Sure. This is wild. I love that we’re diving into the ethical code. This is one of those things I feel like that we forget or perhaps just assume that any test that has made it far enough in the market to be on our radar as something we might use has gone through whatever rigorous amount of testing and development that it needs to and we can use it because it’s out there and a publisher decided to go forward with it.
Dr. Chris: That’s right. For me, in my forensic work, I might get cross-examined in court about the reliability or validity of data. I am likely to write about the strengths and limitations of the tests that I use in anticipation of being challenged regarding whether they meet Daubert criteria or [00:18:00] something like that in court, but in our clinical and therapeutic evaluations, I don’t think it’s very common for us to talk about this.
I think it’s really important for all of us to learn about the tests that we’re using, not just see the nice brochure or the nice webpage and quickly jump in and use a test. We need to figure out how was this test developed. How was it normed? Whether or not it can be used with the group of people that we’re working with and what the strengths and limitations are because those strengths and limitations are going to impact our interpretation and our application of the tests.
Dr. Sharp: Absolutely. I like that we’re laying this groundwork just to provide some context for this discussion. These are things that we take for granted that we need to dig into.
Dr. Chris: If you’re reading or [00:19:00] if you’ve read the comments over the years, oftentimes in The Testing Psychologist, you’ll see discussions of whether or not CPTs have sensitivity or specificity. So before you jump on Google and start researching what those are, Jeremy and I’ll talk about those for just a moment.
Sensitivity refers to a test’s ability to identify the presence of an actual deficit or condition or whatever it’s trying to measure. So the sensitivity of a COVID test is how well it measures whether COVID is present whereas specificity refers to the test’s ability to identify the absence of whatever it’s trying to measure. So it’s a negative result, the accuracy of that.
Those pieces are important because CPTs don’t measure ADHD. They measure symptoms of attention or symptoms of [00:20:00] inattention or inhibition or some factor, but it’s not ADHD as a construct and it’s important for us to differentiate that.
Dr. Sharp: Yes, I completely agree. And this is one of the big problems with CPTs that comes up often. My understanding anyway, is that a lot of them are pretty sensitive but not necessarily specific. Is that right?
Dr. Chris: That’s right. On different websites, they sure market themselves as having certain sensitivity and specificity. We’ve dug into that quite a bit. I still have some questions, is my polite way to respond to those claims? But if we keep going through, our good friend of the program, Jordan Wright’s hypothesis testing model, step 3 is testing. Jordan writes in his books that no test is [00:21:00] perfect and no measure is without flaws.
Super important for all of us to remember that science in general and psychology are probabilistic. It’s not 100% true that somebody has ADHD. We have confidence intervals in all of our data. And so the science is probabilistic. And so we need to understand the strengths and weaknesses of each test and the limitations before we interpret each test.
Meyer writes, and this is an interesting quote, “No self-report is made without bias and blind spots. No test has perfect reliability and validity, and no single method of measurement should be taken as gospel. The validity of every single test in existence has been challenged”.
So we want to think about validity as incremental validity; that we’re gathering [00:22:00] multiple sources of data from different people. We’re gathering information from the teacher, the parents, and the kids themselves, and then we’re observing the kid in our office. The more the data aligns, the more incremental validity we may have or the more confidence we may have in our findings. Of course, the challenge is that sometimes the data goes in different directions.
Step 4 of Jordan’s model is the integration of all the data. So in the hypothesis testing model, the test results in behavioral observations are combined with the initial clinical assessment data to address each of those hypotheses. We developed the hypotheses early on in the process and then we’re testing hypotheses throughout.
Dr. Sharp: I think that’s an important thing to emphasize just in case it’s flying by, [00:23:00] folks, that difference of testing different hypotheses versus confirming one single hypothesis. Stephanie Nelson talked about this in her episode, a year or two ago about biases and what we bring to assessment. And this is a big one, is confirmation bias and getting locked in and having tunnel vision about one particular rule out.
This is why, just as an aside, I’ve tried to move our practice away from the idea of a referral question for our evaluations and say, we’re just doing comprehensive evaluations. We’re not doing ADHD testing or autism testing, at least internally. Externally, marketing-wise, you got to work through that and how you present yourself in the community but internally, at least, we’ve talked a lot about getting away from referral questions to try to start with as much of a blank slate as we can for this very reason.
Dr. Chris: That’s right and a great [00:24:00] example. I appreciate you mentioning Stephanie. I think about her work and love every time I get to talk to her. She’s a real blessing to our community.
Thinking about how sometimes if we’re just focused on the referral question, it is likely the case that they need or they’re requesting testing because they don’t have insight into all the factors that are going on. So they may think it’s ADHD, the parent or the teacher may think it’s ADHD when it could be a sleep issue or it could be trauma or it could be anxiety. What do you do if you only respond to the referral question and you don’t open it up to investigate all the potential hypotheses, you may not look in the places or try to answer the questions that you should try to answer.
As we’re [00:25:00] going through the hypothesis testing model, I think we need to remember the difference between nomothetic and ideographic approaches. Those may be new terms to folks. Essentially what we mean by nomothetic approaches or nomothetic methods are that the psychologist uses a descriptive meaning that can be applied to a score based on a normative group finding.
So in a nomothetic approach, the normative meaning of the score is associated with the normative group. So little Billy gets a certain score on a CPT. We then compare that score to the normative group. It may or may not be true for little Billy. It may or may not be true in the real world. It may not have any ecological validity, meaning it may just be that that’s how little Billy did [00:26:00] on this very boring task in your office.
So nomothetic approaches in the world of ADHD evaluation include the CEFI, the BRIEF, or the BASC-3 has attention scales. So we’re getting that score and we’re comparing little Billy or whoever we’re testing score to the normative group.
Idiographic methods or approaches are when we then consider that data in the context of the client’s history, the referral information, and the observed behaviors, to understand who that person is, to understand the referral questions, to understand how we’re putting together the pieces of this data in the world, those types of measures.
So idiographic measures include the DIVA or the DiSC [00:27:00] or a lot of our clinical assessments or clinical evaluations or interviews. When you interview a teacher and the teacher gives you descriptive information, that is sometimes the best information that we get in evaluation and that’s different than nomothetic approaches.
So the key here though, the reason that I want to describe these two is that what I see it, when I’m consulting with colleagues, they will get a CPT score. That’s a nomothetic score that may or may not apply to the rest of the kid’s life. They will go to the bank or buy the farm with that score and say, oh, it’s ADHD because of this one score. We need to remember that that score is comparing it to the normative group. We’re going to talk about normative data. It may or may not apply to the ecology and to who little Billy is in different environments.
[00:28:00] Dr. Sharp: Right. So this is getting at that phenomenon of kids coming into the office and either tanking a CPT and doing terribly when their parents didn’t describe any issues with attention or impulse control or vice versa, where they do super well in spite of everyone in their real-life describing lots of struggles with attention and impulse control.Dr. Chris: The CPTs market as getting this real-life objective measure of attention. I think that for the listeners of the podcast, I encourage you to go back to Joel Nigg’s interviews with Jeremy, so great to discussing how children present differently over the years when you do assessments.
Jeremy, you and I’ve talked about this off-air over the years. It’s fascinating now that we’ve been in private [00:29:00] practice for as long as we’ve been in private practice, we get to see kids over the years and you get to see those kids come back in and they’ve developed in different ways and they’re different than they were a few years ago. And that is a critical piece of understanding how they may present differently to us at different times.
Dr. Sharp: It’s very humbling. I’ve made a lot of peace with being wrong over the last 8 or 10 years. Like, oh, that assessment is not written in stone. This is like a snapshot of that kid and now things are different. What does that mean? Oh my gosh.
Dr. Chris: I used to be so confident. It’s a good dose of humility. So let’s talk about CPT scores in general. CPT scores provide different [00:30:00] information than what can be gained from clinical observation or behavioral rating skills. They have the potential to incrementally improve our evaluations but we are not simply relying on the CPT scores.
We talked about a good friend of the program, Jordan Wright’s book, but Jordan Wright also in 2021 wrote an article called Psychological and Neuropsychological Underpinnings of Attention Deficit Hyperactivity Disorder Assessment. It’s an excellent article, including in the article is an integrative model for identifying ADHD. He walks through some of the nomothetic versus ideographic approaches.
He has this statement that I’m going to read aloud, while continuous performance tests have not emerged as diagnostically useful alone, they have proven to improve diagnostic [00:31:00] accuracy when there are discrepancies or other elements that render parent and teacher reports inconclusive. Two things to pull out of that so that; CPTs aren’t giving us diagnoses alone, but they can help us differentiate and help us gain more information when we have those complex cases.
So before using CPTs, we want to think about how we use them appropriately. Before we talk about that, I want to spend a moment discussing the history of CPTs because Jeremy, I find this fascinating. For those of you listening, I hope you check out the recent article Haley Goller, William Rice, and I wrote for the Journal Pediatric Neuropsychology. It was a review of the Nesplora Aula, which we’ll talk about at the end of this [00:32:00] podcast. I had so much fun writing the history part and I could have written a whole book on it.
By the way, speaking of books on CPTs, the Bible on CPTs is a book called Clinical Applications of Continuous Performance Tests by Cynthia Riccio, Cecil Reynolds, and Patricia Lowe. It’s 23 years old, and I haven’t found anything written since 2001 that comes close to it. It’s wonderful. It describes a lot of the history. I think it’s still applicable. If you want to learn more, I highly recommend diving into it and grabbing that book.
History of CPTs; CPTs aim to obtain qualitative and quantitative data regarding an individual’s ability to sustain attention over time. Their goal is to measure selective attention, [00:33:00] inhibition, focus, sustained attention, and response selection. Depending on which CPT you’re looking at, they will describe different categories or scales or measures, but generally speaking that’s what they measure.
I have a question for you, Jeremy. When do you think the first CPT was developed?
Dr. Sharp: Oh, gosh. I’m going to guess it was at some point in the early to mid-20th century and there was some military application.
Dr. Chris: Cool. That’s good. That’s a very good guess. It was 1956. It was developed to study vigilance. Let me know which present-day CPT comes to mind when I describe what it did. So letters were presented visually, one at a time. [00:34:00] The subject was instructed to press a lever whenever the letter X appeared. At the same time, the subject had to inhibit responding to any other letter.
Dr. Sharp: Sounds very familiar.
Dr. Chris: Yeah. Sounds like this Conners CPT 3, except for instructions are a little bit different, but I couldn’t figure out in 1956 how they were doing this. So I dug deep and it wasn’t a computer at that point. It was a revolving drum. They put letters on a revolving drum and then the subject looked through a small window so they could only see one letter at a time, and they saw the letters at a fixed rate of 920 milliseconds.
So every 920 milliseconds, they would see another letter and they press a lever whenever the X appeared. 1956, the researchers found that [00:35:00] the data from the CPT correctly classified 84.2 to 89.5% of younger subjects with identified brain damage. That’s when it begins. CPT stay in the laboratories for the next 20 or 30 years and over time they move from laboratory use into clinical settings as psychologists begin to have computers in their offices in the 70’s and 80’s and by 1993, over 100 different CPTs were in use.
Dr. Sharp: Oh, that’s remarkable.
Dr. Chris: 1993, 30 years ago, over 100 were in use. One psychologist said there are more CPTs in use than psychologists who are using them.
[00:36:00] So we’re going to talk about the strengths and weaknesses of the CPT and we’re going to talk about five or six of them. The field has improved greatly and we don’t have to pick amongst 100 different ones. They’ve narrowed down a lot.There are different modalities. Some CPTs are only visual. Some CPTs are only auditory. There are two that are both auditory and visual at the same time. For auditory CPTs, they have buzzes in tones or they say words or numbers like one or two. The visual ones are fascinating.
So in the book that I mentioned earlier, Riccio, Reynolds, and Lowe have these great tables describing a lot of the CPTs over the years, what stimuli they use, and how they measure things. The different stimuli [00:37:00] included letters like you have with the Conners CPT 3, colors, numbers, squares like the TOVA, tones, fruits, words, non-words, pictures of objects like the Kiddie Conners K–CPT 2, playing cards, or my personal favorite, animal noises.
Dr. Sharp: Animal noises.
Dr. Chris: Animal noises.
Dr. Sharp: This was great.
Dr. Chris: There was also one CPT in which the subject or the client was presented with 4-letter words, and they had to respond by pressing the lever every time they saw a f4-letter noun that [00:38:00] represented anything larger than a Volkswagen.
Dr. Sharp: Oh my gosh, I can understand why there were 100 and why there aren’t 100 anymore. These are completely ridiculous. Could you imagine, as a clinician trying to pick one of these, how do you even determine whether fruits or nouns bigger than Volkswagens is the right thing to be tuning into with your client?
Dr. Chris: Well, that’s right. The big focus here is that different CPTs place different demands on an individual’s attention, executive functioning, and memory systems. Some of the CPTs have minimal memory requirements, other ones have heavy memory requirements.
There’s a lot that goes into these different CPTs. We’ll talk about some of those details, but I think the thing that I’ll repeat a few times and [00:39:00] Riccio et al wrote about this in 2001, that CPTs do not provide useful information about whether or not a person has ADHD, instead, they provide information regarding self-regulatory systems in the brain often responsible for symptoms of inattentiveness in executive dysfunction.
Dr. Sharp: That’s great. That’s an important distinction. So there’s two layers here:
1. It’s not just ADHD, yes or no.
2. It may not even be like symptoms, yes or no.
3. It’s brain systems and maybe weaknesses in certain brain systems.
I just want to highlight that. We can simplify things a lot in these discussions and there’s some nuance here.
Let’s take a break to hear from a featured partner.
Y’all know that I love TherapyNotes but [00:40:00] I am not the only one. They have a 4.9 out of 5-star rating on trustpilot.com and Google, which makes them the number one rated electronic health record system available for mental health folks today.
They make billing, scheduling, note-taking, and telehealth all incredibly easy. They also offer custom forms that you can send through the portal. For all the prescribers out there, TherapyNotes is proudly offering ePrescribe as well. And maybe the most important thing for me is that they have live telephone support seven days a week, so you can actually talk to a real person in a timely manner.
If you’re trying to switch from another EHR, the transition is incredibly easy. They’ll import your demographic data free of charge so you can get going right away. So if you’re curious or you want to switch or you need a new EHR, try TherapyNotes for two months, absolutely free. You can go to thetestingpsychologist.com/therapynotes and enter the code [00:41:00] “testing”. Again, totally free. No strings attached. Check it out and see why everyone is switching to TherapyNotes.
The Clinical Assessment of Attention Deficit-Adult or CAT-A is a comprehensive instrument that assesses symptoms of ADD and ADHD in adults. A self-report tool sensitive to the symptomology of attentional deficits, both with and without hyperactivity is now available on PARiConnect, PAR’s online assessment platform. Learn more at parinc.com\cata.
Let’s get back to the podcast.
Dr. Chris: There is, and there are two things that I want everybody to think about when they think about CPTs. These are the two biggest things for me. Test publishers and other folks may focus on other stuff, but for me, response time and norms are the two things that as I’ve been researching them, are big differences between the CPTs. You can [00:42:00] focus on nouns larger than a Volkswagen or animal sounds but I think response time and norms are some differentiating factors.
Dr. Sharp: I just want to say before we totally dive into that, I have to give some appreciation for your depth of knowledge on the history of CPTs. That little detour was such a delight to hear about CPTs. I love that you’re the kind of person that will go there. So thank you for that.
Dr. Chris: I love the history of psychology and the history of how we got to where we’ve gotten, how these tests evolved over time. Some of them are old. Some of the tests that people have in their office are decades older than they know that they are. I find that fascinating.
Timing precision is super huge to me. It’s a big factor of how these work. [00:43:00] Individuals are presented with the stimuli, how quickly they respond to it is a huge factor in their specific cutoffs.
If you imagine a scenario in your office, Jeremy, or my office where psychologists are using different timing mechanisms to record the kids’ processing speed scores, and some of the timing mechanisms might be faster, some may be slower, or even worse, imagine a scenario where one of your psychologists allows a kid to finish the line on coding and the other stops it right when the time stops. The speed in which we start and stop our tests or respond to rules are very important.
And when we’re talking about CPTs, these measurements are in milliseconds. And so the cut-offs are in milliseconds. I [00:44:00] don’t know about you but we have lots of different computers in our office. Some of them run fast, and some of them run slow. We had one last week that was running super slow and needed to be serviced and there was a bunch of malware on it. Another one that needs to be serviced this week.
Over the years administering CPTs, I have watched computers glitch and have problems. There have been technical issues. We need to think about how the actual mechanisms of the computers may slow down or not grab a response as quickly as possible. The difference between using a keyboard and using something like a micro-switch on the TOVA may be a significant factor.
One of my psychologists, William Rice, and I, he helped write the Nesplora article. We totally geek out on this and look deep into [00:45:00] whether there are delays in the technology. We have found evidence that there are delays and that there’s research for delays that we should be mindful of, but some of the test manufacturers say that actually our new computers are essentially gaming computers that kids can game on them. So they have really fast response times. However, Will and I looking at it, we’re not sure that that’s been proven yet in CPTs.
I’m going to geek out a little bit, but if you think about the latency that occurs in some of these computers, researchers have looked at latency ranging between 23 milliseconds and 243 milliseconds, depending on many factors, including the computer speed and the peripherals like the mouse or [00:46:00] the keyboard or whatever’s being used.
And there are cutoffs on these tests. So if there is a delay in how the input is working its way through the system, that could impact where and how things are scored on these tests. We found, in the research, that latency is most reduced if you’re using a touch screen so something like an iPad, but if you’re using peripherals in which keyboards impact input, we documented and hopefully we’ll publish it at some point in the future, but we documented ways in which that can slow down the process of grabbing that data.
Dr. Sharp: And I just want to jump in and clarify, when we’re saying latency, for anybody who may not be familiar with that [00:47:00] term, that’s the delay between the input and the registering of that input with the computer. So it’s like the delay between hitting the space bar and the computer registering that you hit the space bar. Is that right?
Dr. Chris: That’s right. What I found is that the more apps and the more things that are running in the background, the more delays that can occur because the computer’s doing other stuff. What you’ll find is that some of the tests have a dedicated computer or a dedicated system so that they can control that speed of reaction, whereas in other tests, you can be playing video games in the background and surfing the web and watching YouTube while you’re doing a CPT and I worry about whether or not, and I don’t know for sure whether or not that’s impacting the data that’s being gathered.
Some companies, and we’ll talk about this, have latency [00:48:00] standardizing features where they will force the same degree of latency for every single response. So for example, we’re going to talk about a test that uses the Oculus virtual reality system, and that is a gaming system. And so they have forced latency that occurs at a certain clip so that there can be consistency with gaming. They can then use that same system for the CPT.
So we wrote in the article and hard to define it, but I’ve observed it over the years that there are CPTs in which the data may be significantly impacted by computer speed and background apps running concurrently and thus you might get inconsistent [00:49:00] response time data. There’s some tests that we’re going to talk about it, tests like the TOVA, the Quotient, the Nesplora Aula that use standalone devices to minimize confounding computer speeds.
Normally when I’m responding in The Testing Psychologist forum, I’m often mentioning my preference for response times and how some tests prioritize that. The other thing that we need to think about those norms. As we discuss different CPTs, we’ll see that the norms may be a significant issue.
Particularly, the question is, how does the psychologist justify using a test that was not normed on the population they are working with? How does the psychologist justify using a test [00:50:00] with minorities when it wasn’t normed to work with that population? We need to ask whether gender or ethnic bias studies were done and what those results were.
You may be a psychologist in a different country or somewhere in which you might not have normative tests for the data that you use, so think about ethical standard 9.02b. If there are limitations, we acknowledge those and we share those freely. That’s part of being a scientist, whether you’re a scientist-practitioner or you claim yourself to be a practitioner scientist, we’re psychologists and we identify the limitations of our data.
If you’re in the U.S., I think we need to be asking these questions to identify whether our practice is in line with ethical standard 9.02b. I think it might be time to start talking about the specific tests.
[00:51:00] Dr. Sharp: That sounds great. Let’s dive in. Where do we start?Dr. Chris: Let’s start with the TOVA. It tends to be one of my favorite because they prioritize timing precision and I like that. Let me ask you this, when do you think the TOVA was invented?
Dr. Sharp: Oh gosh, 1993.
Dr. Chris: 1966.
Dr. Sharp: Great.
Dr. Chris: It was designed to evaluate attention and inhibitory control. Nowadays we have both an auditory and visual TOVA, but they are different tests. So you can run the auditory test and then you can run the visual test.
The thing that I love most about the TOVA is it has a separate micro-switch. It connects to a computer and it focuses on precision with [00:52:00] timing. So I love that. I think that’s the greatest strength that it has.
Timing-wise, you can give the TOVA to a four or five-year-old and the test will last 10.8 minutes. You can give it to anyone over five and a half years old and the test will last 21.6 minutes. It is one of the longest CPTs. So that’s something to think about.
A cool thing about it though is the visual boxes, where the boxes are up top or below. It’s relatively culture and language-free. There are some instructions in the beginning, but it’s a relatively culturally free test, which is great. You can get different language versions, so you can get the instructions in Dutch or French, German, Hebrew, Korean, Spanish, and Swedish. [00:53:00] I love that about the TOVA.
While the micro-switch is awesome for timing, setup can be a pain, and set up can be so challenging that if you read the Buros review, they have reviews online and they have their yearly mental yearbook that you can purchase reviews of tests and you should look at these. I’m going to be talking about them a lot. They complained about the setup for two paragraphs in the review. It can be challenging.
I think one of the coolest things about the TOVA is the company that runs it has great customer support. So they help with setting it up. They help with training, with interpretation. They’re quite an active company. They’ll even sell you an old computer if you need an old computer to run [00:54:00] the test.
My concerns with the TOVA are the normative data. So we talked about normative data is important. The preliminary normative data is all from Minneapolis, Minnesota, and the surrounding areas. The kids were tested in the morning. 99% of the kids were Caucasian. They did get additional samples from the surrounding Minneapolis area, 1340 folks. Then they got 250 adult participants from local universities or liberal arts colleges around Minneapolis. 99% of those adults were Caucasian. There may be more data but that’s the data that I’ve been able to find thus far.
Similar [00:55:00] studies regarding the auditory TOVA. So I was just talking about the visual TOVA. The auditory TOVA, they got 2,551 students from around Minneapolis, which is a great sample size. 99% of them were Caucasian. They did not get many adults. So the manual actually says that if you’re using the TOVA auditory with adults, that you should consider it as an experimental test.
Dr. Sharp: Good to know.
Dr. Chris: Another cool thing about the TOVA is there’s been tons of good research over the years or tons of research in general. I think some of it’s good. I don’t know if I can quantify all or qualify all of it, but a lot of good research and a lot of research in general to dig over.
The Buros review is one of the most positive reviews for CPT. It does say that reliability and validity evidence is somewhat scattered with some evidence appearing in [00:56:00] multiple published studies. And then again regarding diagnosis, it says, although the TOVA may have a variety of useful purposes, diagnosis per se is not among them, just a reminder there.
Dr. Sharp: So two questions that come up for me with any of these measures that we’re talking about; method of administration, whether it’s local or web-based, and in the case of the TOVA and some others, is it specific to Windows or Mac?
Dr. Chris: Yeah.
Dr. Sharp: Yes, we’ll start there.
Dr. Chris: TOVA is in person. They don’t have a web-based version. We’ll talk about the CPT-3 and the Nesplora Aula. Those are in person. Web-based tests include the IVA or IVA [00:57:00] has both in person and web-based, the MOXO and the QbCheck, not the QbTest. QbTest is in person. QbCheck is online.
My concern about online is regarding response time. I have not found evidence of how they are dealing with that. I keep asking the publishers how they’re dealing with response time and I don’t think that that has been demonstrated in a way that I can share with our listeners. So I think it goes back to 9.02b. I would love the test publishers to describe how they’re dealing with response time over online. I know they’re downloading it into the browser, but that’s not enough for me to be confident in the online administration.
Dr. Sharp: That’s reasonable. Yes. And is the TOVA still locked to Windows or can you administer it online?
Dr. Chris: It is, that’s a great question for the company. I’m pretty sure that [00:58:00] it is because it launches out of DOS. I’ve had conversations with them about Apple products and whether they could use MacBook. I would reach out to them directly. I don’t remember if you can at this point.
Dr. Sharp: Sure. And pricing. Full disclosure, I started with the TOVA, gosh, probably in 2010 and it was great. And then they, did they switch or did my volume, I think my volume started to grow and you purchase by the test administration and so at this point, at $15 per administration, this would cost us $20,000 a year in our practice just to do the TOVA. So that was my main, like a lot of people, I think, a main reason to move away from it, even though [00:59:00] it’s pretty psychometrically sound, at least relatively.
Dr. Chris: It got a great response time. I love the precision timing on it. That’s my favorite part about it. All these tests are going to be expensive and some of them will have unlimited usage for heavy users like you and me but traditionally, we’re looking at $15 to $45 per use.
We’ll talk about the Quotient in a moment. That was $45 per use for some users. Some users were at $55 per use. Pretty expensive test to use. So for high-volume users like us, we’re likely going to steer towards more unlimited-use options.
And let’s talk about the Conners CPT 3. I believe they do have unlimited use. It is a USB connection to your computer, which works for a while and then needs to be [01:00:00] replaced and the company will replace it, but it’s a little bit of a hassle when it happens.
This Conners CPT goes back to 1992, was launched in 1994. So it’s in its third version. Best thing about the Conners CPT 3 is the norm group. They had 800 kids aged 8 to 17 nicely spread out and identical to the 2010 U.S. census regarding geographic regions, race, ethnic groups, education level. 205 Canadian kids, so some data for Canadian kids.
I like the Conners CPT 3. We used it for years. I also like their kiddie version although the data on that isn’t as strong, but kids loved it. It was cute, short, little pictures of things that they were looking for. It was about the most user-friendly, I think it was seven and a half [01:01:00] minutes, some limited data to support it.
The big concern with the Conners CPT 3 is the Buros review that Conners gives you 14 scores or 14 scales, 14 constructs maybe they’re called and it looks awesome. When you read the Buros review, however, it says that results suggest that scores on half the variables may not be as stable as needed for interpretation.
Test-retest reliability statistics on some of the measures evaluated by the Conners CPT 3 are marginal. If you’re at home and you’re using it, those include response style, perseverations, variability, HRT block change, and those scores or scales may not be sufficiently stable for individual interpretation. So again, [01:02:00] going back to ethical standard 9.02b, know your test. Know the strengths and limitations.
If you’re interpreting commissions and omissions on the test, great but if you’re interpreting perseverations, know the data. Know whether or not it’s strong enough for it to interpret. The Buros review suggests it’s not.
My general feelings about the CPT 3 are that I like it a lot, however, it runs on a computer, and I believe computer speed can be an issue. When I’ve used it, and my advice is to have a computer that’s just dedicated for that, that’s not connected to other things, doesn’t have other apps running in the background that might slow it down. If you’re doing that, then I think it’s a fine instrument in part of a multi-source, multi-method assessment.
Dr. Sharp: Great. [01:03:00] I love this so far. What’s next?
Dr. Chris: The Quotient; Jeremy and I are on Zoom, but you can see the Quotient behind me. I love it. The Quotient was great because it had great research behind it. It was developed by Martin Teicher at Harvard McLean Hospital and just had a lot of good data over many years.
The test was in its own, listeners can’t see it, robotic box. It included a MacBook Pro inside and the MacBook Pro could do nothing else except for administer the stars on the Quotient and run the cameras. So it had two cameras that took 30 frames per second and would track a little crown that the kid wore with a little silver ball on it and Jeremy sees it because I’m wearing [01:04:00] it right now. The kid sat on a swivel chair and it measured the kids’ movement throughout the task.
I absolutely loved that MacBook Pro made sure you were getting good timing because it didn’t do anything else, you couldn’t even get on the Internet. Safari didn’t even work on it. Nothing worked. You could just send the results to a cloud.
However, Pearson was charging $45 or more for it. I at least have great challenges even paying Pearson for what I agreed to pay them for my Q-interactive and other tests. They had a hard time running it and they finally shut it down. I’m told over the years that Martin Teicher or McLean Hospital has been trying to revamp it. It is not alive right now. [01:05:00] And so I grieve the Quotient.
Dr. Sharp: It’s a shame.
Dr. Chris: It is. Early on in its history, it was going to be marketed by QbTest and then QbTest left the U.S. market and was in Sweden for many years. They had key elements of the same Quotient programming, which to totally geek out was called the OPTAx. They have key elements that are still comparable between the Quotient and the QbTest.
So QbTest now is marketed in the United States, and you can get the QbTest. It includes a camera to track a child’s movement. I like a lot of things about the QbTest.
A few of the limitations are that I believe that their [01:06:00] normative studies are from Sweden and Germany. They’ve shared those studies with me. They look like good studies. I am not sure if they share the original normative data that they had with the Quotient years ago, and so if you’re using the QbTest, I don’t know about how well it compares to U.S. norms. And that’s a big question and I would like to hear from the publisher and figure out whether or not they have U.S. norms but in general, I like the QbTest because it has a lot of the same foundations as the Quotient, and I loved the Quotient.
Dr. Sharp: Okay. And that is available here, you said.
Dr. Chris: It is, yeah, it’s available here in the U.S. and in many other countries, especially in Europe.
Dr. Sharp: And is it the same method of administration with the self-contained box and the paper?
Dr. Chris: No, they use a computer and it’s [01:07:00] different. They do have a long pole in the cameras from very high up. They have some differences than the Quotient that we could spend more time on but I think for general listeners to hear, there’s a lot of good foundation there in the QbTest and I have some questions about how well their normative data compares to the folks that we work with in the United States.
Dr. Sharp: Sure. Great.
Dr. Chris: The next one to discuss is the IVA. The IVA is auditory and visual stimuli at the same time. So numbers ones and twos and you’re responding when you hear one of them but not the other. It uses a computer administration, so you download the program and you run it from your computer.
[01:08:00] My own personal experience is that computer speed has been a really big issue over the years. The other big issue is norms. I reached out to the company and asked them about norms. They’re on the IVA-2 right now is the latest version. I asked if they had norms for the IVA-2. They wrote to me, “since the actual test format has not changed from the IVA+Plus, the research that applies to the IVA+Plus applies to the IVA-2 as well”.Buros review on the IVA+Plus, which you can purchase it on Buros online describes the norms as being from a group of 26 children who were diagnosed by a physician or psychologist as having ADHD and a second group of 31 children ages 7 to 12 and in one [01:09:00] 15-year-old. And that’s the normative group.
Dr. Sharp: That seems small.
Dr. Chris: So the Buros review describes, “reliability and validity data are not sufficient”. “Reliability and validity data are incomplete”. And also in the Buros review, “for the most part, psychometric soundness of the IVA+Plus has yet to be demonstrated”.
Dr. Sharp: Okay. That’s not great. Anyone listening, that doesn’t sound great.
Dr. Chris: The other thing that I haven’t mentioned, Jeremy, is these tests have a lot of research that has been happening outside of norm development. So there’s a lot of research that includes the [01:10:00] IVA or it’s related to the IVA that folks could look into and maybe they’ll learn more.
Same with the MOXO. We’ll talk about the MOXO next. The MOXO is normed in Israel and Spain and they are in the online tests, so they’re grabbing data all the time. They’re getting this huge database and I think they’ll have some cool stuff.
I reached out to them to ask them about U.S. norms and they said that they don’t have plans for U.S. norms at this point or developing U.S. norms at this point. I also asked them about equipment; what equipment were the normative individuals or the normative tests done on and they wrote back and said, “we don’t know the specifics on the equipment they were [01:11:00] using”. So I have concerns about that. I want to know more about the equipment that was used in the normative studies.
And then lastly, it’s an online test, so I wrote them about response time and asked them about that. They wrote back, “regarding response time virtually, all computer devices have been produced in recent years have an immediate key to response time, mostly thanks to the gaming industry. This is what we rely on”.
So that’s what I was referencing earlier. That could be good enough for some computers. It could be good for all computers. I have questions about it, and it’s something that I’m diving into trying to learn more. I think if they were to have U.S. norms, then I would want to dive in more with the response time and learn more about how they were describing that and whether or not that would then meet the [01:12:00] 9.02 hurdle for us.
Dr. Sharp: That’s fair.
Dr. Chris: Last test to talk about, the Nesplora Aula. We recently wrote a review, which I’d encourage you to listen to. The Aula is very cool because it is the only CPT that I think has ecological validity, meaning that the kid is using virtual reality glasses and the kid is essentially in a classroom listening to a teacher. So on the face of it, that sounds perfect because we, as psychologists, unless we go to the school and we get to observe the kid in the classroom, we may not have that data.
And it is really cool. It’s both auditory and visual, and the kid has to respond to stimuli from the teacher. I think in design, it’s excellent. The issues though, are [01:13:00] that the graphics, think about how long it takes for something to get normed and developed. The graphics are a bit outdated. So you have these really cool virtual reality glasses but the graphics aren’t quite up to snuff compared to other games that the kids might be interested in.
And honestly, Jeremy, one of the coolest things is when the kids see the glasses, they get really excited about it, which doesn’t happen for any other CPT. If you have the glasses then in your office, the kid can take a break and play games. You get the benefit of having the Oculus glasses and games in your office. So outdated graphics.
I started using the Nesplora Aula in 2016 after Sam Goldstein discussed it at an APA conference workshop, and I’ve liked it a lot. Kids love it [01:14:00] until maybe they’re in the classroom and then they realize that the task is a bit boring, but it’s supposed to be. It’s broken up into seven-minute sections, which I think are more doable.
We’ve had a very good experience using it and trying it out. Some concerns come back to those norms. So the dorms are from Spain. They look good. The data looks good. We analyze that data for the review. The Nesplora folks have told me that they are working on developing U.S. norms, so that’s something that we can look forward to in the future. They’ve been very responsive to my critiques and my questions over the years and they continue to do more research, which I think is very positive.
The other thing that we’re working on reviewing is some of the other tests [01:15:00] that Nesplora has for the virtual reality glasses. So they have an executive functioning test in which the kid is behind an ice cream parlor desk, it has to serve ice cream and the folks come in and they give complex requests and orders, and the kid has to remember certain rules to apply, and it’s fun for a little bit, and then it gets hard and kids don’t always love it.
Nesplora also has an adult version of their CPT called Aquarium in which the adult is visually in an aquarium and has to watch different fish swimming around and respond to some fish but not to others. It has a working memory component and we have not yet reviewed that formally but [01:16:00] we’ve started to write that up. So I don’t know if we’ll ever get around to publishing it, but it’s something important for people to know.
Regarding pricing, it is expensive because you have to buy the glasses and then you have to pay Nesplora for the license, but they do have unlimited licenses. So I think once they get to having a U.S. normative data sample for us here in the US, it’s going to be a very attractive CPT for a lot of us.
Dr. Sharp: Sure. We have talked about a lot of CPTs. This has been fantastic. I just want to make sure that I am tracking all this. Maybe recap for folks who are listening as well, the big three to me, basically, there’s only three worth considering essentially, the TOVA, which is great for response time, the isolated micro-switch environment, but the norms were not good. It’s all white kids from [01:17:00] Minnesota.
The CCPT 3; good norms but problems with administration and response time because it’s just local to the computer and you don’t know what the computer is capable of or interfering with and whatnot. And then the Nesplora, pretty good all-around but again, the norms were collected in another country and we don’t have U.S. norms at this point. Is that a fair summary or would you add or change anything?
Dr. Chris: I think folks are going to look at QbTest and look at IVA. I think IVA has been popular in the past because it has some unlimited administration licenses. My hope is while reviewing and talking about these tests, that they’re going to keep pumping out research and improving the quality of the tests.
So I think summarizing the tests, I think that’s a fair summarization. [01:18:00] I will say that I want everybody to remember that CPT results, just like all the results that we get when we’re testing a kid in our office can be impacted by sleep and motivation. I know for Jeremy and I, our results would likely be impacted by whether we exercised right before.
Dr. Sharp: That’s a good point.
Dr. Chris: Or the day before, what test you did before, and when you administer the test in your battery. What the kid has going on afterward can impact results. So I think it’s important for our listeners to remember if you’re using CPTs, use them within a multi-source, multi-method assessment in which you’re gathering data from the teachers and from the parents and you’re using other tests. Know the norm data, know-how response time is accounted for, and grab a good friend of [01:19:00] the program, Jordan Wright’s book, and use that hypothesis testing model. It’s tried and true.
Dr. Sharp: Sure. The thing that I’ve appreciated about this conversation; is you are exquisitely prepared, as always. I knew that you would come to this discussion with so much information and research and that’s necessary in a discussion like this, where we got to sort through so many different options and nuances and so forth. The hope is that this cuts through some of the confusion for folks, there’s a lot out there, and bringing a balanced perspective to it is super important.
You gave us so many resources. There are going to be a lot of resources in the show notes; the books, the articles, all the things that you mentioned, we’ll make sure to list there in one place so that folks can go [01:20:00] check it out for themselves, which I imagine a lot of people will do.
Dr. Chris: That’s great. Remember that some of those folks that we mentioned, like Jordan Wright and Cecil Reynolds are part of this cycle of The Testing Psychologist forum and frequent contributors. It’s just such a wonderful environment for us to have these conversations and discussions and talk about the strengths and limitations of what we do and work towards improving our practice.
Dr. Sharp: Yeah. I love that. I will maybe close with a little note. I would love to hear how you describe these limitations in your reports because we’ve leaned on that pretty heavily. Like, hey, there are limitations to all of these measures. So what does that look like in your reports if you wouldn’t mind sharing quickly?
Dr. Chris: Sure. So let’s start most importantly by, it should impact how we’re interpreting the data and the weight that [00:21:00] we’re placing on the data when we’re interpreting it. So I think what it looks like in our reports is that we’re not overemphasizing one score over another. We’re talking about the ecological system and how many pressures and factors are impacting this child.
I use Stephanie Nelson’s report writing format and I am so appreciative to her for providing us with that. So it’s providing the strengths and limitations or strengths and vulnerabilities of who this child is in understanding the child from as many perspectives as possible. Regarding the tests themselves, I think that if you’re using a model in which the test results are on the back, I think that you can then have descriptions of what the test is and [01:22:00] potential limitations to the test.
The bigger challenges, whether or not, if you’re doing that throughout your report, meaning that you write the testing results throughout in a traditional way, writing IQ scores and memory tests and so forth throughout and having charts, then I think you can include descriptions of the test, including those limitations.
I will tell you how I do it in a lot of the forensic reports that I write for the court is I use footnotes. The footnotes will include a description of what the test is and the strengths and potential limitations of the test. I find that if I document those clearly, I’m less likely to experience an attorney trying to trip me up on [01:23:00] the reliability or validity statistics about a certain test.
Dr. Sharp: That makes sense. That’s great. This is great. This has been a fantastic conversation. It’s aggregated information that a lot of people are interested in and you’ve done a great job pulling it together from all the different places across the Internet and others. So I really appreciate it, Chris. Thanks for being here.
Dr. Chris: Thanks for having me, Jeremy.
Dr. Sharp: All right, y’all. Thank you so much for tuning into this episode. Always grateful to have you here. I hope that you’ll take away some information that you can implement in your practice and your life. Any resources that we mentioned during the episode will be listed in the show notes so make sure to check those out.
If you like what you hear on the podcast, I would be so grateful if you left a review on iTunes, Spotify, or wherever you listen to your podcast.
If you’re a practice owner or aspiring practice owner, I’d invite you to check out The Testing Psychologist mastermind groups. [01:24:00] I have mastermind groups at every stage of practice development; beginner, intermediate, and advanced. We have homework, we have accountability, we have support, we have resources. These groups are amazing. We do a lot of work and a lot of connecting. If that sounds interesting to you, you can check out the details at thetestingpsychologist.com/consulting. You can sign up for a pre-group phone call and we will chat and figure out if a group could be a good fit for you. Thanks so much.
The information contained in this podcast and on The Testing Psychologist website is intended for informational and educational purposes only. Nothing in this [01:25:00] podcast or on the website is intended to be a substitute for professional, psychological, psychiatric, or medical advice, diagnosis, or treatment.
Please note that no doctor-patient relationship is formed here and similarly, no supervisory or consultative relationship is formed between the host or guests of this podcast and listeners of this podcast. If you need the qualified advice of any mental health practitioner or medical provider, please seek one in your area. Similarly, if you need supervision on clinical matters, please find a supervisor with expertise that fits your needs.