Blog

[00:00:00] Dr. Sharp: Hey everybody, this is Dr. Jeremy Sharp, and this is The Testing Psychologist podcast episode 24.

Hey everybody. Welcome to another episode of The Testing Psychologist podcast. I’m Dr. Jeremy Sharp. I am really excited to be talking with our guests today. I have with me Dustin Wahlstrom and James Henke. They are going to be talking with me all about Q-interactive.

I’m sure a lot of you have heard of Q-interactive. It’s the digital platform for administering a lot of the tests that we use. And like I said, I’m so excited to have these two guys here so that we can talk some ins and outs of the platform, what it’s all about, where it’s headed in the future, and [00:01:00] all sorts of cool stuff. So welcome to the podcast guys.

Dustin: Hey, good morning. Thanks for having us.

James: Thanks so much. We’re looking forward to it.

Dr. Sharp: Oh, good. Same here. Let me do a quick introduction for each of you and then we can dive into it because I think there’s a lot to talk about this morning.

Dustin Wahlstrom is the product owner for the Q-interactive project at Pearson. He has a PhD in clinical psychology from the University of Minnesota and completed his clinical internship at Children’s Hospital of Minnesota. He joined Pearson in 2009 and was a research director for the WPPSI-IV and WISC-V prior to working on Q-interactive.

James Henke is the Q-interactive Product Specialist and National Trainer for the digital system. James’s background is in education, graduating from the College of Education and Human Development from the University of Minnesota in 2001. Prior to joining Pearson, he worked in Japan teaching English and also as a 3rd grade [00:02:00] teacher for the Minneapolis Public Schools. James lives in Minneapolis with his wife and two children.

Welcome, guys.

James: Hi.

Dr. Sharp: Did y’all know each other at the University of Minnesota?

Dustin: We did not. No, I don’t think we were even there at the same time. I don’t think we probably were.

James: We’ve never actually had that conversation before.

Dr. Sharp: Oh, okay. So just a coincidence, it sounds like.

Dustin: Yes. Pretty much everyone in the state goes to the University of Minnesota at some point in their lives, I think.

Dr. Sharp: Got you. That sounds good. One of my professors in graduate school, Bryan Dik, was at the University of Minnesota for graduate school. He does a lot of vocational research, meaning and that kind of thing. I know there’s a great program up there.

So glad to have y’all on the podcast. Between the two of you, it sounds like you cover a lot of the goings-on with Q-interactive. I [00:03:00] think we have a lot to cover today. So I’m just going to jump into it.

My history with Q-interactive, I feel like, has been a long one. As someone who loves technology and tends to be an early adopter of technology, I was so pumped when I found out that we were going to be able to do psychological testing using iPads.

I remember that day, I called my wife and I was like, I really need to buy two iPads for the business, and she’s like, okay, can you explain this a little bit? And then it’s like, no, of course, this is the real deal. We’re able to do tests on the iPad. That was a really exciting moment for me back then.

Dustin: It is a great excuse to buy iPads. I will give it that.

Dr. Sharp: It’s great and always a good reason to have technology in your practice. I find that the kids really like it. We test a lot of kids and they love touching things on the iPad. It’s a nice [00:04:00] transition for them.

Anyway, long history with y’all and you’ve been helpful over the years with getting things set up and troubleshooting some issues here and there so I wanted to give a little bit of an opportunity for others to learn more about it now that things have come along and matured in the platform.

Dustin: It’s funny you say that because this year is going to be the 5-year anniversary of when we initially launched our beta. We would have launched the beta for people at APA in 2012. It’s already been 5 years. It’s amazing how fast time flies.

Dr. Sharp: Oh, I’m sure. Do you just feel like you blinked and that time was gone?

Dustin: Yeah.

Dr. Sharp: I know it’s been busy for y’all.

James: It’s also been nice to in the context of, Dustin and I have been together on Q-interactive for almost all five of those years or four, give or take. By and large, [00:05:00] we have a lot of consistency on our end of people who are working on it on the back end, on the sales and marketing to a certain extent. It’s a nice group, it’s a nice working environment. We’re all excited about how this product has grown and where it’s going to go.

Dr. Sharp: I’ve definitely noticed that from my side too. Your names have been very consistent as some of my points of contact over the years. And that’s honestly surprising. That’s not always the case with a lot of companies and so it’s been really nice from the user side as well.

Just to get started, could y’all maybe just talk about what is Q-interactive exactly, how’d that come about and what’s some background on that?

James: Sure. [00:06:00] Q-interactive basically it’s iPad administered tests. The tests that we’re talking about would be one-to-one administered instruments like the WISC, the WAIS, the WPPSI, achievement tests, some speech tests, and neuropsychological tests.

The one thing that they all have in common is it’s one-to-one testing environments. We’re not talking about group-administered tests here. One iPad is going to be in the practitioner’s hands and the other iPad is in front of the client, and that’s where visual stimuli will appear.

On a test like Matrix Reasoning, for example, you’ll have the picture show up, and the options show up on the bottom half of the iPad screen, and the client [00:07:00] touches their response, it shows up on your side instantaneously as the practitioner. Automatic scoring is integrated into the system. You’re able to move through the content in a very seamless way. There’s a nice flow as you move through your batteries.

From a more technical perspective, Q-interactive basically has two fundamental pieces to it; it has the iPads where you are administering and capturing the data and then it also has a website where your data is organized and stored long term. It’s where you’re able to generate your reports and manage your account.

You have a website where data is stored, you have your iPads, you’re mobile, and you’re able to administer these tests in any location. Q-interactive doesn’t require [00:08:00] you to have a Wi-Fi connection when giving these tests. Even if there’s a power outage, your testing needs, they’re not at risk. You can still maintain those appointments.

There’s a lot of great features within the system that take the feel to the next level. Things like iPads have microphones on them. We’re able to make audio recordings of what the client’s responses are to aid you in the review of the data post-administration to ensure accuracy and that type of thing.

Dr. Sharp: That’s been really helpful.

James: You’re able to instantly get your scaled or standard scores, your index scores. As soon as you swipe off the last item in your battery, you have access to a comprehensive score report [00:09:00] for whatever test you’re giving immediately.

We can talk a little bit about time savings. We’re not changing the actual duration of the tests themselves, because we’re still maintaining some equivalency to our paper counterparts but when we talk about the work you do setting up the test session, or in particular, after the test session is over, there can be a significant savings in time so that you’re not doing as much math and more just taking the data that’s been calculated and working with it, interpreting it and applying it to your diagnoses and your practice.

It’s basically two fundamental pieces, a website and an app that’s on two iPads, and that’s where you’re able to administer the tests.

Dr. Sharp: That’s fantastic. Were either of you around in the beginning when the idea for Q-interactive was being developed? Can you speak to [00:10:00] that at all?

Dustin: Yeah. I think that’s an interesting question because the origin of Q-interactive wasn’t necessarily one around how we digitize tests. The goal wasn’t necessarily just to come up with a way for us to computerize the WISC or do anything like that. The charter was what can we do to transform assessment and make the practice of assessment better for all of our customers.

There are a lot of things on the table for that and not all of them were digital necessarily. So it wasn’t the sort of thing where we wanted to do technology for technology’s sake, just because it was being made available.

As the original team, and neither of us was on that original team, but as that team went through the various options that they had available to them, especially with iPads becoming available at that time what [00:11:00] ended up becoming Q, I bubbled up is the idea because of all of the possible benefits that it could provide us in terms of accuracy, efficiency and yielding better results and ultimately better outcomes for people.

Dr. Sharp: That’s wild. Can you remember, this is just out of my curiosity, what some of the other options for making testing easier were that got cut or didn’t get pursued?

Dustin: That’s a good question. I don’t know what they are because neither of us were around at the time.

Dr. Sharp: Got you. Okay.

James: Tablets first came out when the first iPad came out in 2010. When we look back to those initial stages where Q-interactive was just an idea, from a hardware perspective, there weren’t really a lot of options out there. Laptops were pervasive out there, but in terms of a test-taking tool, it was really when [00:12:00] iPads came out that it started to gel and make a lot of sense.

Dr. Sharp: I think that’s a good example of when preparation maybe meets opportunity, like you were ready, the iPads were ready and just going for it.

James: That’s a very good way to put it.

Dr. Sharp: Sure. I know that a lot of discussion that I’ve heard around Q-interactive happens about how can we translate the paper and pencil version to a digital version, it seems like those are very different on the surface. I’m curious about the development, the research, standardization that went into translating these tests from paper and pencil over to a digital platform.

Dustin: There’s two different ways to think about that. One would have been all of the initial work that was done originally when we were getting Q-interactive up and running, and then [00:13:00] what we’re doing now as part of our standard development process now that it’s been around for a while and incorporated in everything we do.

I don’t know the exact figures on this but when we started, we would have done in the order of hundreds of interviews with people and spent a lot of time talking with psychologists before any test was designed before there was any prototype or concept created a single line of code written.

I think that’s one of the most important things that the Q-interactive team has done in the development cycle was get out there, talk to people, and not even just talk to people, watch them do their work, whether it be the testing itself, whether it be setting up an assessment session, analyzing their data after converting that data into reports. So really understanding that whole workflow so that we could ensure whatever it was that we [00:14:00] built and how we built it was going to match onto that.

Once we did all of those interviews, we came out with a list of things that we thought Q-interactive was going to have to be. So it was going to have to be design-focused. It was going to have to make people more efficient. There are various pain points that we saw watching people do their work that we knew had to be addressed by the system.

It needed to be consistent. So if you know how to give Block Design on the WISC, you know how to give it on

the WAIS. If you know how to give Vocabulary, you should know how to give Comprehension because while the content differs, the mechanism for administering those tests is the same.

Once we had all of those design goals, there was a lot of iterative designing and prototyping. So we’d come up with concepts, [00:15:00] we would go out and test those concepts, based on that feedback, we would go back and redesign and so on and so forth.

A good example of that we talk about a lot is the CVLT. For people familiar with the CVLT, it’s a list learning task. It’s relatively difficult to administer on paper because you need to write down verbatim all of the examinees’ responses as fast as you can, and some people provide those responses very quickly and so it’s hard to keep up.

Dr. Sharp: I remember those days, that was a nightmare.

Dustin: It is. And so you create a workaround, so you write the first three letters of the word or whatever it is, so you can keep up. And so initially we created a design where you simply just use buttons but that doesn’t really work because you need to be able to capture intrusion errors.

And those intrusion errors are important because there’s scores that are dependent on them and there’s scores that are based on the [00:16:00] ordering of the words. And so we then had people just hand write on the iPad. That was a nightmare. No one could keep up.

It was very obvious very quickly that wasn’t going to work. So we went back and we did a hybrid approach where you had buttons and a little area to handwrite. We even noticed there that it was hard to keep up and go fast enough.

Eventually what we did was a hybrid where you have the buttons and a dual handwriting area where you can jump from box to box to write, and instead of hitting a return button to make a word go up in a list, simply by switching boxes, the computer puts it up in the list for you as it was finally was that piece of the interface that we were able to obtain the speed necessary.

All that work was done before a single data point was collected on that test at all. We have stories similar to [00:17:00] that for a lot of the subtests that we ended up creating for the platform.

James: One thing that I could add to that too, Dustin, is the design elements that we developed, for example, the CVLT, were able to apply to several other tests, as Dustin alluded to. This helps significantly from a training and learning perspective, meaning as you become familiar with one instrument from top to bottom, you are going to become familiar with other instruments in your library that maybe you haven’t seen before or that you haven’t administered in a long time.

So you don’t need to spend necessarily a ton of time learning every single test over and over again because of these consistent design elements and a lot of that groundwork was laid in [00:18:00] the early stages. It helps from a training perspective and that is a benefit to our customers and that they’re able to pick things up potentially faster than they maybe initially thought.

Dustin: That’s a good point that at no point where we really designing a subtest, we never really designed WISC-V Vocabulary. So the first step in that process was to group all of our subtests that had similar response demands for both the examinee and the examiner and then look across those and to the best of our ability, design a single interface that could support all those different subtests.

Dr. Sharp: I definitely noticed that with the WPPSI. The WPPSI just came out. I think it was one of the more recent tests to go to Q-interactive. We’ve given the WISC a lot and then the transition of the WPPSI was seamless. The first time through it was certainly doable. [00:19:00] I think that speaks to the consistency that y’all have been talking about working to develop. So that makes sense.

Dustin: And so all of that leads us up then to the point where we started collecting some of that equivalence data. It’s important to note as we start to talk about equivalence, that it wasn’t the case where we went out and did all of these designs and then at the end cross our fingers, do an equivalent study, and hope to God that they come out the same.

We started that design process with equivalence in mind. Knowing that that was where we’re going to go, we made design decisions accordingly and so in that respect, the equivalent studies at the end really become a confirmation of what we were trying to do the entire time as opposed to just a study that helps us figure out what’s going on in terms of the differences.

And so you see that [00:20:00] the equivalent decisions throughout the platform. Block Design, for example, still uses the blocks. You can imagine creating a completely digitized version of that task, but it’s obviously not going to be equivalent and it’s not going to be measuring the same thing either. I don’t know what it would be measuring and how it would be different, but I think it’s fair to assume that they wouldn’t be exactly the same construct.

Dr. Sharp: I don’t want to maybe get bogged down in the statistics, the standardization, and all of that, because largely I would end up sounding like a fool, and ask those questions.

Dustin: Me too, potentially.

Dr. Sharp: I am curious, I think that’s the main question is, can we trust that the scores we’re getting and the performance we get from the Q-interactive interface is the same as what we would get using paper and pencil with these tests? Can you speak to that?

Dustin: Yes. At a high level, the [00:21:00] goal of equivalence is to allow us to use the paper-pencil norms. So for the most part, none of the tests on Q-interactive have been normed independently using the Q-interactive version. And so what we’re doing is using the paper norms but then establishing equivalence through these studies that then allows them to be applied.

And so you’re right, I won’t bore you with the different methodologies. There’s a lot of different types that we’ve used, such as test-retest or equivalent group designs that depend on the test and the construct being measured.

The high-level idea is that we’ve set an a priori threshold for equivalence at the outset. And for us, that’s an effect size of 0.20. So that’s going to be 1/5 of the standard deviation, which ends up being about half of a scaled score point. So essentially we’re saying that when we do these studies, if the paper digital [00:22:00] format effect is less than 0.20, we’ll assume that these tests are equivalent and that the norms can be used interchangeably.

I think there’s maybe two interesting things to point out about the study, that is the studies that aren’t necessarily statistically based but can help give people some confidence in them. One is examiner training. The huge thing for us was making sure that the examiners were trained properly on Q-interactive prior to doing any of the data collection.

The reason obviously is that if we found a format effect, we wanted it to be due to the interface itself, not due to the fact that the examiners felt much more comfortable in one format than the other and thus did a better job in paper than they would have in digital, which could happen. People have decades of experience testing the WISC in paper so you could [00:23:00] imagine that just that familiarity alone could introduce some format effect.

The second was the importance of video recording. So we recorded most of, I won’t say every single session, the equivalence testing sessions. The reason was that there were instances where we found nonequivalence between two subtests.

And in those cases, when we found that, the first step would be to go back to the data and then the videos to see if we couldn’t identify what was introducing the nonequivalence. And then once we did that, to the best of our ability, we would change the interface and then retest it until we could get that equivalence established.

And so through all of those processes, if you were to go and look at our equivalence tech reports, which we have published on our website, helloq.com, you’ll see that almost all of the tests within Q-interactive are under that 0.20 threshold. [00:24:00] There were two subtests early on in the WISC-IV, I want to say it’s Matrix Reasoning and Picture Concepts that had an effect size slightly above 0.20. So it would’ve been like maybe 0.22 and 0.28, both in favor of Q-interactive. So scores were coming out a little bit higher on digital.

Through the videos and everything else, we looked at the data, and were unable to ascertain exactly what was causing those effects. So we don’t know for sure. We think it could potentially just be engagement. Those were two of the only tests on WISC, for example, that required the child to really interact with the tablet device. That’s an educated case guess, so we don’t know for sure.

It is interesting to note that when we redid the WISC-V study, which has both of those subtests in it, we weren’t able to replicate those results. When we did the WISC-V study, both of those subtests without really there [00:25:00] being much change to the design at all, both came back with an effect size below our 0.20 threshold.

People can go check out the research. I think they’d find that the methodology is very thorough. It’s a pretty impressive research program. Almost 100 subtests have been tested now with thousands of examinees. So the database is getting pretty big for us.

Dr. Sharp: That’s great. That’s fantastic. I’ll ask one more quick question then move to a little bit of a different topic. Are there plans to do standardization with the iPads?

Dustin: Yes, some of that stuff is already in progress. For example, hopefully, very soon, we’ll be releasing the WISC-V Spanish. [00:26:00] The WISC-V Spanish will be, I believe, completely standardized with the iPad version. And so we’ll see more and more of that going forward and perhaps the equivalent studies will then equate that standardization back to a paper form as opposed to what we did originally.

Dr. Sharp: Right. Oh, that’s fantastic.

Dustin: One other maybe interesting thing for people is that just because we establish equivalence in a normative sample doesn’t mean necessarily that the results apply to an individual with some sort of clinical condition. I think the next phase after equivalence is establishing the validity of Q-interactive through some of the clinical studies that we typically publish with our paper tests.

We’ve started to do that with the WISC-V, for example. We have tech reports on our website that show the performance on Q-interactive for various clinical samples; [00:27:00] ADHD, learning disabilities, intellectual disability, autism, things like that.

What we found so far is that the patterns of performance are very similar. I don’t think, for any of us, it’s much of a surprise. The reason it’s not is we purposefully tried to keep that test content and the interfaces as similar to paper as possible throughout the development process in order to support that construct equivalence.

Dr. Sharp: Sure. That’s interesting. I like that we’re talking about this a bit because the main thing I would think about is maybe motivation with kids, maybe individuals with ADHD, I don’t know, I’m just guessing, it might be more motivated when there’s an electronic device present. So the fact that y’all are looking at that, I’m trying to account for that, I think it is important.

Let me switch [00:28:00] just a little bit and maybe talk more about day-to-day practical uses for clinicians. I’d be curious who do you think clinician-wise would be a good fit for Q-interactive? When should somebody start to consider using Q-interactive instead of paper and pencil? What are some benefits of switching, that kind of stuff?

Dustin: If we start with maybe the benefits and then go into who would maybe be the ideal type of person to switch. One of the things you just mentioned around motivation and ADHD, for example, points to what I think is one of the best benefits of Q-interactive and that’s the engagement piece. We all knew when we started the project that people, especially kids are probably going to like iPads more than paper but we’ve been pleasantly surprised by how big of a deal this has been to people.

[00:29:00] I think it’s really important for two different reasons; one is test validity. One of the major threats to the validity of something like an IQ test, for example, is going to be motivation. We don’t want Susie getting a low score on the WISC because she doesn’t feel like completing the items, we want her to be able to demonstrate her best ability. By being engaged and finding the test exciting, people are more confident that kids especially are doing that.

I think it decreases effort on the part of the examiner. For anyone who does testing with kids, you know that, maybe I was just a bad clinician and that’s why I’m here at Pearson now, but I remember being under the table, for example, especially with the WPPSI, or bribing someone with an M&M to get them to finish off a subtest. That can be exhausting emotionally when you’re doing that for two hours.

I think [00:30:00] there’s a benefit to the clinician doing the work if the person is easier to manage and they’re having more fun. And then I think there’s something about making the whole experience with the psychologist more positive.

I remember an anecdote from someone lately who was working in a school who said, I’ve had kids stop me in the hallway and ask if they can come back to my office to play with the iPads and that never happened before. I think kids can have legitimate test anxiety and so to make the whole idea of coming to see the psychologist and playing these games or whatever a more positive experience is a good thing.

James: Let me also piggyback off of that a little bit and relate another anecdote, which is, I was talking to someone who’s been using the system for two years earlier this spring. [00:31:00] He said something that I hadn’t heard before, which was, when he would be giving a WISC in paper form to one of his students that he’d have to schedule the test session around having two or three different breaks because of not just test anxiety, but just test fatigue. A WISC can put your head through the wringer as you move through the content if you’re being administered it.

After adopting and using Q-interactive, he said that he doesn’t schedule breaks anymore when he sets up the test time because the kids aren’t having that fatigue, and that engagement with the iPads has kept their energy up as they move through it. You’re having a more engaged client in the testing [00:32:00] process versus paper.

That’s just anecdotally. I’m not referencing any studies along those lines, but just anecdotally that in general, that engagement gives you a more motivated client, which you would think might yield more solid data.

Dr. Sharp: Yeah, I could see that. That’s great. What are some other benefits do you think to using Q- interactive? Are there any others that come to mind?

Dustin: Two of the other big ones that resonate with people who are using it right now are going to be efficiency and time savings. A lot of that comes, as James said earlier, from the scoring piece. It may be because of engagement like we were just talking about, or maybe the ease of just swiping through items that you save some time while you’re actually doing the testing, but not having to score a protocol entering that data into a [00:33:00] scoring program to get your report, et cetera, saves a lot of time.

We’ve done some data collection with some customers around how much time they’re saving to see anything on average from around 30%. Some people report more than that and some people less obviously but I think a lot of it is coming from the activities that don’t necessarily happen while the tests are being administered.

Another big one is portability. You’re not carrying around a bunch of kits. We have a lot of people, especially in the school psychology space who, and this was news to me having only worked in clinics, configure the trunks of their cars and turn them into a storage unit so that they can fit all of their kits because they’re going from school to school and then you got to take all those kits out and bring them back into your car and so on and so forth. So the idea of having two iPads, blocks, and [00:34:00] two pieces of paper for response booklets or whatever, and putting them in your bag is a much better thing.

We had someone tell us in New York that they rented space to keep all of their kits. And so by being able to get rid of those, they were saving money by not needing the rent or whatever it was to do all their storage. We hear funny anecdotes like that all the time.

What one person, this was a really good story, was in Alaska in a tiny little prop plane would fly out to these areas. She was on the coast and then she would fly inland to these various places to do testing. These planes are so small and they need emergency materials like food or stuff like that in case they crash, I don’t know, and they have a weight limit. [00:35:00] She had said, gosh, to the extent that I don’t have to bring all these kits with and I can replace it with two iPads, I have more weight for a coat and food. That’s a little dramatic but saving lives.

Dr. Sharp: Yeah, Q-interactive can save lives.

Dustin: That’s right.

Dr. Sharp: Oh, that’s great. We’ve definitely noticed all those advantages here in our practice. It does make a big difference storage-wise. I remember those days in graduate school, I worked for this neuropsychologist and he had three graduate students and we would have to trade these materials between testing days. And so we’re lugging around these two rolling carts and having to coordinate tradeoffs. It was terrible.

Dustin: Or if you want to change what you’re doing in the middle of a test battery and you’re all sharing materials, that can be really hard to do. So now I want to give something from the WIAT but James is upstairs [00:36:00] using the WIAT with someone else.

In Q-interactive, one of the great things is that you’re getting scores as soon as the subtest is over. Even if you’re testing without Wi-Fi, you have the ability to add in new subtests or take subtests out of your battery at any time right there from the iPad. So you’re able to use real-time data to make good decisions as you’re testing and then seamlessly add in new tests so that you can really do a better job of personalizing in an efficient way, the tests that are being given to your examinee,

James: Which also means that, let’s say you were administering a test in a clinic to a client in paper, once that’s done and you take a look at that data, you may have to schedule a second test session with that client a few days later whenever scheduling works out.

[00:37:00] So when you’re in the process of doing an evaluation of a client in private practice, you could be looking at this over several test sessions, spread over a week or however long, but with the ability to act in real-time on this data and pivot on the fly here, it potentially could mean that you don’t need to have that second or third test session where you have that interim of trying to figure out what those scores mean.

I think that ability to pivot quickly, that time efficiency is a key piece to using it effective. A lot of our “power” users tout that flexibility piece repeatedly, which is great.

Dr. Sharp: An example of that just for us here very simply is that, say we have two subtests on one of the [00:38:00] WISC indexes that are vastly different and we want to give one of those supplementals to try to clarify things a little bit, the ability to circle back around and add that additional subtest helps a lot, whereas before it’d be the end of the day, we’d be scoring and then it’d be like, oh, wait a second, we need to do another test and have to bring them back. So just a very specific example. It’s been helpful.

One thing that y’all haven’t mentioned is cost. I would imagine that it is potentially beneficial for folks who maybe don’t do a ton of testing to jump in with Q-interactive so that they don’t have to buy a full test kit and commit to that. Is that right or do you have different?

James: Absolutely. If you were looking to hang your own shingle, open up your own practice, for example, money’s tight. If [00:39:00] you want to open up your doors and offer some testing services, the out-the-door costs of Q-interactive can be as low as basically $200 and that can give you access to a multitude of tests. On day 1, you can offer achievement testing services or cognitive testing services or whatever arena you want to go into so it’s very low upfront cost but allows you to span a wide range of services.

If you were to try to do the same thing in a paper environment, I’d have to pull the raw numbers in terms of what each individual kit costs, but you’re talking thousands and thousands of dollars to [00:40:00] offer a similar array of tests. So that upfront costs for people in private practice being so low.

And then as business picks up, you’d be basically invoiced based upon your prior month’s usage. Once the money starts to move a little bit, that becomes a lot easier to manage. So that out-the-door cost can be really low for people in private practice, and it allows them to offer the full services that they want to, which in a previous lifetime could have taken them several months or even longer to amass the funds to purchase all of those materials. I think it allows people to get started faster, wouldn’t you say, Dustin?

Dustin: I would agree. I think one assumption people probably make is that it’s mostly young people who already have iPads and iPhones and are starting their practice who are going to be the ones most [00:41:00] excited about this sort of thing. That may be true in general but we’ve found people at the end of their careers who see a similar benefit.

For example, a new test comes out and they know they’re only going to be practicing for maybe another 3 to 4 years and they don’t want to buy that test kit and make that huge investment because they know they’re not going to be using it for that much longer. And so they see Q-interactive as a cheaper way to get access to that content and be a more cost-efficient option for them at the tail end of their career.

Dr. Sharp: Sure. Could y’all speak just briefly to how the cost actually breaks down; if someone wanted to just get started, what would they be paying upfront and then ongoing? How’s that work?

James: Sure. Yes. Basically, there’s two components to purchasing Q-interactive. There’s a license piece, if you purchase a license to Q-interactive, [00:42:00] it’s 12 months of access to the system. That can range from as low as around $150 up to maybe $250, give or take. I’m just going from memory here, based upon the tests that you want access to. So there’s a license fee.

And then there’s the usage piece. So how much are you using? When we say using, we’re talking about it on a subtest-by-subtest basis. Billing isn’t going to be based around how many WISCs you give, but more how many subtests within the WISC are you giving.

Typically, if you wanted to get an FSIQ off of a WISC, that would be the first 7 subtests of the instrument, you’d be paying for 7 subtests and then the additional subtests to gain additional [00:43:00] indices. However, any more subtests you administer, that’s what you’d be billed off of. Generally speaking, you have a license fee that goes 12 months annually, and then it’s how much you use. And it’s on a subtest-by-subtest basis.

Dr. Sharp: And how much is each subtest?

James: That can vary a little bit. Not all subtests are created the same. For our achievement tests of the KTEA and the WIAT-III, those are priced at $0.75 a subtest, for your cognitive instruments, we have a lot of subtests within them like a WISC or a CELF, those are $1.50 per subtest.

And then we have some instruments that are large, single beasts like a PPVT, for example, or a Goldman-Fristoe 3, and those are priced at $4.50 per administration. That’s because those aren’t made up of [00:44:00] component subtests. They’re single instruments. So $0.75, $1.50 or $4.50.

That is probably what would resonate if you were looking at this from a private practice perspective. Alternatively, if you anticipate being a heavy tester doing a really healthy volume, you can also purchase a volume of subtests upfront. The cost per subtest can get lower the more subtests that you buy. Volume purchases will have discounts applied to them, otherwise, it’s $0.75, $1.5,0 or $4.50. Does that make sense?

Dr. Sharp: It does. I’ve run into at least one clinician who said that they talked with folks at Pearson and they were willing to do a cost [00:45:00] analysis of Paper versus Q-interactive and when it would become beneficial, is that something that’s widely available or did she somehow just work the system a little bit?

James: That type of cost analysis varies from person to person because some people may already have their WISC kits but not their KTEA or not their WIAT and vice versa. We will definitely work with frankly, anybody to help break down what the true costs are.

Q-interactive doesn’t have any hidden costs. It’s how much you use and your license fee. That’s pretty much it. Periodically, you might need to buy a handful of more response booklets if you’re given a lot of WIATs or KTEAs but by and large, that’s it.

[00:46:00] Some people come to the table with different tests already in their bags. So that cost analysis varies on a customer or person-by-person basis but if you just reach out to our sales staff, any one of us would be more than willing to help break that down a little bit and give a better sense of what those true costs would be annually.

Dr. Sharp: Oh, that’s fantastic. That sounds great. I’m curious, it sounds like there are a ton of benefits. We’ve certainly seen benefits here in the practice. I am curious, from y’all’s perspective, is there anyone who might not benefit from Q-interactive or that you would encourage to stay away from it? I guess that could include disadvantages.

James: I don’t know how to put it; iPads aren’t necessarily for everyone. My brother, for example, does [00:47:00] some training in a different company. Sometimes his training sessions boil down to here’s how you right-click, and here’s what that allows. iPads are really easy to use once you embrace it a little bit but for people of a certain generation, sometimes they get so frustrated with typing in usernames, passwords, and this technological revolution.

Generally speaking, some people can be tech-averse regardless of what the tech is. When it comes to all the different tablets that are out there, in my opinion, iPads are probably the easiest to use. Apple does a really good job of making things really [00:48:00] intuitive and how Q-interactive is designed is really intuitive.

So if you just give it a shot, the system is fundamentally sound. And so being able to overcome that anxiety pieces is usually not as big of a hill when you look at it in hindsight, as opposed to thinking about how scary it may be.

We did an interview earlier this year with a customer who was talking about anxiety. She was a little bit on the older side of things. The biggest thing that she was really nervous about when it comes to using digital tools like this centered around passwords.

And so this is not just a Q-interactive concern that she has, but just across her entire life is everything that she does; her bank, [00:49:00] her credit card, her testing purposes with Q-interactive and other things, everything requires a different password, and just keeping all of that stuff organized and situated is her biggest concern, and so it’s not a Q-interactive issue, it’s how do I use all these tools and make sure I have continued access to them through login credentials and that kind of thing.

Sometimes it’s not the tech itself, but it’s just the ways in which you access it, usernames and passwords. No one ever said that the tech revolution would require 50 different passwords.

Dustin: That does highlight some of the other things that you do need to be cognizant of if you’re going to use something like Q-interactive instead of paper. I think in the paper world, if at the last minute, all of a sudden where someone were to walk in, you could grab a record form and just [00:50:00] write the name down and go.

And so there are some other things they have to do to stay on top of things in Q-interactive. You have to set the person up ahead of time, which doesn’t take a long time, but it is a step. We make updates to the platform relatively frequently. And so you need to make sure you know how to do those updates and you stay on top of those updates and take the time to do it.

I think that’s a strength to flip it around. I think our ability to make continuous updates as we go allows us to or at least we try to stay very close to our customers, get feedback from them, incorporate suggestions and improvements whenever we can, and then push those out immediately rather than having to wait for the next revision of a test but that does mean [00:51:00] that people have to do updates and stay on top of that maintenance.

We talked about engagement earlier, I think engagement can cut both ways. For every kid who’s super engaged and loves the iPad, there might be one who’s touching the buttons too frequently because they’re too excited. Overall, the engagement story is one of being a big benefit, but I do know that there are people who have reported back to us that in younger kids, especially, for example, they’re so excited by the technology that you have to reign in that excitement.

As psychologists, we know how to do that. You are, as I talked about earlier, always managing the assessment session, and so this just becomes a piece of that but it is something that’s probably different than in the paper world.

Dr. Sharp: Got you. That makes sense. That sounds good. Gosh, [00:52:00] we all shared a lot of information with this. I know we’re getting a little bit close time-wise. I wanted to transition a little bit and maybe do something that’s different here for the podcast, but maybe do a little bit of a rapid-fire question-answer thing.

I crowdsource some questions from The Testing Psychologist Facebook group and those folks have some very specific, but I think important questions. So I’m wondering if I could just throw those at you before we wrap up?

Dustin: Yeah, let’s do it.

Dr. Sharp: Okay. So the first one, one person asked, will the WISC-V be available in Spanish? And you addressed that earlier. So that sounds great.

James: Yes. That release is imminent coming this summer.

Dr. Sharp: This summer. Okay. That is imminent. Fantastic. Another question, will the personality measures ever be available in Q-interactive to do the input right on the iPad?

Dustin: Never say [00:53:00] never, but right now there’s no concrete plans to do that. So those are available on-screen within our Q-global platform so that you can email links to people and have them do it that way using a standard laptop or a tablet device. So because they’re available over there right now, there’s no plan to have them on Q-interactive but that could always change.

Dr. Sharp: Okay. Sounds good. Do you have any plans for the entire NEPSY to be added to Q-interactive?

Dustin: I think we’d like to have the rest of the NEPSY complete but there’s no concrete date that I could give you about when those subtests would be complete and they’re not in active development right now.

Dr. Sharp: Okay.

James: In general, there’s a broader question embedded in there. So I want to say this briefly here, which is, tests, they have a life cycle, maybe it’s every 10 years, give or take. When it comes to [00:54:00] making decisions about what’s going to come and when it’s going to come, we have to look at a lot of different factors; where is the current instrument and when is it up for revision?

For example, we wouldn’t be bringing a test that’s on its last legs to Q-interactive, we would look at the revision or the next generation of that test. Does that make sense? We have to look at where things are in the life cycle how popular of a test is it and so on. In general, we want to bring as much content to the platform as we can. We have to juggle the resources appropriately.

Dr. Sharp: That makes sense. Another question, will the full interpretive reports that are available on Q-global ever be available on Q-interactive?

[00:55:00] Dustin: Yes. The WISC-V already is. So yes, if you’re a WISC-V user right now, you can get the interpretive report. What you need to do is, for any WISC-V administration that you give, when you go and generate your report, there will be an option there to click on a box and get the interpretive report.

It’s an extra usage or $1.50 depending on whether you’re paying upfront or getting billed as you go but that is available right now. And then based on how much interest there is in that and how many people are using it, we’ll look at putting some of the other stuff like the WAIS or the WPPSI on in the future.

Dr. Sharp: Okay. That sounds great. And then last question, will it ever be possible to only run Q-interactive from the iPad without having to log on to the website as well on a computer?

Dustin: Ah, good question. I can’t tell you when, but I think what people are essentially asking is can I create like a client, for example, from the actual app [00:56:00] itself? I would say, yes, that’ll be in our plans somewhere. I couldn’t give you an exact day of when that’s going to release, but we do hear that request a lot from people and it’ll be something that we work on.

Dr. Sharp: Cool. Okay. Just before we close, I know that y’all spoke about the long-term vision for Q- interactive and where you see it going. I would love to hear your thoughts on that as we wrap up. I think that’s a nice note to close on.

Dustin: I think you mentioned early on, we were talking about how long James and I have been on the project that maybe that’s rare that you had the same point of contact for so long, and not speaking for James, but I can say me personally, I’m invested in the platform in large part because of all the things that we’re going to be able to do with it in the future.

We talked a lot about equivalence and how right now we’re making purposeful design [00:57:00] decisions that allow us to test the same constructs that we’re testing in paper. I think as you look forward into a future where we’re developing tests specifically for the iPad, you can imagine all sorts of real excitement in terms of us being able to create brand new tests that are possible in paper, measure scores that aren’t really possible in paper.

A basic example of that might be really fine-grained reaction time data. There’s information out there in the basic science literature around reaction time variability being related to frontal striatal circuits and the importance of that and things like ADHD. You can’t measure fine-grained reaction time or reaction time variability at all or at least not very easily in paper.

Those sorts of things start to open us up to all [00:58:00] sorts of things that change the field of psychology. If you think to the extent that in psychology if you think of Paul Meehl and all the construct validity work, our understanding of a lot of these constructs is based in large part on the measures that we’re using to tap into them. And so for us being able to use check technology to make those measures bigger can have a huge impact on the field.

And that’s what’s really exciting to all of us. We have the ability to change those tests, to get new types of data, to partner with customers such as yourself, to share that data, and to provide that data back in order for you to make better judgments and have better insights into people. I think there’s all sorts of really good opportunities that technology is going to afford us in the future. It’s really exciting.

James: It’s also exciting to [00:59:00] see, just earlier this week, Apple had this big keynote address talking about some of the new technologies that are coming on the Apple products, with iPads, for example. You look at an iPad from 7 years ago and an iPad now, and they’re getting more and more sophisticated. Their computing power is getting broader and grander, touch sensitivity and touch ID, and all these different things.

There’s a certain unknown here too, where is the hardware going to take us as well? So we definitely want to think about how can we best take advantage of the tool that we’re using to capture this data as well. There’s things that we can do on the content side, but also maybe there’s ways that we can take advantage of the hardware, the iPad itself in ways that try to make you a more efficient or effective practitioner.

[01:00:00] Dr. Sharp: These are great points. That is exciting. Like I said in the beginning, as someone who loves technology, it’s really cool to hear y’all talk about that. I’m looking forward to see what happens next there at Q-interactive.

Dustin: The tech changes so fast too. Apple comes out with new ideas every single year. And so yeah, it’s a lot of fun to be in such an active environment like that working with tech like this.

Dr. Sharp: Yeah. Thank you guys so much. I feel like we packed a lot of really good information into this hour and hopefully, folks will take away a pretty good idea of what Q-interactive is, what it looks like, how much it costs, how to get started, and all those different pieces.

If anyone is interested in learning more, what’s the best way to learn more about Q-interactive or get in touch with someone there, how should they do that?

Dustin: I’d [01:01:00] say the best way is through our website which is helloq.com. When you come there, you can come to the Q-interactive page where we have free trials, we have a constant stream of webinars giving people a good idea of how the system works and contact our sales staff or anyone within the Q-interactive team, and we’d be more than happy to talk further about how the system works.

Dr. Sharp: Okay. That sounds great. We’ll have links to that in the show notes, of course.

Dustin: Great.

Dr. Sharp: Dustin, James, thank you guys so much for spending the time with me and being willing to talk through all these different pieces of Q-interactive. I appreciate it.

James: My pleasure.

Dustin: Thank you. We appreciate it.

Dr. Sharp: Of course. All right, guys. Take care. Hey, y’all. I hope you enjoyed that interview with the guys from Q-interactive. I was so appreciative that they were willing to take so much time to talk [01:02:00] with me about the ins and outs of that platform.

Like I said, I’ve been working with Q-interactive for many years at this point and have really seen it grow from where they started there in the beginning and have just seen them add so many measures and tweak things interface-wise and software-wise. I think it’s super helpful and we use it a lot here in the practice these days.

So if you have any questions, like they said, you can go to the website, which is helloq.com. We have that in the show notes as well. You can learn more about pricing, availability, and things like that to see if it is appropriate for your practice.

As I mentioned the last two weeks here, I have another cool interview coming up with the author of Feedback That Sticks, which is a wonderful book about doing hard feedback sessions. I’ll be talking with Dr. Karen Postal here this coming week [01:03:00] and I should have the podcast released next Monday with her. So that’ll be fantastic.

In the meantime, if you’d like more information or to check out past episodes, you can go to the website, which is thetestingpsychologist.com. You can also join us in some good discussions about testing-related topics in our Facebook community, which is The Testing Psychologist Facebook community. Pretty obvious there. You can search for that in the bar at the top of Facebook and it should pop up.

Note that there is a page for The Testing Psychologist, which is the business page, but then there’s also The Testing Psychologist community, which is the group where we talk about all things testing.

Hope you’re having a great summer. Summer is fantastic. Goodness. Like I said, I’m definitely a summer person and just got back from a great week-long trip to the beach on the East Coast in South Carolina. I’m ready to [01:04:00] hit the ground running and continue on with building things here in the business and the podcast and keep bringing some excellent testing information to you. I hope you’re enjoying your summer and take care until next week. Thanks. Bye bye.

Click here to listen instead!

Hey everybody. I’m Dr. Jeremy Sharp. This is The Testing Psychologist podcast episode 23.

Hey, everybody. Welcome to another episode of The Testing Psychologist podcast. I’m Dr. Jeremy Sharp, and I am here today with someone I am really excited to talk with. Dr. Bryn Harris is a professor at the University of Colorado Denver. We’re going to be talking all about culturally and linguistically responsive assessment. This is a huge, super important topic, and Bryn has a lot to say on this. This is where her research is and she’s focused on this for a long time. So, we’re going to have a great conversation.

Bryn, let me just say, welcome to the podcast, and then I’ll do a formal introduction for you, okay?

Dr. Harris: Thank you. Glad to be here.

Dr. Sharp: Glad to have you.

Bryn Harris, Ph.D. is an Associate Professor in the School Psychology doctoral program in the School of Education and Human Development at the University of Colorado Denver. Her primary research interests include the psychological assessment of bilingual learners, health disparities among bilingual children particularly those with autism spectrum disorder, culturally and linguistically diverse gifted populations, and improving mental health access and opportunity within traditionally underserved school populations. She regularly conducts international research, primarily in Mexico.

Dr. Harris is the director and founder of the bilingual school psychology program at the University of Colorado Denver. She is also a bilingual (Spanish) licensed psychologist and nationally certified school psychologist.

So again, welcome.

Dr. Harris: Thank you.

Dr. Sharp: Absolutely. Like I said, I’m really excited to have our conversation today. I have to comment. It’s just such a small world here in the psychology world. We initially connected because one of my graduate student interns had you as a professor and she spoke so highly of you and the course, and then we got to talking and emailing about maybe doing a podcast and it turns out that you did an internship with someone who was in my graduate school cohort and you know another woman who was in my cohort. I’m just always struck by how small this world is.

Dr. Harris: It really is.

Dr. Sharp: Yeah. So there’s some familiarity there already, even though we haven’t actually spoken before, which is always nice.

Dr. Harris: Yes, definitely. Well hopefully, we can meet in person at some point.

Dr. Sharp: Oh, I would love that. Yeah, absolutely.

Well, so for today, I think we have a lot to get to. You obviously have a wealth of experience with what you call culturally and linguistically responsive assessment. I would love to just jump in and start chatting with you about that.

Dr. Harris: Sure.

Dr. Sharp: Generally speaking, I’m really curious how you got into this particular area. Can you speak to that?

Dr. Harris: Sure. I think a lot of it started because I have lived abroad. I’ve lived in different countries growing up: Argentina, Costa Rica, Mexico, a little bit of time in Guatemala. And those experiences really, first of all, I was able to achieve competency in Spanish from those experiences. But secondly, it just gave me a lot of different perspectives in terms of different ways that educational and mental health contexts operate in different countries. And so it’s always been fascinating for me to learn about that.

In college, I double-majored in psychology and Spanish. And I was really struck. In college, I was doing some research around eating disorders, and we started interviewing teachers around some of the issues that they were seeing in the classroom, even at the kindergarten level around some of the red flags around body image that started so early on. At that point, it really struck me that I wanted to take a preventative look at how we can implement intervention and best practice assessment, et cetera if we can at first at the school level. And so that’s been my entree into psychology.

And then I did my master’s degree and Ph.D. in school psychology at Indiana University. I did a lot of clinical-type work as well. And then I did a clinical internship at the health science center in Memphis where I did a lot of autism assessment and intellectual disability assessment, as well as some other rotations. So I’ve really focused a lot on children, mostly around underserved populations, and how we can really improve their access and care. 

Dr. Sharp: It sounds like you had a nice mix of research and clinical work going through grad school and internship. Where are you at these days in terms of the clinical versus research balance?

Dr. Harris: That’s a great question. I am on a tenure-track professor position. I’m an associate professor. My job is, technically, it’s supposed to be 40% research, 40% teaching, and 20% service. I do a day a week of clinical work. I have a grant right now so I’m not teaching quite as much. I’m doing a little bit more research than usual. So I would say right now I’m at about 60% research.

Dr. Sharp: Okay, that’s heavy, right? I’m just thinking about all those deadlines and all that writing is challenging for me. So, what does your one day a week of clinical work look like?

Dr. Harris: I am doing one day a week at the Denver Language School, which is part of the Denver Public Schools. It’s an immersion school that’s a complete immersion in Spanish or Mandarin. And so, I’m doing an assessment for special education placement or not, of course. So a lot of the kids that are coming my way are possible rule-out autism. And since the curriculum is done entirely in Spanish or Mandarin, I have been able to utilize my expertise there in providing culturally responsive assessment. I really enjoy that work. That absolutely guides my research and my teaching. So I can’t imagine not doing the clinical work as well. I think it all goes together quite nicely.

Dr. Sharp: Absolutely. I think it’s nice to have both sides, certainly. I know that in our program and I’m not sure if this is just a national push or what, but with a lot of the Ph.D. programs, it seems like many of our professors are not licensed as psychologists, and that maybe leaves something to be desired when it comes to supervision and the actual clinical training. So, that’s really valuable to have both of those sides as you continue to develop the professorship, of course.

Dr. Harris: I completely agree. I know that APA really wants faculty members to be licensed as a psychologist. And I think we need to create a better way to incentivize licensure in academia because right now it’s not considered part of teaching, research or service in most settings. So we need to figure out how we can integrate that and basically prove to our leadership why it’s so important.

Dr. Sharp: Well, that could be a whole other conversation, I think. We’ll shut the lid on that can of worms for now. It sounds like you’re doing a lot of good work. I wanted to check in just as we’re getting going, you politely corrected me as we were emailing back and forth. I was using the term culturally competent assessment, and you said, no, I like to say culturally and linguistically responsive. So, I’m just curious. What does that mean to you, and is there a difference between those terms that is semantically important?

Dr. Harris: Yes, that’s a great question. And it’s a hot topic in our field because cultural competence is still written in the literature and it’s not considered incorrect. It’s more of where you are just personally in terms of what appeals to you. So, the reason that I don’t align with cultural competence is because the definition of competent entails that somebody would achieve a particular level of competence and then they would be competent to do that practice forever. So you obtain that information and you’re done basically.

So the people that are trying to use the word responsiveness, it’s really because this is an ever-evolving professional development endeavor and just like any area of psychology, you’re always going to be learning. So, you’re never going to be fully competent. And so being responsive is being individualized, personable to that particular family, child, whoever it is that you’re working with, and what their needs are.

Dr. Sharp: Yeah, that totally makes sense. I haven’t thought of it like that, but the way you frame it, of course, you’re never going to be 100% there. Things are always changing and you have to adapt. Well, I appreciate that.

So, very basic question, it may be a dumb question, but I’m just going to ask it because that’s what I do sometimes. Why would you say culturally responsive assessment is important?

Dr. Harris: I think there are some legal and ethical issues around it. Of course, first, we have, depending on your area, but we have APA ethics or different professional association ethical obligations that we need to provide culturally responsive assessment, and also the type of assessment that we’re doing needs to provide accurate and valid results. So we need to make sure that we’re providing that for every type of person that we’re working with.

And then, there are also legal issues. We’ve had situations where, for example, children that are English language learners were given cognitive assessments in English, and English was not their native language. They weren’t fluent in English. And these assessments, of course, you and I know, if you’re going to give a child an assessment, they don’t understand, they’re probably going to score low. So, they qualified for ID and that was inaccurate. And so there have been multiple situations like that from a legal standpoint that have shown us that it is not ethical and we can also lose our license if we don’t comply with some of those ethical recommendations. So that’s really important.

And then the other reason is that we want to make sure that we’re accurately assessing every person that we work with. If we aren’t providing culturally and linguistically responsive assessments, we can be misidentifying people, we could be missing identification in general. We could be missing out on early intervention services if we do that and really change the trajectory for this child or this person. So, I think those are the main reasons in my mind why we need to make sure we’re doing this.

Dr. Sharp: Sure. It sounds like you’ve actually been involved or had contact with cases where someone administered assessment in the wrong language and that turned out poorly. Is that right?

Dr. Harris: Yes, absolutely. There’ve been multiple cases like this, and it’s absolutely unfortunate because a lot of the times, especially when you’re thinking about, for example, immigrant populations or really underserved populations, they’re not as likely to know the ethical legal obligations and they’re not as likely to advocate for themselves. And so, that just puts another layer on this that we need to be filling that role as well, and be their advocates to make sure they’re getting the right assessment services. 

Dr. Sharp: I wonder if that flows into, I’m really curious about ways that clinicians might stumble into these mistakes. I would imagine none of these clinicians set out with the intent to get involved in a lawsuit and do the wrong thing. So, I’m curious, do you have ideas on blind spots or ways that we might make these mistakes unintentionally and not be providing appropriate assessment?

Dr. Harris: Absolutely. First, I wanted to mention that I teach an entire class on this. So it’s hard to whittle it down to a few minutes, but I will definitely try to give an overview of the main areas that I think are problematic.

So first of all, there are two main areas in an assessment. I’m going to really focus on children but I think that this is also absolutely applicable to adult populations. The two main areas are acculturation and language proficiency. So, if we’re looking at culture and language and their impact, we need to make sure that we are putting that into our body of evidence, into our assessment practice when we are evaluating these children.

Regarding acculturation, there are standardized measures of acculturation, but generally, I think that is really a hard thing to measure. The research behind it doesn’t show that there’s a lot of validity or reliability with these acculturation measures in general. And so I think the important thing is to evaluate acculturation in some way.

I like to do that through interviews. So interviewing the child, interviewing the parent, really finding out about what their day looks like. For example, what kind of music do they like to listen to? What kind of TV shows are they listening to? Who are they hanging out with outside of school for example? What level of engagement does this family have with certain community groups?

I think that’s really important because we need to understand the cultural influences that these families have, and also the cultural expectations that these families may have too. Just to give you an example, when we’re measuring adaptive behavior, we need to make sure that when we’re asking if a family has given the child the opportunity to do something or the expectation to do something on adaptive behavior, whether that has some cultural relevance as well.

We have very little research on this in terms of how particular cultural groups might fair differently than others on measures of adaptive behavior. But we have lots of research saying that there should be differences. And so, we need to make sure that we’re really looking into whether that score could be a factor of cultural beliefs around some behavioral expectations, for example. So I think that a thorough interview with a family, a lot of background information about that child is going to be your most important factor in that interview.

Another thing that happens in the acculturation process when a child or an adult moves to the US or moves from one area of the country to another area, or even just another community within the same city, there’s an acculturation process that occurs. And for some people, it’s much harder to acculturate than others. And those symptoms can look a lot like mental health distress when it’s in fact part of a typical acculturation process.

So you need to make sure you need to be asking questions about that child or that family or whoever it is, what their perspective was around moving to another location or learning English for the first time. Those are really big changes for people. So you want to make sure that you’re evaluating the impact of those.

And then, of course, the language proficiency piece. So we need to make sure that we’re understanding what level of English language proficiency as well as native language proficiency that person has before determining what assessment measures we’re going to give.

The most common example given, and the one where there’s been the most legal impact has been around cognitive assessment. A lot of people will tell me, “Well, I can just give them a non-verbal assessment and then it won’t be an issue. And I do advocate for non-verbal and in some ways, but I want to make sure people know that all assessments including non-verbal assessments are not void of culture. We’re still creating the non-verbal assessment within our US mainstream culture if you will. And so we still have a lot of cultural components, not to mention the way in which we use nonverbal assessment.

We give pantomime instructions in a non-verbal fashion. And some of those pantomime instructions are problematic for certain cultural groups. Thumbs up, for example, is different. And in some cultures, it’s rude to give a thumbs up. So we need to also be careful of knowing certain nonverbal gestures and whether those are culturally appropriate. But language proficiency will give you really good information about what type of cognitive assessment to give. So, if you’re trying to figure out whether a child can get a very language-loaded assessment or more of a nonverbal if you’re looking at it as a continuum, you need to know the language proficiency of that child.

You also have an obligation to know what the level of linguistic demand is of the assessments that you’re giving. So for example, a WISC or a WAIS, those are going to be some of the most heavily language-loaded assessments. They require more language demands, so probably not the right choice to give to somebody that’s learning English. But we have other options that have less language and cultural loading. For kids, we often talk about the DAS and the KABC as being some of those choices.

I think it’s important to look at the manuals of these assessments and understand the theoretical underpinning. The people that created the DAS and the KABC created it in a way to try to minimize language and culture and the impact of prior schooling on the effects of cognitive assessment. So I think those are the big things.

And then the last thing I wanted to mention is that a disability if a child is an English language learner, will only occur in both languages. You can’t have a disability in English but not have it in a native language. So, that’s why it’s really important to get information about native language development.

I do a lot of work with autism. So for example, a child not speaking until the age of 3 is definitely a red flag, but I’ve had situations, I’ve looked into prior records, and so the child hasn’t spoken English by age 3 but they were only exposed to English for the first time at age 2, they were speaking a native language before that. So we should really be asking about native language as well in that regard because that really changes how that parent might respond to that.

Dr. Sharp: Of course. These are great points. So I have two maybe dumb questions, but I just resigned myself to asking dumb questions during the podcast. One thing you talked about, you have to have some sense of language proficiency. Is that something that you would formally evaluate somehow before you decide how to measure cognitive, before going forward with the full assessment or is that just through an interview? How would you…

Dr. Harris: That’s a really great question. So if the child is younger, it’s pretty hard to evaluate any kind of language proficiency except for what the parent is telling you. So, I’d be asking questions about what percentage of the time is English spoken at home and the native language. Who’s the person or who are the people speaking that native language?

I’d really be trying to get a context for how much language input that child is getting in their native language. And if it’s more than 50% of the time, then that’s when I would start to think, I need to either bring in a bilingual psychologist or an interpreter depending on what you’re trying to do. And so, that’s where I would start at an early age.

Once the child is 5 years of age, in Colorado and nationwide, we have a federal law that every year if the child reports that another language besides English is spoken at home, then the school is required to give them a language proficiency assessment in English.

In Colorado, we use the WIDA ACCESS, and it’s used in over 30 states. And so you can always request the results from that assessment if you want to learn more about the child’s English language proficiency. I think in general, it’s a hard thing to research in terms of how quickly someone acquires English, but the research shows us that usually, it’s about 5 to 7 years, but that really is if they’re in an English immersion environment. So when we think about kids that are in school and they’re learning English but then they’re coming home and the input is the native language, it might take longer for them to learn English.

And there’s a big myth out there that learning two languages is confusing or might stunt language development, and that’s a huge myth. So we really want to encourage families to keep speaking their native language. It’s such an incredible asset for children.

Dr. Sharp: Oh, that’s good to hear. I’ve heard anecdotally from families whose native language was not English that there was some concern about that. So, that’s nice to pass that along. I did want to check in. You mentioned the options of getting a bilingual psychologist or an interpreter. What situations would each of those be appropriate?

Dr. Harris: There’s another legal situation that you want to be careful with an interpreter, and that is that an interpreter cannot interpret assessments. I’m sorry if I’m preaching to the choir here, but we have had situations where, for example, the family speaks Russian and there’s no WISC that’s been standardized in Russian. So the interpreter interprets every single question while a psychologist is administering it as well but of the English WISC into Russian.

That’s problematic for a lot of reasons. It voids standardization. It also changes the level of complexity of the question when you translate a question into another language. Let me give you a really easy example from an academic assessment perspective. If you’re asking the child the Spanish word, I’m going to put you on the spot here, do you know the Spanish word for dog?

Dr. Sharp: Perro.

Dr. Harris: Yes. Okay. So the word dog in English is one of the first words that a child learns. It’s pretty easy to learn, and usually, by 18 months, most children are saying something around, dog, but Perro in Spanish is much harder to say. It has a rolling Rs. It’s a word that children don’t usually learn very early on at all. And so if you’re trying to translate that word into Spanish and measure whether that child is able to say that word or know that word, it’s a completely different question, right? So we don’t recommend interpreting assessments. We need to use assessments that have been standardized.

That being said, we have a long way to go in terms of test Publishers really need to be more inclusive in the standardization practices, even if they want to standardize with subgroups of populations, that would be helpful. We don’t have very much information about how many groups do fair on certain assessments. So that’s something that we need to advocate for the test publishers. But that’s a big area.

When you’re using an interpreter, an interpreter really should be used for interviews with the family interviews, interviews with the children, or whoever it is. More of the informal measures an interpreter is are really great for that. And then, a bilingual psychologist would be brought in when you believe based on the history that you’ve obtained that the person is more dominant in their native language and that you would be getting more information from them through a native language assessment, that’s when a bilingual psychologist would be best.

I would definitely recommend having a resource bank of some of the people in your area that are bilingual psychologists and using them also as consultants at times. When you’re not sure whether a bilingual assessment is warranted, hopefully, you can reach out to one of them and get some more information.

Dr. Sharp: Yeah. Well, I know at least in our area here that bilingual Spanish psychologists are in high demand for doing testing. I get a lot of those requests and really don’t have anyone to send them to at least in Fort Collins. Denver is relatively close.

Dr. Harris: We have a very similar situation. I can’t even believe how few bilingual psychologists we have. I think, as we train future psychologists, we really need to tell them about this area and their need, but we also need to, as psychologists that are monolingual, we can’t just say, oh, well, this person should just go to a bilingual psychologist because I don’t speak Spanish or whatever it is.

We have an obligation to those children or families to really figure out whether they do need a bilingual assessment or not, and whether you could work in collaboration with a bilingual psychologist, maybe the bilingual psychologist just needs to do the cognitive testing, but you could do everything else, but we really need to make sure that the onus is on my monolingual psychologist to be culturally and linguistically responsive as well.

Dr. Sharp: Sure. So I know we’ve talked a lot about language, which is super important obviously, but I think a lot of us maybe get stuck in more gray areas where the language piece seems intact, maybe as best we can tell, English, they’re very proficient and that’s okay. What are some other culturally responsive ways to do an assessment or maybe things to be aware of that fall outside the language realm that are maybe less obvious? Does that make sense?

Dr. Harris: Sure. I think that it’s important to learn about the cultural experiences of the groups that you’re working with. It’s hard to generalize any kind of tips because, for example, there are some textbooks, you’ve probably read many of them, that we’ll spend a chapter on African-American populations, a chapter on Latino populations and that’s always been a big issue for me because there’s just like, for you and I, we might be very similarly… Our background might be very similar, but we might have very different cultural expectations. And so, we need to make sure that we’re not generalizing any of the families we work with.

I think if you’re specializing in an area, for example, since I specialize in ASD, I think you need to really understand how different countries and different cultures have beliefs around social reciprocity, for example.

So for example, in the research, when would a parent first come to you with initial concerns? Well, in the US, initial concerns around ASD are almost always language-based. So the child hasn’t spoken by 2 years and parents are concerned. Well in other countries that actually is very different. In India, parents are often reporting first initial concern around social reciprocity. So, I think it’s important to know what the values and expectations are of that family before moving into your assessment and your intervention recommendations, all of that.

Dr. Sharp: Okay. I know you do a lot of work with ASD. I was doing an ADOS the other day, and this has happened before, but there’s the birthday party activity. This particular assessment was with an Arabic family, and it just happened, as I was setting up the birthday party, I just thought and I turned to the mom and I was like, “Do you celebrate birthdays? Has this kid ever seen a birthday party?” And she was like, “No.” All of a sudden it’s like, well, we need to consider that then. And that’s happened in different scenarios with different activities there in the ADOS, particularly.

Dr. Harris: Yes, absolutely. I’ve had lots of conversations with people about the ADOS in that very same way. And I think that the fact that you’re thinking about it and even asking parents about this, puts you miles beyond a lot of people because the way, and I’m not trying to make other people feel bad by any stretch of the imagination, but the way that the ADOS is portrayed in the literature as being the gold standard, really, I think makes people question it less. And so I think the fact that you have that awareness and are asking those questions is awesome. So keep that up.

Dr. Sharp: Very well, thank you. I’ll take that. Sometimes, I have my moments.

Dr. Harris: Sure.

Dr. Sharp: I know that, gosh, our time has gone by really fast, which just means we’re talking about some important, pretty good stuff. I wanted to just check-in. Do you have any thoughts on writing culturally responsive reports? Is there anything to consider there? And then, we can maybe move to just ideas for training or resources and that kind of thing.

Dr. Harris: That’s a really good question. So, the culturally responsive reports, I think the most important thing is, who is your audience, and for most of us practicing, that would be the client or the parent. So making sure that your reports include really parent-friendly language. I’ve had lots of families come to me with reports that they’re like, can you please let me know what is in this report? I don’t understand it. And that usually there’s a lot of acronyms included, a lot of high-level professional language. We want to make sure that we’re writing the reports for future intervention, so it needs to be understandable.

The other thing that I would really want you to do in being culturally responsive is to understand your own biases and stereotypes that might impact you in that report writing. Unfortunately, there’s lots of research showing that people have lower expectations for certain cultural groups to be able to perform certain tasks. We have research showing that, for example, the same child, a white child, and an African-American child, the same vignette, the African-American child is more likely to be seen as having ADHD versus the white child that seemed to have behavioral issues, but not to the extent of ADHD.

And so, why are we thinking this way? We’re thinking in a deficit-based lens. And so, how can we write our reports that are really strength-based and really talk more about the symptoms or the behaviors or whatever it may be and not necessarily focus on all of the problems, right? So, I think that would be my overarching recommendation.

Dr. Sharp: Okay. So if individuals are interested in learning more about culturally responsive or linguistically responsive assessment, what would you recommend? Maybe we could take it in two parts: a beginner-level resource list and then, someone who’s been in the field and has some experience but would like to take it to the next level, so to speak.

Dr. Harris: Sure. So if you’re a beginner, then I think one of the best things you can do is look into a university in your area or a lot of people are doing online programs as [00:39:00] well, but taking a class on multicultural considerations. There are lots of different titles, but the focus of the class would really be on understanding your own experiences and how they can impact the work that you do with families because we all have biases, we all have limitations, racism, lots of different things. So I think the first thing is to make sure that you have a foundation in that area.

Definitely, the next thing really depends on your area of psychology. There are lots of different professional organizations that I would recommend. So for example, The National Latino Psychological Association is affiliated with APA. If you’re a neuropsychologist, maybe the Hispanic Neuropsychological Society. There are tons of different interest groups within your particular field in terms of your professional organization.

So, I would recommend getting involved with those, going to conferences, and then going to the special sessions that are hosted by these interest groups or Division. So Division 45 of APA, for example. Division 45 of APA also has a journal: The Cultural Diversity and Ethnic Minority Psychology journal. That would be a great place to go and see some of the recent findings related to this, and a lot of other associations also have journals. The National Latino Psychological Association does too.

And then, if you really want more advanced knowledge, I think the key is doing some peer mentoring and consultation. So if you could arrange some ways to maybe monthly have even a call with people in other areas that have similar interests that are doing similar work and really talk about cases, talk about how people have looked at culture in this regard, that kind of thing, I think that when you get to be more advanced than you have that foundational knowledge, you really need that practical application.

People at universities like myself, I’m always happy to get emails from folks. I have this case, this is what’s going on, what would you suggest I do? Please, don’t hesitate to contact people. And if you read an article and you think, oh, this is so interesting, contact the author, ask them if you could talk to them for 15 minutes. Or if you’re going to a conference and you see this person is presenting on this particular topic that fascinates you, contact them and see if they can have coffee with you for a little bit during the conference.

I think the important thing is reaching out because we don’t really have tons of research in this area yet. And so really finding ways to improve professionally within your own skillset is what’s going to be most important.

Dr. Sharp: Sure. Thanks. I feel like that’s super helpful. Those were very concrete ideas on how to pursue some more training. I think that’s a nice segue actually with reaching out. If people want to get in touch with you or learn more about what you’re doing, what’s the best way to get in touch with you?

Dr. Harris: I would love to get emails from anyone, any questions, I love hearing from people. So, the best way to reach me is my email, which is, bryn.harris@ucdenver.edu.

Dr. Sharp: Okay. Awesome. And I’ll definitely have that in the show notes, along with a lot of the other resources that you mentioned here during our talk today.

I said it before, but I’ll say it again, I feel like this time went by super fast. There’s a ton of information that could have followed up and asked more about, and I’m sure you had a similar experience in trying to convey a lot of this info.

Dr. Harris: Yes. Well, I really enjoy talking with you and I just want to commend you for broaching this topic in your podcast because I know it’s not easy. A lot of us go into psychology, we really like assessment because we know it’s black and white, they either get a two, a one or zero and we can score it. And in this regard, there’s a lot of gray area. I think some people shy away from this topic. So thank you for broaching it.

Dr. Sharp: Yeah, of course, I think it is important. And honestly, a lot of this, the desire to talk more about it comes from my own recognizing that I’m not incredibly well versed with it. And I think, if it’s happening for me, it’s probably happening for others and we got to be talking about this. So I really appreciate your time. This has been a great conversation. And I appreciate all the resources you’ve shared with us.

Dr. Harris: Any time, please email me. And thanks for continuing on your own journey as well.

Dr. Sharp: Oh yeah, of course. Well, take care of Bryn. Thanks again.

Dr: Harris: You too. Take care. Bye.

Dr. Sharp: Bye-bye.

All right, everybody, thanks again for listening to that episode with Dr. Bryn Harris. I was really impressed. Obviously, Bryn has been doing this work for a long time, and I really appreciate that she was able to share so many concrete tips, strategies, and ideas around culturally and linguistically responsive assessment.

As I said, I have two really cool interviews coming up over the next two weeks. I spoke with the guys from Q-Interactive and they shared a lot of really interesting info about that digital platform. I think that interview will be coming out next week. And I will also be speaking with Dr. Karen Postal over the next two weeks to talk about doing feedback and delivering hard feedback to families. So look for that as well.

In the meantime, thanks as always for listening, and check us out on Facebook if you want to join the testing psychologist discussion- that is The Testing Psychologist Facebook Group. You can also check at the website for past podcast episodes and a bunch of other resources.

All right, I will talk to you next time. Thanks. Bye, bye.

Click here to listen instead!

[00:00:00] Hey, everyone. Welcome to The Testing Psychologist podcast episode #22. I’m Dr. Jeremy Sharp.

Hey yáll, welcome back to another episode of The Testing Psychologist podcast. Great to be back with you this week. I took a little bit of a break last week for Memorial Day. I have to say it wasn’t exactly a planned break, but I did give myself permission at the beginning of the weekend to say, we’ll see how this goes. If I have some time to put together a podcast episode, I’ll do it, but if not, I’m going to relax a little bit and spend some time with the family. That’s what I ended up doing and it turned out nice.

We finally, I think have gotten to summertime here in Colorado. We’ve got a nice run of about [00:01:00] 80-degree weather here for the last few days and the next few days. It’s super sunny and that’s pretty awesome. Yesterday was our first pool day. So that’s cool to see the kiddos jumping around in the pool and having a good time. For me, summer elevates the mood and puts me in a nice space in general.

I hope things are going well wherever you are at and getting some summertime sunshine and maybe taking some vacations. All those things are super important, especially when you’re working pretty hard. That’s where my break came from last weekend. I think it was a good one. It gave me some time to reflect as well on the business and the podcast and come back with some great content, I think.

Today, I am doing another solo episode. I’m talking to y’all about kind of an extension of the vulnerability episode. I did an [00:02:00] episode on the vulnerability of psychological testing back in episode 13. During that episode, I mentioned at the beginning, a story about how a family had come in for an evaluation, and for various reasons, I didn’t feel like I gave them the best service that I could have and how they were pretty upset about that.

At that time, I didn’t finish that story or discuss the outcome on the podcast. So I got a lot of inquiries from folks about what the outcome was. That got me thinking about this topic of how to handle it when you mess up. That’s something that happens to us from time to time. That’s a pretty human experience, especially here in testing, going back to the vulnerability piece, this is something that folks put a lot of stock in, families come in with a lot of [00:03:00] investment of time and money and energy. It’s a big deal to get themselves evaluated or their kiddos evaluated. And so there’s a lot of risk for them there. Not to say that that does not happen with therapy by any means, but I think when there’s this concrete product, the evaluation, and the report that comes out of the testing process, it just heightens that sense of responsibility that we have to deliver something quality to the family. 

Along the way, mistakes are going to happen, certainly. I’ve had a number of mistakes over the years, but I got to thinking about some of the most typical ones that come up and how I’ve developed means of handling those. I thought I’d chat with you all about that today over the next several minutes. Let me go ahead and dive into it.

One of the things, I’ll actually [00:04:00] just to back up, I’m thinking about walking through the testing process, how mess ups can happen at different stages, and how I’ve learned to handle some of those mess ups.

The first one that I wanted to talk about is what happens during the initial scheduling process or even the initial contact and what happens if you mess up there. I use the term messing up pretty generally, but when I say messing up, I mean, anything that doesn’t go exactly as you might want it to or as the client might want it to.

For me, the biggest pitfall initially is missed emails or phone calls or not getting back to people in a timely manner. My hope and my expectation for getting back to people is that I always return phone calls or emails for new clients within 24 hours.

When it was [00:05:00] just me here in the practice, that became increasingly more difficult and it got to a point where I had on my outgoing voicemail that I may not return phone calls for 48 hours. Looking back on that now, that seems crazy because I’ve moved to a place where I’m fairly militant that we get back to our new clients within the same business day. That seems like a long time ago, but I think it speaks to how things can get busy and it is hard to set aside time to return phone calls.

There are two things proactively to do with that to try to make sure it doesn’t happen. One is that I have shifted my schedule to have a half hour or 45 minutes at the end of each day, whenever possible, to sit down and return phone calls and emails. Another thing that I’ve done is that we brought on an in-office [00:06:00] administrative assistant who answers the phones full time. That has helped a lot. I’ve had to go back and almost redistribute my contact number so that people are calling the main administrative number now instead of my personal number, because if people leave me personal voicemails, that increases the likelihood that I may not get back to them.

I don’t know about y’all, but I have a really hard time with voicemails. For some reason, I would much rather read an email or a text, but when there are voicemails, it feels hard to sit down and listen for some reason. With the voicemails though, because I do still get a few calls that go to my direct line, what I’ve done is, I have an iPhone, I can’t speak to Android and whether this works there, but on the iPhone, there is a function in the voicemail when you’re listening to it, [00:07:00] it looks like a page with an arrow icon, it’s basically the share icon. I have a practice management system or project management system that I share with my admin assistant that allows me to share the voicemail directly from my phone to his task list, and that has helped greatly.

So, in the past, what I might do is I would listen to a voicemail and then not have time to return it or say that I’ll do it the next day or something like that, and that doesn’t work very well and it’s not great for clients. So, I figured out a way to share that directly with him if it’s someone that he can call back and schedule, which is the majority of the phone calls.

Now, that still leaves the situation where you might have a missed phone call or an email that you don’t return for two days. That does happen certainly. As you’ll see as we go along [00:08:00] with a lot of these examples, my general response to that is to be direct and honest.

Usually, there is some reason that I have not been able to return messages. Maybe I had to leave to pick up my kiddos early. Maybe there was a crisis here in the practice where I ended up stuck on the phone with someone solving another problem.

Usually, there’s some legitimate reason and I will usually share that with the client when I email them back. I always try to do that in a way that does not feel like it’s making an excuse or anything like that. I’ll just say, “Hey, thanks for reaching out. I apologize for not getting in touch within a business day. That’s fairly atypical. We had an emergency yesterday and that ate up more time than I was expecting. Thanks for your patience. I’m excited to talk with you at this point.” Just being direct and [00:09:00] explaining what happened and usually people respond pretty well to that.

The next part in the evaluation process that people get frustrated is often with scheduling. Scheduling is challenging sometimes for us. I always try to… We tell people that we try to wrap up the evaluation within a day and do all the formal testing in one day. I always try to meet with the kiddo or the client that day as well to do the individual interview, but sometimes that does not happen.

So again, I do my best. If I am not available to wrap up the testing and complete everything that we need to, I’m just very direct with the parent or with the client and say, here’s where we’re at. It looks like our time is running low or it looks like I’ve had a scheduling conflict come up, and again, usually try to explain [00:10:00] some aspect of that like what led to that scheduling conflict, of course, without revealing any other client info or anything like that, but just being honest and letting them know. And then I always try to get them scheduled as soon as possible.

To be honest, that typically involves shifting my schedule a little bit and bending a little bit from some of my scheduling boundaries, but that’s just my philosophy that once people are in for the evaluation, it feels necessary to take care of them. Sometimes I do have to bend a little bit to make sure that they get in in a timely manner so that we can wrap up the evaluation. So, just being willing to flex your schedule a little bit, not to the point of being resentful or anything, but being willing to flex just a little bit so that you can get people in a little bit sooner, I think helps.

A variation of that is [00:11:00] with the report turnaround. At least in our clinic, we do a feedback session and then write the report afterward. Our expectation is that the report is always delivered to the client within 2 or 3 weeks, 3 weeks is max most of the time. But again, sometimes, that initial timeline is not acceptable for clients and they get frustrated with that, or sometimes it does take a little longer than that for one reason or another. Maybe I’m out of town on a vacation or have some other clinical issues that come up or something that eats into my report writing time, or again, a sick child. That’s a theme of time that gets taken unexpectedly.

One thing that I’ve tried to do that helps ameliorate that a little bit is, I’ve put together a template [00:12:00] for what we call an evaluation summary. It’s a one-page document. It hits all the important bullet points of test results, diagnosis, and recommendations. I can usually put that together pretty quickly within 10 or 15 minutes. We have started to let people know that they can have that evaluation summary pretty quickly. I can sometimes do it that day or the next day. That’s pretty straightforward. People seem to respond well to that. It’s like a bite-size acceptable version of the evaluation that doesn’t take much time on your end but gives them something to hold onto and latch onto. 

I also make sure as people take off from the feedback session that they have something to hold in their hands. So oftentimes, I’m giving a lot of referrals to different services out in the community. I will always give them [00:13:00] a referral sheet with some options checked off and some directions for the next steps so that even though they don’t have that full report, they can walk out and make some phone calls or take some action if they would like to.

Speaking of the feedback session, one of the things that comes up at some points over the years, it’s not super common, but it does happen sometimes. So again, we do a feedback session where I deliver the results and then write the report afterward. What happens sometimes is, that you’ll be talking with a parent and delivering some of that feedback and they may share new information or they may ask questions or ask about results or offer something that makes you think a little bit differently about the evaluation.

I typically come into the feedback session with, let’s say a 95% certainty, [00:14:00] maybe 98%certainty of the diagnostic picture and tend to deliver that fairly straightforwardly, but sometimes this new information will say things a little bit. So if we get into one of those conversations, then, one of two things happens. I might adjust on the fly in the feedback session and say, “This information sounds different than what I was working with before. I think this is important. So, I’ll tell you what, I’m going to back off from this diagnostic certainty and go back and try to integrate all of this data and see what comes out of it.”

Parents, I think, appreciate that a lot of the time because it often is a reflection of my being willing to hear and understand their concerns and integrate new information as they feel is important. So, usually, they react pretty well to [00:15:00] that.

The other situation that can come up is that, I’ll do the feedback session and be pretty convinced of the diagnostic picture, but then as I go back and write the report, pull everything together, and synthesize the information, sometimes what happens is my brain processes things differently. I see the data a little differently, or maybe I consult with a colleague and things change. In that case, I do think that we have some obligation to re-explain the results in the context of this new diagnosis and recommendations.

So, I’ll often write an email to the parents or the individual and say something like, “I was going back and putting the final report together. As I did, things shifted for me a bit. I think that the diagnostic picture and recommendations are a little different than what we talked about in the feedback session.”

Now, if it’s a major [00:16:00] shift, then I will implore the client to schedule another feedback meeting, which I typically will do complimentary since it was my priority, I suppose, or my mistake, to have to bring them back in. So I’ll do that complimentary. If it’s not such a major shift, if it’s something like maybe I’m adding a writing disorder to an existing reading disorder diagnosis or something like that, I’ll just write out an email and say, here’s the new picture. Here’s the information that I used to make that. It’s all explained well in the report. Let me know if you have any questions about that. I always offer to get together again for another meeting if we need to discuss those extra results. And that seems to work pretty well.

Now, the situation that came up back in episode 13 that I discussed was probably, I would say the worst [00:17:00] scenario that has happened here in our practice and certainly to me personally and that’s why I did want to talk about that because it was the best example of how to fix it when you mess up because I think I messed up pretty badly on this one.

I will say, not to keep you on the edge of your seats, it turned out great. So that’s all good. I worked with this family, there’s a typical evaluation with their kiddo. What happened is, they came to the feedback session. This is a family who drove from a significant distance away, and that maybe lent a little more importance to it for me, even though, it probably shouldn’t, but just knowing that they were putting so much effort into coming here made it super important for me.

So, they came for the feedback session and what ended up happening [00:18:00] is, the night before, I think our little girl was up sick and had kept us up a lot of the night. Basically, I was not on my game during this feedback session. I used some of my downtime during the day to work on some other things and to try to save some energy and rest a little bit.

 I didn’t save as much time as I typically do to prepare for the feedback session and look over the records and that kind of thing. And so when we got to the feedback session, this is hard to admit or acknowledge, but I ended up doing an okay feedback session. It did not feel super personal to this family or this particular kiddo.

I usually speak [00:19:00] pretty specifically to recommendations that are appropriate personally for each kid. This time I spoke more in generalities. I asked the parents some clarifying questions that I had already asked them in the initial interview, which was certainly embarrassing. I could just see, as the feedback session was going along, that the parents were disappointed. Their faces were falling. They eventually voiced this disappointment that they thought the feedback was going to be more specific to their kid and it felt like it was not personalized and that I’d missed some important information. Of course, all of those things were true at the moment. For me, that’s my worst nightmare. A big reason that I do this, and I think a big part of our [00:20:00] reputation here in the community is being fairly attentive to families and providing really good service. So this was pretty heartbreaking for me.

In the moment, I did apologize. I didn’t say a lot, but I apologized and acknowledged that I could understand that this was not exactly what they were hoping for, and said that what I would do is look back over everything and get back in touch with them with some recourse. I let that let that ride for two days. I think that was a Friday. Over the weekend, I thought about what would be appropriate here.

What I ended up doing is I wrote a pretty genuine, heartfelt email to the family. [00:21:00] I was honest. I did not go into extensive detail about my daughter being sick and all of that, but I did say, yes, I totally understand how you could have been disappointed. I was admittedly not as thorough as I should have been. I did say. “I was not at my best for a number of reasons, and I apologize for that. You’re completely right to feel the way you did. Here are the things that I would like to do to change things a bit.”

In the email, I offered some additional explanation in the context of this kiddo’s diagnostic picture. I won’t go into detail about that, but I offered some more information there. I did give them one of those evaluation summaries and beefed it up a little bit to make sure that they had some concrete info to take to the school for their school meeting.

[00:22:00] The main thing that I did was, I said, “Listen, I’m going to look through all the data. I’m guessing the report is going to be a lot more comprehensive and contain a lot more information than what we talked about in the feedback session. So I am happy to do another meeting with you to address any concerns. That will be complimentary. We can have as many conversations as we need to sort through all this information and make sure that we’re all on the same page.”

That went over really well. We did end up meeting again. We had a great conversation. One of the parents was extremely well-versed in the diagnostic picture and special education and that sort of thing. So we had a great in-depth conversation about little nuances of the report and what the [00:23:00] evaluation looked like. 

I also put a lot of attention into that report. I try to do a pretty personalized report no matter what, but with this particular kiddo, I would say that I maybe even went above and beyond just to make sure that I was speaking to each of the little components that were particularly important for that kiddo. So, when I delivered the report to them, it was maybe a little bit more thorough than even typical. And I think that was really appreciated.

So I think the theme with a lot of these instances of messing up is owning it. The times, maybe in the past that I honestly can’t recall, but I know that I’ve done, I think there’s that inclination to get defensive and maybe brush it off or say  [00:24:00] the client was too demanding or something along those lines, that never goes well. If you’ve found a situation where it goes well, let me know. But for me, owning it is super important.

This is nothing revolutionary necessarily, but at least I know for me, when I get into this situation, owning it can be one of the hardest things to do because it means admitting that you made a mistake and running the risk that someone is going to be disappointed or maybe leave a bad review or something like that. But, at least in my experience, it seems like owning it and offering some matter of recourse can help out greatly and has always been received pretty well.

The last piece that I wanted to talk about is billing. That wraps up the whole eval process. We do take a lot of insurance. I’ve talked about that here and on other podcasts before about taking insurance in [00:25:00] testing. With that, we do run into situations sometimes where the bill comes back differently than was expected. We do have an insurance verification team that checks benefits. We always try to give people an estimate of their out-of-pocket costs before they get started, but sometimes that is different than what we were quoted for whatever reason. In those cases, I always… when I was doing this on my own, this is how it worked, and now that we have an admin assistant, I trained that person to do the same thing, but I always give people the benefit of the doubt basically when it comes to billing.

You may have been on the other side of this maybe with one of your own medical bills, but when you’re calling and there’s money involved, things can get tense, and defensive and can go downhill pretty quickly. [00:26:00] That’s often the case. When people are calling us, they’re on the defense or even maybe think about being offensive, where they’re trying to talk us down or negotiate a balance or something almost like they expect you to be defensive in return. What we do is we always give the client the benefit of the doubt. We totally employ those active listening skills and try to understand their perspective. If it’s a really straightforward issue like deductible versus copay or something, we’ll just explain that and try to do that pretty clearly: Here’s your deductible. Here’s what that means. Here’s why the charges went to that and it was more than you were expecting.

The other piece is that, if it’s relevant, we will always do one appeal on the client’s behalf. Most insurance companies will allow you [00:27:00] to submit paperwork or give them a phone call and try to do a retroactive authorization or something like that. That’s part of our, I guess you’d say customer service is that, I will let our staff do that, have our office admin assistant do that is just do one appeal. It might take 15 to 20 minutes, but I think it goes a long way in the client’s mind.

If it doesn’t work, we have a form letter that we send that basically says, “Hi, so and so, we just wanted to get in touch about this balance. We know it’s different than you expect. We did perform an appeal on your behalf and unfortunately, that was not granted. We do not have the time or the resources to continue to appeal this, but we’re happy to provide you with any documentation you might need to appeal on your own. In the meantime, we kindly ask that you remit your payment for this balance.”

When you [00:28:00] try to go that extra mile just to assure people that yes, we’re on your side, we’re not just trying to collect this money and send you on your way, that makes a big difference. We’ve had a lot of folks who, just by virtue of that conversation and that letter, will say, “Thanks for doing that. I really appreciate it. Let me just pay the bill right now. I’ll contact you if we need any documentation to appeal it.” So. I think just being human and being kind goes a long way and acknowledging that that’s really inconvenient for the balance to be different than what they expected. That’s not a nice surprise. Of course, we’ll help them out.

Let’s see. I think that’s it. That’s all that I have in terms of ways to handle it when you mess up. There are any number of examples of how this might come up when you’re doing testing. I would love to hear if anybody else has any stories of how you [00:29:00] handle it when things do not go exactly right, or how you handle it when clients are upset because this is something that we all deal with to some degree. It’s helpful to have some tools in your tool bag for how to work through that.

As we move along, this will be, I think, my last solo episode for a while, which is pretty exciting. I have some cool interviews coming up. I think over the next 2 or 3 weeks, I will be talking with 2 guys from Q-interactive. If you don’t know Q-interactive, that is an iPad-based test administration software platform. A lot of people are jumping on board with it. It’s getting pretty popular. I’m talking with 2 guys here soon about the development of that, how to implement it, cost and pros and cons, and all those different pieces. So, that’s going to be pretty exciting. I’m [00:30:00] really looking forward to that one.

I’m also going to be talking with two practitioners that I’m really excited about. This coming week, I’m going to be talking with Dr. Bryn Harris, down in Denver, who specializes in cultural competence and assessment specifically with autism spectrum assessment. She’s going to be talking with me about cultural issues. Then two weeks down the road, I’m going to be talking with Dr. Karen Postal, who is the author of  Feedback that Sticks, which is a fantastic book about how to do a feedback session, delivering difficult feedback, and how to work through pretty tough diagnostic pieces with parents and families. I’m really excited. We’ve got some really cool conversations coming up.

In the meantime, I hope that all of you are having a great summer, maybe taking a cool vacation or two. We are headed next week to South Carolina, which is where I grew up. [00:31:00] We’re going to spend a week at the beach, let my kids play with their cousin, see some family, and get some sunshine. So that sounds incredible. 

I hope all of y’all are doing well. If you do want to connect or talk with other psychologists who are doing testing, we have The Testing Psychologist Community on Facebook, which you can search for there in that bar at the top of Facebook. The website, thetestingpsychologist.com also has resources and links to past episodes and things like that.

Of course, if you want to think about or move toward growing testing services in your practice, I am always happy to talk with you. We can do a 20 or 30-minute call just to get a sense of what’s going on and what you’re thinking about. Then we can figure out if consulting is, is the right direction for you. Or if you’d like to maybe explore some other options, which I could talk you through. So don’t hesitate to give me a call if that feels [00:32:00] relevant for you.

Take care, y’all. We’ll talk to you next time. Bye. Bye.

Click here to listen instead!

[00:00:00] Hey everybody. I’m Dr. Jeremy Sharp. This is The Testing Psychologist podcast episode 21.

Hello again, everyone. Welcome to another episode of The Testing Psychologist podcast. Hope you’re all doing well today. I am just in a great mood this morning. It’s a Monday morning. And the reason is because I think we have finally had our last snowstorm here.

Colorado is notoriously crazy for having late spring snowstorms. I thought we had escaped it this year, it’s obviously mid-May, but last week we had two straight days of this heavy, wet snow and ended up with probably 6 to 8 [00:01:00] inches of snow in the middle of May, which is not ideal as far as I’m concerned.

Growing up in the south, I’m definitely a summer person. I like the heat. I like the humidity. So, that was a bummer to have that thrown upon us all of a sudden, but as is typical with Colorado, the next day it was first about 55 degrees and then the next day it was 60 or 65 degrees and all the snow had melted and now it seems like we are actually, I think done with the snow and we’re really cruising into the summertime, which I’m really excited about.

Both my kids have learned to ride their bicycles. They love it. So we have been taking some family bike rides and it has set the stage I think for a really cool summer. So hope all is going well wherever you are and maybe you’re getting into the summertime as well.

I’m going to be doing a solo episode today talking all [00:02:00] about how to hire and train psychometricians. I get a lot of questions about how psychometricians operate in our practice and why you might want to hire a psychometrician. I’m going to talk all about that today and try to give you some info if you’re considering that or maybe even have some psychometricians in your practice, maybe some ideas to streamline that process and get everybody on the same page and make it work as effectively as possible. So I’m going to dive right into it.

One of the first things when you’re thinking about hiring a psychometrician, and just to back up a little bit, I should say the term psychometrician is a mouthful. Some people will call these individuals techs or testing assistants or technicians, but either way, the [00:03:00] psychometrician is someone who administers the tests and scores the tests for you.

A good analogy would be, for example, radiologists are not the ones who actually put the vest on you and get you into the radiography machine, whatever that might be. They are not the ones that do that and they don’t print out the x-ray or anything like that, but they are the ones who interpret the x-ray.

So for us, the psychologists are the radiologist and the psychometrician is more like the x-ray tech; the person who plays this really important role to get everything set up appropriately and make sure the x-ray is taken precisely and then relaying those results accurately and effectively back to the radiologist. That gives you some idea.

When you’re thinking about whether a psychometrician [00:04:00] may or may not be a good fit for your practice, I can say that the main reason that I ended up taking on psychometricians way back, in the beginning, is because I needed more time. 

I think I’ve told the story here and maybe on two other podcasts about how back when I launched the testing part of the practice, it was due to getting probably 5-6 times my typical amount of referrals all at once. I went from getting maybe 2 or 3 referrals a month for testing up to, I think I had 15 or 20 all at once at one point due to a change in policy here at our local university. So I had to figure out how to find the time.

Now, I had served as a psychometrician when I was in grad school. I worked under a clinical neuropsychologist for two years. So I was familiar with that model and I was pretty open to it right off the [00:05:00] bat just because of my experience there. So when I knew that I was getting extra busy and physically could not see all these testing cases, then I started to think it’s time to look for some psychometricians.

I went to our local graduate school- the department that I did my doctorate and just put out the word that I was looking for two students to come and get trained and be psychometricians in the practice. And that ended up working out really well.

A big reason that I think a lot of psychologists hire psychometricians is to have more time. If you think about doing the testing, actually administering the testing, depending on the battery you’re doing, that could be 4 hours, 6 hours, maybe 8 hours. So that’s a lot of time that the psychologist could otherwise be using for other things. And that’s why I think a lot of folks do hire a psychometrician to help with that.

So, if you’re deciding, [00:06:00] I’d really sit down and think what could you be doing with that time: Will that allow you to be free and maybe not work as much but still be able to make some amount of passive income? Would you be able to see more cases? That’s a big motivation. So you could certainly spend your time more on intakes and feedback and interpretation rather than just administering the tests. So there are two factors that you may consider.

Now, two things that are a little bit tougher about having a psychometrician though is that it does certainly take some time to invest in training depending on the level of experience of that person.

Now, I typically bring on advanced doctoral students, so these are individuals who are in the latter stages of their PhDs. They typically have at least a year’s experience administering and scoring tests already, and they all have master’s degrees already. So in my [00:07:00] training, there is quite a bit that happens and I’ll talk about that here in just a bit, but they really are ready to hit the ground running for the most part. So that’s something that you would want to take into account.

We will talk a little bit more as well about the board of certified psychometrists. This is an organization that’s moving to create standards for certifying psychometricians. I’ve met 2 folks who have had some experience with that board, and they just have an incredible experience with testing and are ready to almost just like plug and play in practice because they are so skilled. So the level of training is going to vary, but that’s going to be an investment in your time. I also do ongoing supervision with my psychometrician. We meet for an hour every week. So that’s an investment as well, just on an ongoing basis.

The other piece I think that’s really crucial if you’re thinking about hiring a psychometrician is that you [00:08:00] have to let go of a certain amount of control. Now, as a practice owner and as someone who likes to do testing in general, I think many of us really enjoy being precise, looking at data, facts, and standardization. So this can be challenging. It was certainly challenging for me as well.

Now, you can address that with some clear training processes, which I will talk about here in just a little bit as well. But I think that’s just something to think about for yourself, like, would you really be able to let go of that amount of control? And if so, how do you do that?

Let’s start to talk about how you actually do that.

For me, the interview and selection process is huge in selecting the psychometrician. Now, of course, I think this makes intuitive sense, but I start from the very beginning when I’m considering who might come on as a psychometrician [00:09:00] here in the practice. When I put the advertisement out, obviously you get applications and emails and that sort of thing. And I am looking right off the bat at the quality of those applications.

One of the skills that I think is just huge for a psychometrician is attention to detail. They have to be accurate in their scoring, in their writing, in their test administration. Attention to detail is huge. So, I am actually really strict and very attentive to those initial emails that I get. If there are any errors in those emails, I mean, capitalization, grammar, the wrong agency name on the cover letter, any errors at all in the application or on the resume, for better for [00:10:00] worse, I take that as a sign that this person may not have the best attention to detail. And so, I rule out those applications pretty quickly. 

Once you get past that, then you can really start to focus on the applicants and how they might be a good fit. So in my interviews, I am really upfront with the applicants about what the job entails: what it means to be a psychometrician and what qualities are important for that. Of course, I’m kind. I don’t drop the hammer right away and scare them off, but I’m very clear that two of the biggest things that you have to consider are:

1) Attention to detail like we’ve talked about, but

2) Efficiency is a big one as well.

The way that our practice is set up, we do typically all of our testing in one day. So that tends to be a lengthy day. It’s probably 6 [00:11:00] hours face to face with the client when all is said and done with a lunch break and everything like that. And then it takes at least 2 hours to score everything and do the writing. That’s all the psychometrician’s job.

I’m very upfront that if the individual is not good at using time efficiently and working fairly quickly and accurately, then it’s going to be really easy to get behind, the cases stack up and then that can be a bad situation. So I’m pretty clear with that. I talk about timeliness very clearly. So making sure that the individual is on time for appointments, on time for supervision, and on time with the reports.

So, again, I’m very clear about the expectations here in our clinic where my ideal expectation is that the scoring and writing is done at the end of the day after testing. There are some cases when [00:12:00] that might extend out a little bit just working primarily with graduate students- they have schedule issues that they have to work with and sometimes they have exams and that kind of thing. So I’ll give them a little bit of leeway sometimes to make sure that they’re on time, but generally, the expectation is that the writing and scoring is done on the day of testing.

Now, other things that I really talk with them about include their willingness to ask for help. My worst nightmare is a psychometrician who might think that they are doing things okay but they just cruise along doing it the wrong way rather than ask me a question about it. So I really try to emphasize that I have an open-door policy and I’m very willing to talk with them about any issues that might come up.

And then another piece that I talk with them about is their ability to take feedback. I’ll touch on this a little bit more as we go along, but a big part of [00:13:00] the report writing process is feedback for me. I use Track Changes to correct and edit their reports, especially initially. And I’m also giving ongoing feedback as we go along and as they are learning and even later stages, sometimes things will come up. And so, I always make sure that the folks are open to taking feedback and hopefully not going to be super defensive. And of course, I try to be kind as well. And that helps, of course.

So the interview process I think is really important. Generally speaking, I feel like administration skills can be taught fairly easily. Again, I’ll talk about that here and how we do that, but my philosophy is that personality and these other skills that I’ve discussed: timeliness, efficiency, attention to detail, that kind of thing. I think those are way more important than administration skills when someone comes in.

[00:14:00] Now, of course, they need to be familiar with the different measures, but I’m not drilling someone during the interview about what do you do if the person does not give an answer within so many seconds on the WISC. I’m not talking about that kind of stuff. I just feel like that stuff can be taught and can be taught well. So I’m really focused more on personality and how this person is going to connect with others in the agency, how they’re going to connect with my clients during testing, and how they’re going to get the job done.

So just to emphasize that, I think the personality is huge. And again, connection with the client. I should have maybe said more about that as I went along, but that is actually really, really important for me and I think for the client too. Testing is just such an intimidating process that you need to have someone who can connect with your client and help people feel comfortable during that process.

Now, one other aspect of the, I guess you’d say interview processes that I [00:15:00] do ask for sample reports. As I said, everybody who comes in has already done a fair amount of testing. So they always have sample reports ready. I look at those not so much for the content of the interpretation, but really more just to see what their writing styles look like, do they have good grammar and good punctuation? Again, do they have attention to detail? So I’m looking for any mistakes in those reports. That really helps to give me a decent idea of what they’re capable of and how they pay attention to those important pieces.

Once people get here, then we are talking about training. So during the interview, I always ask them, are they familiar with these tests? I show them our typical battery and try to get some sense of what they are going to be bringing in. Most people have some familiarity with the battery, but not complete familiarity.

So I always ask: Is there any way you can do some [00:16:00] research on the internet? Can you go to your local clinic? Does your grad school program have these measures? And I ask them to look over any measures that they can just to be somewhat familiar. So it’s not the very first time they’re seeing that particular measure when they come into our practice. That just happens right off the bat during the interview.

Now on my side, I think the training documentation is really important. And so for you all, if you’re thinking about hiring a psychometrician, I would start right now writing down literally everything you do during test administration. Now I’m not talking about, you don’t have to rewrite the Testing manual or anything like that, but all of those little things like what tests you give, the order that you give the tests, what you do in between, why you’re not giving the tests, [00:17:00] what you say when you greet the client, how you explain the testing day, when you take a lunch break, what order you score things in, how you write, where all the scoring software is, do you do it by hand or do you do it on the computer? You get the idea.

I would go through your typical testing day and operationalize virtually everything that you can. So just write everything down and try to start. This is going to be the basis of your training manual. Even the little things. That’s one of the big things that I noticed is that I was doing a lot of little things that were second nature to me that other people might not think to do. For example, asking for the parent’s cell phone number so that we can text them when the testing is over, something like that. And that’s just a little piece that’s important that you might skip over.

Anyway, I [00:18:00] would start and just write down everything that you do, take two testing days. It’ll take you a little extra time, but I think this is really important. 

Once you do that, you can start to build your training manual. For us, our training manual at this point is pretty lengthy. We have what I would call a macro document that has I’d say a big-picture checklist for the testing day. It’s about 1-2, it might be 2 pages now- a 2-page little document that basically has a 50-item checklist for the actual testing day.

The idea with this macro document is that someone could come in, they could open this document and as long as they know how to administer the particular tests, they could follow this checklist and get through the testing day and [00:19:00] administer a battery effectively. So it has big picture stuff like greet the client, give them the schedule for the day, ask for their phone number, tell them what the testing day is going to look like, so on and so forth.

And then it gives the order for the battery that we typically administer. It tells what time to do lunch, how to determine when to do lunch, and then it describes what order to write things in most effectively, where all the scoring software is, and how to do the note in our EHR system, which is TherapyNotes. So, big picture checklist. 50 items sounds like a lot but that includes every test that we administer. So there’s a lot that I think is pretty simple and straightforward on there and just makes up for a lot of items. So that’s our macro document.

But then we also have micro documents for each test. These are housed in a [00:20:00] binder though we also have them electronically on Google Docs as well, which I would definitely recommend because then you can edit them at will. So, micro documents for each test.

And these go into a lot more of the nuances of test administration. I don’t really duplicate anything that’s already in the manual, but we do talk about little things to keep in mind that might be buried in the manual or things that you might look over or just little eccentricities that happen when you’re administering that particular test and things to look for and just be aware of. And then, of course, we talk about how to score it and typical questions that might come up and things like that. So we have little micro documents, like I said, for each test that we administer.

That really makes up our training manual. It’s, like I said, pretty lengthy at this point. The individuals get access to this from day one. I will actually provide access to the [00:21:00] training manual before they come on board. And I really encourage them to look over it even though it’s all theoretical, but just to get some familiarity with our process and with what we do.

In terms of the training schedule, when someone first starts, we have a fairly structured schedule for how we “onboard them”. The first two days, they observe either myself or another advanced grad student who’s been here for a while. They observe one of those individuals administering all of the tests. So they just sit in the room back in the corner. We, of course, tell the client and everything like that but all they do is observe, and then afterward they stick around and we score everything together. We talk about it, answer any questions, let them look around on the computer, and familiarize themselves with the [00:22:00] scoring software and things like that. So that’s the first two days where they observe someone else administering.

The 3rd and 4th day, I observe them, or again, another advanced psychometrician would observe them doing the administration. They would do the scoring themselves with supervision, of course. So we’re sitting side by side this whole time. If they don’t need anything, then I can work on other work. If they do, then of course I’m there to help them or the other grad students there to help them. And then walk them through the writing process, and answer any questions. At the end of that fourth day, they’ve either observed or conducted their own assessment for about four days in a row. If they’re good to go, then that’s great. Then I turn them loose and if I feel confident, then they’re ready to go.

One thing that I [00:23:00] know some other practices do, and this is what I’ve done when I worked for a neuropsychologist was, they do test administrations. So the new psychometrician would have to practice administering on another employee, for example, before they get turned loose with a client. So that is an aspect that I’ve certainly considered adding. I haven’t done it yet, but I know that other practices will do that.

Now another aspect of this training is that I always have them just shadow me out to the waiting area. So they get a sense of how I interact with the client, talking with the parents if it’s a kid, also modeling interaction with the kids. I think generally speaking, here in our practice, we tend to err on the side of being informal whenever possible. So joking around with the kid, say something like, all right, let’s go back to the torture chamber, something like that, just [00:24:00] to lighten the mood and break the ice a little bit. You get that idea.

I’ll be pretty informal, of course, as long as it doesn’t interfere with actual test protocol or test administration or anything like that. So joking around, being casual and friendly with the parents and the kiddos. I like to model that really clearly for the grad students just to let them know. I think sometimes they’ll come in and, of course, it’s a new job, they want to do a good job and be pretty precise and professional, but I’ll let them know right away it’s okay to joke around, keep people in a lighter mood. So I think that’s really important.

Another aspect of training is that we are really big on behavioral observations here. I think this is one of the biggest things when hiring a psychometrician that it’s a big hurdle to get over is, people say, Oh, I’m going to lose all these observations. [00:25:00] Interacting with the client for all those hours gives you so much data. And I totally agree. So that’s a big one. That’s hard to get over. I’ll just put that out there.

The way that I’ve addressed that, there are two ways. There’s the documentation standpoint. We have actually a pretty lengthy behavioral observations document that I put together that addresses all the different aspects of test-taking behavior, all the way from gate and motor skills to eye contact, to mood and affect, to their effort, response to failure, how much caffeine they drank, were they excited, happy, sad, nervous, all that kind of stuff.

We have that larger scale document that has a bunch of check [00:26:00] boxes. I try to make it easy where folks can just go through and check off certain characteristics or observations. So that’s pretty important.

And then I also emphasize taking notes during the actual test administration. So lots of notes in the margins of the scoring booklets trying to document as much as possible what the actual test-taking behavior was like.

For example, if you’re familiar with the WISC or the WAIS, on the block design, I really want to know, are people missing these items because they are running out of time or because they worked quickly but got the wrong answer. Just a small example like that. With like reading tests, I want to know what kind of errors they’re making and how they responded to that, were they upset, were they not, different things like that. So a lot of notes on the margins.

Now, one thing that I am working on is an [00:27:00] even more detailed behavioral observation sheet where we have a separate sheet for each test. So let’s say there will be a WISC observation sheet with spaces or blocks for taking notes on each particular subtest so that you don’t have to cram them into the margins. But that’s an ongoing project. So, that wraps it up in terms of training for administering the tests.

The next part of our process is writing reports. Again, I do a lot of quality control so to speak with scoring to make sure that scoring is being done effectively and accurately. And that leads of course, to writing an accurate report to make sure that the scores are correct. So, there is that piece.

With the actual reports, it’s pretty simple. We have cloud-based electronic records, so everyone can access the reports that are [00:28:00] written. I just use Track Changes or the editing feature in Google Docs, which is great, for shared document editing. And I using one of those methods make it really clear what changes I’m making to the reports. This tends to be more relevant at the beginning while folks are learning how to write the report.

And I will say, actually, I should back up just a little bit, that I have honed our report template, particularly the results section where you’re just talking about scores and what range the score fell in and that kind of thing. I’ve really nailed that down to where it’s very structured. There’s not a whole lot of room for creative writing so to speak. And it’s, like I said, pretty structured and straightforward. So that was some work just on the front end that was put in place to make sure that there is standard language in the reports.

[00:29:00] Either way, I do go through, I edit the reports. I use Track Changes just to let people know little tweaks and nuances here and there that I am making and things that I like to address in the reports.

And then again, on an ongoing basis, we do supervision each week. So we meet for an hour talking about specific tests, things to watch for, and answering any questions. So we use supervision a lot and of course, talk about interpretation of the tests and things to watch out for there. So those pieces tend to… We tend to focus on that over the first month or maybe 6 weeks of supervision, and then after that, we transition to the work and case consultation and that kind of thing.

That’s our training process in a nutshell. I know that there are many other models out there. Like I alluded to at the beginning, there is, I think an increasingly popular [00:30:00] push to get your psychometricians certified. So there’s now board of certified psychometrists. I would really love to get a professional from that board to come on the podcast. So I’m going to be working on that. But there is a push to certify your psychometrists. And this is, I think like any other certification, you have to get a certain number of practice hours or supervised hours, there is an exam and then there is a certification that you can obtain. I think the opinion coming out of this so far is that it’s really nice.

If you do a lot of forensic work, you are probably aware that they will question the qualifications of your psychometrician in court. And so, having a certified psychometrician can really help. And then it’s also, I think just nice for standardization period. You know that you have someone who’s quality and is [00:31:00] administering the tests effectively.

A step short of that is that I know in children’s hospitals, for example, where they might have a team of psychometricians, they will often have a lead psychometrician who is in charge of supervising and training all of the psychometricians just to ensure standardization as well. So there are a few steps that are possible to make sure that you have a standardized administration process. But like I said, hopefully, I can get someone on to talk about that board of certified psychometrists and really give us more detail on that.

Thank you as always for listening. This is really cool. I love doing these podcasts and sharing some of this information with you. We have some really cool interviews coming up over the next few weeks. I’m going to be talking with Dr. Bryn Harris down at CU Denver about culturally competent assessment. [00:32:00] I am also going to be talking with Dr. Karen Postal, who wrote the book, Feedback That Sticks, an amazing book about giving feedback and difficult feedback. So I’m really excited for those interviews.

In the meantime, if you want more information or want to learn more about The Testing Psychologist or testing in general, you can go to the website, which is thetestingpsychologist.com. You can also check us out on Facebook. We have a nice Facebook group going with some discussion talking about all sorts of things testing- that is The Testing Psychologist Community on Facebook.

And if you are interested in growing your own testing services or need to chat with somebody about how to do that, I would love to talk with you about that. You can schedule a call on The Testing Psychologist website. We can chat and see where you’re at and if it would be helpful to do some consulting around testing in your practice.

Take care, enjoy the oncoming summer, and I will [00:33:00] talk to you next time. Bye. Bye.

Click here to listen instead!

[00:00:00] Dr. Sharp: Hey, y’all. Welcome to The Testing Psychologist podcast, episode 20. I’m Dr. Jeremy Sharp.

Hello, welcome back to another episode of The Testing Psychologist podcast. Hope y’all are all doing really well. I am doing another pre-roll this week because we have a little bit of a special episode this time. Here in just a minute, I am going to play you the audio from an interview that I did on the Insurance Answers Podcast.

So the Insurance Answers Podcast, if you haven’t heard of it, is a podcast aimed at mental health clinicians who want to learn more about how to bill insurance in their practices. I got to know the host, Danielle, and Katia, through Facebook, which seems like the place [00:01:00] that I meet everybody these days but with good reason. I have made some great contacts.

Danielle reached out to me and asked if I would be a guest to talk specifically about insurance billing for psychological testing. I was happy to do that. I think the interview turned out really well. They asked some great questions. We really break down a lot of the small nitty-gritty questions about how to bill insurance for testing.

My hope is that you will walk away with the idea that it is totally doable if you want to do it. And that has definitely been my experience. In the meantime, I hope you get a nice introduction to Danielle, Katia, and their podcast, and maybe follow up and listen to a few more of their episodes because they’re great.

So without further ado, here I am on a reverse interview with the Insurance Answers Podcast.

Danielle: This is episode 19, Insurance Billing for Psychologists, with guest interview, Dr. Jeremy Sharp.

Katia: Our guest today, [00:02:00] Dr. Jeremy Sharp. He is a licensed clinical psychologist. He’s a Clinical Director at the Colorado Center for Assessment and Counseling, a private practice that he founded in 2009. It has grown to include five other clinicians.

He earned his undergraduate degree in Experimental Psychology from the University of South Carolina before getting his master’s and PhD in Counseling Psychology from Colorado State University. Jeremy specializes in psychological and neuropsychological evaluation with kids and adolescents.

He is also the host of The Testing Psychologist podcast, so everyone will have to check that out as well. In this podcast, he provides private practice consultation for psychologists and other mental health professionals who want to start or grow testing services for their practice. Jeremy lives in Fort Collins, Colorado, with his wife who is also a therapist, and his two young kids.

So welcome Jeremy.

Dr. Sharp: Thank you. Glad to be here.

Danielle: Welcome. [00:03:00] Jeremy, tell us a little bit about your role as a licensed psychologist and what types of services you offer with clients.

Dr. Sharp: I think of my practice in two phases, what I think of now as my former life as a psychologist. I had a generalist practice specialized in couples. So I had a lot of training in emotionally focused therapy for couples.

Two years after I founded my practice, things started to shift and that’s when I started to focus primarily on just doing testing and evaluation. So I’d say over the last 6 years, something like that, I’ve only done testing primarily with kids and adolescent.

So that’s what my world looks like these days personally. I do, like I said, psychological testing and neuropsychological testing for kids, a lot of IQ, academic, memory, learning, executive functioning, that kind of stuff, personality assessment.

And [00:04:00] here in our clinic, we offer a variety of services. So we have a few other therapists who do therapy with kids and with adults. We also do assessment with adults as well.

Katia: Okay. All right. When people come for psychological testing, what are their main concerns?

Dr. Sharp: For me, with seeing kids primarily, I get a lot of referrals from pediatricians, from schools and just other parents around town. So I’d say the big three referral questions for us or for me right now are probably, does my kid have ADHD? Does he have a learning disorder or do they have autism? Those are probably the top concerns.

Katia: Okay.

Dr. Sharp: And then there’s usually some mood stuff wrapped up in there as well. So trying to separate out like how much of this might be anxiety or depression or even more serious bipolar, psychotic stuff as well, differential diagnosis of most of those things.

Danielle: Can you tell us a little bit more [00:05:00] about what testing looks like when accepting insurance?

Dr. Sharp: Oh goodness, yes, testing and accepting insurance. I’m joking. Like everybody, there’s something about testing and insurance, and it’s supposed to be a hard thing but it’s actually not that tough.

I assume we’ll get into some of this stuff as we go along, but insurance has been pretty easy to deal with in the testing world. There are a few particular insurance companies that are really strict about preauthorization and might limit the number of hours, but for the most part, as long as you have your documentation in place for medical necessity of testing, most insurance companies are fairly easy to deal with.

Katia: That is really helpful to hear.

Danielle: That’s really good to hear because I’ve always assumed that it was a huge mess as far as getting testing with insurance. So that’s great to hear that it’s not that difficult.

[00:06:00] Dr. Sharp: Yeah. I think there was, I don’t know what you’d call it, kind of a holdover from years past. I think it used to be a lot harder. I think it is a lot tougher maybe in a hospital setting, but private practice has been okay. They shifted some of the guidelines for reimbursement for testing, I forget when it was, maybe 2006 or 2008, and that made it a lot easier to get insurance coverage for testing. So I think that’s made a big difference.

Katia: Interesting. Have you always accepted insurance in your practice?

Dr. Sharp: I have. Yes.

Katia: Wow.

Dr. Sharp: Totally. When I started my private practice, I didn’t even think about not taking insurance. This was …

Danielle: Right on.

Dr. Sharp: Yeah. I’m sure y’all are big fans of that. So like I said, I never even considered not doing it. I feel like I didn’t know back then, to be honest, that I could not take insurance. I just thought that’s what you did. [00:07:00] So I did, and it has worked out well.

And over the years, we’ve gotten busier and busier, and I think have a decent reputation here around town. So I think we could go off insurance, but every time that I revisit it, it just keeps coming back to how important it is for access for certain clients and really for most clients because testing is expensive without insurance. I feel like it’s really important to provide those services for folks through the insurance process.

Danielle: That’s great. So when you first started accepting insurance, what was the credentialing process like for you?

Dr. Sharp: Gosh, I feel like I have blocked it out.

Katia: It’s easy to think.

Dr. Sharp: I don’t think it was too bad. Maybe y’all could speak to this, but our community has gotten more and more saturated over the years and so I keep referring back to the good old days when I started, which is really not that [00:08:00] long ago, but it felt like it back then that I don’t know that there was a whole lot of competition to get on insurance panels. So it went fairly quickly, like 2 or 3 months.

Danielle: Oh, that’s very quickly.

Dr. Sharp: I applied to probably four or five panels; the major ones in our area right off the bat and I haven’t added any insurance panels really since the beginning.

Katia: That’s helpful. So then how is billing done for your practice?

Dr. Sharp: We have a combination. Maybe 2 years ago, I brought somebody on, so I contracted out the insurance verification, so we have a full-time admin person who does all the scheduling, accounting part, and sending out bills, but I also have a separate insurance verification team. So we just relay the client benefit information to that team, and then they run it through their system, whatever magic thing that is, and then they [00:09:00] get it back to us quickly.

Katia: Do they also cover authorizations then for you guys?

Dr. Sharp: No.

Katia: No. You do have that in-house then.

Dr. Sharp: Yeah, we do it all in-house; either I do it because it requires specific clinical information or for some of them, my admin person can do that. It’s more general.

Katia: And so then your admin person is the one that does the billing for your practice then?

Dr. Sharp: Yeah, he does all the accounting and keeps track of patients’ bills, sends them out, and collects them.

Danielle: Payments. So what are the main differences between billing for counseling or therapy versus testing and assessments?

Dr. Sharp: I think the main thing is probably the hours involved and it totally depends on what kind of assessment you’re doing, but for us where we do pretty comprehensive evaluations, we’re billing, 8, 10, 12 hours of testing per client. [00:10:00] And so I think that’s the thing is like with counseling, you probably have two codes you might bill like the 90791.

Danielle: Right. And one unit.

Dr. Sharp: Therapy code and one unit. But for us, you got the interview code and then you have the testing code and that can be different depending on what kind of testing it is. And then some insurance companies prefer you then bill a therapy code for your feedback session. So that gets a little more complicated.

Danielle: Oh, interesting.

Dr. Sharp: Yeah. And the documentation. So you have to keep track of the hours that you spent on each test and how much time you spend writing reports. And all of that goes into the testing note that is submitted for the insurance reimbursement.

Katia: Wow. So that’s very different than the way the counseling piece works then with all of those codes.

Danielle: Wow. Yeah.

Katia: The number of hours that you have to put in.

Dr. Sharp: Sure. And it’s taking a while to get it all [00:11:00] lined up. What was that? How do I keep track of it all?

Danielle: Yeah, how do you keep track of it all?

Dr. Sharp: That’s where my EHR system is so helpful. I use TherapyNotes. I picked it initially because it was really well set up for testing in particular.

So the template that they have for a psychological testing note or neuropsychological testing note is really detailed. It forces you to specify all the tests that you administer and how much time you spend on them, and how much time you spend writing the report. It totals all the time for you and puts it all right there. So I like that structure and that just helps immensely.

Katia: Wow. Okay. Can you tell us about authorizations for psychological testing? Is that required or is it dependent on the insurance company?

Dr. Sharp: Totally dependent on insurance. I should back up too, there is also some difference between whether you’re billing for psychological testing or neuropsychological testing, [00:12:00] and it depends on the insurance company. And of course, this is the joke with insurance, you can’t nail it down. Of course, it’s not consistent between …

Danielle: Of course, that would be too easy.

Dr. Sharp: Of course. Some insurance companies require preauthorization for psychological but not neuropsychological testing and vice versa.

Katia: Okay. So can you explain a little bit the difference between what the neuropsychological testing looks like versus the psychological testing?

Dr. Sharp: Yeah, that’s a little bit of a hard question.

Katia: I just threw you a curveball. Sorry.

Dr. Sharp: So depending on who you ask, this answer probably will differ, but generally speaking, when you’re trying to decide if it’s neuropsychological testing or psychological testing, a big question is usually the referral question.

This is a generalization, but if you’re talking about what a lot of folks will call a mental health diagnosis like depression, anxiety, mood stuff, [00:13:00] even ADHD; a lot of folks will call that psychological testing. Neuropsychological testing typically follows when there’s a consideration of a medical diagnosis involved. So maybe it’s epilepsy or cancer or brain injury, concussion, that kind of stuff.

Danielle: TBI stuff.

Dr. Sharp: TBI, yeah. So that’s probably a short answer.

Katia: No, that’s helpful.

Danielle: Can you mention different CPT codes? Do you know off the top of your head, what the CPT code would be for, this is another curve ball, sorry.

Dr. Sharp: Hey, bring it on. Yes.

Danielle: The CPT code for the initial assessment testing and then the follow-up testing, are they different then?

Dr. Sharp: I still bill 90791 for the initial interview.

Danielle: Okay. Would that be the same as a therapy code?

Dr. Sharp: Mm-hmm.

Danielle: Okay. [00:14:00] The initial interview where you’re figuring out what they’re coming in for, what sort of tests you would be administering and then you would do the testing codes after that and they come back for the actual testing?

Dr. Sharp: Exactly. I should say too, I don’t know how much detail y’all want around all the new nuances, but if you are doing neuropsychological testing in specific settings, sometimes they’ll bill on 96116 for that initial interview and talking with other family members and collateral sources of information. There are a lot of nuances to it, but I generally do a 90791 here in private practice.

Katia: Interesting.

Danielle: Okay.

Dr. Sharp: And then the testing codes, like you said, they differ depending on what kind of testing we’re doing and who’s doing it. So that’s another piece that I didn’t even mention. A lot of folks who do a lot of testing will have a supervising psychologist and then have testing be administered by a technician or a psychometrician. And [00:15:00] that changes the CPT code as well.

Danielle: Okay. At least in Illinois, we have lots of students that are doing the psychological testing for their diagnostic practicum, that would be a different code then?

Dr. Sharp: Yes. That’s a little bit of a can of worms. As far as I know, there’s a little bit of a prohibition against students doing testing only for student learning. I should probably say that, but that’s the general model. Anyone who’s not a licensed psychologist would bill under a technician code.

Danielle: Okay. So there’s a different CPT code? Interesting.

Katia: That’s interesting. Do you have people that work in your practice that are those technicians that administer them or are all of your clinicians licensed psychologists?

Dr. Sharp: We do have technicians here in the practice. Yes.

Katia: Okay. I’m wondering what would be behind the decision of who actually administers the test [00:16:00] then?

Dr. Sharp: The driving factor is probably, there are probably two. One is that a psychologist’s time can be better spent not administering the testing. If you have someone who is really well-trained, like a well-trained technician who really gets the ins and outs of administration and takes good notes and all of that, then that frees up the psychologist for, that could be 8 to 10 hours per evaluation to be doing other things. I think that’s a big part of it, to be honest.

Katia: Other things like analyzing the results then?

Dr. Sharp: Exactly. That’s the model that we run here in our practice where I do the interview. I do portions of the testing with each person. So I’m in there for probably 25% of the time in most cases. And then I also do all the interpretation, pulling the data together, writing the summary, [00:17:00] the recommendations, the diagnosis, and producing that final report. I think a lot of folks run a tech model to free up time to interpretation.

Danielle: That reminds me of like when you go to the hospital to get an MRI, you have the MRI tech doing the actual test, but then you have the radiologist that is interpreting it and writing reports.

Dr. Sharp: Exactly.

Danielle: So similar.

Katia: Right.

Dr. Sharp: Yes. Very similar.

Danielle: Oh, makes sense.

Katia: I’m just curious because I did not realize they even had this. So what kind of experience do your techs have to administer the tests? I’m just fascinated.

Dr. Sharp: For sure. That’s an interesting question. So the folks that I hire are all advanced doctoral students. And so they’re a year or two away from their PhDs. They at least have master’s degrees and they’ve had at least a year or two of prior testing experience before they come in our [00:18:00] practice. And so that’s their level of training. And then we do on-the-job training once they get here.

There’s a big push, I forget the official name of it right now, but there’s a certification process for psychometricians that’s gaining steam and that’s starting to become more important, the Board of Certified Psychometrists, I just looked it up.

Katia: Interesting.

Dr. Sharp: So there’s a little bit of a push to get certified as a psychometrician to make sure to maintain the quality of administration.

Katia: Okay. Wow.

Danielle: Wow. Neat.

Katia: I had no idea.

Danielle: I wonder if this varies by state, I wonder. I’m going to have to check it out.

Dr. Sharp: Yeah, check it out.

Katia: That is fascinating. Okay. Wow.

Danielle: What’s the term psychometrician?

Dr. Sharp: Psychometrician.

Danielle: Psychometrician, like oh, what do you do? I’m a psychometrician.

Katia: I had no idea. This is the mind-boggling piece is that from our [00:19:00] perspective as just clinicians doing the counseling part, it’s like, how do you do all of the different pieces? And that’s where we said we really needed to get somebody to interview because it’s just so many different components here with the actual administration and then the analysis and then the feedback. It would be really hard to be a one-stop shop and have to do all of that work alone. That would just be intense.

Dr. Sharp: Yeah, it’s taken a lot of time, a lot of years, a lot of trial and error, and phone calls with insurance companies to try to nail all of it down. I should say too, this might be a good time to mention that, I don’t know if you’ll do show notes or stuff like that, I have to refer to the current president of APA. His name is Antonio Puente. He has a ton of presentations and information out there about appropriate billing for psychological and neuropsychological testing.

Danielle: Oh, great.

Dr. Sharp: So if you can link to that.

[00:20:00] Katia: Absolutely.

Dr. Sharp: I know he has a website and some cool resources out there.

Katia: Okay. I’ll make sure we add that because I think that’s going to be a really helpful link for psychologist listening and even some other clinicians that are allowed in their state to do some of the testing, I think that would be really helpful.

Dr. Sharp: Absolutely.

Katia: Okay. It’s definitely more complicated.

Danielle: You’re claiming that it’s not but I bet once you do it for a while, it’s not so much, but this seems very complicated to me.

Katia: Right.

Danielle: That’s saying a lot because billing in itself isn’t complicated to me, but this whole different codes for different tests. You mentioned writing reports and everything, is that also covered by insurance?

Dr. Sharp: Yes, it is. That was one of those major shifts that I was talking about a few minutes ago, back in whatever it was 2008. Report writing is now covered under the typical testing codes which helps a lot [00:21:00] because I spend between the history and the interpretation probably at least two hours, maybe three or four, depending on the evaluation. So that’s a big chunk of time.

Katia: Sure.

Danielle: Depending on the test and everything.

Katia: Yes. Are there specific tests that are not covered?

Dr. Sharp: Yes. At least in my world, the ones that are typically not covered are academic tests. So the insurance companies in my understanding is that they operate on the assumption that any academic testing is going to happen through the school district which is a bummer.

Schools do the best they can and support students really well. There are a lot of students who don’t qualify for testing through the school who could still benefit from it. So that’s tough, but the academic tests are the ones that are typically not covered.

Katia: Okay. So if you had someone who wanted to come and get services to get that academic testing, [00:22:00] does your company still do them, but it would just be a non-covered service?

Dr. Sharp: Yeah. There are two different ways that we do that. One is that I cannot administer the entire academic battery. There are different subtests that are maybe a little more relevant for the presenting concern without having to do the whole battery. So we do that sometimes.

But if someone is coming specifically and they’re only saying, hey, we have a question about a learning disorder. That is purely an academic concern. If there are no other complicating factors then I have to say, okay, well, here’s the deal; insurance doesn’t typically cover testing for learning disorders. So that’s an out-of-pocket expense.

Katia: Right. Okay, that makes sense.

Danielle: And then would you possibly refer them to their school? If they would say, oh, I can’t afford that. Would you say, well, your school psychologist would probably do the same [00:23:00] testing or?

Dr. Sharp: That’s tricky. I was saying that a lot of kids don’t typically qualify for testing through the school unless they’re falling pretty clearly outside the average range for either behavior concerns, disruptive behavior, or academic concerns. At least here, and I think this is pretty consistent, they have to be at least clearly two grade levels behind to qualify for an evaluation.

Danielle: The squeaky wheel gets to the oil. That’s unfortunate.

Dr. Sharp: I’m cautious about telling parents what the school will or will not do. I’ll say, you can request this and it might not happen.

Danielle: Right. Sort of you could look into it.

Katia: That is interesting. That does make it really challenging if you’re trying to make these assessments and trying to make sure that these kids are not falling down [00:24:00] to the two grade levels behind. If you catch something sooner, then later you would just think that preventatively that would make a big difference to provide appropriate accommodations if needed. That’s definitely a huge challenge that your industry must face then in this respect.

Dr. Sharp: Yeah. You’re opening a big can of worms there, services in the school.

Katia: That would be an interesting episode for your podcast because you deal with that side of things; how you guys navigate that. That sounds really challenging.

Dr. Sharp: It’s interesting you mentioned that. In my next interview, I’m talking with a school psychologist who’s now in private practice to talk about bridging that gap.

Katia: Wow. Okay, neat.

Dr. Sharp: It’s a good chat.

Danielle: You mentioned doing multiple tests in one day. I guess you might have answered this earlier, how you handle the multiple tests is that you [00:25:00] don’t necessarily do them all yourself. You pass them off to the psychometrician. I can’t even get that term right. It’s so new to my lexicon. How do you handle multiple tests in one day if you would be doing it themselves? Do you do a whole battery and are all those tests in that battery billable?

Dr. Sharp: There are different models in different practices, but we tend to try to knock out the whole battery in one day. So it ends up being probably 4 to 5 hours of actual testing. We take breaks on a lunch break and all of that. We try to try to keep the person engaged as best we can, especially kids.

So all of that is billable. The limit that I’ve run into as far as insurance billing goes is that some insurance companies do limit you to billing no more than 8 hours of testing in a given day. That’s probably the main limitation but that only [00:26:00] happens with a couple that I work with.

Katia: Would that include your report writing?

Dr. Sharp: Yes.

Katia: What if the testing takes 8 hours and you still have to do the report, you just have to bill it for another day?

Danielle: The next day?

Dr. Sharp: Right.

Katia: Okay.

Danielle: That makes sense. The insurance companies are saying, you don’t work more than 8 hours a day.

Dr. Sharp: In a sense, they’re doing us a favor, right?

Katia: Yeah, sure.

Danielle: Like go home

Dr. Sharp: Thank you.

Danielle: Not to write this report.

Dr. Sharp: Right. The time that you spend on the report is pretty easily shifted around depending on the date.

Katia: Okay, so how are rejections handled then? Do you have to deal with rejections often?

Dr. Sharp: At this point, I don’t get a whole lot of rejections. Having done it for so many years, I know the ins and outs and what insurance companies need, what information and how, what authorizations, and how to write the authorizations.

Probably the biggest rejections that we get are if [00:27:00] the initial quote of benefits and coverage was wrong for whatever reason and then we have to go back. A lot of insurance companies are decent to work with. If we call and say, oh, we accidentally billed the wrong code for this, can we just resubmit the claim? Usually that works fine.

Danielle: Okay. That’s similar to the therapy world. If you call and you say, oh, I put in 90837 instead of this, they let you adjust.

Dr. Sharp: Exactly.

Katia: I have another question that we didn’t already provide but I’m curious a little bit about what the follow-up session looks like when you’re providing the results for the testing. What is that like for clients?

Dr. Sharp: So working with a lot of parents, I’ll bring both of the parents in. If the kid is in high school, I’ll have the kid come too. I think [00:28:00] that’s really important for them to have some agency over their testing results and their positive qualities in areas of concern.

In the feedback session, typically I’ll start and just go over things that went really well and really play up strengths for the client or the kid, talk about how those will serve them and come in handy, and then transition to things that were a little more challenging which often gets at the referral question, the diagnostic picture and how all that fits together.

So then I present them with the diagnostic picture and then that flows into a conversation about recommendations, what to do, what will be helpful, and what kind of interventions would be appropriate, answer parents’ questions as we go along and try to send them away with a really clear plan of how to move forward and why we’re doing what we’re doing.

Katia: Okay. What types of things would you end up recommending in general?

Dr. Sharp: It totally depends on the diagnostic picture, the [00:29:00] kid, the parents, resources, and that kind of thing but I’d say the main recommendations are, I refer a lot of kids for medication consults. That’s a big one. So partnering with psychiatrists and physicians around town. I talk a lot about how to bridge the gap to school and maybe pursue intervention in the academic setting. I talk about that.

I do a lot of referrals to things like biofeedback or occupational therapy, individual counseling, and family counseling. Those are the main recommendations, social skills groups, that come up a lot.

Danielle: Oh, sure. Autism diagnosis and things like that.

Dr. Sharp: Right.

Danielle: Are there any other tips or tricks that you have for psychologists listening regarding billing?

Dr. Sharp: I think the biggest thing is having your ducks in a row as much as you can before you start doing testing and billing insurance. To know it’s totally doable. I’ve totally built a practice around it and we’re [00:30:00] successful.

And like I said, I keep deciding to take insurance, we aren’t going off, so totally doable and there are some nuances to consider and to make sure that you’re doing everything ethically and legally as far as you’re billing and have all those ducks in a row, I think that’s important.

Katia: That’s really helpful. This was so informative. I did not realize how this all went down and it sounds like you know and figured out the best type of structure that works. You’re providing a wonderful service by structuring it properly and being able to utilize your skills appropriately. So this is really fascinating.

Dr. Sharp: Oh, that’s great to hear. It’s fun to talk about and I’m happy to spread the word about testing.

Danielle: What I also like too is that you know when to outsource too. With the technicians doing the testing and everything, I was thinking, oh my goodness, how does he do all these tests, do all this billing and everything, but you’ve recognized your [00:31:00] strengths and you’re the overseer. You’re the radiologist, so to speak.

Dr. Sharp: Sure. Hey, I’ll take that comparison.

Danielle: Because doing all the testing yourself would be exhausting and tedious.

Dr. Sharp: Some psychologists like to do their own testing and I do some, I really like to be in there for some of the testing. I was a psychometrician when I was in graduate school for two years and I enjoy doing the interpretation and the recommendations a lot more at this point. So that’s why.

Danielle: But you’re there for the initial assessment and things like that, so it’s not like you’re totally removed. You come in and people don’t know who you are, they know who you are.

Dr. Sharp: Right. Oh, of course. I still consider myself the point person and the main contact for their evaluation.

I’ll throw this in there just because we’re talking about it, but if you are billing insurance under a psychometrician code, one of [00:32:00] the mandates of that code is that the overseeing psychologist is checking in with the technician throughout the day to make sure that you can adjust the battery if you need to.

And then they introduce the psychometrician and then they also say goodbye to the client at the end of the day. So there are some guidelines for how to bill under a psychometrician code that are probably important to be aware of.

Danielle: That makes sense. So it’s not really the psychometrician running their own show. It’s like, okay, we’re tag teaming this.

Dr. Sharp: Yes, exactly.

Katia: Exactly, cool. Wow. This is really neat. I’m so glad that you were able to come on and answer our questions about billing for a psychologist. This is just one area where neither of us knows anything.

You helped to do what we are so passionate about, which was bust these myths about the difficulties with insurance as a provider. And that is one of the biggest things is that the language has been out there that scares [00:33:00] people away. And so I like that you are pro-insurance and have that similar stance and have demystified how intense it is for psychologists. So this is really helpful.

Dr. Sharp: Absolutely. Thank you so much for giving me the opportunity to come on and talk about all this stuff. I love spreading the word, like I said, and it’s a fun thing. It’s totally doable.

Danielle: It seems like with any type of billing, trial and error, you learn all the kinks and then once you know how to do it, the ship runs pretty smoothly.

Dr. Sharp: I think that’s true. I totally agree.

Katia: Great.

Danielle: So not to get discouraged by the trial and error piece of insurance billing which I’m sure you’ve gone through.

Dr. Sharp: Oh, my goodness. Yes, I can’t. It’s like we all say, if we had the money for the time spent on the phone with the insurance companies, it would just be…

Danielle: Sure.

Katia: All right. Thank you so much, Jeremy.

Dr. Sharp: Of course. Thanks for having me.

[00:34:00] All right. Thanks for listening to my reverse interview with Danielle and Katia on the Insurance Answers Podcast. Hopefully, you took away some helpful information there. If you are billing insurance, maybe there are a few things that you could tweak. If you’re not billing insurance and want to, hopefully, it gave you a little more clarity as to how to do that. Like I said, I think it’s totally doable and has certainly been fairly easy here in our practice once you get those standard operating procedures down.

So as always, thanks for listening. I’m having a great time doing these podcasts, getting to talk with folks, answer questions, and connect with other people. This has just been an amazing experience so far. I can’t believe that this is already 20 episodes. I remember back before I started thinking, I had no idea what this is going to turn into or end up like. I have to say that it’s been a really cool journey so far. I’ve had such a good time connecting with other folks and learning so much along the [00:35:00] way.

I hope that you’ll continue to tune in and be part of this community. If you do want to join the Facebook community, there’s some great discussion there on testing preferred measures, processes, billing, different things like that. You can search for our community on Facebook at the bar at the top. So search The Testing Psychologist Community, and you should be able to find us.

And if you are interested in doing any consulting around building psychological assessment services in your practice, I would love to talk with you about that. We could do a complimentary 20-minute conversation just to see where you’re at, and what you want to do. I’ll give you some ideas about whether consulting could be helpful and if not, where you might go instead.

So I hope to talk with you next week. We have lots of cool interviews coming up and I’m excited. In the meantime, enjoy the summertime and take care. Thanks, bye-bye. [00:36:00]

Click here to listen instead!