[00:00:00] Dr. Sharp: Hello, everyone. Welcome to The Testing Psychologist podcast, the podcast where we talk all about the business and practice of psychological and neuropsychological assessment. I’m your host, Dr. Jeremy Sharp, licensed psychologist group practice owner, and private practice coach.
This episode is brought to you by PAR. The TSCC and TSCYC screening forms allow you to quickly screen children for symptoms of trauma. Both forms are now available through PARiConnect; PAR’s online assessment platform. You can learn more at parinc.com.
Welcome back, everybody. Glad to be back with you for another episode here as we start to wind down 2020. Hopefully heading it to bigger and better things in 2021. Before we get there, though, I’ve got a couple more conversations, maybe just one more conversation. And I think this is it. [00:01:00] So today I’m talking with Dr. Anna Kroncke and Dr. Marcy Willard all about the autism assessment tool that they have developed called CADE.
So you may not have heard of this. I did talk about it a little bit back in my episode on the Remote Assessment of Autism. But today we’re taking a deep dive with Anna and Marcy co-developers of this tool. It’s a software tool and I think of it as a lengthy autism-specific measure that really digs into the nuances of autism. And it’s based on a fair amount of research and a framework that they have developed over the years in their work. So I think this will be interesting for you.
We talk about many things. But we touch on certainly the origin of their company and the [00:02:00] software, the research and how they have validated this tool, how it’s different than other questionnaires out there. We talk a little bit about the remote assessment of autism in general, and then we talk about future directions and where they’re headed with this project. Now, you will hear it briefly in the episode, but I do want to highlight that Anna and Marcy are very open to expanding the research base with this measure and they put out a bit of an open call for anyone affiliated with hospitals, universities, who might want to conduct larger-scale research using their measures. So if that is you, don’t hesitate to reach out and get in touch with them. Their contact information is in the show notes.
Okay, before I totally transition, let me tell you a little bit more about Anna and Marcy.
So Anna Kroncke is a licensed psychologist and [00:03:00] nationally certified school psychologist. She’s the lead author on an autism assessment textbook which is linked in the show notes. Anna loves to meet the needs of families in her assessment practice.
She was a presidential fellow at the University of Georgia, which is where she got her master’s and Ph.D. She has experience as a school psychologist working in a variety of districts and settings. She’s been in clinical practice for the past 10 years, and she does a lot of work with certainly conducting psyche vows, but also training others, consulting with families, and helping the community providers and schools.
She’s trained graduate students in comprehensive assessment across her career, through her expertise in autism, anxiety, depression, and assessment tools. Now she and Marcy worked together to develop their framework, which is called CLEAPE and the [00:04:00] CADE assessment tool. And she is the co-founder of Clear Child Psychology.
Okay. Marcy is a tech entrepreneur, licensed psychologist, nationally certified school psychologist, and a published author as well. She also loves helping children and their families navigate different challenges and learn to thrive. She has a history of conducting diagnostic assessments at local clinics and as a fellow at JFK Partners in partnership with the Children’s Hospital of Colorado.
She got her bachelor’s at the University of Colorado, a master’s from Pepperdine, and a Ph.D. from the University of Denver. So Marcy aspires to partner with and guide parents who have concerns about their child’s development or mental health by providing a customized roadmap in supporting their unique child.
She’s a big believer that no family should suffer or any kid’s potential got wasted when there are so many resources that can genuinely help. [00:05:00] So before founding Clear Child Psychology with Anna, she worked as a school psychologist for several years. She has trained many other psychologists and autism assessment in both the clinical model and the school certification model.
All right. So if you are still on the fence about joining a mastermind group in 2021, I would consider the Advanced Practice Mastermind for Testing Psychologists. As of the time of recording here, we have one spot left. It starts on January 7th. And this is a group that is really aimed at practice owners who need some accountability and just some support from other psychologists in the same place as you reach for those bigger goals in your practice.
So we focus a lot on accountability, streamlining, hiring, things like that, that come up after you pass that beginning stage of practice. If that sounds interesting to you, you can go to the testing [00:06:00] psychologist.com/advanced and schedule a pre-group call to see if it’s a good fit.
All right. So these women are true entrepreneurs. They have a wealth of knowledge about measure development and clinical practice specifically around autism. So I hope that you get a lot from this conversation.
Hey, Marcy, hello, welcome.
Dr. Marcy: Hi, thank you so much for us.
Dr. Anna: Thanks so much for having us.
Dr. Sharp: Yeah, I’m so glad that you were willing to be here. You are fellow Coloradoans. It’s always nice to have other Colorado folks on the podcast. And you’re doing some really interesting [00:07:00] work. So I just am grateful that you were able to take some time to sit down and chat with me. So thank you.
Dr. Anna: Yeah, thanks, Jeremy. We’re so excited to talk about what we’ve got going on and it was just need to be able to connect to the wonderful community that you’ve brought together here.
Dr. Sharp: Sure. I’m guessing that some people might recognize if not y’all’s names, the name of the tool that you’re developing from the previous podcast where I was talking about Remote Options for Autism Assessment. I mentioned y’all’s tool there. I think this will be great to have some time to really dig in and talk in more detail about what y’all are working on and this amazing piece of software, I suppose, that you’ve got going on.
So without further ado, you know, we were talking before we started to record about one of the cool things about the two of you is that I see you as normal people which I mean, in the [00:08:00] most complimentary sense possible, in the sense that you’re in private practice or you’ve been in practice, you’re not researchers or like academic whatever ivory tower people you’re just practicing clinicians. And you’ve really dedicated a lot of time to develop this measure. So it kind of gives me hope that a normal person could do it without these crazy backgrounds.
So, maybe tell me, start from the beginning. How does this even happen? Like going from, you know, you get into your career and you’re doing your job, and then all of a sudden you’re developing a measure. What happens in that whole process?
Dr. Anna: Yeah. This is Anna. I would love to jump in and tell you guys a little bit about that and then maybe Marcy can dive deeper into CADE, which is our tool, and talk about what it does. [00:09:00] So we connected in 2011. We were in a very traditional clinical practice setting when we started working together and we had some near opportunities over the years to dive into different projects.
We were able to work on a textbook together. And one of the things we wanted to do in that textbook was to really break down autism assessments in a way sort of not just relying on like, okay, let’s look at the GARS and let’s talk about the reliability and validity of this measure. Let’s look at this scale here. This is the eighth house and this is how it breaks down. We wanted to talk about the actual nuts and bolts process of what it’s like to work with a family and to work with kids and what it’s like when you’re in that evaluation room.
And so we spent a lot of time working with a great team on that project. And that’s been really fun for us. And so we have an autism assessment textbook that some of you may be familiar with. That’s out there on the scene. Another thing that we were able to do is work [00:10:00] together in a sort of clinician school partnership. There was a time period there where I was doing clinical services and Marcy was doing school-based services. And so we had a neat opportunity to follow families after their journey and to collaborate. And Marcy was able to really guide them in how to implement some of the things that came out of our evaluation process.
And so one of the things we learned is that this is a lot for families. And sometimes when you’re sitting down to feedback and you’re showing them a WISC score report and you’re going over the WJ and you’re diving into this fell for whatever else, it’s a lot. It’s overwhelming.
And so in Cate Arts School, we really wanted to focus on symptom profiles in these areas that families could really understand. And what could they implement and make that really tangible at their fingertips? So that was super cool for us. I’ll say that a few things that came out of our lungs with career and working together [00:11:00] before Marcy started Clear Child Psychology, is that we had a couple of things that came up time and again that drove us nuts.
And one of them was that families would walk in the door and say, where were you five years ago? Where were you eight years ago? I was concerned about my child when he or she was three or five or seven, and I didn’t know what to do at that time. And we felt the parent education really isn’t out there the way that it should be in guiding families to the services that they need.
So they go to their pediatrician and they hear well, boys develop a little more slowly than girls, Oh, let’s wait and see, give it a little bit of time. Or they go to their teacher who says, Gosh, maybe you should try some play therapy or they get prescribed something to try that’s not really fit their child’s profile. And it takes them years to fall into our office. They would say, Oh, I live two miles from here and I didn’t even know that this is where I needed to be. And so certainly we talk a lot about waitlists and that’s hard too, but at least [00:12:00] those families know they’re on a path to get some supports, right? And we’re talking about families who were floundering with these challenges for a long time before they found us.
And then the second big thing we noticed, I think being in Colorado uniquely we’re surrounded by areas like the Westerns Slope by States like Wyoming, that people sometimes call the Frontier. I love Wyoming. It’s beautiful. But the idea is that families don’t have resources. And so they come to find us in Denver or in Fort Collins where you are Jeremy. So they don’t have the resources to go back to after the evaluation. And so we wanted guidance for families and recommendations that they can implement on their own. And they also, they don’t always have the ease of travel. And they have a child maybe who obviously we know with kids with autism struggle to transition new places, new people can be a hard adjustment and here they are having to come a [00:13:00] long way to see us and to get the help that they need.
And so Marcy is this brilliant business person and she’s creative. She thinks outside the box. And she’s like, “What can we do here to address this in a way that’s more virtual, that’s more oriented to family education?” And that’s where we really started working together and developing tools for families because we really wanted to tackle that for we’re passionate about families having access and having the support they need.
Dr. Sharp: Yeah, that’s amazing. I think we’ll talk about this as we get into the tool. But one thing that really jumped out and that I heard from my peers who were using it early on was the recommendations or the suggestions that come from it are way beyond anything we would expect.
So it’s like it’s very grounded. It’s very concrete. It’s focused on giving families ideas more than anything which is really cool.
So I’m [00:14:00] curious then, I mean, so the way you described that it’s like, okay, we had a business idea and Marcy had this thing to work on and then all of a sudden it’s happening. But I am curious about the nuts and bolts. How do you actually do that? Go from I have a great idea to I’m developing a measure now.
Dr. Marcy: Right. Yeah. There were a lot of fun and exciting and scary steps along the way and we still have all the above in our existing practice and business that we’re building.
One of the first things that we did was to get great people who could answer some of the questions that we knew we couldn’t answer ourselves. And so early on, we had multiple meetings with large groups of experts that we diligently recruited. So we did something we called a Gemba interview. The concept of Gemba come from the [00:15:00] Japanese idea that they used to have factory workers, the bosses, or the big managers of these factories would do something called Gamba where they would go and be on the factory floor and find out what problems that needed to be solved.
And then actually, Co-develop those solutions with the people who need that solution. Right? So it’s the opposite of the ivory towers philosophy, or you have to do get a special pass to go to the C-suite. It’s the opposite of that thinking, right? And so we wanted to do the same thing in developing this measure.
We said, we want to ask the people who are in pain what is it that you need a clinician to make your life better? And what is it that you need family in order to get access to these resources? And as you can tell that that’s a pretty complicated question there, right? Because we really want it to be an ecosystem and a marketplace and a place where people are interacting and sharing resources versus how it has been, which is [00:16:00] very siloed. And you get a stroke of luck if you bump into somebody that happens to have what you need. And even for clinicians, we want to work with each other. Anna and I have always been like that. Gosh, I’m not an expert in every single thing that could be going on with a kid. I’d love to have access to those people and those resources and I gladly pass them along. And so we really wanted to create that.
And actually one of the first things we ever did was a big course on autism assessment and our framework that we had developed and we got to meet you, Jeremy, at that day and a bunch of other great clinicians that we still partner with now, in terms of how do we improve our measure? How do we make sure that it’s accurate and valid? How do we make sure that it’s useful?
Dr. Sharp: Right.
Dr. Marcy: And so my dad who is an expert in software development was a key asset in that process too. And understanding how do you develop a software program that [00:17:00] works and that solves the problem. So one of the big things, that’s kind of the opposite of this, this lean agile method we use is that you basically go in the back office and design the whole solution and then you launch it. And what happens with that is it doesn’t work.
Dr. Sharp: Right.
Dr. Marcy: You have to go to your stakeholders and say, “Hey, what do you want?” And then you try it. And then they say, ‘Yeah, I like that little part but I don’t like any of those other parts.” And then you go back to the drawing board again and you do those hundreds of times until you say, what do you think of this? And they’re like, I would use that. And then you start to get it out there.
So that was really the journey that we went down from validating the measure to getting it into a software program that could automate it and not only automate it, but also deliver the results in a way that would be really useful. We want it to be useful to the clinicians and to the families [00:18:00] that they could see this profile versus hear about it. What are the big things that we run into as clinicians is that feedbacks are roaming. Your families sometimes look at you with these big deer eyes saying, Oh my gosh, like, thank you. And now know what I’ll do.
And there’s a lot of things, right? We have a lot of things we want them to do. And so how can we make that more clear for them? So that was really a huge piece of what led us down this path.
Dr. Sharp: Yeah. I love the idea of being agile and responsive to what people want and need along the way. And I’m just holding back from asking all sorts of questions about the software and the development and everything because I love that stuff, but I don’t want to lose our audience in that whole process, but suffice it to say it sounds like a nice marriage between you had the fortunate circumstance of someone who was a software expert and you had [00:19:00] your clinical experience and you did a great job sort of recruiting folks from the community to give you feedback. And then you’re off to the races, right?
Dr. Marcy: Right.
Dr. Sharp: Yes. So that might be a nice place to transition to talking about the measure or the tool. What do you y’all call it first of all? That’s probably important.
Dr. Anna: We’d probably call it more than one thing. I call it a tool most of the time and I call it a collaborative questionnaire it’s a way to gather feedback and to share information. But I think when I’m talking to people, I generally call it a tool. What if I tell you more about it in case somebody didn’t listen to your other podcast a few months ago?
Dr. Sharp: That would be great.
Dr. Marcy: I think Jeremy is looking for the name of the measure.
Dr. Anna: CADE, the name of the measure. And so, yes, I like that it has a little catchy psyche name to it. Some of our software people are like, No, we don’t like what you name it. We just call it the clear tool. I’m like, no, I [00:20:00] like it. So it’s called CADE.
And so we have 80 questions with our rigorous process of getting that information in there. So we started out with a framework that we developed and we used wonderful supports in speech pathology, occupational therapy, pediatrics, board-certified behavior analysts, other psychologists to really develop the meat and to write about all of these different symptom areas that appear in the tool.
So we have 11 different areas that we’re assessing. So anything from socialization, communication, cognition to emotions and behavior. And we use parent-friendly terms. So understanding, socializing, communicating. When we put that out there, right? So, but beneath that, we do have a lot of articles and deep information that came from a variety of clinicians who have expertise in these different areas.
So we took that as the basis and then as Marcy said, we did a lot of Gemba [00:21:00] we’re in the community. And then we also brought in a panel of psychologists and developmental pediatricians and school psychologists. And those people were able to give us feedback on these different areas that we’re assessing and say, well, I don’t think that’s relevant here to an assessment that’s more focused on autism or I think you should add some more depth and more detail and let’s add items and information about these different areas. So we were able to use a nice item, response process there to go through. And so when we came to validate the tool, we have 80 items total in the tool.
So we went down to like 50 and back up to 80. And so we’re all over the place to get the items that really fit and went through a rigorous process to do that. We were also able to have some nice meeting with meetings with some experts along the way. One of the fun ones for me was sitting down with Bill Reynolds. We had the items and he was talking to us about just being really inclusive, right? Bill does this depression and suicide risk measures and things. And he was impressed that we were really broadly inclusive of a [00:22:00] lot of different symptoms. We were asking them about their sleep and about their emotions.
And even though we’re a tool that was looking at autism, we wanted to really get a broad picture. And so having those 80 items I think it’s needed in a few different ways. Like for our families, it allows them to get a really complete picture of their child’s symptom profiles because we know autism looks so different.
And so we do want to dive into gender identity and sleep profiles and whether the child has depression and anxiety or attention problems. And so we can do that and then we can give recommendations that aren’t just very autism-specific, but really address the whole profile that the child exhibits. So that’s been good.
And then we put a lot of visual imagery with everything and Marcy is totally a visual person you can see behind us that we have these pictures. Jeremy can see you guys can’t. But that relates to different symptoms, right? So we’re looking at fluid reasoning and auditory processing behind us, but we put visual imagery [00:23:00] and we use radar charts to display the symptoms. So you can look at a child’s socializing profile for example, and say, okay, well, this child really has a lot of difficulties with perspective taking and with conversations and with shared enjoyment, right? And so then you’re diving into the symptoms, not so much just saying, okay, this child has autism.
So we feel like we can get more meetings with the families, but at any rate. So we just took these 80 items and all the resources that we developed and built behind it, all the recommendations and articles, and then our software team was able to build that into a tool so that we can generate an autism score. And then these just individual profiles for domains that are gonna determine what kinds of recommendations come out of the tool and where we guide families to go next.
Dr. Sharp: Yes. I have so many questions from all of that. So let me try to remember back to my first question, which is kind of a reflection too, but one thing that [00:24:00] I like about your tool is that it sort of goes beyond the DSM-V criteria, right? Like you are pulling in these domains that aren’t explicitly implicated in the DSM-V. Some of them, I don’t know if you call them soft signs or just other related factors or whatever. But I guess there’s also a danger there, right? That you’re venturing into uncharted territory when you’re saying like, how much does this associate with autism?
So I’m curious what process you all maybe went through as you were fleshing out these different dimensions and these questions that got at the soft signs that maybe weren’t part of the diagnostic criteria. Can you speak to that at all?
Dr. Anna: Do you want to answer, Marcy?
Dr. Marcy: Yeah, I’ll take that one. The process we went through, we had a multifaceted research-based for it. And then [00:25:00] both qualitative and quantitative analysis that went into those items. So as you mentioned, we really wanted to get at those ancillary signs of autism, the ones that the clinicians see and feel before you even go through your checklist. The kid walks in and they sit down, you maybe spend 10 minutes talking to them and you say, okay, this kid’s on the spectrum.
And then we started asking ourselves, “But how do I know I don’t have data. I don’t have all my measures collected, but I still know.” So we wanted to say, what is that? What are those other signs? What are those little subtle things? And so we thought through just noodles and noodles of questions from other measures. And we thought, where is this falling short? Where are these little pieces? I’ll give you a great example. This is something I totally geek out on, which is narrative coherence. That’s one of my favorite things. [00:26:00] So narrative coherence is a big one, right? So their storytelling skills. Yes. We have certain pieces of some measures that barely touch on that, but we wanted that to be a piece of the measure. And we ask ourselves how is it that your kid at telling stories? We ask the parents that, we ask the school that, we directly assess it. And then the question becomes, well, yeah, we know it’s related in our clinical practice, but is that just our opinion? You can’t write a measure on your opinion.
Dr. Sharp: Right.
So we did two steps to really dig into that. And the first one is in our own validation study, we met with a whole bunch of expert psychologists and said, “Hey look at this item. Is this worded the way you would want it to be worded is it asking what we think we’re asking? How specific is this item? And then how sensitive is this item? If a kid has this issue, is the tool going to pick this up? [00:27:00] And then how clear is it? Do you know what the heck we’re saying?”
And so we went to these large groups of experts and asked them to really rate these items. And then we did a quantitative process to measure the accuracy of each of these items on those three criteria. From that, we went into a factor analysis. So that process, we looked at what are the factors that these items are loading on? And then we did a deep look at the ones that are loading on an autism factor, what are the factor loadings? And we only took items that loaded very highly on this autism factor.
We found multiple factors, but the main one on the items had to load highly. They didn’t get included in the autism score. There are also items on the measure that aren’t included in the autism score that we kept. And the reason for keeping those is for those visual profiles. So maybe let’s say that narrative coherence didn’t make the cut. It did, but let’s say that it didn’t [00:28:00] we still care about it. And so we would want that to be noted in their profile for intervention. And so that’s a long-winded answer for that. But the point of that was there really was a rigorous process in finding those ancillary extra little symptoms and signs and making sure they get captured in the measure.
Dr. Sharp: Right. I love that. And tell me, would you call this a diagnostic tool or do you see it as something different?
Dr. Anna: Yeah, that’s a good question. Because we’ve had some questions in the past about that. Oh, my Gosh, we have been through the FDA and all of this stuff? So it’s a collaborative tool, I think and it definitely gives you a diagnostic profile. But we feel like it’s a compliment to what the clinician is doing. It allows them to draw information from various sources and have a place to put their thoughts and perceptions. And so if they [00:29:00] answer those questions, it is going to generate an autism profile, but it’s coming from their input.
And then she was able to back-check on that. And we did that originally in the validation, but now we’ve been able to build a piece into our tool where it will ask the clinician. Is the child diagnosed with autism? Is a child educationally identified? So that we can compare two to see how is the tool doing compared to what the child’s actual diagnosis or educational classification is?
And we’re at 98% accurate there which is quite consistent with the study we did before. And we obviously want to get a lot more data. But yeah, so I think that… I don’t want someone to have a parent Duquesne and just say that’s the answer. There it is autism diagnosis. And then the tool is not really set up that way. Right?
So the parent column, then you can put teacher ratings, you could have your psychiatrist, your occupational therapist any number of the providers that you feel like is relevant that can give a nice perspective on the child. You can analyze that profile as the clinician, but ultimately you’re [00:30:00] making the call.
So when we meet with the child and spend time with them, we’re deciding, and certainly having the fact that a parent and teacher have given us input, the parents telling us how’s the sleep? What are the patterns there, how’s this child eats, and the teacher’s telling us, well, how organized is this child? How good are they at reading and writing? So we have those pieces that we can use as a way into our decisions, but we are as the clinician making the diagnostic decision. And you can only generate a diagnostic report out of the clinician response column. So you can use the other response columns and generate concerns, reports, and recommendations, but the diagnosis does require a clinician to be a piece of it.
Dr. Sharp: Got you. Yeah. I wonder if it would be helpful for folks to talk through the role of this tool in an evaluation, right?` So should we think of this as sort of on the same level as a bask or an SRS or something like [00:31:00] that? Or is this like a pre-interview thing? How do y’all incorporate this into a comprehensive eval?
Let’s take a quick break to hear from our featured partner. With children currently exposed to conditions including a global pandemic, social injustice, natural disasters, and isolation, you need a trusted tool that can screen for symptoms of trauma quickly.
The TSCC screening form allows you to quickly screen children ages 8 to 17 years for symptoms of trauma and determines a follow-up evaluation and treatment is warranted. The TSCYC screening form does the same for children ages 3 to 12 years. Both forms are available in Spanish and support the trauma-informed care approach to treatment. These screening forms are now available through PARiConnect; PAR’s online assessment platform which provides you with results even faster. Learn more at parinc.com/tscc_sf or [00:32:00] parinc.com/tscyc_sf.
All right, let’s get back to the podcast.
Dr. Anna: Do you want to take that Marcy?
Dr. Marcy: Sure. Yeah. So we use the tool in two main phases. So as you mentioned, is this a diagnostic tool and a screener? I would say both. In terms of the screener aspect, yes, pre-interview pre-initial. We send out CADE to families and Raiders. The Raiders can be anyone in the child’s life. And then of course the parents.
So those Raiders and their input, we analyze that before we even sit down with the family for the initial. That guides the initial, right? So now I have areas to dig into and extra questions to ask. Also, gives me a little bit of an idea for where is the family with their [00:33:00] understanding of what could be happening with their child, and is that consistent or inconsistent with what the school is seeing or other therapists are seeing?
So we use it as a screener there. Then we use it as we’re doing our assessment and we’re gathering data. We’ll answer questions. So this is a live assessment. This is not another assessment where you answer all the questions, generate your report. That’s it. We use this as a live assessment ongoing during the evaluation. And then at the end, once we have all the questions answered, it generates responses, right. And it tells us the autism level, the autism profitability score, and then some profiles, and then those are used in feedbacks and in the report.
Dr. Sharp: Got you. Let me ask a very detailed sort of granular question which is if you’re sending this out prior to the intake, how do you handle like as the person on a client at that point, are you getting releases [00:34:00] for teachers and pediatricians and that all that before they even come into your office?
Dr. Anna: Yeah. So they’ve signed. So we do a discovery session with families prior. So we spend about 15 to 20 minutes on zoom to see if it’s a fit telling them about what we do. And then after that, they’re going to sign the consent and start our paperwork. And that’s when we can give them CADE. And in that consent, they’re asked to provide consent and send this to various… they can decide who they want to. And so we have all that, then we can send them CADE. We have to follow all up with reminders, but we make sure we have it in hand so that we can have a look before that initial console. So that’s what we were able to do that.
Dr. Sharp: Got you. So let me make sure am understanding this right? That you will send this out pre-interview to different stakeholders in the child’s life. You get that info back, that guides the interview, and then you, as the clinician are filling [00:35:00] in sort of the clinician side throughout the assessment. And then are you having those folks do it again, or is it a one-shot deal before the intake?
Dr. Anna: Yeah. So it’s that interesting because it is a one-shot deal. But one of the things that we’re doing, we found has been really helpful, is doing ongoing coaching and support. So the family will do it again. Marcy does a lot of great schools advocacy stuff. So say she’s working with a family. They’re going to meet with the IEP team. It’s a few months down the line. Well, then the teachers will do it again. The parent can do it again so we can chart progress over time and see how are we doing now? And sometimes you’ll find obviously through the course of the evaluation, the family has learned a lot about their child seeing these different profiles.
And so their ratings actually in that very next one might actually go down a little bit because they’re really aware of more of the symptom profile that they’re seeing, but we think that’s great because we start to see more reader agreement between the clinicians working with the child and the family.
And [00:36:00] that’s actually part of a research study that we’re working on now that we can talk about. But yeah, so the idea is we can do it again. But in the evaluation process, we just do that one time through the assessment. We try to make our assessments pretty expedited, get people through quickly. And so, yeah, it’s a pretty tight process and they can come out really quickly with their report and recommendations.
Dr. Sharp: Sure. Since you brought it up, I’m curious what your evaluation process is looking like these days especially with the remote layer that we’re all working with here over the last several months. Yeah, could you talk through what your evals look like now?
Dr. Anna: Sure. Do you want me to answer that? As I mentioned, we do a discovery session just a quick zoom meet and greet, get questions answered, see if it’s a good fit and if a family wants to work with us. And then we’re using IntakeQ as the portal, and that’s been really great for us, I think.
And so we then send the paperwork, we [00:37:00] send CADE and we get that back, get a chance to review that. We do a consultation with the parents or we do some work with adults too, so with the adult, whoever is doing the evaluation. And then we’ll set up sessions. And so it’s different depending on the age of the client.
So, you’re familiar with the ASP’s and the research they’re doing at Vanderbilt. We’ve actually been using a process similar to that for some time for our autism evals for younger kids. And before COVID, when kids were out of state it’s what we would do. So we’re collecting after the initial and we’ve got a chance to see CADE and understand the parents’ concerns, we’re starting to collect some video clips, right. And to look at different aspects and then we’re formulating. So if we’re going to be doing something that looks like a MODE-1 and MODE-2, a toddler module, what are the kinds of things we’re going to prepare the family to have ready for that engagement.
And then it’s really parent-led, right? So we’ve had fantastic families doing a beautiful job of leading these sort of modified eight [00:38:00] offices. And we know we’ve listened to Cathy Lord and we know she doesn’t love maths and she doesn’t love virtual, but we’re not trying to generate an autism algorithm through the eighth office. We’re trying to in some cases pull some activities that we can really use to get at the shared enjoyment and the social presence and the creative play and communication and things like that. So we’re able to do that and so with our younger kids, we may have just a couple of sessions where we’re sort of live on video there doing various things. We’re doing some aspects of the WISC-C or doing a developmental profile with the family. Certainly, we have the parents do evals to give us some feedback. So that’s what it looks like for the young kids. With our school-aged kids, we’re doing more in these sort of hour-long chunks.
And so that’s going to be varied based on the case. But it might be in three to five-hour-long chunks with the child via zoom where we’re really digging in and gathering that information that we need. As well as rating scales that we might additionally [00:39:00] want to add again with the Vineland or something like that, the Basques. And then with our teenagers and adults, we can sometimes do a lengthier chunk of time.
So we might spend an hour and a half zoom or even two hours really, depending on the client and what’s their attention span and their schedule. But we found that the zoom, well, I don’t think been challenging to identify autism or to do this evaluation this way. And two for the families, the parents and teens and adults, they like zoom.
They like being on and being able to connect just from their office or they’re doing school this way, so they can just click over and okay, we’ve scheduled that. They’re going to miss this one class today and come on and meet with me. So I think that there has been some ease to it taking some of those travel pieces and having the concerns of COVID in the offices and that kind of thing.
So a lot of our families are being very careful. And their kids are doing virtual school and they’re not doing a lot of community-based things. And so the [00:40:00] fact that they can dive in and do an evaluation virtually has been something they’ve liked a lot. So that’s been good.
Dr. Sharp: Yeah. It’s been amazing how we’ve adapted to this whole scenario. And I know for y’all, it’s maybe less of an adaptation. You’ve been doing remote autism assessments prior to COVID and it was less of a transition for you, but yeah, it’s been really cool to see these different methods pop up and get utilized.
Dr. Anna: Yeah, and then back to the CADE piece of it. So then for feedback, right, it’s really nice to be on a virtual call where you can screen share and we’re showing the different profile graphs and talking through that and so that’s worked really well so that we bring it back to the CADE in the end, and are able to walk them through that. And then from there, often we move into a journey package where we’re, again, revisiting the recommendations, the next steps, and revisiting CADE periodically with the family as we guide them through the journey.
Dr. Sharp: Yeah. [00:41:00] One little question since you brought it up a bit ago, what ages is it appropriate for?
Dr. Anna: Yeah, absolutely. So CADE generates an autism score for 3yrs and up. So we do have 41 items that are appropriate for two-year-olds. And so you can get profiles for it for a two-year-old. It’s a briefer piece of the measure there, and we’re definitely working on building out and collecting more data on those profiles so that we can dig deeper into that. And so that’s one of the pieces that we’re working with some early childhood programs. And so adding a place for those to tell us, is this an autism profile or not will help us to build that algorithm stronger overtime for the two-year-olds?
Dr. Marcy: Yeah, I was going to add to that with the age groups that we serve. One of the things that we really wanted to do was actually not focus as much on the little guys which it’s great that a lot [00:42:00] of organizations have focused on that. And when we went to the NCR conference, there were a lot of new measures coming out for toddlers. I think that’s fantastic.
But most of the kids that I’ve worked with over the years that are suffering the most due to either an autism diagnosis or lack thereof had come in and in kindergarten, first grade, second grade, and then we’ve got these really complex middle schoolers, middle school girls. And so we really wanted to be able to address those profiles most of all.
And so the measure really targeted that school age, teenage population, most of all. And we were proud of the span that we can do. And I would say this is an ideal measure when you’re bumping into these really complex 13-year-olds with all these different things going on. One of our hardest kids ever in terms of getting to add a diagnosis was a teenager [00:43:00] that was referred for spelling. And ended up having one of the most comprehensive, really elaborate profiles. And we felt like, CADE did an excellent job at differentiating autism, ADHD, looking at what’s going on with this? Is it a dyslexia profile? What’s happening with this, the spelling and reading, and just a really very interesting profile. And CADE did an excellent job of differentiating that. And we feel like there’s a lot of power there for these kids that they come into your office and they’re in third or fourth grade. And they’re like, yeah, I don’t know why I’m like this. And we can really help them too.
Dr. Anna: Yeah. Marcy said to me, we’ve got a spelling referral. I don’t know about that. And she said actually, but why don’t you take a dig into the CADE profile and have a look? And then you tell me if you want me to look. I was like, “Oh yeah, this is interesting.”
Dr. Sharp: Oh, that’s wild. I’m curious then. Now that opens this whole can [00:44:00] of worms of like, so it the measure, is it able to differentiate between different disorders? And there’s the gender question as well. Have y’all done, I’m not sure what the word is, I’m not a researcher clearly, but the validation or standardization or whatever you may call it to have like gender norm, or anything like that.
Dr. Marcy: Yeah. So we have a lot of good things happening with that. So with regard to gender, we in our validation study had a good percentage of the population that were girls. And we did some nice measures to show that it’s doing a nice job differentiating those girls from girls that don’t have it, for example.
And then in terms of your other question about the other disabilities and differentiating between those, CADE has done a fantastic job on these really complex kiddos. [00:45:00] It does not point to a diagnosis of ADHD, for example, in the way we have it set up right now. What it does though, is those profiles jump out at you.
So ADHD jumps out in these profiles. So the executive functions will be impaired. We have a whole focusing domain that will be impacted and then you’ll have working memory jump up. Whereas with these autism profiles, you’re getting a bunch of deficits in the social domain and communication domain. And then we also have learned and we have a cognitive domain, right. So those are really showing you signs that you need to go dig over here versus just these autism profiles.
Dr. Sharp: Right. Yeah. You said that there’s some stuff in the works with all of that. So I take you’re trying to move forward and gather more data around those areas.
Dr. Anna: Yeah. It’s been super promising. We’ve had so many girl profiles in there and some really tricky ones and we’ve been really pleased to [00:46:00] see CADE differentiating that. It is a neat area it would be fun for somebody to take CADE and run with it and look at the gender pieces.
That would be really cool. But yeah, we did analyze our validation sample and look at how that was performing just with females versus females and males. And we found that it was consistently like predicting autism at the same rate and everything. So that was good. But yeah, there’s definitely more than it would be neat to dig into there.
Dr. Sharp: Yeah. So tell me, how does that work? So if someone did want to conduct the research with it, how does that work? Can they do that? And are you open to that?
Dr. Anna: Yes we are very open to that. So a few different things. So, CADE, we can just grant access for clinicians if they have some work going on. And that’s some of what we’re doing now and some of the projects they can have access to CADE and use that to collect [00:47:00] data in their own setting hospital or clinic setting. So that’s possible. We also have either some neat things that we’d like to look at in the system just breaking down the profiles and the items.
So like we can look in the profile and see what’s a significant autism score. And then these children with sleep problems. How did they perform differently and behavioral context versus the kids without sleep problems? Right? So you could take and look at different relationships between different items.
The one that Marcy’s brought up a lot of, sort of the gender piece, right? Because we run into so many teens with gender identity when we do ask about that in the measure. So that’s something, we can look at and how does that correlate with autism or not autism or how does it relate to other pieces. So I think there are a lot of neat ideas that could be found there and a lot of new data to look at.
Dr. Sharp: Sure. It seems like you could make a whole job out of just doing the research with this sort of thing. And y’all are [00:48:00] also doing clinical work and running the business, right?
Dr. Marcy: Yeah. They’re our triple projects all the time; that ongoing research, the clinical practice, and the business.
Dr. Sharp: Yes, Oh my Gosh. Where do you see it… actually, before I transitioned to that, What other applications or ways are you using it that we may not have touched on so far?
Dr. Marcy: So there’s a lot of neat things that we can do because we have CADE and we have these CADE profiles. So what we’re doing in our business overall is we partner with families to provide diagnostic assessments, clarity, and a customized roadmap for supporting their unique child. So unpacking that a little bit.
We start with this diagnostic assessment piece, right? So we use CADE to determine [00:49:00] what is going on with my child. Number one, and then number two, and more importantly, what am I going to do about it? So we want to make sure these families really understand this profile, not just from a one-hour feedback session but from ongoing support and interactions and that’s the clarity piece.
And then we go to this customized roadmap. So families that decide to stay on with us. So we’re able to provide this ongoing support for them in terms of understanding what services are going to be helpful and what interventions and those interventions are based on those profiles. Right? So it all starts with here are some of the most impactful symptoms in this child’s profile and this is what we’re going to do about that one.
Once we’ve got that one a little way down the road, we can come back and revisit some of the less impactful issues that come up. One of the things that are really neat that we’ve really jumped in with is this idea of really strengths-based assessment [00:50:00] and strengths-based intervention.
So the research shows over and over again, I really like Sam Goldstein’s work around this, that if we build on what he calls these islands of competence we have much more success in terms of our long-term outcomes than we then just remediating deficits all day long. And so we really take these profiles and say to the family, listen, you’ve got an awesome kid. And here are all the things that are working and we’re going to build on those most of all. And then we’ve got these deficits over here and this one is causing the most damage to this child’s day and his functioning. We’re going to hit this first. Once we’re knocked that one down, we’re going to keep on going.
And like an example could be if a kid’s aggressive. Let’s say he’s an aggressive kindergarten or first grader. Well, we’ve got to deal with that first. I’m not going to be able to work on your spelling when you’re aggressive, right? We’re going to get that out of there first.
And so really helping [00:51:00] these families understand what is it that I need. And for that clinicians listening, one of the things that we really want to do is go really beyond an after-the-assessment. So a lot of times families will come to us that have a good assessment already completed, and we take that and can use that to guide these roadmaps and to help these families ongoing.
And one of the ways that we do that for example, is in the schools. Both Anna and I are trained as school psychologists as well. So we’ll take some data that we get from an excellent assessment from one of our colleagues in the community and say, okay, based on this, here’s what the school could be doing and we’ll help you get that started whether it’s on a five Oh four and IEP or just interventions, right? So those are some ways that really extending this beyond just the assessment and into how do we help them help their kids.
Dr. Sharp: Right.
Dr. Anna: Another piece of school partnerships. So we are piloting the tool [00:52:00] with school teams and a number of districts now. And a neat thing about that has been a school psychologist telling us they’re able to have the key teachers use CADE, the parents have CADE, their team and they’re looking at it as, from an eligibility standpoint, but they’re really able to guide a family and even particularly young children to say, Oh, well, these are some of the symptoms that could mean autism.
And then they’ve come back and told us well, now some of these families are out there seeking that evaluation. So even with the children, with the IFPs and just moving into IP. So again, just catching those families and the school team can use this tool to say, well, yeah, we are noticing these challenges in this child. He doesn’t have a lot of social presence and they’re not using language yet. And so instead of those families waiting, they are realizing we can take this profile from CADE the school has given me. This is not a diagnosis, but this is an educational piece. And then that’s going to help me find the services I need. So [00:53:00] that’s been a neat too. We’ve got some nice feedback for that use case.
Dr. Sharp: Yeah. It’s got me thinking. So, you’ll see this as something that would be used outside of a formal evaluation for even in a therapeutic context, if someone wanted to get more information or more insight into a kid they’re working with or an adult they’re working with, something like that.
Dr. Anna: We find it really useful in that way. And so yeah, Marcy is building these coaching packages and definitely is using CADE there. And I think some of the clinicians that we’re working with have worked in a psych slash ABA practice. And so looking at how CADE can guide the creation of some of those initial ABA goals.
And one of the things we’re looking at with Children’s Hospital and enrich with J of ACTA as RPI is looking at how Raider agreement overtime shifts when you’re using CADE to take these measurements and you’ve got your [00:54:00] speech pathologist, your OT, your BCBI, your ABA therapist and the families and using CADE to guide those interventions and, and take some benchmarks across time to look at that. So that’s something neat that we have going on that we’re excited to get data from.
Dr. Sharp: Yeah, that’s funny. The business part of my mind was like, Ooh, maybe you could license this to ABA clinics or something, these other places that do this kind of ongoing… that’s great to hear that you’re moving in a direction like them.
Yeah. So a funny question or maybe difficult question, but I think hopefully helpful for folks who are listening is, at this point in the process, what would you say are the gaps in the measure or the areas that you’re trying to improve, or shortcomings feel like a harsh word, but I’m sure people are like, okay, what’s the deal with this?
Dr. Anna: We ask ourselves sometimes. We want to make the best thing possible. And like Marcy said before, we work in these iterations [00:55:00] and it’s not going to be perfect. We’re going to make improvements. But one that we have been talking about recently is these AI implications, right? One example would be so in working with some psychologists in incentive-based programs where perhaps kids have a lot of these symptoms, right?
So CADE is coming out a 60 of 80 of these symptoms are present for this child and the clinicians are coming to me and saying, “What do I target first?”And so we’re pulling up the socializing profile and I’m looking at it and saying, okay, well, we’re not going to be working on perspective taking until we work on social presence and we work on some cooperative play and we’re addressing some of the repetitive languages.
So we are breaking it down and saying, here’s the way you should address this. Start here, move to this next. And we would love to build that machine learning and we know we can do it. Our tech team has said we can do it, but just to build that into the tool so that it can be even more and more specialized in creating here is this treatment plan, here’s this next step process. So even a [00:56:00] family could jump into the tool and know what should we be addressing first or a school team could, or a clinician could. So we would love to build more AI implications into the tool where I can do more without automatic machine learning.
Dr. Sharp: That’s super exciting.
Dr. Marcy: Yeah. I wanted to add that we have some neat connections in the AI community in terms of the knowledge we need. We have an embarrassment of riches in terms of people that have a lot of experience with that. One of the potentially really cool things is we developed that framework that everything is built on. And there’s a lot of symptoms and hundreds of articles in there. And what we wanted to do was use natural language processing for people to be able to essentially ask their phone these questions and get back some really nice immediate feedback on really some of the things that we as clinicians [00:57:00] think of as basic that we take for granted that people know that they can ask pretty quickly and answer pretty quickly.
So some really neat potential implications there with interactive journals that families could do where they can put in my child did this and get something back pretty immediately in terms of support and response. The really cool piece of this is that being cloud-based, we’re hosted on Amazon web services and we have access to a lot of great engineers. We can take this and make it more and more accessible for families and more and more easy to use for the clinicians.
Dr. Anna: Potentially for other medical providers too if you think about different settings like emergency room settings. We’re very interested in that broader medical community. And obviously, we know in all of our work with pediatricians that they don’t always get the low down on autism. They don’t always have all of this knowledge [00:58:00] base. That’s not a huge part of their training. So there are other times other ways where autism intersects and other things too, but where we could provide information and quickly put things at a clinician’s fingertips too if they were in a different part of the medical field.
Dr. Sharp: Yeah. I love this. You all are speaking my language. This intersection of assessment and technology and AI. I’m just fascinated with all. I don’t know anything about it really, except enough to just be curious. But yeah, it’s awesome to know that y’all are pushing into that frontier and trying to come up with some cool solutions to help families.
So, are there any other future directions or just exciting things on your radar that would be fun for people to know about with the tool or otherwise?
Dr. Anna: One thing just for clinicians, we do have a subscription model. Clinicians can subscribe to use the tool. We are rolling out a paper use model. We’ve talked to clinicians who do a variety of things in [00:59:00] their practice. And they’re saying, oh, I don’t have a subscription. I’m not going to use this five times a month, but I’d love to be able to pull it out and use it here and there as assessments come up. So we are working on rolling out a model that clinicians can go in and use CADE one-off and that kind of thing. So we’re doing that. Let’s see, what else, Marcy?
Dr. Marcy: I would say we’ve also had some just neat recent happenings. One of them is we won at start-up of the year in the healthcare category for… they have a People’s Choice Award. And so we just earned that. We’re really excited and able to get our information out there to more and more potential partners. And through that event and others that we’ve been involved in, we are meeting with a lot of potential corporate partners who would be interested in piloting the tool. A lot of them have innovation arms where they want to take this [01:00:00] information and say, how do we apply it to really help our patients or our clinicians and researchers streamline their work?
And so a lot of just exciting things coming off the press as we speak, in terms of potential large partnerships with large healthcare organizations. I mean these integrated delivery network that exists out there are really growing in the healthcare world. And we think this is a great place for this because it’s a collaborative tool. So the different clinicians with different expertise and the parents at home and the people in the community can all collaborate through the use of this tool. So those are some neat things that are just coming out of our camp really right as we speak.
Dr. Sharp: Oh, that’s so exciting. And congratulations, by the way, that’s a big deal. You got to be thrilled.
Dr. Marcy: Yeah. We are really excited about all this stuff that’s happening. We were just talking about how it’s been a rough year for everyone in a lot of ways. And we are not excepted [01:01:00] from that. However, our business has been fantastic. One of the reasons is because we were in remote assessment already. We already knew that virtual was going to be a huge important access thing for families. We had no idea COVID was coming, but we knew that virtual was going to be important especially for like rural areas. And for this access issue, we thought if we can do this virtually, we could do it so much faster. We could get these families through versus going, okay, so when’s the next time in our schedule that we have six hours where we can have this family come in. So we can really get families in. We knew that would happen. But then COVID comes along and it’s like wait, this is one of the only ways that we can help families.
And so clinicians started showing up and saying, “Hey, you told us about that thing and we said like, we’re just going to stick with what we’re doing, now we need to know what you’re doing.”
[01:02:00] Dr. Anna: That’s amazing to connect in the field. And like we said, in the beginning, just making those connections so powerful. And I think COVID has allowed us to have conversations with clinicians and teams all over the country. And I don’t know, it’s just been really neat to have those connections and see how they are doing things and for us to be able to offer supports. That’s been super just super cool during such a hard time. Just those connections are so valued. It’s just so nice.
Dr. Marcy: And speaking of which I wanted to throw out there to the listening audience that one of the things we started doing during COVID and we’re continuing to offer through COVID is that clinicians who sign on to use CADE have access to a clinical consult with us to walk through their first one or two cases just getting comfortable with the tool and during COVID, they can ping us as often as they want with questions. And what’s really nice about that is this is really neat peer to peer collaboration [01:03:00] which we all wanted to do when we were in grad school, this idea of what we were going to have this network of people were going to bounce things off of and consults on. And it’s hard, right?
It’s hard to get those connections outside of any practice you’re working in. And we’ve really launched that. And the clinicians it’s been just a pleasure to hear the way people are assessing and what are they using and what do they do if the parents and the clinician really aren’t seeing this child in the same way, how do you deal with that? And does that bear even on your diagnosis? And so really interesting. And a lot of us have all different kinds of models on how we’re assessing. And so it’s been really great. And we’re continuing to offer that to clinicians who want to use CADE.
Dr. Sharp: That’s fantastic. Yeah. So for folks who are interested and I would imagine there will be some folks who are interested, what’s the best way to find the tool, to find y’all. And I can link to all that in the show notes.
Dr. Anna: Yeah, [01:04:00] fantastic. People can email me directly at anna@clearchildpsychology.com. They can visit our website clearchildpsychology.com and connect to resources there. So yeah, and we can point them in the direction that way from the website. They can just click to try CADE. They can do it that way, or they can reach out directly if they’d like to see a demo, get a little bit of information, and check it out before they jump in. We can definitely do that too. So yeah, we look forward to hearing from folks. And also referrals if you’re doing assessment and you’ve got families who could use some ongoing coaching and it’s not part of what you’re doing., we love doing that with families and really guiding them through those steps as well. So we’re around for that too.
Dr. Marcy: Yeah. So just adding to that on the website, which is clearchildpsychology.com there’s a professionals page. And if you scroll down on the professionals’ page, there is a signup page. And [01:05:00] on there you can sign up for subscriptions or you can get a demo. So they can click to schedule a demo and a current link pops up in the zoom link and we can do that demo with clinicians. So that would be two ways they could access us. If the families want to come and reach out for coaching, they would go to our website again, clearchildpsychology.com and it would link them to a signup page as well for our discovery session.
Dr. Sharp: Awesome. Yeah, I’ll make sure to put those links in the show notes so people can find those when they’re ready. So thank you all. This was great. An hour flew by.
Dr. Anna: Yeah. Thank you so much. It’s been a pleasure to talk to you. We really appreciate it.
Dr. Sharp: Yeah. Well, it’s cool too just to dive in and hear a little bit more. I met y’all I guess two years ago, maybe more now at that workshop and I know you’ve been working so hard on this. Before that and since then, and it was [01:06:00] a pleasure to be able to hear more about it and see what you’re up to. So, thanks.
Dr. Marcy: We really appreciate this opportunity to talk to you more and we appreciate also your scholarly collaboration over the last couple of years has been great that you’ve stuck with us and supported us along the way.
Dr. Sharp: Well, as Coloradoans has had to stick together. We’ve got to support each other. And it’s a cool thing. It’s really cool and really exciting. I want to make sure that’s clear. It’s no pun intended. It is a pretty awesome thing you all are doing. So thanks for letting me be a part of it.
Okay, y’all thank you for listening to this episode with Anna and Marcy about the CADE assessment tool for autism. A lot of resources in the show notes. If you want to learn more certainly about the research or their approach or their framework, I would invite you to go check that out. You can also get in touch with them. Like I said, in the beginning, if you’re [01:07:00] interested in partnering in any sort of research venture, I know that they are super interested in that.
Like I mentioned as well if you are in need of a support and accountability group to help you reach those goals in your practice that may have been challenging to reach so far, I would invite you to check out the Advanced Practice Mastermind Group. So the current group that’s in progress right now is just knocking it out of the park, holding each other accountable, setting goals, supporting one another. It’s amazing to watch. So we have a new section starting on January 7th.
As of this recording, like I said, there’s one spot left. So you can find out more at testingpsychologists.com/advanced and schedule a pre-group call to see if it’s a good fit.
All right. So we are winding down 2020. I think the next episode in the queue is going to be our [01:08:00] best of, so stay tuned for that. I think it’s going to air on new year’s Eve, the last day of the year. Top five podcasts of the year and we’ll go into detail about those and what made them so great. And look forward to 2021. So thanks for tuning in as always and take care.
The information contained in this podcast and on The Testing Psychologist website is intended for informational and educational purposes only. Nothing in this podcast or on the website is intended to be a substitute for professional, psychological, psychiatric, or medical advice, diagnosis, or treatment. [01:09:00] Please note that no doctor-patient relationship is formed here. And similarly, no supervisory or consultative relationship is formed between the host or guests of this podcast and listeners of this podcast. If you need the qualified advice of any mental health practitioner or medical provider, please seek one in your area. Similarly, if you need supervision on clinical matters, please find a supervisor with expertise that fits your needs.
Click here to listen instead!
