Many of y’all know that I have been using TherapyNotes as our practice EHR for over 10 years now. I’ve looked at others and I keep coming back to TherapyNotes because they do it all. If you’re interested in an EHR for your practice, you can get two free months of TherapyNotes by going to thetestingpsychologist.com/therapynotes and enter the code “testing”.
This episode is brought to you by PAR.
The new PAR training platform is now available and is the new home for PARtalks webinars, as well as on-demand learning and product training. Learn more at parinc.com\resources\par-training.
All right, everybody. I am back here with a podcast [00:01:00] episode about a question or topic that gets a lot of airtime but not a lot of conclusions or follow through in the testing world, and that is, how do you consider and possibly set up an internship and/or a postdoc program in your private practice?
I think a lot of us are struggling with hiring licensed folks and so many people are trying to turn toward trainees or pre-licensed folks to serve as almost a pipeline for building your staff and for many other reasons but internship and postdoc has gotten a lot of attention over the past few years, but not a lot of us know how to go about that process.
My guests today are from Clover Educational Consulting Group. It’s a group that helps entities set up and evaluate internship and postdoc programs. My guests are Dr. Erin Foley and Dr. Juliana [00:02:00] Rosa.
We get into a lot of topics related to internships: we talk about advertising for internships and postdocs and what that even means in this world, we talk about the bare minimum requirements to apply for APPIC membership, we talk about the difference between membership and accreditation, we talk about the differences in internship and postdoc requirements as far as training and supervision, we talk about program evaluation and the key variables that you should be measuring and then how to use that data if you do measure it and many other things.
This is a pretty rich discussion. I enjoyed talking with them, especially because Clover was the group that helped us set up our consortium way back when. I feel fortunate to be able to circle back and share some of their knowledge with all of you.
If you are a [00:03:00] practice owner and you’re looking to build your practice in any kind of way, I would love to support you with that. At this point, when this is released, we’re maybe a week or two out from the next cohorts of The Testing Psychologist mastermind groups getting started. Those are starting the second week of July. It may have a spot in one or two of them.
So if you’re looking for group support, group coaching, accountability, homework, and folks to help you through the process of growing your practice, could be worth checking out. You can go to thetestingpsychologist.com/consulting and schedule a pre-group call to see if it’s a good fit.
All right. Let’s jump to my fantastic conversation with Erin Foley and Juliana Rosa from Clover Educational Consulting Group.
[00:04:00] Erin, Juliana, welcome to the podcast.Dr. Erin: All right. Thank you.
Juliana: Thank you for having us.
Dr. Sharp: Absolutely. Thanks for being here. It’s a surreal moment, I’ve talked on the podcast before about how we used Clover to help set up our internship program however long ago, 7 or 8 years ago, and then I bumped into Juliana, at this alumni event a few months ago, and here we are. I’m glad to be chatting with y’all and sharing a lot of what y’all do with the rest of the community here.
I will start with what I always start with, which is, my focus on this, why do this? Out of all the things you can do with your time and life in this world, why this? Erin, I’ll let you go first and you can say a little bit about what you do as well just to help people orient to who you are.
[00:05:00] Dr. Erin: Absolutely. I oversee Clover’s work in helping training programs either develop or become accredited or problem solve, just having a macro or big picture impact on the psychology training community. I do it because you can have that big impact.So this chance to work with people across the country and to think about how to improve our behavioral health workforce at a different level than I had when I was providing direct services as a psychologist years ago. So it’s a different focus and it’s exciting to be able to work with folks who are interested in developing future psychologists.
Dr. Sharp: I feel like it gets to that ripple effect, you’re affecting a lot of trainees’ lives and psychologists’ lives. That’s really cool. Juliana, how about [00:06:00] you?
Dr. Juliana: I am the Director of Research and Evaluation at Clover. What we do or what I do is oversee program evaluation and research services to support a lot of behavioral health programs and workforce development, as Erin mentioned, as well as community programs, and capacity building. So we do a lot of different evaluation and research projects in the behavioral health realm.
I’ve been doing it for about three years and enjoy the work because I love data. I love nerding out about data, especially when we’re using data to inform improvement and decision-making. That’s where my passion is.
Dr. Sharp: I love that. We’re going to tackle both sides of this. We’re going to tackle the setting up of the programs but then the other side of that is evaluating them and making sure that they’re doing what they’re supposed to be doing and serving everyone as best they can. So [00:07:00] I’m really grateful to have both of y’all here to tackle both of those sides.
We can launch into the front end of that and get right to it. Training programs, I think looking through this lens of how training programs are maybe available for folks in private practice would be interesting but we can talk about training programs in general, maybe just a big picture overview of where the training is coming from and how we might start to think about bringing them into our work wherever we are.
Dr. Erin: I’m happy to jump in on that. When I think of training programs, I’m either thinking of internships, that doctoral level internship, that culmination of their doctoral training or the postdoctoral fellowship and thinking of drawing trainees [00:08:00] from programs based upon what their training history was, what their area of focus was, and especially if you’re getting into private practice, thinking of identifying and finding those trainees who want to build a practice of their own someday or want to be able to specialize in a particular area, especially at the postdoctoral training level.
Programs can, if they do it right, find and access potential trainees across the country, and recruit nationally with different resources that are available to bring people into their practice or their setting. The trick is to advertise right and to make sure you’re finding those right matches. Somebody who has the same professional interests or things that align with what you’re doing in your practice or your setting.
Dr. Sharp: I [00:09:00] know we’re going to talk about internship and postdoc separately here as we go along and there’re different processes with each of those but I key into the word advertising right away. I don’t know that I think of it as advertising in how we recruit our trainees. So I’d love to hear your thoughts on what “advertising” looks like when we’re looking for interns and postdocs.
Dr. Erin: I think programs need to sell themselves just as much as potential trainees or prospective applicants have to sell themselves to programs. If we’re going to be successful in finding the right people, we have to be able to show what there is to offer.
When I was a training director in my former life before I started with Clover, one of my goals was to let prospective applicants know what we could offer that was unique and interesting and exciting to pull in [00:10:00] somebody who shared that excitement and passion. I worked in juvenile justice and I wanted to advertise very clearly, this is what we offer and this is how we can help you go down a path in this type of justice setting.
That was done through the written materials that we provided, our engagements at APA, and talking with others at APPIC or different settings. Even talking at the high school level, we would do advertising early to get people thinking about a career in psychology that focused on our justice setting.
I could see that type of advertisement happening in private practice or in hospital settings and other ways to create that pipeline effect of, oh, this is where I can go in my career, and here’s a place that offers that type of training or that type of work.
Dr. Sharp: That’s interesting, [00:11:00] I don’t know that I would think to go to high schools to start to recruit our trainees.
Dr. Erin: The pipeline starts early. I get to do training for high school interns and talk about all the different paths they can take in psychology. And then years later, those folks reach out and they’re like, you talked about consulting work and I’m still interested. Now I’m in a doctorate program. So laying the groundwork can start early if folks have the time and capacity to do it.
Dr. Sharp: Sure. I talked with neuropsychologist Lynette Abrams-Silva, and she does a lot of work, you’re both nodding, do you know of her? She’s done a lot of educating around diversity in the field and such, but we talked a little bit about that, how do we set up this pipeline particularly for more culturally diverse or marginalized [00:12:00] groups to get into neuropsychology specifically? She also talked about like we have to start early somehow and model what this field could look like for younger folks. So there’s a lot of energy around that, I think.
I think we can dive into this whole training experience. I might start with a general question, though, of everything that we’re talking about, we’re going to talk about internship and postdoc, how much of this is realistically happening outside of the APPIC umbrella, especially with internship? Is there even a non-APPIC market out there that folks could consider or should we just set our sights on somehow getting under the APPIC umbrella as we try to recruit interns specifically?
Dr. Erin: There could be, but I think it would be pretty limited to go outside of [00:13:00] that APPIC umbrella. I think that the most typical route is to obtain that APPIC membership because with that, you can participate in the National Match that many of us did when we were selecting our own internships years ago, or for me, it was years ago.
And with APPIC, there is a necessity for APA accreditation. Those go hand in hand now, which they didn’t always, but they do now. And that’s the most typical route that folks take. There could be other routes if someone had an exclusive agreement with the university to take their doctoral students as interns and it was a one-to-one match that happens within that setting, but most typically, it’s going to happen with APPIC membership and APA accreditation.
Dr. Sharp: That makes sense. I think you answered this question but I just want to make sure to clarify for folks who are listening. Is it possible to be an APPIC member and [00:14:00] participate in the National Match without being APA accredited?
Dr. Erin: Not long-term. Once you become a member, you have three Match cycles to obtain your APA accreditation. So in the long haul, you have to be an APA accredited internship to be to continue with the Match. Let me correct that, you could stay an APPIC member, I believe, but you could not participate in the Match and you want that Match opportunity to be able to find interns for your training program. So it does not make any sense to do one without the other.
Dr. Sharp: Yeah. Talking about advertising, it feels like 80 to 90% of the advertising is just being able to participate in the Match. I don’t know. I do get it. It’s otherwise. So let’s talk about that. What’s the barrier to entry to just gaining APPIC [00:15:00] membership? Could anyone listening go to a website and become an APPIC member tomorrow or how does that work?
Dr. Erin: Anyone could apply tomorrow. When people apply, they have to have some things in place before they go through the process to make it worth your time. You want to make sure you have a training program, essentially sketched out and propped up so you know what your training goals are, and what competencies you’re training your trainees in. You have certain policies in place.
APPIC wants you to have a due process and grievance policy in case there are troubles that come up within the training program. Be able to demonstrate that you have didactics in place. So you have a curriculum going and they’re going to need to see that before they consider you for membership.
So you could apply tomorrow but you want to make sure when you do that application that you have all of [00:16:00] those pieces that meet their membership criteria. They have 16 criteria that have to be met. And looking through those programs, we’ll want to make sure that they meet all of that, have the right number of psychologists, offer a stipend, provide the right amount of supervision, that kind of thing.
Dr. Sharp: I got you. It seems almost like there’s a little bit of a chicken or egg phenomenon here. I’m getting the sense that if you want to apply, you should have some demonstrated track record of these components but how do you get that without having interns? How have you seen programs navigate that? Or am I making an assumption, is it okay to just have a theoretical training plan?
Dr. Erin: It is now okay to apply for APPIC membership without having trainees on site or in training, you can create all the pieces of the puzzle and just not have those trainees there. And that’s a provisional membership.
And so you can [00:17:00] apply for provisional membership. After that, you will have to get trainees on site and in training in order to transition to full membership but you can start participating in the Match with that provisional membership to help you get those trainees there. So just looking at the intern level you.
It is a little bit of a chicken and egg thing. It feels funny to write a training manual saying our interns do X, Y, and Z when you’ve never seen an intern, but that’s part of that early stages of the process, that you are envisioning what it is that we are offering and how we’re going to offer it and then you’re providing that full package to APPIC to consider.
Dr. Sharp: Okay. That sounds great. I wanted to circle back just to touch on some of the components that you mentioned and emphasize there are some things that you probably need to have in place. I think in our pre-call, you said folks have to [00:18:00] have at least two psychologists on staff. Is that right?
Dr. Erin: That’s right. At least two full-time equivalent psychologists, you have to have a stipend that is fair and equitable and matches what would be expected in your region or your jurisdiction, so you can’t be well below the average in your area. APPIC wants to see that it’s fair and equitable.
You need to be able to provide two hours of individual face-to-face supervision with your trainees, which can sometimes be a burden. You have to provide two hours of didactic training per week as well, that goes into demonstrating that training curriculum. This is not just on-the-job training, this is a training program, and we are doing this for the purpose of training future psychologists.
Dr. Sharp: Right. I know in our materials, I think we’re up for reaccreditation this year, we’re having to fill out a bunch of [00:19:00] information, which is super fun. One of the main things that they keep emphasizing is, hey, this is a training experience first and not a money-making experience for the practice and so making sure that the training is in place and the stipend is in place.
I will say this, it’s funny, I had a bit of a boomer moment or something, we were talking about average stipends for interns, and my understanding, you can correct me if I’m wrong, but our training director was saying that the average stipend is creeping toward $40,000 at this point. Is that right?
Dr. Erin: Yeah, that sounds about right. Well different from my $14,000 when I was an intern.
Dr. Sharp: Oh my gosh. Okay. So you have me beat, I thought I was struggling at $25,000 but I definitely had this moment. I was like, what? $40,000, who gets $40,000? But then I went back and [00:20:00] did the inflation calculator and it totally makes sense. It should be about $40,000, but it just goes to show things change and you got to adapt.
I think that’s good for people to know, too, that if you are thinking about taking on an intern, you should plan to pay them around $40,000 a year, even if that seems like more than what you made on the internship, which is true for me.
Dr. Erin: The other thing that sometimes comes up with private practice is you can’t, in your materials, either for APPIC or APA, talk about it being based on clients served. So it has to be a stipend, and that can be challenging for folks to plan out that regardless of how many clients my trainee sees or what the billable hours are, they will receive this stipend and that it will be equal across those who are in the training program.
That can be a little bit tricky to [00:21:00] navigate for folks. You can bill, of course, depending on where you are, for your interns or your postdocs services under the supervisor or you can do some different things, but their stipends have to be equal and out ahead of time.
Dr. Sharp: Yeah, I’m actually glad you touched on that. I noticed that in the materials, it’d been a long time since I’d filled out these materials. So I think I forgot, but it was super clear too, that we can’t say, required to bill a certain number of hours per week, it’s more like a target and you have to be explicit that they’ll get their stipend no matter what. Essentially, it can’t be hourly employees, it’s a salary and they get paid that even if they billed hours or not.
Super important. Anything else as far as the components that people need to have in place or make sure that they can provide before thinking about applying for an internship [00:22:00] program.
Dr. Erin: Those are the key ones, that having the right staff, the right stipend, and the capacity to do the supervision and the training activities. So if you have those pieces, you can start to build the additional requirements around that.
Dr. Sharp: Fantastic. This might be a good place to dip into accreditation a little bit. It is a process. I think that’s why we use y’all to set up the internship and help with our accreditation, make sure, and coach us along. Could you give a general overview of then what people would be looking at two or three years down the road when they go through the accreditation process?
Dr. Erin: Sure. The accreditation process begins with a self-study, so anybody who is applying for accreditation will complete a self-study that covers five standards [00:23:00] of accreditation and all of the pieces therein. They will be looking at things like their setting, their resources, their budgets, their training curriculum, their training aims, their staff, their internship data.
They’ll be looking at policies and procedures, all of the written materials that go with that, and compiling that into this big thing that a lot of the folks I work with equate to completing their dissertation. So it feels like a lot, there’s a lot that goes into it.
And with that, they’ll submit a self-study. It’s reviewed in a preliminary review way by the Commission on Accreditation and they will receive potentially approval for a site visit or request for additional information. So you self-study, site visit, then there’s a site visit report that you have to respond to, then all of that information goes to the commission on accreditation for them to make that decision.
[00:24:00] The length of time for that varies for programs, but it can take several years, honestly, to get from point A to point B because there’s a lot, there’s some back and forth, there’s responding to review, there’s waiting for the site visit, there’s making sure you have the trainees on site and in training at the time of a site visit. All of those pieces have to fall into place in order for you to get from application to accreditation.Dr. Sharp: Yeah, it’s a lot. So this is where I bring in the possibility of a consortium because that has helped me personally immensely because I didn’t have to do any of that. We’re part of a consortium and we have a training director, I suppose, who is over the whole thing and she completes all of those materials.
What’s the likely, I’m not sure what question I’m trying to ask, I suppose, but it’s more around what a consortium [00:25:00] looks like. How does that work, if there are private practices out there who might want to join an existing one or build their own? I’ll leave it open and see where we take it.
Let’s take a break to hear from a featured partner.
Y’all know that I love TherapyNotes, but I am not the only one. They have a 4.9 out of 5-star rating on trustpilot.com and Google, which makes them the number one rated electronic health record system available for mental health folks today. They make billing, scheduling, note-taking, and telehealth, all incredibly easy. They also offer custom forms that you can send through the portal.
For all the prescribers out there, TherapyNotes is proudly offering ePrescribe as well. And maybe the most important thing for me is that they have live telephone support seven days a week, so you can talk to a real person in a timely manner. If you are trying to switch from another EHR, the transition is incredibly [00:26:00] easy. They’ll import your demographic data free of charge so you can get going right away.
If you’re curious or you want to switch or you need a new EHR, try TherapyNotes for two months, absolutely free. You can go to thetestingpsychologist.com/therapynotes and enter the code “testing”. Again, totally free, no strings attached, check it out and see why everyone is switching to TherapyNotes.
I’m excited to tell you about PAR’s all-new PAR training platform, an elevated online learning environment with everything you need in one place. This is the new home for PARtalks webinars, many of which offer APA and NASP-approved CEs, and also houses on-demand learning tools and PAR product training resources. Best of all, it is free, totally free. To learn more, visit parinc.com\resources\par-training.
[00:27:00] All right, let’s get back to the podcast.Dr. Erin: A consortium is a coming together of different organizations that are agreeing to operate under the same overarching training umbrella. It’s under the training program name and there are certain policies that are specific to the training program. The organizations are coming together and signing a consortial agreement that we will all train under this umbrella in the same way.
We will pay our interns the same stipend that it’s equitable across our different training sites. We will engage in didactic training in particular ways. You’re sharing resources and you’re sharing responsibility. And if you have the training director at one of those sites or at a [00:28:00] university site, a lot of places will engage with universities and there’s consortium that include that university support and the training director might be housed there, but then you have these other settings, the benefits of being able to resource share across all of those spaces is tremendous.
To get that going, it might be a matter of seeking other champions in the community who are wanting to have the opportunity to provide training to future psychologists, but no, they cannot do it on their own and seeking out potential private practices or other agencies that are willing to come together to collaborate.
We’ve seen in the past two years, especially in some more rural areas or in areas where there’s just not enough resources potentially, where private practices come together and create a private practice consortium, which is cool. And then you have all of these interns who are doing [00:29:00] similar work but in different settings and they can come together to talk about what that experience is like. And the supervisors, those providing the training can share the resources and responsibility of offering that to those interns.
Dr. Sharp: Yeah, that’s great. It’s been a real benefit for us to be able to share the training burden, I suppose, not that it’s a burden but to spread out the training responsibilities and the didactics in the supervision and have partners in the whole process.
It’s been really nice and I think it’s an attractive feature for interns because they have a cohort. It’s not this coming to a private practice and being the only intern. They have a cohort of folks that can make the experience a little better.
Dr. Erin: They do. We didn’t mention this earlier but for any [00:30:00] internship, it’s planning development, the plan should be to have at least two interns in your program. There is a need to have that socialization and communication amongst the trainees. And having more, maybe not getting wild and having 20, 30, 40, but having more interns creates an environment where they can learn and grow together.
Dr. Sharp: Yeah, that’s fantastic. I’m going to ask a granular question that comes up sometimes. We talked about supervision and needing to have those psychologists on staff. There is the potential guideline of needing psychologists to be licensed for a certain amount of time. Is that accurate? There’s the 3-year mark deadline, people have to be licensed for three years to be able to supervise interns within APA accreditation. Is that true or [00:31:00] is that not true? That floats around out there.
Dr. Erin: I don’t know if that’s entirely accurate. I would have to dig back through the standards to find that.
I’m actually wondering if that is more of a requirement by jurisdiction rather than by APA. In certain settings or in certain states, you have to complete formal training in supervision before you can provide supervision and that is not true in all states. I don’t know that that’s correctly captured as an APA requirement.
Dr. Sharp: Okay, that’s good to know. Well, let’s pivot to postdoc a little bit. So postdoc feels a lot more like the Wild West unless you’re doing a formal fellowship like a neuropsych fellowship. Those seem to be more well-defined but otherwise, I hear a lot of stories about postdocs that vary greatly, let’s say, in terms of training quality and experience.
[00:32:00] So talk to me about postdoc. Is there an accreditation for an internship? Is there an APPIC membership? Is that still relevant? What do we need to know if we’re trying to set up a postdoc?Dr. Erin: There are different options that you have if you’re trying to set up a postdoc, and some of it depends on what you’re trying to accomplish. So if you are looking to provide a space for people to just receive the required supervision they need to become licensed, that might not be a postdoc. That could be your on-the-job training with a supervisor available to sign off on your hours and oversee your work.
But if you were looking to provide that next level of training that people are seeking and you want to do that in a more formal way, then that’s a postdoc and that might be where you dip your toes in the water and say, okay, I’m going to apply for APPIC membership so that I can have something a bit more formal to [00:33:00] offer to trainees.
There’s a role for APPIC membership. There’s a role for APA accreditation. It depends, though, on how much you want to offer and what you’re looking to set up. Are you looking to create something that’s highly competitive? Are you looking to create something that’s highly specialized? It depends on where you want to go with a postdoc. It is much more variable than an internship and much less clear, but there are different options for you.
We’re seeing a lot of folks right now who are coming to us who have a postdoc in place but they have trouble recruiting and so they’re electing to apply for APPIC membership so that they can demonstrate more clearly that they’re meeting some basic criteria of training and that they can offer something particular, a certain amount of supervision and training [00:34:00] activities and that they’ve had some vetting by … There’s this place that we become members of and you can see our materials here and that helps them a little bit more.
They might want to become APA-accredited for the same reason. Become a bit more competitive and be able to demonstrate at that level that they have an accreditable program with a certain level of quality that they’re seeking to meet.
Dr. Sharp: You mentioned that people are having trouble recruiting postdocs. Do you have any more insight into that? Because I hear that as well. Everybody wants a postdoc. It’s so hard to hire licensed folks these days for any number of reasons and so people are turning to postdocs and interns, but mainly postdocs to start to fill staffing needs, but we’re also having trouble finding postdocs. So I don’t know. What are y’all saying from your side in terms of difficulty filling those positions?
[00:35:00] Dr. Erin: I wish I had great insights to offer around postdoc. It’s a mystery to me, but what we see folks saying or what they come to us with is that they’re not getting the applications, that they’re trying to figure out how to reach applicants and that’s a struggle.And then we sometimes see people that aren’t advertising themselves like I mentioned earlier on, in a way that would attract somebody. Just missing, wow, you are doing this really cool thing for your postdoctoral trainees. You’re helping to create and prop up their potential private practice by teaching them all the skills that go into it, but you’re not advertising that you do that.
And somebody who wants to start their own private practice someday, if they learned that you provided this training, this background, or maybe they could keep their client base when they left, that was a part of the [00:36:00] training, you might get more applicants that way. So some of it is just not knowing what to advertise or those gems within your program to put out there. And some of it is not knowing where to go to seek applicants.
So a lot of times folks don’t know, for example, that APPIC offers the universal directory. So it’s a universal postdoctoral psychology directory that people can put their information into whether they’re an APPIC member or not. You don’t have to be an APPIC member to put your postdoctoral training program information into the directory and not everyone knows that. There’s some things that people might be missing in terms of getting the word out.
And then sometimes maybe postdocs are looking for on-the-job training and somebody to sign up on their hours so that they can go do what they want to do, and they’re not seeking the more formal route. It [00:37:00] depends. It could be any number of reasons, I suppose.
Dr. Sharp: We talked about salary information with interns, do you have any numbers or statistics on average compensation for postdocs these days?
Dr. Erin: I don’t have those statistics. I can say though, if you’re going the route of APPIC or APA, it’s the same deal with postdoc as it is at internship, but you have to advertise upfront and it has to be these stipends that does not depend on the work that’s generated by the postdoctoral fellow or trainee. And so it can create a challenge in a similar vein that it would for your internship level.
If people were curious, one of the things that folks can do to see what’s being offered in their area is go to the APPIC directory, look for postdocs in your, to say you’re in Denver and you [00:38:00] look for postdoctoral fellowships in Denver and you see what the average is, that’s one way to find out a little bit more about what people are providing as a stipend in your areas, to go right to a source that can give you the actual current stipends for the postdocs where you live.
Dr. Sharp: That sounds good. I know what I’m going to be doing after our interview is over, make sure we’re paying everybody appropriately. Before we wrap up the postdoc discussion, you mentioned the training requirements, that’s a decision between an actual postdoc and on-the-job training. Are the requirements for training for postdoc as a rigid or well defined, I suppose, as for internship or what are we expecting if we want to go that training route?
Dr. Erin: There’s a little bit more flexibility, but there is still an expectation that you’re doing two things; you’re providing [00:39:00] that two hours of individual face-to-face supervision per week, and that can be a sticking point sometimes at the postdoctoral level to provide those two hours.
And then you have to provide two hours of training activities. It’s not defined as didactic lectures necessarily. At the postdoc level, it could include group supervision, case consultation, grand rounds, seminars, different things. There’s a little bit of flexibility there in terms of what those training activities might entail but you have to have those offered each week throughout the whole training.
Dr. Sharp: Fair enough. That’s super helpful. Okay. So we’ve laid a lot of groundwork. I think folks should have a decent idea of how to apply for or maybe pursue internship and postdoc.
Juliana, you’ve been waiting very patiently to talk about program evaluation and what happens on the back end. Let’s open it up, I’d love to hear a general description of what that means when you say program evaluation.
[00:40:00] Dr. Juliana: Sure. It even starts before the program evaluation. A lot of times we do feasibility assessments for individuals or organizations who are interested in knowing whether or not they should start a training program. Are there other training programs around? What are their resources? Is there grant funding available? And so we do some of that legwork right before we even start.As folks start their own training programs, there’s a lot of required data points, if you’re accredited through the ARO system that we assist with collecting. It’s not just that, it’s also thinking about evaluating your program, in this case, your training program, for continuous quality improvement.
You want to make sure that your training programs are effective. You’re training the next generation of psychologists, are you doing a good job? Are they prepared? Do they feel a sense of confidence and autonomy as they go out and look for a job?
[00:41:00] That’s where program evaluation comes in. We help with the required data collection and reporting, of course, with both accreditation and for grants, if you have a grant, but it’s also a matter of keeping in touch, a temperature check of what’s going on with your internship or your postdoc. It could be simple surveys or focus groups or interviews.And so assisting with that is something that we do. It’s something that programs should do in general. Being able to have real time data that they can use to pivot, especially in that developmental stage, as you’re starting off a training program and you’re experimenting with different didactics or different conference attendance or different elements, it’s so essential to have a perspective on how things are going in case you want to revise it and improve it next time. And so it can be impactful if you’re using that data continuously and collecting it continuously.
[00:42:00] Dr. Sharp: I know that y’all do this independently. I’m curious about the integration with APPIC or APA, what is required or what’s optional or fun, I suppose, in program evaluation? When do we have to be worried about this and when should we probably do it because it’s a good thing?Dr. Juliana: Erin, I’ll let you speak to the required. Can I speak to the fun?
Dr. Erin: Yeah, so if you decide to become an accredited program, you will have to evaluate your interns.
That is a full-stop requirement. You’re going to have to evaluate their competency development over the year. That’s called your proximal data, the data that’s happening in the program. Are they reaching some minimal level of achievement that you would expect a trainee to reach during the training year and at completion? We have to assess that.
You have to also assess [00:43:00] distal data or alumni data. So after they graduate when we reach out to them and ask them how the training program did, how well prepared they were in all of the profession-wide competencies that we expect trainees to be prepared to operate within, did the program prepare them in the way that they expected? That’s going to be going to part of your distal or long-term data. Absolutely required.
So you have to look at those things each year. There’s also this annual reporting online, Juliana used the acronym ARO, that’s Annual Report Online that goes to APA where you are providing details like your applicant statistics, how many applicants you received, how many you interviewed, and how many are there. And providing some information about the professional experiences of your trainees and of your supervisors.
So there’s a lot of moving parts that go into the required data. It’s generally around your interns and your [00:44:00] supervisors and all the ins and outs of what you’re doing to train them and how your supervisors are qualified to do that. So looking at that whole picture in the data.
Dr. Sharp: Got you. And the fun part.
Dr. Juliana: The fun stuff, we do all sorts of stuff. We’ve worked with programs looking at their diversity and inclusion, equity and inclusion work and recruitment, looking at demographics of interns and other trainees that are in their program, looking at any initiatives or training that they deliver in that area, how are interns reacting to those trainings? What are their feedback around that? So we collect a lot of feedback. That’s one thing that we do.
Another thing too, a little bit bigger is looking at cost-effectiveness as well as cost impact for training programs. Sometimes we work with programs in looking at how did the intern impact the site. [00:45:00] How are the postdocs? How many individuals did they serve? How much did that affect the cost of service delivery for your organization?
In many cases, there’s definitely a cost benefit to having a trainee in your organization supporting the services you’re delivering. So we do cost-benefit analysis in that case but also just temperature checks, throughout the year we do several surveys or we recommend doing at least two surveys during your training year with any of your trainees, postdocs, staff, also students and getting a sense of how things are going.
Are we seeing an increase in their confidence, in their autonomy, and their satisfaction with their program and discussing it with the training committee or training directors? And trying to think of ways that we can improve anything that we notice that might need some improvement or some work. So not waiting till the end of the training year to get that [00:46:00] feedback, but having temperature checks throughout the year to be able to get that information.
As you start off, you have a small internship or a small postdoc with two people. It’s more one-on-one, but as your program grows too, sometimes those surveys can be helpful in getting that information. So those are the fun things especially when delivering new services or new training activities.
We have programs that want to try a different training model or during COVID switching to virtual training and getting some feedback around that, evaluation, research, we do a lot of research on best practices; what are some training that are out there for students that we can incorporate in a program?
Evaluation and research can be incorporated any stage of your training program from developing a cost-benefit analysis to just looking at your efforts as you’re rolling things out. So it really can be incorporated at several stages.
Dr. Sharp: That’s fantastic. I didn’t realize that y’all [00:47:00] were doing a cost-benefit analysis as well. I think a lot of practices would probably find that helpful, especially private practices, but other entities too. That’s fantastic.
I know you’ve seen a lot of data from a lot of programs. I would love to hear both sides, what are some of the characteristics of a program that increase positive outcomes, and then what are some of the biggest mistakes or train wrecks you have seen in training programs?
Dr. Juliana: Absolutely. I think the programs that have the best outcomes, a lot of what we’re seeing is about relationships is a huge part of it. Having effective relationships with your supervisor, with your training director, and having open communication. So a lot of those things that play out in a regular job too, you’re seeing that in the [00:48:00] surveys that we do.
Having concrete training goals and agenda, having clear goals for students, keeping them informed of what they’re going to go through, and not pivoting or being surprised about that. So those are all the feedback that I’ve received from students in good quality programs are the ones they have positive relationships and it’s very well structured.
And they use data. I would argue that those programs use data to pivot when things aren’t working so that students feel heard, that they feel their concerns are being heard.
Disastrous, I haven’t worked with disastrous programs but I do know one of the things that a lot of programs do is they have so much data and they don’t know what to do with it. How do I start? What’s important? How do I make comparisons? And so that’s losing sight of the fact that you have these data requirements, but that’s not just it, you might have other things as well.
How do you get that all together? How do you use it, not just as a training [00:49:00] director but if you’re in a consortium across sites, how do we present this information across sites? If you have a training committee, how do we convey this information to the training committee in the way that’s helpful?
And that’s something that programs struggle with. And if they ignore their data and they’re not pivoting, sometimes that could lead to more negative outcomes for those students because we’re not paying attention to those outcomes.
Dr. Sharp: I’m glad you brought that up. It makes me realize, I don’t know that we have talked a lot about our data. Maybe it’s generally good and so it feels like we don’t have to, I don’t know, but I can’t think of a detailed discussion where the data was presented in any kind of way.
It sounds like that is relatively common or maybe the right question is, how have you seen that done well? How do people take the data and then operationalize it and make it impactful?
Dr. Juliana: Right. [00:50:00] One of the things that I’ve seen coming up is dashboarding, just having data in dashboards that are easy to use, that people can navigate. I think in general, as you’re looking at data, it’s one thing to just hand people data and ask them to digest it. It’s another thing to help with creating questions or looking at trends in the data. If you can get closer to those actionable steps with your data presentation, that’s usually the most effective way of using the data.
I know there’s people who want to nerd out like me on all the significance testing and all that stuff, but that’s typically not busy training directors, that’s typically not how that works. So thinking in what can we do with this? What are the trends? What are the basic things where you’re seeing? What are the next steps? So a lot of action planning and strategic planning around the data use.
Dr. Erin: I can give you a real example from a developing program how they incorporated that [00:51:00] use of data. This was in a university counseling center and as they were setting up their internship, they planned out for the year that as a training committee, so the committee is all of the people actively involved in the planning and implementation of the program; your primary supervisors, your training director, et cetera.
We’re going to commit to meeting throughout the year, but every summer we will have a mini-retreat and we’re going to pull all of our program evaluation data or supervisor data or didactic training information data and we’re going to look at that and decide, did we do what we set out to do last year? Do we need to make changes before we enlist a new cohort in this program?
So taking time to sit down and look at that together and using it for conversation. That’s the ideal, to be able to plan out time that we’re going to look at this [00:52:00] together so we can make changes that are immediate and meaningful to our cohorts.
The programs that don’t do that find themselves in a bit of a pickle when it comes time for reaccreditation and they realize that they haven’t been sitting down and using feedback to improve the program and make changes along the way. So it’s using the data year over year so we don’t forget about it that’s meaningful.
We do see programs that do forget about it, though, and 10 years later, they’re doing their reaccreditation and gosh, they wish they had looked at this 9 years ago. That can be disastrous for a program sometimes.
Dr. Sharp: I could see that. That sounds like a complete nightmare trying to go back 5, 8, 10 years and gather data or make some conclusions or decisions. Okay, goodness.
Is this [00:53:00] a part of what you’re talking about, this data support program, is that what we’re describing when you’re talking about, almost like coaching around what to do with the data?
Dr. Erin: Yeah. Clover recently launched a data support program. It’s a subscription service for psychology training programs, both internship and postdoc. What we do is we work with programs, one, to make sure that their evaluations correctly collect what is required. So meets the standards of accreditation and we make sure that those evaluations are disseminated and that the information is collected, aggregated, and presented in a way that’s useful.
So we’re looking at your interim data, that proximal and distal data that I mentioned in the training program and after but we’re also looking at things like your didactic training data and providing reports back to the program on [00:54:00] that highlighting if there’s a concern in competency evaluation, like look here, things seem to be a miss, want to make sure we pay attention to this, same for program evaluation and et cetera.
So we’re looking at that data, collecting that, aggregating that, putting it together, and creating data workbooks and the data dashboards for programs so they have everything they need at their fingertips, but we’re helping to relieve some of the burden. I know you didn’t want to use that word earlier, but the burden on training directors or training committees that when it comes to collecting data, it can be a lot of work reporting, putting that together can be a lot of work. So we’re trying to lift that burden, make sure people are doing things up to standard and that they have the information at their fingertips when they need it.
Dr. Sharp: That sounds super helpful. Just to be super clear, is it a software program or is it coaching or is it both, not [00:55:00] coaching, but personal support, what’s it looks like realistically dated?
Dr. Erin: We offer individual consultation with that during the year. Our team, so you’re assigned to one of our consultants, and that consultant is building out and sending the surveys, the evaluations on your behalf, collecting that information, putting it into our data workbook for you, and then aggregating that into a separate workbook that’s easier for the program to navigate because everything is set up to match or align with the APA tables that you will someday have to submit for self-study or to align with what’s required for the annual report online, the ARO data.
So we have consultants on our team who are working that for you behind the scenes so that you don’t have to but then [00:56:00] part of that annual subscription is to be able to sit down with a consultant to talk about your evaluation-related information throughout the training year.
Dr. Sharp: Got you. It sounds super helpful and it’s a lot. Like I said, I have not had to handle a lot of this. Honestly, I’m just filling out a lot of information but it’s a heavy lift for our training director and any support in that regard is super helpful.
I really appreciate it, y’all. This conversation went surprisingly fast. I know there’s a lot more that we could dive into in either of these areas, but I’m grateful to be able to speak with you for a little bit and hope that this has maybe inspired some private practice folks out there to consider application for membership and those of us who are doing internships already or postdocs, being willing to look at the data and [00:57:00] put it into play a little bit more. So thanks for being here.
Dr. Erin: Absolutely. Thank you so much. It was fun.
Juliana: Thank you.
Dr. Sharp: All right, y’all. Thank you so much for tuning into this episode. Always grateful to have you here. I hope that you take away some information that you can implement in your practice and in your life. Any resources that we mentioned during the episode will be listed in the show notes, so make sure to check those out.
If you like what you hear on the podcast, I would be so grateful if you left a review on iTunes or Spotify or wherever you listen to your podcast.
If you’re a practice owner or aspiring practice owner, I’d invite you to check out The Testing Psychologist mastermind groups. I have mastermind groups at every stage of practice development; beginner, intermediate, and advanced. We have homework, we have accountability, we have support, we have resources. These groups are amazing. We do a lot of work and a lot of connecting. If that sounds interesting to you, you [00:58:00] can check out the details at thetestingpsychologist.com/consulting. You can sign up for a pre-group phone call and we will chat and figure out if a group could be a good fit for you. Thanks so much.
The information contained in this podcast and on The Testing Psychologist website are intended for informational and educational purposes only. Nothing in this podcast or on the website is intended to be a substitute for professional, psychological, psychiatric, or medical advice, diagnosis, or treatment. Please note that no doctor-patient relationship is formed here and similarly, no supervisory or consultative relationship is formed between [00:59:00] the host or guests of this podcast and listeners of this podcast. If you need the qualified advice of any mental health practitioner or medical provider, please seek one in your area. Similarly, if you need supervision on clinical matters, please find a supervisor with expertise that fits your needs.