Dr. Jeremy Sharp (01:18)
Hey folks, welcome back to the Testing Psychologist. I am happy to be here with you as always, and I’m thrilled and honored to have a guest today who honestly needs no introduction, but just in case, I will tell you who Dr. Kevin McGrew is. Kevin is a practicing school psychologist, educational researcher. He’s interested in individual differences in learning. He’s a professor of applied psychology, and he’s a co author of The Woodcock Johnson 3, 4.
And five, where he was actually the lead author. intelligence and applied psychometric expert. he’s an expert witness and consultant in Atkins Intellectual Disability Death Penalty Cases, and he’s a director of the Institute for Applied Psychometrics. So you have likely heard Kevin’s name any number of places, whether it’s from test authorship for one of the most well-known intelligence and academic measures.
in our field or just you know in the literature. He is a prolific publisher and he’s done tons of presentations. He’s been, you know, an active member of our field for about 50 years now. And as you’ll hear in the episode has just a wealth of knowledge
and expertise to share with us. So we’re talking about we talk about a lot of different things here, but the overarching theme is what we’re calling beyond IQ. So we talk about
Kevin’s journey through grad school to becoming involved in development of the Woodcock Johnson and IQ theory and his involvement there. we talk about just different, you know, mathematical reasons or statistical reasons why we can’t really rely solely on IQ scores to predict academic achievement. we talk about his CAML model, C-A-M-M-L, cognitive affective motivation model of learning.
to account for the percentage variance that IQ cannot explain when we’re looking at achievement. So we really get into motivation, self-beliefs, self-regulated learning, how these things drive real-world achievement.
And we talk about his theory on why G or general intelligence is maybe better understood as an emergent property rather than a biological entity in the brain.
we also close with a discussion of digital innovation and how digital assessment can really up our game in terms of psychometrics and what we’re measuring in intelligence assessment. So very, very broad conversation here, but there’s so much to take away from our discussion. I hope you enjoy this conversation with Dr. Kevin McGrew.
Dr. Jeremy Sharp (04:03)
Kevin Hay, welcome to the podcast.
Kevin McGrew (04:05)
Thank you. It’s nice seeing you today.
Dr. Jeremy Sharp (04:08)
Yeah, yeah, good to see you too. Yeah, thanks for taking the time to to jump on here. I know you’ve got a lot going on and you’re pretty well known in in our field. So yeah, it’s an honor to get an hour of your time or so. Thanks.
Kevin McGrew (04:19)
Well, I am in s I am in quote semi retirement, so these are the kind of the things I like to do. I’m finding I got too many of these I’ve agreed to, so
Dr. Jeremy Sharp (04:27)
huh.
Kevin McGrew (04:28)
they could it comes in batches though. So but this I’ve been looking forward to doing this.
Dr. Jeremy Sharp (04:33)
good. Good, good, good. That sounds good. Well, I’ll start with the the question that I always start with on these podcasts, which is, you know, of all the things that that you could care about and spend your time on and invest in, you know, why why this in particular? You know, why intelligence theory and and that kind of thing? Why does this matter?
Kevin McGrew (04:51)
it I don’t know how long answer you want, but it w it was back I I’ve been thinking about this a lot. Back in my childhood when I was in like some ten to twelve, thirteen years of age, I had a kind of inherent interest in individual differences amongst children in my neighborhood. I can explain my why if you want me to, because it was a v very unique n setting w it had almost had a normal distribution of different type of k no kids, I’m serious.
And also with k kids with very serious disabilities and probably the first poster child for ADHD I’ve ever seen, but I didn’t know about it at the time. but there was a really
Dr. Jeremy Sharp (05:25)
Yeah.
Kevin McGrew (05:26)
interesting mix of people, and I was always an observer. I wasn’t kind of a social isolate, quiet, kind of introverted kid. So I was always inter
Dr. Jeremy Sharp (05:33)
Mm-hmm.
Kevin McGrew (05:34)
interested in individual differences. And when I got into school psychology training, I really got intrigued with cognitive assessment and how people differ and how people approach tasks differently.
And I had a course during a upper level graduate course in psychology on individual differences and that just resonated with me. And I just kind of and I’ll tell you more when we get to another point you want to talk about. there’s some things that happened to me between nineteen seventy seventy eight and eighty four which are kind of magical that really got me going in the area of intelligence, intelligence testing and testing and why people differ and
in the in the theories. it just it was a ma it was a magical period in my career. It magical start and then kind of it just continued after that. But I can give you a little historical background on that when you when we move to the next point in our kind of structured outline or semi structured outline. If you want me to talk any more about
Dr. Jeremy Sharp (06:25)
Semi structured ally. Yeah, yeah.
Kevin McGrew (06:27)
any of that stuff I just talked about, let me know.
Dr. Jeremy Sharp (06:31)
Yeah, no, of course. I mean, I think like a lot of us it sounds like personal experience is driving driving your interest in in this work. Yeah. I think I mean it’s funny, I’m just laughing that you’re kinda looking back and retroactively saying we had a normal distribution in the neighborhood. That’s kind of that’s a funny thing to say. But I get it. Yeah.
Kevin McGrew (06:47)
Yeah, I mean we had w we had yeah
in post hoc I can look and say, well, one of my neighbors two doors down. It was clearly a very bright, extremely bright kid that was clearly on the aut autistics high functioning o autism spectrum. We would walk round on the sidewalk just doodling like this and think with very thick glasses and
Dr. Jeremy Sharp (07:04)
Mm-hmm.
Kevin McGrew (07:05)
then we had the poster child for our ADHD down there just bounced around like and just was just just it was kind of
Not shocking, just intriguing to me. And then we had to d a couple of doors down from me there was a couple of kids that were in the old vernacular trainable mentally retarded that I would you know, talk with
Dr. Jeremy Sharp (07:23)
Mm.
Kevin McGrew (07:24)
and such. And we had gifted kids and we had you know, average and we had a whole you know, diversity and I just sort of always kinda ponder it and not and just embrace individual differences and they were there from the beginning and I just kinda got
interested in it but I didn’t realize w how deep my interest was until I got into school.
Dr. Jeremy Sharp (07:45)
Yeah, yeah. It might be a good segue. Yeah. Tell me about this, I think you said kind of magical experience that you had.
Kevin McGrew (07:52)
Yeah, okay. Well the the it’s basically how did I I got to where I am in being a kind of CHC thought leader and author of the Woodcock Johnson three through the five. I spent my first two years after training i in at that time in Minnesota you could you could be a master’s level school psychologist. And I went down to Iowa ’cause they were hiring and they needed a lot of school psychologists ’cause it was the beginning of ninety four one forty two, the first special educ federal special education law.
And after two years I knew I I didn’t want to stay where I was in I in Iowa. but when I was down there, I was in a service center. The main office was in Sioux City, and I was in one of the satellite centers in a very rural area. And I was all by myself with a speech clinician that was also in the office. And occasionally a journal would come through in inner office of delivery, journal learning disabilities, a few others, you know, stay from the main office. And I I stumbled
Dr. Jeremy Sharp (08:41)
Yeah. Mm-hmm.
Kevin McGrew (08:43)
across two articles by Alan Kaufman.
in one in nineteen seventy five and one in nineteen seventy seven. And my training the training in in school psychology at that time, and I graduated with my degree in in school of psych in nineteen seventy five, was Sattler’s first book. It was nineteen seventy four and then he did a reprint and added a an appendix on the WISCAR. And there was only one and a half pages of of of discussion of theory and like on the and it’s talked about Cattell theory that the fluid crystallized that was it, it was five sentences.
Every theory was like five sentences
Dr. Jeremy Sharp (09:15)
well.
Kevin McGrew (09:16)
and there was no classification of the tests. Uh and it talked about the WISC, the WISCR, the WIPC, and the Stanford Binet, with the exception of at that time Sattler was looking at all the tests and measures from Guilford’s structure of intellect model, which has basically been it’s in all psychology textbooks that it shouldn’t be. It it it had it served its purpose, but it was really a a very abstract, dense kind of thing and it didn’t
didn’t didn’t work. So we we were running around with training that didn’t didn’t really have a framework. And when I saw Alan Kaufman’s articles, one was on base rates of scatter profiles and one was on G loadings and subtest specificities of the WISCAR tests. I would I it was like an epiphany to me because it was like I made work
made this will date me a bit to have my secretary make mimeographed worksheets so whenever I gave somebody a WISCR, which was the instrument of choice at that time,
I do I’d do this analysis that was and I felt like I’m finally doing something that’s norm, norm based, data based, and helped me with interpretation. after leaving Iowa, I came to St. Cloud, Minnesota, the St. Cloud schools, and so I g I so I had my exposure to Alan Kaufman, didn’t know who he really was. Uh just interesting articles.
Dr. Jeremy Sharp (10:26)
Yeah, yeah.
Kevin McGrew (10:27)
And I came to St. Cloud in about seventy seven in nineteen seven fall nineteen seventy seven.
And not soon after I got there, I was the lowest psychologist on the totem pole in terms of seniority. There were only two only two of us, now they have like eight. and there a book came out, an advertisement for a a book called Intelligent Testing with the Wiscar by Alan Kaufman. It was the most
Dr. Jeremy Sharp (10:52)
Mm.
Kevin McGrew (10:52)
brilliant it’s he still he’s he updates the book w whenever there’s a new edition of the Wexlers and now it’s a huge book, but this was a really thin book.
And it was like a religious experience for me to read it because he talked about there’s no theory behind the Wexler really, but here’s how you apply the fluid and crystallize, here’s how you do Luria DOS, the neuropsych model, here’s a Vernon’s model. And it was all databased and psychometrically based about you need to know factor results of of tests, you need to know their specificities, the reliabilities. And that really resonated to me as a database psychologist who came out of the animal lab as
Dr. Jeremy Sharp (11:28)
Mm-hmm.
Kevin McGrew (11:29)
an undergrad. And so
I I read that book probably in one night and it’s underlined. I still have it here. I I I I took it with me when I left the school district. And it it had a it was like an epiphany like it was it was the most pivotal book in my career. So that happened and I really got turned on to that kind of what he called the intelligent approach to intelligence
testing. And then
Dr. Jeremy Sharp (11:50)
Mm-hmm, mm-hmm.
Kevin McGrew (11:51)
about the same time the state of Minnesota was struggling with what is learning disabilities.
And the state was moving towards an ability achievement discrepancy model where you had to account for regression towards the mean. And this chance would have it. When when I showed up at the in St. Cloud Schools, about halfway into my first year there, these free tests showed up called the Woodcock Johnson 1977 and the technical manual. They were given to me as this the least senior guy to you know when you had time to take a look at them and I put on the shelf, just left them there like,
No, everything’s the Wexler. You got the WISC the WISCR was the gold standard. And when they started to say, well, we were going to move to ability achievement model, I said, you know what? I think that test has got ability and achievement, and they’re co-normed, and they actually have regr
Dr. Jeremy Sharp (12:38)
Mm-hmm.
Kevin McGrew (12:38)
account for regression to the mean in the norms. So I cracked it open and started really studying the technical manual and the test. And got the research of director of the research and development office who was affiliated with the
special ed, he was the ex special ed director, convinced him that, you know, we’ve got to take a serious look at this thing. And next thing you know, next year we the next year we were running a study where my colleague Paul and I would go do our regular assessment with the WISCARs and then we would we train two psychometrists who were senior students in psychology at nearby universities. and you know all he did is give the test. They didn’t do any interpretation. And we did that
Dr. Jeremy Sharp (13:15)
Mm-hmm.
Kevin McGrew (13:16)
and they followed us so it wasn’t counterbalanced but we got to up
fifty-four subjects, I think. And we found no statistically significant difference between the the the two tests in terms of average scores. And they correlated, I think, probably 0.79 or 0.8 something. And that that was really basically then the school system decided we’re going to the Woodcock Johnson and we basically put our WISCARs on the on the shelf. we still use we still use them for follow-up assessments or if somebody was being considered for at that time called EMH, educable mentally handicapped for intellectual disability.
but we were one of the first school systems in the country to basically pivot away from the Wexlers. And the reason this
Dr. Jeremy Sharp (13:55)
That’s remarkable.
Kevin McGrew (13:56)
is yeah, this it’s what’s interesting is we found no dis statistical significant the difference between the two Wex Westcar and the in the Woodcock Johnson broadcognitive mobility score. But there was a huge controversy in the field that there were some articles published by two n federally funded
Institute for Research and Learning Disabilities. And these articles came out in the journals and they said watch out for the Woodcock cognitive. It scores significantly lower than the WISC R by about ten or twelve points.
Dr. Jeremy Sharp (14:26)
Mm-hmm, mm-hmm.
Kevin McGrew (14:27)
Okay, ours didn’t. And it was it basically it was a kiss of death on the Wood for the Woodcock Johnson for a while in psychology, school psychology in particular.
Dr. Jeremy Sharp (14:35)
Mm-hmm.
Kevin McGrew (14:35)
And I went to a conference, I was trying to make this short. I went to a conference in in Minneapolis from it was
sponsored by the Minnesota IRLD Institute. And there was a guy s sitting behind me who was big in Minnesota at the time in special ed in school psychology called Dr. Jim Eiseldyke. And he had somebody sitting next to him, didn’t know who he was. And another guy was at presenting one of those studies is saying the Woodcock Johnson scores ten to twelve points lower on average in learn in learning disability sub populations, because they used referral samples of learning disabilities. all the studies that came out, that there were like two or three of them, they said the Woodcock
Scored too low was were based on LD subtypes. And after his presentation, the gentleman behind me gets up and stands up and very nicely, very professionally said, I’m Dr. Richard Woodcock. And I was like, my goodness. and
Dr. Jeremy Sharp (15:22)
Ha ha.
Kevin McGrew (15:23)
he he quietly, gently told him what was wrong with the design of his study. He said, You can’t compare two tests when one was used to pre-select the sample to select the sample because it distorts statistics statistics.
And of course the second one will score lower and there’s there’s mathematical reasons for that. Statistical reasons. And he s he said what you need to do is have a normal sam r random sample or referral sample. And I go, that’s what we did. We did referral sample and didn’t select and and he said, Do this, do this and and I go, We did everything he said and so I got up with the nerve during coffee break. I had a lot of imposter syndrome at the time.
went up and introduced
Dr. Jeremy Sharp (16:00)
Of
Kevin McGrew (16:00)
myself and said, you know, we just finished a study that found what you would say that there should be no statistically significant difference, except for difference in maybe in norm sample dates. And he said, Well, you should write that up. And that started my interaction and aff affiliation with Doctor Doctor Whitcock. And it it basically after that time that he and I got we’re there was no computers to do this kind of stuff. It was
Dr. Jeremy Sharp (16:25)
Mm-hmm.
Kevin McGrew (16:25)
w he
he would call me occasionally and and I submitted the article and it was published in nineteen eighty three. And in the meantime I I got the school system, our special ed system, it was very data based, to open up an account for me over at the n neighboring university to have access to a mainframe computer. and I basically said, Yeah
Dr. Jeremy Sharp (16:45)
That’s a big deal.
Kevin McGrew (16:46)
it was. and I did it when I had time in office hours or after work basically.
And I and I said, we need to figure out the same stuff Alan Kaufman had in his book Intelligent Testing at the Whiscar for the Whitcock Johnson, G loading, specificity, fa factor analysis stuff. So I taught myself factor analysis, because if you have a technical manual, you can suck the correlation matrices out and basically factor analyze. So I started teaching myself and I started publishing articles, kind of just left and right of the practicing school psychologist. And
my c connection with Woodcock got more I sent I’d send them results, I send them manuscripts and we start talking and I’ll I’m gonna skip the part that that ’cause I think I’m talking a little too long here. at one point I was doing cluster analysis looking for subtypes of learning disabilities on the Woodcock using
Dr. Jeremy Sharp (17:35)
Mm-hmm, mm-hmm.
Kevin McGrew (17:36)
the Woodcock Johnson twelve cognitive tests. And he informed me that sorry Kevin, but those scores are all wrong.
Because he didn’t publish the test with subtest uh scale scores, he used RAW scores and had a brilliant, elegant way of profiling them with confidence bands. So you talk about strengths and weaknesses. And he said I I I somebody did those, I consulted with them, told them what to do, and they did it differently. He said, All your
Dr. Jeremy Sharp (17:58)
Mm.
Kevin McGrew (17:58)
data’s wrong, so I was crushed. But then and he looks up at me, and
Dr. Jeremy Sharp (18:01)
Great.
Kevin McGrew (18:03)
this is at a personal meeting, and he said, How would you like to do the new subtest norms? And then that really
Next thing I know, I was on a train, Amtrak train out to Oregon and I lived with him for a week and he mentored me on how to fit curves and develop norms. And that was the beginning of our long term basically apprenticeship. So that happened all in the you know, between nineteen seventy eight and eighty four. And I was so enamored and excited about doing this stuff. I’d stay up late at night till two or three in morning analyzing data and writing articles and basically started to write a book. And that’s kinda like what’s
basically was the impetus for me doing what I’m doing now, but moving from Iowa to a place that had a free woodcock sent to them. I did before I was even there. And reading Alan Kaufman’s book, nineteen is in Alan Kaufman’s nineteen seventy nine book, it was like, this is meant this is what I’m supposed to be doing. And it was kind of very almost don’t know, just the energy level was so high for me at the time. So that’s how I got involved.
Dr. Jeremy Sharp (19:05)
Yeah.
Yeah.
Kevin McGrew (19:07)
Originally.
Dr. Jeremy Sharp (19:08)
Yeah. I mean, it just makes me think like we have these moments in our careers, you know. I think of them like guideposts, right? And you are like walking through the forest or whatever and you like find the book and that like sends you down a certain path. And then, you know, you end up in this conference and you talk to, you know, Doctor Woodcock and that sends you down a path. And then before you know it, you’ve like changed your life, like the entire trajectory over five or six
Kevin McGrew (19:29)
Yeah, I yeah.
Dr. Jeremy Sharp (19:30)
years. That’s amazing.
Kevin McGrew (19:31)
My motto is professional motto so my webpage is purpose, passion, and serendipity. And to me, serendipity is not
Dr. Jeremy Sharp (19:37)
No.
Kevin McGrew (19:37)
the happy accident. It’s Louis Pasteur’s notion of that luck favors the prepared mind. I
was starting to read a lot
Dr. Jeremy Sharp (19:44)
Mm-hmm.
Kevin McGrew (19:44)
of stuff, teaching myself some statistics I d hadn’t learned. And I was I could see see opportunities or see, this is important, this is important. And then I w when I got doc I di Woodcock asked me to be involved with the revision of his seventy seven original.
The one that came out in 1989. And I was in just starting my doctoral training. And I ended up being the first author in the technical manual. And so I was doing the work on the project and learning the coursework at the same time at the University of Minnesota. And the really important thing there about the guideposts, walking through the forest, is at the first meeting in 1986 on the revision for the revision of the WJR, to the WJR.
John Horn was the prim was on the advisory board, was there invited to the meeting, was asked by Dr. Woodcock to present his theory of extended GFGC theory. And then just by luck, the president of the company invited another expert that he worked with on a dictionary. I don’t know, we didn’t know really
Dr. Jeremy Sharp (20:43)
Okay.
Kevin McGrew (20:44)
who he was. Turns out it was John Carroll. So the H and the C of the C
Dr. Jeremy Sharp (20:49)
my gosh.
Kevin McGrew (20:50)
A C
were at that nineteen eighty six meeting and I got exposed to both of them and I was kind of like, whoa, this is and then for seven up to seventeen to twenty years, depending on what w which one you want to talk about first in terms of when they passed away, I ended we attended a lot of meetings with both of them, both either with Horn or with Carroll, or both of them when we were doing the WJR revision and the WJ three revision. So there I was as an another imposter
kinda learning from the you know, from the sh on the shoulders of giants. It was really kinda and that just took took career and just pitched it to a train and off it went.
Dr. Jeremy Sharp (21:29)
Yeah, yeah, it’s incredible. Right place, right time with a lot of preparation and like you said, yeah, yeah. So at some point, I mean, you have this realization that you know, intelligence or IQ tests could only only account for what forty to fifty percent of variance in academic achievement. So we have to make up for the rest somehow. and that’s where this like idea of beyond IQ is kind of coming from. Can you talk about that a bit?
Kevin McGrew (21:53)
Yeah, well it it started when I was a practicing school psychologist because I would meet with parents and you know I and I gave either the w Jr for while. I mean the WISCAR and then and the WJR and you’d have to present results. And you know, kids
Dr. Jeremy Sharp (22:04)
Uh-huh.
Kevin McGrew (22:04)
who are referred typically, you know, the bulk of your work as a school psychologist are kids who tend to score below one hundred. I mean, the the
Dr. Jeremy Sharp (22:11)
Mm-hmm.
Kevin McGrew (22:12)
mean of L D populations was like in the mid eighties or eighty eight or something. Or you got the so called slow learners, and or
kids who qualify for disabil intellectual disability services. And I always felt extremely uncomfortable presenting the results. and I was very o honest with the parents. I would I would say, you know, I could test your child forever because there’s this n is not, you know, I gave them twelve subtests from the d Woodcock Johnson revised. I didn’t, you know, that’s the name of the test. And I say I I drew I draw a circle and I say, I had an hour, hour and fifteen minutes with him and I went in and gave and I didn’t make these little dots in the circle. We just sampled
certain abilities, but there’s all this white space around there that didn’t touch because they don’t have time. And it just gives us an estimate of what their their aptitude for school learning is. I preferred the word aptitude instead of intelligence. And I said, but there’s a lot of stuff we’re not measuring here that you might be seeing elsewhere or and because parents would always come back with that. Well you know, but he does this on this and when he’s it’s his interest or his hobby or something. And I’d say, yeah. And also and and I was very much a strange
statistical had a strong interest in statistics and psychometrics as an undergraduate and then got some more graduate training. I knew the correlations, the best correlations between cognitive measures and achievement measures, concurrent correlations. We’re not talking predicting the future, which is weaker. w were be between point six zero to point seven So I took the highest value. If you take a correlation of point seven and you square it, that’s forty nine. Point four nine actually
and it’s
multiply it by a hundred, forty nine percent variance. And I knew that. That basically I tell parents though well these tests only explain fifty percent half of what we’re concerned about and we’re here today about achievement. The other things are things like motivation, interest, maybe some personality factors, which things you do and help with at at home. So I just kind of went there intuitively and um I and also because I knew like
David Wexer used to talk about non intellectual factors, which were motivation and persistence and such. So I I had this, you know, I
Dr. Jeremy Sharp (24:08)
Mm-hmm. Mm-hmm.
Kevin McGrew (24:12)
didn’t want to bring the parent down because I I knew the fallibility of our instruments, that they that there is no such thing as a real true IQ score that’ll be there and that’s stamped on their brain and it’ll determine where they’re going. So I was very realistic with them about, you know, limited, this is what we found. and this is where he’s achieving right now, and there’s a discrepancy and he qualifies for service or
She qualifies. but I do did say a few things in some reports I’d like to have retracted, and they’re probably gone by now. But most school psychologists, if they saw somebody with an a 90 or an 85 full-scale score or broad cognitive ability, would say, well, they’re going to be achieving you know below their peers in achievement, uh and we’ll probably see, you know, they can’t really achieve at or above their ability. And I did put that in some reports, and I’m upset that I ever did that because I knew.
from reading and from my own personal experience that I shouldn’t be talking to you based on my I my IQs and measured aptitude tests. I got here somewhere and it was the motivation, self beliefs, that kind of stuff. So
Dr. Jeremy Sharp (25:14)
Yeah.
Could we address that a little more directly? I feel like you know, folks still wrestle with that to some degree. Like I I see this question come up fairly frequently. maybe you do too, where, you know, people ask, well, you know, this this kiddo has an you know an IQ of I don’t know, ninety-seven, for example. how are they how are how am I seeing this one eighteen on the reading, you know, basic reading scales or whatever, one twenty-one on math.
Calculation. Like yeah,
Kevin McGrew (25:41)
Yeah. Yeah.
Dr. Jeremy Sharp (25:43)
talk about that just for a second to to make that clear for folks.
Kevin McGrew (25:46)
Okay.
Well that kind of leads into a s one thing we we talked about earlier is between nineteen ninety and nineteen ninety six I was one of the principal investigators or senior researchers on the grant that was at the U of called the National Center for Educational Outcomes. And in parentheses it was for students with disabilities, the NCEO. And this whole idea idea of beyond IQ became more focal when George Bush’s
policy of no child left behind hit hit the hit the country. And it was it was a big big change in education and it was all about accountability, test driven state assessments, accountability systems and some national the net like the NAPE assessment. And the the conversation started to turn to well schools didn’t want to include kids with disabilities in the in the stats because it was by basically reflected on schools, because there were school results. And
Dr. Jeremy Sharp (26:38)
sh ruinless, yeah.
Kevin McGrew (26:41)
And I was you know, intelligence expert, developing I mean a beginning intelligence intelligence expert and getting some psychometric skills. And I didn’t really like doing policy research, but I went to a I was invited to to go to a meeting in DC with some other N CEO staff that and they they knew what I was gonna present. And the conversation started to often that there were all these people around there from different offices of the Department of Education and and other related offices.
and some Bush appointees there and you know ch used to be called Chapter One or Title One, OSAP Office of Special Education Programs, NIDER, and other affiliated things. And I could tell immediately that that there was there was a division in the room about, yeah,
Dr. Jeremy Sharp (27:22)
Mm.
Kevin McGrew (27:23)
we gotta include kids with disabilities, those are the disability or advocates. And there were some parent groups uh there and the N C E O was pushing that a agenda.
And so I put together a presentation which I thought would made a lot of sense. And it was called the Forrest Gump presentation, that’s what I called it. And bas basically, I presented it two bits of information. I presented with well, if we if the if you remember Forrest Gump is the first it opens up with Sally Fields in the principal’s office where it’s it’s soon that’s one of the first scenes. And principal says, Your your child is different.
He’s got an IQ of seven seventy-five and he shows a normal curve with where he’s at. And Sally Fields, mom goes, Well, I know he he’s different and he’s special, but he’s not going to go to a special school to retread tires. and as as he went on through life, he he he accomplished a lot. So it was to he was just a metaphor for you can achieve above your IQ. And so then I brought out the ammunition. And one of them is if you have an IQ score and you want to predict what their current achievement’s gonna be.
You you predict it and then there’s a a standard error of estimate. So because that prediction is an estimate. There there’s all kinds of standard errors in statistics, but you take a correlation of point seven and predict achievement for s for somebody, with a correlation of point seven, you you take you calculate uh a value of taking fifteen standard deviation times the square root of one minus the correlation squared. Simple. And you do that, and I’ve got it right here.
And the the standard error of estimate for predicting achievement for somebody with an IQ of seven with a with a correlation of s seventy is a is approximately eleven plus or minus eleven points. So you got a twenty two point range. You know? That’s a big range.
Dr. Jeremy Sharp (29:00)
It’s a pretty big range. Yeah.
Kevin McGrew (29:02)
And then I took up since I had access to the W J three norm data, I pulled out anybody who had a score from seventy to eighty, and then I plotted their achievement scores. It was on a graph X w XY axis.
And the IQ scores for between seventy and eighty went from forty-three to a hundred and nine. really that graph that showing it I said, these are real people. These are real kids. And it basically means just because something has got a high correlation in group data, like a point seven, that sounds very impressive in psychology. But if you were to try to predict an individual, for mo for a lot of the people you’d be pretty close, but it’s a it’s a normal distribution around there.
And it can range up to plus or minus eleven points at if you want to be sixty eight percent confident, and you double that if you want to be ninety-five percent confident. The people in the room were getting extremely uncomfortable, and the other ones were
Dr. Jeremy Sharp (29:58)
yeah.
Kevin McGrew (29:58)
getting extremely excited. And it
Dr. Jeremy Sharp (30:00)
Mm-hmm.
Kevin McGrew (30:01)
I’m not the only one that helped, but but it’s I know that the NCEO director for a long time credited it with me that that the Forrest Gump presentation had a lot to do with making sure that.
Kids with disabilities were not carte blanc excluded. They basically had to have alternative assessments, accommodations, and so it spun off a whole bunch of research about how do we include kids with disabilities. And part of the problem for a lot a lot of practitioners in accepting the fact that you can have a plus or minus twenty two is is selection bias. In schools you’re the kids only are referred that are having achievement problems. So somebody’s got an eighty five.
is referred and of course they’re ach achievements at or below, but there’s probably other kids with
Dr. Jeremy Sharp (30:44)
Mm-hmm.
Kevin McGrew (30:45)
the same profile cognitive scores in the classroom have got high motivation, high conscientiousness, high self regulate regulatory learning mechanisms that don’t ever get referred because they do pretty good and they get the assignments done and they do decent work. So it’s an eye opener I’ve been finding even to today that’s a lot of psychologists are are kind of gobsmacked when you say, Yeah, it’s a plus or minus twenty two.
It’s just uh an achievement.
Dr. Jeremy Sharp (31:11)
It’s just hard to wrap your mind.
Well, yeah, I mean we like certainty and we like predictability and you know, we kinda overreach, I think, in our efforts to make things make sense. Right.
Kevin McGrew (31:23)
Mm.
Dr. Jeremy Sharp (31:24)
Because that creates it makes things messier when numbers don’t line up.
Kevin McGrew (31:27)
Yeah,
but I I am still shocked how often that that belief still is kind of rum running around out there in s in some with some psychologists. And it’s it’s and it’s
Dr. Jeremy Sharp (31:37)
Yeah.
Kevin McGrew (31:38)
it’s huge in the in the lay pop lay population, you know, amongst people. That you know you can’t achieve higher than your IQ and amongst lawyers and and the courts, yeah, it’s really out there. So I got really Yeah, so
Dr. Jeremy Sharp (31:50)
Sure, sure.
Kevin McGrew (31:52)
that’s that’s where I kind of went. Then I move
that program of research into another direction a little later. So
Dr. Jeremy Sharp (32:01)
Okay. Yeah. Well I don’t know if this is where you’re gonna go, but you know, I wanted to jump to I’ve been calling it the camel model. I’m not sure if that’s what you call it, but you know, C A L. can you talk a bit about this this model that you developed
Dr. Jeremy Sharp (32:15)
Let’s take a break to hear from our featured partner. Y’all know that I love Therapy Notes, but I am not the only one. They have a 4.9 out of five-star rating on trustpilot.com and Google, which makes them the number one rated electronic health record system available for mental health folks today. They make billing, scheduling, note-taking, and telehealth all incredibly easy. They also offer custom forms that you can send through the portal. For all the prescribers out there, Therapy Notes is proudly offering ePrescribe as well.
And maybe the most important thing for me is that they have live telephone support seven days a week so you can actually talk to a real person in a timely manner when you need to. If you’re trying to switch from another EHR, the transition’s pretty easy. They’ll import your demographic data free of charge so can get going right away. So
if you’re curious or you want to switch or you need a new EHR, try Therapy Notes for two months totally for free. You can go to the testingpsychologist.com/slash therapy notes and enter the code testing.
Again, totally free, no strings attached. Check it out. See why everybody is switching to therapy notes.
Dr. Jeremy Sharp (33:21)
Hey, everyone. I’m really excited that NovoPsych Psychometrics is sponsoring the show. NovoPsych is a platform for psychologists who care deeply about assessment and testing and want their self-report measures to be the very best. NovoPsych has an extensive library
of 150 standardized instruments with strong coverage across the presentations many of us assess every day, like disability, functional impact, autism, ADHD, and a wide range of symptom measures.
You can also use it for broad personality assessments like the Big Five or go deeper when you’re looking to understand personality pathology. What makes NovoPsych different isn’t just the range of scales, it is the quality of the experience. So I really appreciate the depth of psychometric info that it provides and the clear graphs
and visualizations that make results easier to interpret and communicate. If you want to try NovoPsych psychometrics, you can access a 15 day free trial via the link in the show notes, which is
novopsych.com slash testing psychologist. That’s N-O-V-O-P-S-Y-C-H dot com slash testing psychologist.
Kevin McGrew (34:25)
yeah. it is called the Camel. ‘Cause we’re a psychologist and you have to come up with an acronym. And Alan Kaufman that was looking at the paper I wrote was in published in twenty twenty two in the journal Canadian Journal School of Psychology. And you can find it on my my webpage, the the Mind Hub. that’s you’ll have that up up that information up. and it’s basically I first started with d I said
After the expectation for scump presentation, I was then asked to write a paper for the N C E O. I’d already left, but they were subcontracting with me. And on expectations with kids with disabilities, you know. And I I used the presented the information about, you know, plus or minus twenty two and so what are the things that that have to do with making somebody being able to achieve above their IQ score? And it got really broad into about, you know, there’s family environment, there’s
you know, society’s investment in education that trickles down into the schools. There’s family investment and then there’s personal investment of of the of of the person. And one of the people down at the U of got a discretionary grant from Office of Special Education Programs for you know, here’s some money, use it. Budget year is over, basically. And he turned it over to me and said, do something related to your expectation work. So I I wrote a white paper.
devoted quite a few years to it on what are the other variables besides cognitive ability. And
Dr. Jeremy Sharp (35:45)
Mm-hmm.
Kevin McGrew (35:46)
I did massive literature reviews, had a team of two people working on it. And I stumbled across a book and some writings by Richard Snow, educational psychologist, probably one of the most preeminent educational psychologists that’s ever lived, but he had died early of pancreatic cancer before he was he was just but he had
published a a ton of stuff and somebody wrote some of his ex students and colleagues wrote a book in his honor called Remaking the Concept of Aptitude. It was published in 2002. I wouldn’t recommend it as an easy to read because it’s written written by committee and it kind of reads like a a committee. But he had the skeleton of what is now the Camel model. That you basically have to he talked about cognitive abilities and he talked about the the early GFG C stuff that wasn’t C H C yet.
He talked about motivation and what what what motivation constructs do we did we know about? And he talked about the big five personality. So it was like the the the trilogy of the human mind, which you know has been around for a long time, cognition, cognitive abilities, which is a word people have a hard time with. I ha I struggle with it, cognitive, and affective
Dr. Jeremy Sharp (36:55)
Mm-hmm.
Kevin McGrew (36:56)
or personality. And he basically said this is the way we need to go. And
A person the we need to de emphasize the IQ score, a cognitive part. That’s only one part of it. These other things,
Dr. Jeremy Sharp (37:07)
Mm-hmm.
Kevin McGrew (37:08)
and especially motivation and self-beliefs and other things, are important. And he was working on two books at the time he was die he died. And that basically took me down I want to flesh out the rest of Richard Snow’s model. And originally we called it the Minnesota Snow model of of aptitudes, but that just didn’t didn’t catch on.
And so yeah so I did d
Dr. Jeremy Sharp (37:30)
Sure. It’s not quite as catchy. Camel.
Kevin McGrew (37:33)
did it Camel was a cognitive affective motivation model of learning. and I can go through that a bit. Go ahead.
Dr. Jeremy Sharp (37:38)
Yeah. And for folks who may not know
Kevin McGrew (37:41)
Go ahead.
Dr. Jeremy Sharp (37:42)
can you define cognitive real quick just for folks who may not be aware?
Kevin McGrew (37:46)
Well, cognitive abilities are I actually have a definition here s I could look it up, but cognitive abilities are things well it’s best to give they’re they’re they’re they’re the abilities that kinda are the centrific centripetal centripical force or the magnet that holds together cognition and and personality to to make them kind of work together.
And the the major constructs in there, and I I I there’s a ton of constructs, but primarily under motivation, there is I hope you don’t mind me, I I I want to make sure I’m as accurate as possible here in things so there’s was a concept in the field of motivation and and and motivation is one of the most studied areas in psychology, educational psychology, not all psychology, after intelligence.
And there’s a a broad category called achievement orientations and best way to define that is to tell you what they are like goal orientation. You know, are you you have a mastery goal orientation or you do have a are you no, no, do you have approach bo goals or avoidance goals? Do you want to avoid a unpleasant outcome in education or learning something new
Dr. Jeremy Sharp (38:58)
Mm.
Kevin McGrew (38:58)
or approach, you want to try to get better at something.
Dr. Jeremy Sharp (39:01)
Mm-hmm.
Kevin McGrew (39:02)
and there’s also goal setting.
And do you want it are you a mastery goals? Are you wanting just to learn something for the sake of learning it? Or are you on performance goals? I want to be the best in class. And those are kind of and there’s actually a three by two by two matrix of the different type of goals. So there’s achievement goal orientation, there’s intrinsic motivation, which most of most of us have a c coursework on. There’s interests, especially academic interests, and then the other major category,
There are achievement orientations, and I just named the three that are probably most validated. There’s other ones, but I had to restrict it. Are self-beliefs, and those are like locus of control. You know, do I believe that I’m making the difference? and competence mindset? Do I have a fixed or entity mindset that you can’t change? Or do I have an incremental growth
Dr. Jeremy Sharp (39:50)
Right.
Kevin McGrew (39:50)
mindset that, yeah, I can change? So that’s important. And then there’s self-eth
Academic self eca efficacy. Ecopacy. Efficacy.
Dr. Jeremy Sharp (40:00)
Mm-hmm.
Kevin McGrew (40:01)
and there’s academic self-concept. those are
Dr. Jeremy Sharp (40:05)
Casu.
Kevin McGrew (40:05)
so that’s the kind of things that are but they’re closer to where the rubber meets the road in the classroom than big five personality of conscientiousness and openness to experience and such, which are kind really kind of distal influences on what a person does, but they’re
It’s those kind of things, the things that initiate and direct and sustain behavior in learning is what is what one way to think about what cognitive abilities are.
Dr. Jeremy Sharp (40:31)
Sure, sure. So I guess my next question, you know, bringing it to the to the practical realm is like, how do we assess these things? I feel like we we have, well, pretty good ways of measuring the cognitive, you know, we have a lot of quantitative measures, but how do you suggest we look at some of these these other qualities
Kevin McGrew (40:49)
Okay.
Dr. Jeremy Sharp (40:50)
and traits and characteristics?
Kevin McGrew (40:52)
Can I backtrack this to second? So yeah.
Dr. Jeremy Sharp (40:54)
Course. Yeah.
Kevin McGrew (40:55)
and the idea that you know, if you you can read about the history of cognitive abilities i and it it goes back to Spearman back in nineteen twenty seven, who’s who’s the father of G said the process of cognition cannot possibly be treated apart from cognition and affection. David Wexler said you know, his he he had the classic definition of it’s intelligence of the global or aggregate capacity to th act purposefully
think rationally and and successfully interact with your adapt to your environment. But then he what people seem to ignore is that when he he said when our scales measure non intellectual, cognitive, as well as intellectual factors, then we will more nearly measure what actual in actual life corresponds to intelligent behavior, not intelligence. And I would consider what intelligent behavior is aptitude. So with when you
Dr. Jeremy Sharp (41:45)
Mm.
Kevin McGrew (41:45)
put those together, so
To get back to your question. in terms of instrumentation, there is I haven’t well, I I have done some stuff in that area. There are there are some motivational instruments out instruments that out there that measure goal orientation, intrinsic motivation, locus of control, self-efficacy, self-concept. I haven’t kept up with them, but
There will there will be a handout packet I I sent to you that’s got five pages at the end where I did a lit kind of a a search of the literature and put it all in there with all the links to some of the instruments that that are out there, which are formal instruments like the I used to use the Pierce Harris self-concept scale, the Tennessee self-concept scale. there’s a bunch of goal orientation questionnaires. So there are instruments that are out there. What I I
That I can’t s I haven’t specifically looked at or recommend uh or evaluated. Instead, I have focused questions that need to be asked. And and for example, the achievement orientation big domain is do I want to do this activity? Is is is that and there’s sub-questions underneath there, such as why do I want to do this activity? What are my goals for this activity?
Is this activity of interest to me? Is this activity worth the effort? And what is the personal value? So I would suggest maybe some kind of informal clinical interviewing that you can flesh out maybe some questioning depending on the age appropriateness of the person. And then self beliefs can I succeed be successful on this activity? Am I capable of doing this activity? And can I control my success on the activity? And those are just some questions that I, you know, if I was a practicing psychologist, I’d develop some kind of interview.
scale. I also forgot to t mention earlier part of the camel besides achievement orientations and self-beliefs is self-regulated learning strategies, which is once you decide you know you cross the Rubicon that you’re you’re gonna do something besides wit wish about it, think about it, you develop intentions and you act. And self-regulatory learning are you know the the cycle of how am I doing? Not well.
What do I need to change in my plan? And it’s just this iterative process. And it’s a very it’s a very nonlinear thing that’s going on. so I and some of those questions are what do I need to do to succeed? Well what will be my standard for success? Can I f how can I effectively organize my strategies? How am I doing? What do I need to do differently? So I can’t give recommendations for civic instruments, but would suggest that maybe I’ve heard of motivational interviewing and I have not really d spent any time.
tracking that down. I did, and I don’t know if I’m if I should share this, but between two thousand six and two thousand twenty, I started developing a battery of tests, self report, parent report, and teacher report to measure the model of Campbell model. I still ha have those item sets sitting on my hard drive, but it’s it was always something that when I had time oh not working on CAC theory in the Woodcock Johnson batteries.
So it would c it would go come and go. might be a retirement project that I might do, but I haven’t gotten around to revisiting that yet. So I was gonna put together a colon arm battery of all these constructs.
Dr. Jeremy Sharp (45:02)
I don’t think it’d be helpful and useful. We could always always use that. Yeah. Yeah.
Kevin McGrew (45:05)
maybe. Or I can find somebody who’s interested in it who’s younger than me.
Dr. Jeremy Sharp (45:09)
that’s probably true. Yeah. Yeah. Protect that semi-retirement. So you know, we talked a little bit about how grad programs don’t seem to focus on these factors very much, you know, they tend to overemphasize the I don’t know, the IQ part and miss the other part, the beyond IQ.
What do you think’s going on there and is there you know, is there any way to to change that?
Kevin McGrew (45:28)
Well, y yes, and I actually did some research ’cause I I haven’t been in the tr training for for for for quite a while. But I c I pulled some school psych trainers. Uh in fact I sent a note to the school psychology trainer listserv and asked for you know how do you teach assessment, cognitive assessment
Dr. Jeremy Sharp (45:46)
Mm.
Kevin McGrew (45:47)
and and things about learning dis uh individual differences. And it
Dr. Jeremy Sharp (45:50)
Yeah.
Kevin McGrew (45:50)
and then I I found there was
Brian Farmer published an article in twenty nineteen, a survey of how people teach cognitive assessment. Then Lockwood and Farmer in twenty nineteen did a survey of graduate programs and how they teach it. And the these seems to be the standard model is that cognitive assessment is taught in one semester and the theories
Dr. Jeremy Sharp (46:12)
Mm-hmm.
Kevin McGrew (46:12)
are taught along with it. And to to me what? Yeah. uh
Dr. Jeremy Sharp (46:15)
That tracks. I said that tracks, yeah.
Kevin McGrew (46:19)
And and about the the the Lockwood and Farmer article basically s showed that recently there’s been a big shift towards CHC in training and almost everybody’s getting training on the C H C or Catel Horn Carroll way to interpret tests. So they’re getting that. But I still think that
Well, I’ll just I’ll say it. Doug Detterman started in in an International Society for Intelligence Research, started the journal Intelligence, was the editor for about, I don’t know, twenty, thirty years of that, published a paper in twenty sixteen where he reviewed the major large scale studies on the effectiveness of schooling. And the bottom line that came across all these is that about 10% of the variance in school achievement is explainable by schools and teeth and the teacher.
90, 80 to 90 percent of the variance is explained by student characteristics. Now, to me, I’m an individual difference as a psychologist more than a school psychologist. And for for
Dr. Jeremy Sharp (47:18)
Hmm.
Kevin McGrew (47:18)
training programs to give one semester just to do cognition intelligence, and you gotta learn the WISC, the most recent WISC, the WIFC, the KEBC, and Binet’s probably not being taught because it’s out of date, but it’ll be
be refreshed sometime in the next year or two, and the Woodcock Johnson Four, that’s a heavy dose of information. I don’t think that the and that the s knowledge level and expertise in the ed the primary educational psychology literature on motivation, on self beliefs, and how those things can be manipulated manipulated and be intervened on and make have significant effect sizes. We have for
decades we’ve been trying to tweak intelligence. We it’s basically been a dry well. even the re more recent re research on brain training, like COGMAT and that kind of stuff. It’s the jury’s still out.
Dr. Jeremy Sharp (48:10)
Right, right.
Kevin McGrew (48:12)
could talk a little bit about that. But I it’s just that individual differences is so important for understanding how students do in school. I don’t think schools sky graduate programs
are training enough on intelligence theory. I think they should do more in the C C theory since it’s the predominant model right now that’s being used for interpreting most intelligence tests, save one or two. and to understand more of the things about motivation and self self beliefs and such. the the Canadian Journal of School Psych the issue I had my published my paper in, basically the introductory article by the editors talks about that school psychologists basically don’t have much training at all in motivation.
Which is like if I could have you know five dollars for my child that I was I saw when I was a school practicing school psychologist, who they’d say, Well his his motivation or her motivation isn’t good either, you know. we need to spend more time on
Dr. Jeremy Sharp (49:03)
Right.
Kevin McGrew (49:04)
individual differences characteristics, cognitive ac cognitive and cognitive in particular. affective or big five personality less so.
Dr. Jeremy Sharp (49:13)
Yeah, that’s fair. I mean, I’ll think about too just practically speaking, at least, you know, in our program we had cognitive assessment and then we had personality assessment. And so this sort of raises a question of like, well, where do where do these other factors go? You know, and like do you teach it in personality assessment or do you weave it in with cognitive assessment? And then what would that look like? You know, it just sort of like muddies the waters. And again, like humans don’t really like that as a whole. So
Kevin McGrew (49:38)
Yeah. Well,
you know, excellent quote from tw twenty twenty one by Pratzko and Colomb in the journal Intelligence. Not journal of intelligence, that’s separate one. I love this quote. All the relevant variables preceding ostensible behavior or observable behavior are cooked in the same pot, parentheses, the brain, parentheses. Although we can artificially talk about cognition and con cognitive abilities or motivational constructs and personality.
A person is and even that quote I read earlier from Spearman says, you know, you a person is not indivisible into cognition and connation. And Raymond Cattell back in nineteen eighty seven said that school of psychology made a big error by placing c trying to explain too much of achievement by uh cognitive assessment. I would like to see, and that’s why I wrote that article, the Campbell article, it was more of a thought piece.
Dr. Jeremy Sharp (50:32)
Mm-hmm.
Kevin McGrew (50:33)
I you know, he was given freedom to do. It was the invited paper. I think those things should all be taught maybe in a s a course separate from the cognitive assessments. I mean you might you might talk about CHC theory and and on the the past test you talk about the past model of intelligence briefly, but then you’d spend a semester on educational psychology, individual differences in in these traits in personality.
and motivation and cogni cognition because in my paper I present a kind of a framework of how I think these things schematically relate to each other and like the big five are kind of in the background. They’re like this general can dispositions we have to b kind of behave a certain way. Like they
Dr. Jeremy Sharp (51:19)
Yeah.
Kevin McGrew (51:19)
you tend to be conscientious and that pushes you this way. You tend to be open to experience. That kind of pushes you
They’re gentle pushes that we all have in our life, but they’re not the things that really are the levers that you can change in the classroom. And then there’s a group at down in the Sao Paulo, Brazil, at the Art and Sena Institute, massive institute, doing state of the art research on social emotional skills. And they have taken a big five framework and basically have identified what they call facets of big five, and they are such
things such as under open mindedness is curiosity to learn, creativity, imagination. Under self-management or s conscientiousness is determination, organization, focus, and persistence. So those are getting closer, those social emotional skills that are getting closer. And they’ve got instruments that they’ve developed. You’d have to get a hold of them and see if they you could use them or not. And then they those things, especially openness to experience and in self management or see
influence the the domains that I I put in the Camel model, which are achievement orientation, self-beliefs, and self-regulated learning behaviors. And then those mechanisms,
Dr. Jeremy Sharp (52:30)
Mm-hmm.
Kevin McGrew (52:31)
those are the personal investment mechanisms that a person according to Cattell, there’s societal inve investment, family investment in learning for a kid, and personal investment. And personal investment mechanisms are the motivation, self beliefs,
self-regulated learning. So I think they should be taught kind of all at the same time so you can understand that big five and I I talk about my model also being the 3D model if you don’t if you don’t like the Campbell model. There’s disposition there’s dispositions that push you certain ways. And that’s the Big Five traits. So they just kind of gently nudge you along dispositions. And then there’s the the achievement orientations which is intrinsic motivation, interest, and and such.
and self-beliefs, which are the drivers, so that’s the second B, the drivers of of in initiating and directing goal direct initiating goal directed behavior towards do doing something. And then once you cross the Rubicon or the cross the bridge of you know a point of no return that you’re going to do something about something, then the self-regulated learning strategies, or what’s often called SRL, kick in to monitor how you’re doing.
And they’re they’re the drivers or what I tr just disposition. directors. They direct your activities. So if you don’t if you don’t if this is all too construct dense, think about the general dispositions we have. And if you don’t if you’re low on conscientiousness, you’re probably not gonna have good work work habits and be motivated and organized and that kind of stuff. But it’s the th th th those are kind of pushing you in a certain direction.
But it’s the drivers, achievement orientations and self-beliefs, and the directors that are the the most important thing about how a person invests their cognitive abilities into achieving things like crystallize intelligence, math knowledge, reading, writing, etc. I hope that makes some
Dr. Jeremy Sharp (54:28)
Sure.
Kevin McGrew (54:28)
sense. It’s a heart it’s a lot there, and I did provide a link in those handouts I sent you.
for people who want to look at it to read the actually read the article.
Dr. Jeremy Sharp (54:38)
Yeah, yeah, that’s a good reminder for folks who want to dive deeper. You know, these this material will be in the show notes, so you can check that out, download that. Yeah. I think, you know, there is a lot to to sort through here. But I want to definitely touch on this point. You know, you’ve talked about trying to get unstuck from G, you know, the general intelligence factor. I think that’s worth talking about because it’s
so relevant and interesting
Kevin McGrew (55:05)
Yeah.
Dr. Jeremy Sharp (55:06)
and kind of provocative. Yeah.
Kevin McGrew (55:07)
Yeah, it is it is. And I’ve published two articles where I’ve used the quote. it’s kind of getting associated with me that that that G is a Loch Ness monster of psychology. talked
Dr. Jeremy Sharp (55:17)
Yeah, yeah.
Kevin McGrew (55:18)
about, believed in, we don’t have any proof that it exists. And the the problem is is that people and when they talk i and G
The practical measure of G is the full scale IQ or the broad cognitive ability score. That’s the proxy for it. And there I wrote an article in the Journal of Intelligence in twenty twenty three with the some colleagues, and you can find that on my webpage and get the link. it was a psychometric network analysis, that there are new theories out there. well let me back up.
Factor analysis has been the workhorse of psychometrics in terms of let’s find out what the major abilities are and what to test measure. And basically in Carroll’s nineteen ninety-three seminal treatise, he kind of said this is you know, he reviewed the world’s literature from since the early nineteen hundreds, that this is the kind of the map, this is the taxonomy. And it fit the Cattell Horn model very closely. So we we put them together and called it the Cattell Horn Carroll model.
Carroll believed there was a G, Horn did not believe it was a G. N and factor analysis would always give you a G factor. But the problem is that over there’s there’s statistical G, which is the result of factor analysis, or quantitative G. Then there’s theoretical or biological or neurocognitive G, which is this belief that started
Dr. Jeremy Sharp (56:34)
Mm-hmm.
Kevin McGrew (56:34)
with Spearman that there’s some kind of mystical energy in the brain.
Dr. Jeremy Sharp (56:38)
Hm.
Kevin McGrew (56:38)
And psychologists and neuropsychologists have been
hunting for over a hundred years, over a decade, to try to find out what is the mechanism. Can we find the can we find G in the brain? And we really haven’t found it. We and after over almost 125 years of research, that we haven’t found it. And I am now of the belief of reading the neurocognitive literature that’s been more recent in the last ten, fifteen years, and the factor analytic literature that you have to think about statistical G, the thing that comes out from factor analysis. And
kinda is above the you got you got all the tests down here, you know, and there’s a big G and little arrows going down to the to the them all. That that that is similar to horsepower of an engine. Now,
Dr. Jeremy Sharp (57:23)
Mm-hmm.
Kevin McGrew (57:24)
if you take your car in to get fixed, you don’t say fix the horsepower thing.
You know, there’s carburetors, there’s you know, cylinders and there’s electrical system and all another mechanic but and but horsepower is a what’s called an emergent property. It emerges from the interaction of all the things going on in the engine and it’s something that we can quantify to say how efficient is the engine going? How you know, and
I am of the opinion now, and it’s it’s being supported by people in cognitive psychology, cognitive psychologists, not psychometric psychologists, that a lot of cognitive psychologists are now getting into uh theories that don’t have G in it v at all. And there’s one called the process overlap theory,
Dr. Jeremy Sharp (58:13)
Mm-hmm.
Kevin McGrew (58:13)
Conway and Kovacs, and then there’s the dynamic mutualism model of Vander
Bender Moss and and others, and they have models and they have explanations for for models that don’t include G, because they basically say G does not exist, or they give plausible theoretical and statistical proofs that sh explain why it exists. It only exists because there’s a positive manifold between which means when you correlate all the tests from an intelligence battery, they all are positive and correlate. It’s called the positive manifold.
When you’ve got that, a G factor is always going emerge. And John Horn throughout his whole career basically said that’s just a statistical artifact. It’s not a real thing. It’s an emergent
Dr. Jeremy Sharp (58:55)
Mm-hmm.
Kevin McGrew (58:55)
property. And I think that position, I mean, I can’t you know, I’m not gonna be the one that answers it, but I think that position is more accurate that there is not a brain mechanism that what emerges in factor analysis is psychometric G and basic eventually the I IQ score.
is that it’s like the h it’s like the horsepower. It’s the efficiency of a bunch of different brain networks working together and communicating with each other. because we haven’t found a you know the me the the mechanism. And if you follow brain network research, it’s truly exciting what they’re doing in neurocognitive brain network about the different brain networks that there are and how they communicate through white matter tracks. I think that it’s just a s the IQ score, a factored meaning is basically a
A index that emerges from the interaction of all what’s going on in our head and doesn’t represent anything quote real. So we as practicing school psychologists or assessment psychologists in different areas should de emphasize G because it’s just a practical proxy. It’s a very good one for prediction. It’s got a very practical purpose in terms of prediction. It’s probably the best predictor in all of psychology. But we’ve got to get away from the idea that.
you know, talking loosefully and had a lot of loose lips about G in talking about well this this child’s general intelligence is this. You know, talk more about, you know, he scored within this kind of range, which is a b kind of a general c practical index of the efficiency of how the person’s is organizing different cognitive abilities and making them work through different brain networks at that particular time.
and get away from more deterministic language which I I think is wrong. We sh gotta get away from G is and full skill IQ as much as we can, I think.
Dr. Jeremy Sharp (1:00:49)
Yeah, I think that’s where I get hung up. You know, I’m such a I tend to be a pretty concrete person, you know. And so I just think about, okay, if we start to move away from from G, you know, what is the practical implication then of how we interpret IQ tests and results? Yeah, w where does that leave us?
Kevin McGrew (1:01:04)
Yeah. Yeah. And I
I know ’cause I get that question and be and there are practical purposes of of an emergent property G score. There are. intellectual disability. You can’t ignore the fact that there are laws, regulations, AEID’s manual, that you know, seventy plus or minus five. You gotta you gotta work within that framework. And the score i i if you use it, just be careful with it. Realize that it’s imperfect, that if you’re trying to predict learning and achievement
There’s a big band of a prediction around there. and don’t be s so deterministic about this is the way that this is basically what this person is gonna be able to do or not do for the rest of their life. but there’s sometimes you needed a score for eligibility and I I gotta acknowledge that. But I’m now an ivory tower type person. So I I am more interested in and I proposed in my paper, doing what Richard Snow was was iterating towards in his work.
what’s called an aptitude trait complex model. That is you got cognitive abilities and in let’s say it’s in ma somebody’s referred for math problems in math, probably fluid intelligence. We got the research that shows this. Fluid intelligence, working memory, processing speed, and visual spatial are probably the the most critical from intelligence tests. Just test those abilities for now. And you’ll get a g reasonable e estimate of general intelligence if you kinda look at them all and use your clinical judgment. And if
If it looks like there you need to do a full scale I score, then you need to do the whole thing and then get get that. But just get those abilities that are important for answer the referral question. And then we need to get m better measures of motivational constructs, achievement orientation, self beliefs, self regulated learning strategies, and some of those social emotional skills that are related. And the they form not a factor, they form an amalgam.
or a constellation if you want to think about looking in the sky. They’re kind of connected. They
Dr. Jeremy Sharp (1:02:56)
Mm-hmm.
Kevin McGrew (1:02:56)
hang together and see because they are important for mathematics. And this constellation is important for reading. And there’ll be some overlap, but I I would like to see what’s more of a aptitude trait model. that’s got a lot of practical implications for special education.
Dr. Jeremy Sharp (1:03:14)
Sure. Sure. I think that starts to cross into some of my closing questions, you know, just around where are we headed from here? You know, like what what do you want to see in this in this research going forward over the next, you know, three, five, ten years?
Kevin McGrew (1:03:28)
Okay. Yeah, um I’ve thought about that a lot and I don’t ’cause I get that ask that asked that occasionally.
Well my my p primary sandbox has been it it’s in education. So I’m most
Dr. Jeremy Sharp (1:03:40)
Yeah.
Kevin McGrew (1:03:40)
attuned to special education in particular. I would like to see a more non categorical approach to mild disabilities where we s and this could p a lot of people have proposed these models. I’m not the one that’s proposing it, that we start with screening in kindergarten, first grade, second grade, and third grade with known markers for being at risk. Doesn’t it’s not diagnostic, it just says this person’s at risk, this kid’s at risk for learning problems.
in math or reading learning problems and reading. We can we can assemble a quick screening battery of those and then put those kids in special ed non-categorical special ed, just mild, you know, I don’t know what you’d call it, but for the
Dr. Jeremy Sharp (1:04:15)
Mm-hmm.
Kevin McGrew (1:04:15)
mostly the learning disabled and slow learners and some intellectual disability at the upper end at those ranges. And then monitor their progress on curriculum based measures through it what’s called response to instruction, RTI.
And keep manipulating the instruction until you can you tweak it until it works. And then those there are s kids from the RTI literature that basically are treatment resistors. That basically after a while
Dr. Jeremy Sharp (1:04:40)
Mm-hmm.
Kevin McGrew (1:04:41)
they just don’t make progress. That’s when you do the big blow big comprehensive cognitive assessments. You don’t do it to and I think we gotta do do less knee jerk assessments, you know, comprehensive assessments and do it do it to kinda
mass screening in the first two, three years. And then if they fall out, we do comprehensive and then maybe you might have to start categorizing kids into to the special ed categories because we just have to get get them service, but maybe get lobby for a period of time for either at your state level or at the federal level, through K through two in particular, maybe three. so I that’s I I’d like to see the special ed system change.
In terms of what I think are is coming or what is coming or should be coming, first getting back to the network analogy, I should I I would like to there’s gonna there’s clearly a lot more s what’s called psychometric network analysis research going on now right right now, where people are using intelligence tests and cognitive abilities and achievement tests. I have another article in twenty twenty three in the Journal of Intelligence that you can get on my web page. that
W what what they do instead of having like a factor up here and then four factors here pointing down to the test, like and it that basically makes you know says this test is under fluid and this one’s under fluid, but this one’s under working memory, this one’s underworking memory. We gotta be realistic and understand that everything’s cooked in the same pot and cognitive abilities interact. You
know,
Dr. Jeremy Sharp (1:06:11)
Right.
Kevin McGrew (1:06:11)
and the factor models don’t do that. And network models.
Dr. Jeremy Sharp (1:06:14)
Mm-hmm.
Kevin McGrew (1:06:15)
give you it will give you a network. It looks like a phishing net basically of like here’s G
Dr. Jeremy Sharp (1:06:19)
Mm-hmm.
Kevin McGrew (1:06:20)
F, here’s G C, here’s G L, here’s G R, here’s G S, and here’s reading and writing over here. And you look at the strength of the relationships which are are numerical or bolded by their size of the lines. And they suggest those psychometric network analysis is I would like to see more of that done and more interpretation based on that. It’s it’s not at a practical level yet that sa suggests that you know what?
These abilities seem to interact this way, and and explained maybe why this child is having difficulty. And for example, there’s a well-known phenomenon in the developmental psych literature in cognition called the developmental cascade. Working memory, processing speed impacts working memory, working memory impacts fluid reasoning quite a bit, fluid reasoning results in GC or GR, GQ, etc. And network.
analysis has shown that. And so we need to find out what the interactions are between cognitive abilities and achievement abilities. And I I’ve got one right here that is in this research is so that’s one thing I think we need to get away from G. and then we need what are there’s just this is just happening in the last two years. It’s what are c what I’m calling complex dynamic meta structural equation models. So what the heck is a meta structural equation model?
Dr. Jeremy Sharp (1:07:33)
Right.
Kevin McGrew (1:07:35)
Don Flanagan has got the cross battery approach. The fourth edition of that book
Dr. Jeremy Sharp (1:07:38)
Uh-huh.
Kevin McGrew (1:07:38)
book just came out. And she’s worked together with some really state of the art st people with high stature and statistics in structural equation modeling that have look have developed models of reading and math that looks at the relationships between cognitive abilities and achievement abilities. And those have been just published in the last two years in the Journal of Intelligence. You can look look them up.
And they’re basically taking correlation matrices from all the major test manuals and then synthesize them into one meta analytic correlation matrix organized per CHC. And with achievement things.
Dr. Jeremy Sharp (1:08:15)
Okay.
Kevin McGrew (1:08:16)
It’s it’s it’s that correlation matrix is is in the back of the book of the fourth edition of cross battery assessment. So I we’re getting finally uh and those things don’t model G they only have model G is just kind of it’s out here to take care of the correlations between every
abilities, but it doesn’t have an arrow to the the achievement at all. And I think we’re finally getting some good research in the last five years on how do abilities, cognitive achievement abilities interact and how do they interact developmentally. And the final thing I’ll say is and I got this and I got permission to s to share this, that I think there’s the the CHC assessment has now gone digital with the WJ five which I was
lead author. I get no royalties of it. It was a different type of contract, so I don’t get anything for s from the sales.
Dr. Jeremy Sharp (1:09:05)
Okay.
Kevin McGrew (1:09:05)
it’s a it’s it’s a tablet, iPad and a laptop. Wax flares
Dr. Jeremy Sharp (1:09:10)
huh.
Kevin McGrew (1:09:10)
have been have been, you know, that way for a while with the Q Interactive. I I don’t wanna get into all the controversies about the scoring that’s going on. that’s on your and that’s it’s on your page actually, your Facebook page.
Dr. Jeremy Sharp (1:09:19)
Yeah, that’s okay.
We ca yeah, yeah, we can save that for another day.
Kevin McGrew (1:09:23)
there I’ve been consulting informally, have no ra horse in the race or anything with a group that people need to pay attention to. And they’re called Marker Learning. I’ve been I’ve been in been I mean I’ve been contacted by a number of different people and who are working on instruments. It’s the next the next iteration of digital based assessment. And I’m extremely
Dr. Jeremy Sharp (1:09:44)
Mm-hmm.
Kevin McGrew (1:09:44)
impressed with these people are doing marker learning. If you go to their webpage,
Dr. Jeremy Sharp (1:09:48)
Mm-hmm.
Kevin McGrew (1:09:49)
they do they’ve done other stuff.
Kind gets in the way, but go to the research tab and then it’ll take you
Dr. Jeremy Sharp (1:09:54)
Mm-hmm.
Kevin McGrew (1:09:54)
to a page and then look for the box which says look at the test blueprint. They basically are now just starting norming like 30, I think 39 test battery of cognitive achievement, oral language, and psychomotor tests. And what is really
Dr. Jeremy Sharp (1:10:09)
Interesting.
Kevin McGrew (1:10:10)
interesting is what they are doing, is what I alluded to in my prelude, not prelude my preface to the cross battery book. I was asked to write the preface. Is
What if you have a digital based platform, you can record everything down to the millisecond. And
Dr. Jeremy Sharp (1:10:26)
Mm-hmm.
Kevin McGrew (1:10:27)
the Woodcock Five has got that stuff recorded, but it’s not easily accessible and you can’t integrate it into a report. It’s kind a it’s it’s been captured but it’s it’s not being utilized. These folks who
Dr. Jeremy Sharp (1:10:37)
Sure.
Kevin McGrew (1:10:38)
who who come the core technology people come out of technology and innovation, they’re building AI
into the test interpretation. Into
Dr. Jeremy Sharp (1:10:50)
Mm-hmm.
Kevin McGrew (1:10:50)
it. Not going out to f to use Grok or Gemini or something. And they have got some really cool concepts about cognitive process signatures based upon millisecond timing, you know, or when a person makes a a mistake, how how how long does it take them to self-correct or does do they so people come back and self correct afterwards? when somebody makes
Dr. Jeremy Sharp (1:11:11)
Sure.
Kevin McGrew (1:11:12)
a mistake, are they are are they they are they is it reflected an impulsive pattern of
or a reflective cognitive style. And it’s really intriguing that they’re going to be able to mine this millisecond data and type up errors and are not going to let it just sit there in the database on the platform. They’re going to make it part of the interpretation. It may not be in the first iteration, which I think is l scheduled maybe for next fall. But I am really excited that they’re taking the the digital assessment to the next level.
Dr. Jeremy Sharp (1:11:41)
That’s impressive. Yeah. I’ve seen them on a couple of conferences, I think, if we’re if we’re talking about the same company. And I wasn’t aware of the the measure to
Kevin McGrew (1:11:49)
Yeah, and I think that they started off doing some other stuff like helping people with assessments and assessment packages or uh report writing and and that kind of stuff. Yeah,
Dr. Jeremy Sharp (1:11:55)
Sure. Yeah. It’s like an AI report writer, yeah.
Kevin McGrew (1:11:59)
yeah, yeah. So I when I went to their webpage the first time I was like disappointed then it but now they got have it up under research and then you gotta go and
Dr. Jeremy Sharp (1:12:06)
Casa.
Kevin McGrew (1:12:07)
then you click on the test blueprint. so they’re kind of gone they’re letting the information out and they gave me permission. so they’re branching out and I’m I I think they’re really
Well, and the psychometrician who’s involved with it was the psychometrician and co author of the Woodcock Five. You know, s she s she knows a lot.
Dr. Jeremy Sharp (1:12:22)
Yeah. Yeah.
Kevin McGrew (1:12:25)
so I’m really excited about that.
Dr. Jeremy Sharp (1:12:26)
Impressive.
That’s great. That’s great. Yeah. Yeah. I mean, we there’s so much to talk about in terms of digital just innovation and tech in assessment. And I love that stuff. I’m like pretty tech forward as a you know, as a person. So yeah, I’m I’m excited to see where we go with with all those things in the in the next few years.
Kevin McGrew (1:12:42)
Yeah. I’ll I’ll give you one more just and then
you can you can clip that out case don’t because I just like to talk about this. There there’s a thing called ROSH scaling or IRT technology where that that ever all test developers now use for scaling items from easiest to most difficult. And you look for bad fitting items
Dr. Jeremy Sharp (1:12:58)
Yeah.
Kevin McGrew (1:12:58)
and get rid of them, you or or fit or item fit. There’s always been a statistic in in IRT called person pit. And what it does that does is it looks at the vector of a person has on subtest for the the pattern of ones and zeros and it flags
Does this pattern make sense or is it an outlier? The problem with making use of that is you have to run stuff through an IRT or a ROSH program or a program called Big Steps and set a flag for this is what the criteria is for bad person fit. They are are going to be using person fit statistics on item vectors from the subtests to say this one is unusual, it doesn’t fit and it’s it’s a whatever percent, it’s rare. Take a look.
You know, is that a sign of ADHD? Is that a sign of splinter skills? Is that a sign of something
Dr. Jeremy Sharp (1:13:46)
Mm.
Kevin McGrew (1:13:46)
something else? You know, it’s so they’re they’re actually gonna be mining. And I’ve been calling for this for like fifteen, twenty years that we gotta get this used as person fit stuff. but but it’s and then we got it in the L G five but it’s it’s not being it’s not easily done. These people starting with the idea, we’re gonna do it. Maybe not the first release, it might be the second release. So yeah.
Dr. Jeremy Sharp (1:14:08)
Yeah, yeah. Well it’s cool that the tech is finally catching up with ideas and we can we can implement, you know, some of these some of these ideas that we’ve had for a long, long time. So yeah. Well
Kevin McGrew (1:14:18)
Yeah. Yeah. And some of those and some i that
that I I keep going. That IRT stuff in terms can also flag tests that where you’re looking at possible what do they call it in forensic settings, a person is faking faking Maling malingering. Malingering. It c it can flag
Dr. Jeremy Sharp (1:14:32)
yeah. Yeah. Non credible effort. Malingering.
Kevin McGrew (1:14:36)
it can flag vectors that, you this this just doesn’t make sense.
So I’ll I’ll stop.
Dr. Jeremy Sharp (1:14:43)
Sure, sure. Yeah.
I mean the possibilities. There’s so many possibilities. It’s really exciting.
Kevin McGrew (1:14:47)
Mm-hmm. Yeah.
Dr. Jeremy Sharp (1:14:49)
Yeah. I appreciate the the resource. We’ll put that in the show notes as well for folks who want to go and and check it out.
So yeah, this has been a fantastic conversation. I mean, it’s a wide ranging conversation. but I’m super grateful just for your time. And I think there are, you know, a few voices in our field, you know, folks who have the longevity and the the expertise to
To really speak eloquently on on so many topics. And you’re certainly one of them. So yeah, thanks again for for being here and sharing everything with us.
Kevin McGrew (1:15:20)
My pleasure. I always enjoy talking, as you can probably tell from this, I kinda get wound up. my passion. Yeah, it’s my passion since nineteen seventy four to nineteen
Dr. Jeremy Sharp (1:15:26)
Getting wound up is totally okay. Right? Well that’s what we need.
Kevin McGrew (1:15:29)
nineteen eighty four. Nineteen seventy ninety four. Yeah.
Dr. Jeremy Sharp (1:15:31)
Well, you can tell.
You can absolutely tell. Yeah. Well I appreciate it, Kevin. Take care.
Kevin McGrew (1:15:35)
Okay, thank you, Jeremy. Okay,
Click here to listen to the podcast instead.