SLP Toolkit Podcast, Episode 22, Transcript
Hello, Lisa Kathman.
Oh, wow. We're doing this. Wait, let me fix my hair. I know.
Usually I give you some kind of action that says we're recording. This time I was just like, what's up yo?
Yeah, it's a good thing I'm a consummate.
No, we're not doing any of that in here. This is a confessional.
What's the word? consummate?
Commiserate? Professionals, where I was going with that.
You are a professional. Well, welcome to an episode that's live in recording. Not live, thank goodness.
Live for us.
I could cut that out if I wanted to, but I won't. So you're welcome.
'cause that would be a little work on your end?
Yeah. So Kathman, we are going to talk about something riveting today.
Well it's this time of year that everybody both loves and dreads. Cause school is almost out for most people. I know we end here in Mesa in about a week and a half, and then I think California's a couple of weeks after that. New York, maybe somewhere in June?
Some go into June, but either way we are sliding into the end of the school year.
So there is reason to celebrate. There is a light at the end of the tunnel.
Right after you complete those 9,762 progress reports that you have to do.
That's it. Progress report time. One of my favorite things Ann Paige said to us once was that progress reports are like laundry. Just when you finish it, it's time to do it again.
Yes it is. It's really, really true. That is a very accurate description. And I think that's why they're so awful is because they take a lot of time. They're very difficult. There's a lot of problems that can come when it's progress reporting time, and we'll get to those in our confession. But then on top of that, then it's time to do it all over again.
It's just a lot of work. Frown upon that.
I do too. Paperwork is not the best part of our job.
But, it can be a time-- I think with some of my kids that when I was doing progress reports, it is kind of cool to reflect on where they've been and where they're at, except for if I figure out that I forgot to take data on that goal, then I'm like, oopsies.
What? I was going to con--- Like part of me does things sometimes with these confessions, like, should I? Should I really? And then I'm like, screw it. So here it goes. There are times when I got to progress reports when I did not have data, and I may have made it up.
That's a guesstimate. I'm sure you were close.
It was a smart guess. Here's the deal is I usually wasn't that far off, like once I did actually get some data, I was usually really pretty close because I think we're good at--
Well, yeah, we are full of it. Absolutely full of it, but also internally taking data. So it's not like I was like, I'm going to go with 50% and I just pulled it out of nowhere. I would have an idea of where I thought they were, but I'm not going to lie, it might've been darts to a dartboard sometimes.
Yeah, but the hard thing about progress reports is you're having to zoom out to get some sort of data point on the annual goal. And so I think that's what gets tricky is maybe if I'm still working on some of those precursors, I've got data, but I don't have the data that I need. I mean, there are times that goals can just slip through accidentally where you're like, oopsies, forgot about that one.
Oh there were times where I literally was like, who wrote that goal? Because I've never seen it before.
But I think more often than not it's I don't have enough data or I don't have the kind of data that I need to actually report on that goal. So that's when you start to scramble and you're like, oh crap. That's where I think it's this based on what they've done on this.
Well, or you've been doing a lot of the instruction. And so you haven't maybe been taking as much quantitative data that will be helpful for reporting purposes. Or there was something else I was just thinking of that I used to do all the time. And again, this wasn't because I was a bad therapist and I didn't care, or I didn't appreciate the value of what that data was supposed to be doing for me. It was because I had a caseload of 65 kids who had oftentimes three to four goals each. And it's just what it is. You know, a lot of times we were working on those skills and I just didn't take sufficient data. I just didn't.
Well again, sufficient data for purposes of reporting on an annual goal. I think that's a very different--
That's true. I documented always, cause I billed Medicaid. So I always had documentation.
So we have data, but it's sifting through the data. It's figuring out what data points are relevant and meaningful to report on an annual goal. It's trying to determine if you even have enough information to kind of put it all together, to see how they would do on that annual goal. So it's not, I think-- again, I think there are times obviously that we just kind of, oops, forget a goal. But it's probably more often that you're taking data, it's just not a good translation or an easy translation into how to report that on the annual goal.
I do think some districts report on benchmarks. That's different. I mean, that would make it a little bit easier, I guess. Is if you are just reporting against a benchmark and not an annual goal, we just didn't have that luxury where we were working.
And still, I know in Arizona, you still are reporting on the annual goal. So the benchmarks are kind of the roadmap of how you're getting to the annual goal, but you still have to report on that annual goal. But it's tricky.
And it can be a couple quarters of 0%.
But I think that's where in my comments, I would put things where they were at in this moment in time, even if it was zero. And if it was one of those goals too, like I even think of R, articulation. I would write it for the year for conversation because I figured in a year we can get there. Right now though, it's going to take a lot of work to even build up that skill. So we'd talk to the parents at the IEP and say, you might see some zeros, I'll add some comments to tell you where they're at in those kind of lower levels of the hierarchy but that doesn't mean that they're not making progress.
That's it. That's the key. Well I remember not too long ago-- again, just dreading every time it was time for progress reports. So I usually did a couple of things. One, I would wait till the very last possible minute. So typically you get your grading in the Friday before a break, but they didn't actually report-- reports didn't actually go out til like the Wednesday after we got back, which meant I did my progress reports that Monday and Tuesday before the Wednesday.
And therapy? No therapy then?
Well, no, I would usually still see students. And then I would just in between, and after school and I mean, it would take me hours and hours. Sometimes I'd use my breaks. Not sometimes, a lot of times I used my breaks and all I remember thinking is I felt like I spent an entire day just looking at what data I had and trying to calculate an average.
and figure out what you need. What do I have? What do I need? What can I report? What can I finish?
That's true. And then I would use those couple of days back to get data if I needed it. Which that's always great to try to scramble to get some data because you realize you can't even pull it out of your ass cause you really don't have enough information. And then I'd be up real late getting all that data onto those stupid progress reports.
So what's a better way?
Well there's something called SLP Toolkit. No. It is the reason we created it. The entire program was created for progress monitoring. it's since and got other features that came along as we started to develop the progress monitoring tools. But the idea of progress monitoring like a teacher, a teacher gets the grade she needs for report cards based off of an assignment or a test that the student does. So they're not looking at the student in their day to day based on notes they've made and trying to average that into something that makes sense. They have an actual assignment that they give and gives them concrete data to report on a progress report.
Well, and an actual test. Cause I think it's almost like if they're teaching math, they're not pulling from the math homework in order to assign a grade for the semester. I mean it gets factored in as far as completion, but when they're really looking at what gives them the information of if a student is passing or failing, it's tests. It's not the mini tasks that lead up to that.
Right. So that's ideal, right? Like that quiz that they take on the Friday after a new chapter has been introduced is going to tell the teacher a lot about whether or not they're learning the material. And so based on how they do on that quiz, then you've got some information about the direction they need to go. What's missing? What aren't they getting? What do we need to spend more time on? And so I think that's why this whole idea of why don't we have something concrete to be able to get the data we need to truly measure progress accurately and efficiently.
So I know you were in the same boat as me, that when we started working in the same district, we saw goals that were written as their measurement being a criterion referenced test. And actually it didn't even say that, it said CRT, the acronym. And I was like, well, what is this CRT thing? I feel like I should know it. It's in about half of my students' goals and come to find out, I did some digging and research and it stood for criterion referenced test. And they had created tests in the area of articulation that allowed you to get a baseline on a goal and then pull out that same test to get the data you needed to do progress reports. So it's not a standardized test. It was this test that had 10 items on it, enough for you to probe the student and determine what their baseline was so you can say, okay, this is where they're at, help you determine what level of proficiency you want them to be considered as mastered for the end of the IEP. And you're just comparing the student against him or herself. So it's not like the standardized tests where we look at how they are in compared to same age peers. This is truly how they're taking the strategies that they're learning in their therapy sessions and applying it to the test. And so I loved it because it was super structured, very predictable. I didn't have to worry about scrambling for the data points or whatever. I focused on my treatment, taking notes that were aligning to what I was doing from day to day working with the student. But then for the progress report, I just pull out that binder that had those artic CRTs and felt really confident with that system that I had.
It was game changing. It was game changing, and that's why when we met in the district that had these articulation CRTs, that's when this came out is like, that's awesome. But what about other concrete areas? Like there's vocabulary goals or grammar goals that these would be great for. And so that's really where all of this was born was this need to streamline the way we collect the data for reporting on progress reports.
and they are best for concrete skills. So these tests are designed where it's sort of, if you think of in your therapy, if you can do a plus or a minus, a right or a wrong kind of response, that's the kind of goals that these work best with. And so again, a lot of different language areas that can work with. Some other ones, not so great. If you're doing something like a social goal, you might want to write a goal that is centered around a rubric, but the criterion referenced test, we have hundreds and hundreds of skills that we address from day to day in our therapy sessions that these are perfect for. The only thing that I remember having an issue with when they were in paper form is I had started to create some on my own for some of these language based tests, but I had my little manila folders in the filing cabinet and so when it was time for progress reports, I'd start to pull out and stack all of the kids that needed the testing for the week before. And that would be the thing. I would hope A, that it wasn't my last copy of a test. B, even after I administered it with a student, I had the data, but I would have to score it and get it into my IEP software. Then those files would hang out on my back counter for, I don't know, maybe two, three weeks or until my SLPA got really tired of seeing them and would file them back for me. So it gave me the system, which I really treasured. But the whole idea of it being manual, it still kind of gave me like a little pause in overall management of those.
Yeah. Cause that's what I was going to recommend, is you can create your own criterion referenced tests. I am a huge believer in why reinvent the wheel when other things exist out there? And so that's why we spent so much time creating the tests, so that you didn't have to. But you can create your own. I would recommend doing it in some kind of digital way, like a Google doc or something before I would recommend a paper copy. But even then, yeah, you're right. It's not going to measure it for you. Which that's the convenience factor is not only in Toolkit is it calculating your score for you, but it's also plotting it on a graph so you can see that trend in progress as well.
Well, I remember when we first launched, it's been just over three years, but when you had used the progress monitoring test, the criterion reference test for the first time and you were like, oh my gosh, I finished this set of progress reports for these students before the next group even came into the room.
Yeah. So that's what I did. I got a really great system. And so I knew about a week before progress reports were due. I used my therapy session time to administer each student's progress monitoring tests. So again, most of my students typically had two to four goals. And almost all of them were measured with progress monitoring tests by that point. So either a rubric, which I would score in my own time, based on the data that I've been collecting during sessions. Or the criterion referenced test. I would have some kind of activity that they would work on independently, I would test each student on each of their goals. And literally as they were walking out the door, I was recording those scores in my IEP software system on a progress report. And I was done. And I didn't have to think about it again. And so no more breaks, no more doing progress reports during the breaks. No more waiting until literally the minute they're due before I actually have data entered on them. It was the most relief-- and it wasn't even the time and just the lack of pressure. But also, I was so confident about the data that I was reporting. Probably for the first time ever did I feel really good that this data was accurate and I felt confident reporting.
So I know even when we've been at conferences, people have come up to us asking, how do you administer these? How do you find time to do these within your therapy day if you're already seeing students? I know you and I approached it a little differently. I did mine one-on-one. And so what I would do is if I had my Monday through Friday schedule and on Mondays from 11 to 11:30, if I had three students scheduled during that time, I would just let the teachers know that week before progress reports were due or the last week of the grading period. I would say, hey, I'm still bringing students in, but we're doing testing for progress reports so I'm going to call them in one by one. It will be right around the time that they come. So I would try to get those three students to come in during that 11 to 11:30 window or as close as I could, but that's how I ended up managing it. So then when one was done, I'd say, okay, go back to class and tell Johnny to come next so I can get him done. And that was a system that worked well for me, but I know you did it a little differently.
Yeah. That's what I was saying. I had them all come during their session time. I gave them busy work. I mean, not something completely wasteful of time. Obviously I respect their time. I would give them something meaningful, but something they could work on independently and I would pull them one by one over to the side of my desk
And you had a whole classroom so maybe that was part of the difference too. I had a really tiny speech room. So all I had really was my little horseshoe table or bean table, kidney bean table and not in my desk. So I wouldn't really have had room to have them working independently. But whatever works, as long as you're getting the kids in there. The other big thing with criterion referenced tests, which we kind of talked about, but it's not about teaching these students what's on the test. It is really meant to measure can they take the strategies that they've learned in the therapy sessions and apply it to that test? So sometimes there is some overlap. For example, if I'm working on a set of articulation words that all have the S sound in it, or the /s/ sound, then I might work on the word "soup" in therapy, and then it might also be a stimulus item on the test. But I'm not just going down that and only teaching them "soup, pass, eraser," whatever it is, just those certain sets of words. I'm trying to see if they can take that information and apply it to the test. And that would go for other skills too. So even like a grammar skill, we're doing grammar therapy. Whether that be with books or different activities that we're doing in the speech room, I want to see then can they look at a set of-- or this test and how it's presented and apply those skills to the test?
And I give no feedback when I'm doing it.
No, it's a test.
They don't know if it's right or wrong. I'm just marking whether or not they're getting it.
I mean, sometimes I laugh and point at them. I'm like are you serious? No, just kidding.
So anyway I would never go back to doing it any other way. It really did make this process so much less painful. And I still-- we talk a lot about data because data is important and you still need the data during sessions, as far as you've got to know whether or not your treatment's working.
Right. Where am I going to go next?
Right. And then we obviously have to document, especially if we bill Medicaid billing. So we need data in a session, but I have less pressure to get data that translates for progress reports.
Right. And depending on how often you're doing progress reports too, like I hate when the focus has to be on that annual goal. We're supposed to achieve that goal in a year. So there are a lot of things that we're doing over the course of the year that don't necessarily transfer into that big picture kind of goal. So if I'm always pausing to take data that is specific to that long term goal, I'm just wasting so much therapy time.
Now, rubrics, we didn't talk a lot about that yet. You mentioned that those are good for more abstract skills. So those are going to be anything like our social language goals or goals that are more complex than something that you can measure with a plus and a minus. And so I am obsessed with using the rubrics. It was just a really great way for me to be able to target a lot of things like narrative retell. We have a really awesome rubric for that. And so instead of just writing a goal that says "the student will accurately retell a story with 80% accuracy," which don't get us started on why that's a bad idea. But rather than just that type of goal, I have a rubric now that's going to help me measure do they include all the story, grammar elements? Do they retell a story in the right sequence? Do they use appropriate grammar and story vocabulary from actual vocabulary from the story? And there's just so much more that goes into that skill development and their overall narrative development that I can now measure with one goal and use a rubric and target multiple things at once.
Well, and I think with the rubrics, it's just like the criterion referenced test, but a different sort of "aha" in that they're just freeing. They free you up to not have to put so much stress and worry over the data you're collecting in your therapy sessions. You can really focus on what does the student need? Write a goal specific to their need, not specific to what you can measure. Because that was one of my issues that I've already confessed on here, that there were goals that I intentionally left out of an IEP because I had no clue on how to measure it. So that was my bad, that wasn't anything-- we should have been addressing certain things, but didn't. So when I had that system down for the criterion referenced test, then I recognized that that didn't work for all goals. So what could we do for those more complex kind of needs? And so the rubrics were just really a nice way to not be scared of goals. Cause a lot of times I think we're more scared of the goals and the data than we are the treatment. I could work on social skills, I've got that great curriculum through social thinking or whatever it is that I'm using. But then when I go to measure data, it can get really tricky on what types of data I need, what's meaningful, what will truly show progress, what will make sense to the team? So whether it's a parent or a teacher that's reading that. But then also when you're using rubrics, you're thinking about task analyzing that skill and the progression of it over time. So it does indirectly then end up guiding your treatment decisions too, because if you know they've got this part of the skill, but then you really want to focus on this part. We don't always see that when we're not kind of taking data in a structured way, like a rubric.
If you are not already using SLP Toolkit, I think we've mentioned this before, but you can sign up for free and use it for as long as you want for up to five students. And we did that intentionally, one because we want to make sure that this is something that actually does add value in your life and it's something that's going to help your caseload. So that gives you the time you need to really be able to go in and check out these things. But the reason I'm mentioning it is because if you log in, you have to at least add a student. Even if you want to write test student or something, just because you want to explore, get the student in there and then go to the student dashboard and you'll see something that says "collect baseline data now." That's where that library of progress monitoring tests are. And so then you can search keywords or scroll through. Again, there's hundreds of tests and rubrics in there. You might even want to just search by rubric, you can see all the rubrics we've written. And so that will give you an idea of what it is we're talking about so you can see it in action. We did just add one of our favorite features to date, and that was the ability to add your own custom CRTs and rubrics. So you can either copy existing tests and then tweak them, edit them, make them more individualized to a student's need, or you can start from scratch and build your own. So that's going to be really great for any of those goals where sometimes you have those one offs where you're like this kid needs to work on this, and it's not anything you've probably ever targeted before and so you need to write this rubric so that you can measure that kind of one-off goal. The other one it's good for is if you're using specific curriculum or vocabulary or things like that that you want to use, you want to develop an assessment around that might be another good one for criterion referenced tests. Like I think if we're looking at specific vocabulary from the curriculum or something like that.
Or I have inherited goals, especially like an artic goal where it's one of those mega goals, like the student will produce S, F and TH in sentences with 80% accuracy or whatever it is. So I can go create one criterion referenced test. Instead of doing it every sound, every word position I can maybe just pull some of those words in different positions and develop a test for something like that.
Well, and that was the other reason we added the custom feature too, is because we were starting to get requests for things that were more specific. Again, we tried to think of everything that you might ever possibly target, but that's impossible to be that comprehensive because these IEPs and goals need to be truly individualized to students' needs. And so we wanted to have that feature in there for you to be able to create progress monitoring measures that aligned with your student's specific goals. And so we have had requests for like initial positions or something like that--
or even documents that were in Spanish. We have a lot of requests for Spanish tests. So now you can go in and create those.
That's it. So we wanted to make sure that you guys had the ability to do that yourself in case it's something that we don't get in there right away. So anyway, that's in there. You guys can check that out too sometime. But we would love it-- we always loved whenever we go present, we always talk to people about progress reports and kind of average time it takes to write them. I think we hear anything from what? One to two days?
At least. Lots of tears. Lots of wine.
Lots of, yeah, those would probably make really good progress reports if you drink wine while you write them.
I wouldn't know.
So they are, they're painful. They're not fun. I'm really apologetic that we are going to publish this episode and you're going to be thinking about the ones you have to write in just a couple weeks. But we hope this is a solution that's going to make your life easier. It is exactly the reason why we created SLP Toolkit.
I don't think so.
We might get one more episode in before school gets out. I don't know. It'll depend. We still want to do our Q and A one, so send us those questions still. But if we don't, you guys have a freaking fantastic summer and I'm so grateful that your school year is almost over, cause you all deserve a big fat break.