My toddler refuses to speak and finally admitted it, but I’m not sure what to do now

My son is about to turn three and we’ve always had issues with his speech. We took him to get assessed and they said it was inconclusive. They agreed it was odd, he was advanced in some ways and seemingly behind in others. He can count, read letters and numbers, he knows animals and colors and songs and randomly says a few full sentences here and there, but it’s rare. So we’ve been taking all speech therapy advice we can apply at home. Im a stay at home mom so it’s a lot of me talking to myself (in the hopes he’ll reply and sometimes he does) and reading to him. He even memorizes some of the books I’ve read to him to read back to himself. Yet, he doesn’t speak very much especially when a group of people are around. He doesn’t even try to speak to kids who want to play with him. He just plays without speaking. Today he was getting upset, crying, yelling and finally I figured out what he wanted. My husband goes “you need to talk, buddy. If that’s what you needed you can tell us.” And he breaks into tears again and says “I don’t want to!” It’s been breaking my heart thinking he’s been struggling because he doesn’t understand entirely but he just told us! He apparently doesn’t want to talk. I know he’s struggling still, his tantrums are getting so physical with the way he throws himself instead of speaking is so concerning. Do you have any suggestions for encouraging a toddler to speak especially if they are adverse to it for whatever reason? I really want to make him enthusiastic about it, but I’m not sure how.