I feel horrible after first time sex after breakup...

I'm trying to keep this short. Not sure if this is a rant of if I want advice. Maybe both?

Broke up with my boyfriend of 9 years about 6 months ago. Moved out of the house we bought, left our cats and found a place for myself. Been giving myself time to grieve while also dealing with his stalking and mental health. For a while he manipulated me into believe that I was the only one who could help him. Eventually he finally started seeing a therapist and our contact lowered. I took some time of my job to breath and relax, find my new rhythm in live. But I've been feeling really good. Really good. No crying, no fear, just a lot of excitement for the future.

I ran into an old coworker at a work party. We always had a 'thing' but I never crossed any boundaries because of my relationship. We kiss that night and he stays in my hotelroom, but no sex. I feel save and sexy and good. We stay in contact for the next couple of weeks while we both work at projects in different countries. It's clear that our 'thing' is still there. Same interests, same music taste, same humor. But most of all a lot of sexual tension. He makes me feel so good.

Weeks go by and last weekend we find ourselves in the same city. We text and he comes over. We drink some wine, have a good time and eventually we have sex. Good sex. It was my first time since my ex and he knew it. He was sweet and kind and caring. The sex wasn't rocking my socks of, but it was really good. I would love to do it again.

The next morning he leaves and I stay on this 'high' for a little bit. And then suddenly I get a panic attack. I don't understand why and how but the room is too small, outside too much noise and I don't have the energy to move my body. I suddenly find myself in my car driving almost 2 hours to my ex's house (our house) to cuddle with our cats. They ground me a bit but I still feel horrible. I really thought I was doing great and ready for sex. But now I feel worthless. I don't really know what it is. Am I grieving my relationship right now? Do I regret the sex? I don't think so but honestly, I don't know what I'm feeling. I feel like I can't breathe. Is this what going through a breakup is? Feeling really good and suddenly feeling like total shit? When do I know I'm ready? Because I still think I liked the sex, I would love to do it again. But now I'm also really scared... How do people deal with it? What do I need to do to feel better?