How do you handle students using your office as an escape?

I am new to my school, though I’ve been a counselor for about 3 years. My office is still a novelty to my new students. I’ve had a couple students in my office who were in need of a place to calm down and express themselves. Then I’ve had kids who needed to take a moment to calm down, but now associate my office as a place to go when they don’t want to be in class.

For instance, I have one 1st grader who has come to my office twice for legitimate reasons, but now he’s constantly asking to come to my office. He won’t do work when he comes to my office, doesn’t have anything he needs to talk about, and would rather play.

On the other hand, I have a 5th grader doing the same thing. It’s sometimes easier to coax the 1st grader back to class, but the 5th grader is a bit more stubborn and will start disrupting class in order to get the teacher to call me.

How do you determine if a student /really/ needs to see you or if they’re just trying to get out of class? I know I’m still getting to know my kiddos as the new counselor with a new office, but I’m hoping the novelty will wear off and my kiddos will just see me as another adult who is here to help!