Wizard staff advice

Hello! I'm building a wizard staff with a light-up crystal. I am not sure how to affix the crystal to the wood base. It needs to be removable because there is a battery underneath inside the crystal, but not so loose that it would fall out. I thought of using a brown air-dry clay and pressing it into the wood base, then pushing the crystal into it. Do you all think this would work, and do you think I would be able to remove the crystal easily once the clay dries? Do you have a different suggestion? Thanks!