Where did the "no swearing" thing come from?

Just about every mormon I know treats swearing and any sex reference as some kind of unforgivable sin. Where did this even start? All the scriptures say is to not use the lords name in vain.