Game won't let me change difficulty

I suck at fighting bosses and I can't beat leofrith. I'm trying to reduce down to easy mode, but keep getting prompted "cannot change difficulty during this activity". I've gone back several save points and still get that. Kinda getting annoyed by it.