Are tones that important when speaking Vietnamese?

I’m starting to learn Vietnamese after years of dating my girlfriend (who is viet). I never had to learn it but after picking up a few words here and there, I was frustrated not to be able to make sentences out of them, and say a few random things to my in-laws.

I’m learning on Duolingo, which doesn’t force you to use the right accents and tones when writing. Also, I don’t really care about the tones at this point, and simply learn the words.

Is this the right way to go about it? As a white guy, obviously not native, would a Vietnamese struggle to understand me if I speak with a « flat » accent ?