Too big for a family of four?

I am so ready to pull the trigger but a little voice in the back of my head says that this car might me too big for my family with ‘just’ two little children aged 2 and 5. So I was wondering.. everybody here has at least three kids or are their also owners with ‘smaller’ families. If so; what do you do with the third row? Fold it down? Leave it up? Just wondering..