Surprised by results

How far back in the family tree should I be looking for a non-white ancestor? 2x great-grandfather was from New Orleans, but his father and grandfather are listed as white in the census data.