James Potter had family. Lots and lots of family from both sides. Or at least he did. Then the prophecy happened and Voldemort started pruning. The few survivors refused to take Harry in not because they were evil or because Dumbledore forbid it but because of how terrified they were.