You can't look at the fourth quarter and say "if not for the safety" they would have been down by less than a FG. Hell the safety was in the first quarter. If they don't get it, they might drive down for a TD or a FG. Hell New York might take a pick six to the house later in the drive. Play calling changes, time of possession changes, NY doesn't get the ball immediately back with good field position, so New England's defense isn't as tired.
Not trying to totally bash your argument, but it's like the butterfly effect. You take that safety away and it could have been a totally different game depending on what plays were called in the aftermath.