Basically, if the pitcher is missing his spots by a lot, then you take him out.
With a pitcher, control is always the first to go before velocity, so managers/pitching coaches watch for missing way out of the strike zone before they look at anything else. Pitch count is an individual thing, so it could factor in depending on who the pitcher is. In the postseason, pitch count doesn't matter as much- as the saying goes, you've got all winter to rest your arm.
So, you are correct when you say that Washington saw something he didn't like- and that's what it was. Loss of control. MLB managers are going to realize this and make a move before it gets even close to being a dangerous situation- or at least they should. I can almost guarantee you that had he left Holland in, it would have gotten dicey. Dicey as in runs being scored.
The Cardinals being down like they were- their hitters were going to take at least one strike. When you have a pitcher that is losing his command and is tired and the batting order has seen that pitcher 3-4 times- advantage for the hitters. And odds are real good that they would have gotten into favorable hitting counts- which means that they are going to have to throw fastballs, which a lot of times would be up in the zone because when a pitcher is tired, his arm is going to drop, and when that happens the ball goes up in the strike zone. You get a ball up in the zone to Pujols Holliday, or Freese, and it can change the game real quick.