'Patterson must go'