You've still got action economy against them. What I did (originally I had 5 players, but they backed out until their schedules cleared up) was have each of them handle two characters. This may not work for everyone, though, since some players have difficulty handling one character, let alone two.