You'd already had some fantastic responses but I just wanted to say that I've been in the same position as you, and it ended up pushing me away from leading a guild. Same problems as you, too stressful and made the game more like a job than something I do for fun. I'd say do what makes you happy. If that means a small and focused guild, so be it, and haters be damned.
