Showerthought: let the club pick the next book, or keep my one veto?

When our Tuesday night club at the library in Dayton shrank to five people last fall, I had to decide between letting everyone vote on the next pick or keeping my one veto from the old days. I picked the veto because I was scared a 4-1 vote would drag us into 600 page fantasy books I would never finish. Turns out two members quit by January anyway and said the veto made it feel like my club, not ours. We read The Shipping News that month and only three people showed up to talk about it. Now I am wondering if giving up the veto would have kept them, or if the club was just fading no matter what. Did your group ever survive a rule change like that?