One of the real issues of wearing any zero-drop shoes casually is walking on your pants. Because there's little sole and no heel, you find yourself dragging the back of your pants and wearing through them really quickly. Maybe if you're a skinny-jeans wearer you can get away with it, but otherwise it's a problem.
Take a piece of bungee rope, a spring cord lock, and your S-biner and you have a fix. I use left over bungee from some Lock Laces and found some cord locks in the camping section of Walmart. Make it fix enough to go around your ankle and tie it off. It wraps around and keeps you from ruining your pants or from getting them wet when it's rainy.
Is it nerdy? Yep. But nobody can see it so no one has to know the lengths you're willing to go to wear your Vibrams everywhere.