For me personally, I hate constantly knocking things over with my petticoat (Though, I knocked things over before I started wearing lolita). I hate that I always have to worry about staining it. I also dislike the comments like "Are you in a play?" or "Why the outfit?". Then there is the pricing, and the attitude that some people have in which they think "Lolita has to be this way and no way else."
Besides that I of course love lolita, and would never give it up for things so small.