Nothing, the problem is not with reality (of course :-). The problem is that most people seem to think about freedom as something positive. You can only create freedom for Joe by coercing Jane to not violate it. This enables the kind of scenario that the parent to my original comment describes.
Edit: thanks for the clarification, koningrobot,