I've been dealing with the "Antis" (those who insist that this or that privatization can't work because it does not currently exist) all my life.
I was told that "gold will never be legal to own, since it will bring about the collapse of the monetary system."
I was told that "the draft can't possible be ended, since no one will willingly join the army."
I was told "homosexuality won't be legalized, since it will bring the ruin of marriage."
I suppose that if private insurance companies didn't exist, and some guy proposed that people pool their risks, the Antis would raise wild objections. "Why, that can't possibly work! Only the rich would benefit! Besides, it hasn't already been done, therefore we KNOW it can't work!"
I have absolutely no idea exactly how private justice would work. Probably much the way private justice systems currently work (mutual arbitration). But, maybe things will work out differently in a free market. In any case, it's doubtful it can be any worse than the current government monopoly, in which 1) all participants (police, judge, defense, prosecutor, jailer) work for the same agency, 2) laws are too complex and numerous for anyone to actually understand, and 3) the government actually has a vested interest in people violating as many laws as possible.
However, to be fair, I am not an anarchist, but a minimalist. I do think that--at least for now--a minimal state is needed; one strictly limited to operating the police (though using private contractors, as is done in some municipalities now), a court system (as back-up to private arbitration), and an all-volunteer military (for purely defensive purposes). This should be paid for through non-coercive means. (Which is the subject for another thread.)
I'm simply discribing the theory for a non-government justice system.