There exists in our society a widespread fear of judging that has nothing whatever to do with the biblical "Judge not, that ye be not judged," and if this fear speaks in terms of "casting the first stone," it takes this word in vain. For behind the unwillingness to judge lurks the suspicion that no one is a free agent, and hence the doubt that anyone is responsible or could be expected to answer for what he has done. The moment moral issues are raised, even in passing, he who raises them will be confronted with this frightful lack of self-confidence and hence of pride, and also with a kind of mock-modesty that in saying, Who am I to judge? actually means We're all alike, equally bad, and those who try, or pretend that they try, to remain halfway decent are either saints or hypocrites, and in either case should leave us alone. Hence the huge outcry the moment anyone fixes specific blame on some particular person instead of blaming all deeds or events on historical trends and dialectical movements, in short on some mysterious necessity that works behind the backs of men and bestows upon everything they do some kind of deeper meaning. As long as one traces the roots of what Hitler did back to Plato or Gioacchino da Fiore or Hegel or Nietzsche, or to modern science and technology, or to nihilism or the French Revolution, everything is all right. But the moment one calls Hitler a mass murderer-conceding, of course, that this particular mass murderer was politically very gifted and also that the whole phenomenon of the Third Reich cannot be explained solely on the grounds of who Hitler was and how he influenced people-there is general agreement that such judgment of the person is vulgar, lacks sophistication, and should not be permitted to interfere with the interpretation of History.
Judging defines responsibility, both for the one judged and the the one judging. For the former it demands accountability for an immoral deed; for the latter it demands accountability as the one who witnessess said deed. A witness who keeps silent is irresponsible in relinquishing her judgement. Thus, responsibility lies with the doer of the deed as well as its witness. Arendt's point is that no one is willing to be a witness.
Freedom implies responsibility and demands judgement: being free requires being willing to judge, which is what it means to be responsible. Judgement is not the same as prudence or good sense: it is calling a spade a spade and holding others accountable for immoral behavior. Freedom requires individuals be held accountable, no matter their position or status or historical/social context.
I think there is an element in with-holding judgement that is not about avoiding responsibility, but realizing that the judging of others is futile without first addressing one's own moral culpability. Thus, it is not simply adherence to the metanarrative of determinancy that keeps one from providing witness to the misdeeds of another: but it is the realization that my own misdeeds require a witness too...and until I witness to my own immorality, I cannot responsibly judge another. This judging of others becomes irresponsible because it distracts and detracts from my own errors: and there are really no errors I am better able to correct than my own. Thus responsibility requires judging oneself before the judging of others.