2 Corinthians 10:8

For though I should glory somewhat abundantly concerning our authority (which the Lord gave for building you up, and not for casting you down), I shall not be put to shame.

The parenthesis here is very significant, showing that the strong exercise of his authority, both in this part of the letter and in the impending visit, was not in any manner directed against the great faithful majority. It was solely for the purpose of checkmating the evil, false apostles who intended to put Paul to shame. Notice the distinction between Paul's "casting down" envisioned of the false apostles, and his "not for casting you down" when addressing the whole congregation. Those who read these chapters as a tirade against the whole church have simply failed to read it.