There will be those who will insist on their view, or their accusations, irrespective of what the overwhelming majority of the church think. But in the end, we have to let go of our reputations. Ultimately, it is not before every accuser that we stand or fall, it is the judgement of Jesus that really matters. We cannot be in ministry and avoid accusations and, often, we will not find that we are ultimately able to make ourselves fully understood. Sometimes people will think badly of us and we will not be able to do anything about it.
Everybody wants a good reputation. Nobody enjoys people speaking ill of them. And let’s face it, the Lord Jesus expects elders in his church to be ‘thought well of by outsiders’ (1 Timothy 3:7) and ‘beyond reproach’ (Titus 1:7). Reputation matters.
Of course, that doesn’t mean an elder must be universally liked. As Kevin DeYoung helpfully explains here:
If [outsiders] think my blog is whack, my views are repulsive, and they believe all kinds of nasty things about me (which I hope they don’t, and I think they don’t), that would not mean I have fallen foul of 1 Timothy 3:7. If, however, most of the outsiders who know me from school or from the restaurant or from the pool know me to be rude, untrustworthy, undependable, and hypocritical, then my church should take notice. The key, I think, is that even if a pastor cannot have a good reputation with outsiders everywhere (probably impossible for anyone with more than a handful of Twitter followers), he should be respected (even begrudgingly) by the outsiders who see him up close.
He argues the same is broadly true within the church too:
If the requirement to be “above reproach” focuses on the discernment of the local believers, the qualification to “be well thought of by outsiders” concerns the wider non-believing community. Again, knowing what we do about Jesus’ public ministry, the requirement must not be pressed to mean that the elder must be universally beloved by the unregenerate world. Rather, the issue for us, as it was for Ephesus, is that “the leadership of the church should bring no unnecessary disrepute upon the church through improper and immoral actions” (Mounce, Pastoral Epistles, 183).
Whenever people make accusations against elders, the reality is that these things need to be weighed and judged by the church. Independency demands that these things are not outsourced to others, but ought to be weighed and judged by the church, within the church itself. Being above reproach and well thought of by outsiders i.e. not a hypocrite is judged by the church itself.