Whether or not a pastor stays at a church can only be determined between that man and God. I would not dare to say what God's will is in such a case. However, the reason given does bother me when I hear it. I grew up in a group where church growth was emphasized so much that it destroyed many a good pastor. They did not see the results they had been told were absolutely necessary if they were doing right, so they concluded that they were a failure. Many went from church to church hoping for that magical moment. Often, a new pastor brings an initial spurt of interest. This "growth" lasts for several months or a couple of years, then reality sets in and the church may struggle for a while. During this time, the pastor is convinced that he is failing. In discouragement he goes to a new place to experience the same thing.
Unfortunately, this changing of churches keeps the pastor from building the lifetime relationship that makes for the strongest of ministries. The church I pastor actually declined in attendance for the first four and a half years that I worked as their pastor. I felt the discouragement that comes with such a decline. When the church could no longer afford a full-time salary for me, I was sure that I was the greatest of failures. On one occasion, I offered my resignation to the people if they thought it would be best for the church. They arose in one accord to support me staying. However, the months continued to drag on without any sign of change.
But after the quarter of our lowest attendance, God sent revival. For years, we would grow a bit and then settle into what I called a plateau. However, each plateau was a little larger than the one before. A number of years ago I came back on with the church full-time. Our ministries and leaders continued to develop. Then, in the last year and a half, we have seen the strongest growth in the history of the church. We are still a small church by the standards of many, but we have a thriving ministry that reaches out to our own community and to the entire world. God has done it. I just stayed on for the ride. At one point (as I say), I gave up. All I can say about myself is that I did not quit and walk away. I did not feel that they Lord was allowing me to leave and I was afraid to do so against His will. So, how long has it taken to receive these multiple blessings? I have now been pastor of Antioch Baptist Church for over 20 years. God does bless long pastorates.
Your pastor may have a calling of God to leave the church. In this case, the Lord will surely bless you and provide a new pastor for you. However, he could be very discouraged. If you folks really do not want him to go, then you need to tell him so in strong terms. If he is really called away, he will still go. But if he is discouraged, he may see that God still has something for him there. You might even vote as to whether or not you accept his resignation. If you overwhelmingly refuse to accept it, it might make a difference. In either case, trust in the Lord. He will certainly help you.