Here is an article that looks into what Pastors really think and feel.
The author pretty much nails it!
Here is the link, or find it below:
http://pastormatt.tv/2014/04/09/10-things-pastors-hate-to-admit-publicly/

#1. We Take It Personally When You Leave The Church.
It’s just a straight up fact. We pastors eat, drink and sleep the local church and with that have a deep desires to see it thrive. Therefore when you leave to another church because…

• you’re bothered by a recent decision, but didn’t ask about it…
• the new church has a bigger and better kids wing, youth group, worship team, building space, (fill in your blank)…
• your friends started going there…

… it hits us personally.
For us it feels disloyal, shallow or consumer driven. People affirm that church is a family, thus when you up and leave because the church down the road has Slurpee dispensers, a fog machine or it’s just cooler, well it jams us pretty deep.

#2. We Feel Pressure To Perform Week After Week.
The average TV show has a multimillion-dollar budget, a staff of writers and only airs 22 weeks out of the year; that’s what we feel we’re up against. Where the pressure is doubled comes from the previous point. We know there are churches near by with a multimillion-dollar budget or a celebrity pastor who have the ability to do many more things at a much higher level. From this a sense of urgency is created in our mind to establish the same level of quality, option and excellence to meet the consumerist desires of culture.
Now if this were exclusively in the hopes of reaching new people this wouldn’t be so bad, but increasingly pastors feel the need to do this just to retain people who may be stuff struck by the “Bigger and Better” down the way.

#3. We Struggle With Getting Our Worth From Ministry.
When the numbers are up, the complements are flowing and the people are lively we feel great. When everything is level, it feels like it’s in decline. When things are actually in decline, it’s a full-tilt tailspin in our soul. We almost can’t help but equate the growth of the church with our ability/inability to produce growth. Therefore if there is any appearance of waning we feel defeated and wonder how long before the church board wises up and trades us to another team. The “Idol of Ministry” comes on and off the shelf pretty regularly in a pastor’s office.

#4. We Regularly Think About Quitting.
This comes in two very different forms.
One form is the variation of perhaps leaving ministry all together. While there are some really great things about vocational ministry, there are also less enjoyable realities such as: pastors’ families are noticed (i.e. judged) routinely, pastors’ purchases are observed (i.e. judged) overtly and pastors’ words are weighed (i.e. judged) consistently. Therefore the ability to hide among the masses and not be noticed is very appealing.
The second form comes with the desire for a change of scenery. Pastors are shepherds, thus we love greener grass even more than sheep. To leave for a bigger budget, better building or a place with less difficult people (yeah, we get delusional sometimes) stands out as lush Kentucky Bluegrass when contrasted with the dusty patch of ragged earth called “our current church.” This “Greener Grass Gawking” usually occurs when we become too proud (“My gifts are better than this place”) or too insecure (“I stink and just need to start over”) and flows from #3.

#5. We Say We Are Transparent – It’s Actually Opaque.
Today pastors are generally more open about their struggles than previous generations, but we still sense there is a threshold that is not to be crossed. People want open, honest and real, but not too much. Generally churches want just enough so they feel safe with you, but not so much that it spoils the expectations they have of you. Unfortunately the threshold is a blurry line by which pastors never know how much is too much until its too late. After a couple of infractions we learn that opaque is safe – even if it’s isolating.
When pastors’ wives are polled on how it feels to be the spouse of someone in full-time ministry the #1 answer is one profound word, “Lonely.” They are around hundreds of people every week, but they never feel they can let their guard down because they know people have opinions on how a pastor’s wife should be. Now I know people say they don’t, but literally every church I have served in has shared unflattering stories of the previous pastor’s wife. Many of these stories came from the spiritually mature leadership who considered the pastor and his wife to be their friends. The real irony comes in when later in the conversation I would be told, “But don’t worry, we don’t have any expectations on your wife. We just want to love on her.” Right! Now I don’t blame people for this natural human tendency, but being aware of how things are keeps you relationally opaque. And it’s not merely pastors and their wives who insulate. Pastoral families at large feel alone because there is a certain level of unknown expectations buried like landmines through the field of the church and so there is a constant mode of mostly transparent.

#6. We Measure Ourselves By The Numbers.
Numbers don’t matter! Yeah right. No matter how badly we want to slap that bumper sticker on our Ford the reality is that numbers matter to us. And they matter to us it part because they matter to God. The problem however goes back to #1-3. The absence of growth in our churches can cascade into an internal turmoil by which we begin to scrounge for “The Next Big Thing” that will bring “Radical Growth” “Guaranteed.” So we read books on how to be a “Deep & Wide, Vertical, Purpose Driven, Radical Reformission, Creature of the Word, Big Idea, Center Church.” Then we jet off to a conference with thousands of other pastors who are seeking to glean the secret of success. And what is the first question we ask one another between sessions? “So, how big is your church?” Yep, we measure ourselves by the numbers.

#7 We Spend More Time Discouraged Than Encouraged.
Occasionally people say to me, “Must be awesome to get paid to study the Bible all day.” Every time they do I think to myself, “Must be awesome to be able to give someone the finger on the 520 without people saying, ‘The pastor at Redemption Church flipped me off today during rush-hour.’” I’m not fully sure why that is the comment that flashes across my mental dashboard, but I think part of it stems from what I perceive to be the tone of the comment. Rightly or wrongly I infer they are saying, “Must be nice to have such a cush gig as a paid quiet-time.” In all honestly it is pretty awesome to be paid study the Bible, but it’s a major downer when people:

• tell you – after 2 minutes of un-investigated reflection – that your 30 hours of study and 2 collegiate degrees were wrong.
• tell you that they just couldn’t stay awake today during your sermon, but no offense. (How about I fall asleep at your kid’s graduation and we’ll call it even.)
• tell you how you should have also said…
• tell you how Pastor So-N-So says…

Aside from these particular examples I find that for most pastors it generally feels like the boat is taking on water more than racing with the wind – regardless of size or rate of growth. Lead pastors particularly suffer from this since much of their job is to focus on seeing things get better, which often translates into focusing on the broken, lacking or unfilled parts of the church more than enjoying what is right and working. Many of the most faithful and fruitful pastors in history have suffered deeply with anxiety and depression for the same reasons.

#8. We Worry About What You Think.
We’re human and we want to be liked. Therefore when we know we’re going to do or say something people won’t like, we worry about it. Now when I say that I don’t mean to infer that it causes us to avoid the hard things. There are some of my fellow pastors who avoid challenging topics or decisions out of fear of people, but most of the ones I run with still choose deliver the mail regardless of the popularity of its message. Yet we still worry about how you may take it.

#9. We Struggle With Competition And Jealousy.
We like to hold ourselves above the petty fray and reiterate, “It’s all about the Kingdom,” but in reality pastors are a competitive bunch. As soon as one pastor asks another, “How big is your church?” the game is on if the two churches are within 20 miles of each other (past 20 miles we lighten up a lot and think each other is pretty cool). Within 20 miles however we begin to assess one another’s style, focus, message, sophistication and marketing. We gauge to see if it’s a “Goldilocks Church” – not to deep, not too shallow, but just right (like us). If you’re too deep we benchmark you as internally focused. If you’re too shallow we brand you as consumer-driven. If however we conclude that you too are a “Goldilocks Church” we then figure out how our church is still better than your church. If you have lame amenities, we critique that you will never grow until you reboot that 70’s sanctuary. If you have awesome amenities, we criticize that you grow only because people are shallow and care more about stuff than Scripture.
Yes we know it’s not right. We know that it’s ego driven, but we still fall victim to it. We believe our church is the best church ever and we can’t understand why everyone doesn’t see it.

#10. We Feel Like We Failed You More Than We Helped You.
Most pastors will never be famous. Most churches will never break the 100 mark. Yet we all entered ministry to change the world and reach the masses. With this we know it is the expectation of churches that we accomplish this very thing. Every job posting reinforces the idea with the sentence, “We are looking for a man that will take our church to the next level.” Then when the next level isn’t hit in the way anticipated or within the timeline envisioned – we feel like we failed you. This is especially true in light of the reality that we are our own biggest critics. We came in with expectations higher than anyone in the church. You look to us for direction and when we feel like we failed to produce we feel like we failed you.