We have the '05 XLE, and also have the servo clicking noise.
Supposedly it's not too hard to get in to replace the mechanism on either side -- however someone a while back posted a "quick-fix" of just turning the heat up to full for a minute or so, and the heat would loosen up the mechanism and stop the clicking for several weeks or months. Not a bad remedy for something that could cost you hundreds of dollars for parts and hours of your DIY time.
Fyi there is also a gurgling noise when I rev the engine high coming from a stop sign .. but I assume that's an air bubble in the coolant somewhere within the heater pipes, or water rushing down the radiator pipes, and I'm not too concerned about that.