I had a similar problem on my 04 Sienna LE power passenger sliding door. The door started to sag and was rubbing the paint at many points. I purchased an OE hinge and replaced it in under an hour. I didn't need to remove the door, just the one hinge. Since the hinge is unpainted, I used a spray can of black paint/primer. It matches the black pulley assembly so it doesn’t look bad. The technical service bulletin is very helpful to visualize the door assembly (TSB EL004-04)
1. Turned off the power door switch and opened the rear windows.
2. Manually opened the door about half way.
3. Removed the rear trim (rail end molding) by pulling on the forward area. The rear connection was difficult. I used a screwdriver as mentioned in the TSB, but it partially cracked the piece. It looks like it may slide up off of the rear plastic connector for my 2004.
4. Carefully examine the position of the cables for the power door mechanism (over and under), then remove the front and rear Brackets/tensioners.
5. Removed the pulley assembly bolted to the center hinge using a 10 mm wrench (2 bolts) and loosen form the hinge.
6. Blocked up to support the rear of the sliding door so it will not fall.
7. Loosened the 3 bolts (12 mm) holding the center hinge assembly.
8. Slide the center hinge back while pulling out the cables from the hole and slide the hinge off the rail.
9. Slide the new center hinge on the rail.
10. Pull the cables through the holes making sure they are in the same position.
11. Bolt the hinge on the door and unblock the door.
12. Bolt the pulley on the hinge.
13. Attach the bracket/tensioners (white in rear, black in front).
14. Adjust the door to make sure it slides well and lines up when closing door, then tighten the bolts on center hinge to the door.
15. Attach rail end molding.
16. Turn on power to the door switch.
Hope this is helpful. By the way, since I did not remove the door, it was a one-man job.