I have a lot experience tapering off Zoloft. All psych drugs or drugs that require tapering can use the same method. The safest method is to taper 10% of the previous dose every 3-6 weeks. Since you've only been on for 4 months, the rule is to not take any longer to taper than you were on the drug.
So 4 months (16 weeks) would give you about 5 reductions. So I would suggest doing a trial of reducing by 1/5 (20%) for 3 weeks and see how it goes. If you tolerate it ok, then continue with that. If not, you can reinstate and get stable, then restart with a much slower taper (the 10% one).
The idea is to make it comfortable enough that you can live with the withdrawal symptoms for several months. Some people are especially sensitive, and need longer. It's ok, you'll get there sooner or later.
If you ever decide to get off the Valium, you would do something similar.
ps. I got this information from the Paxil Progress forum. They're great if you need support coming off psych drugs.