Are you running out of memory/resources while you target x86? If not then the question becomes "Is it worth it?"
That was the initial reason, yes
The market is now dominated by x64 machines, only 12% of the windows OS installbase in the steam hardware survey is running a 32bit version of windows on mac and linux there is no more 32bit version in the survey. Another reason to add x64 support is so you find those nasty bugs in your code where you are assuming that a pointer is always of size 4, these kinds of constructs in your code are bad.
One of the other reasons could also be that all 64bit cpus support SSE2 so that SIMD instructions in release builds are enabled by default, this might give you a performance increase for free.