So the question is which is more plausible:
1. You perceive the table get bigger and infer that you are getting closer, or
2. You perceive to be getting closer and infer that the table is getting bigger, or
3. You perceive nothing and infer both, or
4. You perceive both and infer nothing.
I think 2 and 3 are extremely implausible. So it's either 1 or 4.
But there's a problem with 4. Can you really perceive distance? Wouldn't that entail that everyone should know how distant the Moon is? It seems you must move towards the table and check how fast its size grows before estimating how distant it is. If you need observation + movement + observation before perceiving your distance to a mountain, it seems a lot more likely that we are inferring rather than perceiving distances.
Anyway, cool thoughts!