it might be an interesting dabble into science fiction to open another topic and try and fantasize what the arrow of time actually is and why we perceive it the way we do. Because I don't believe anyone really has an idea.
I was under the impression that the 'arrow of time' was simply entropic decay, leading the universe to move from one state to another (order to disorder if memory serves). So given two space-time reference points one in the past (A) and one in the future (B) to go from B to A would require that you could get back to the state the universe was in at point A, which I'm pretty sure is covered as a 'no go' in at least one 'law' (Thermodynamics springs to mind, but dont' quote me, I'm going on half remembered stuff). Or to put it another way; at 4:08pm I break an egg, any attempts at time travel to 4:07pm would require going back to a point where the egg was no longer broken.
This really is an interesting topic and my post will quickly spiral out of control, but that's only because there's no real answer to this question.
First off, allow me to make a small correction to the terminology you used: since entropy is a measure of disorder and the universe started out in a state of almost perfect order, then entropy is necessarily growing, not decaying
. But that's that.
Strangely enough entropic growth cannot be proven to be the cause of the arrow of time as it would actually create a causal loop: entropy cannot grow without time. Unless your example regarding the broken egg is a coincidence, it seems you're familiar with Brian Greene's The Fabric Of Cosmos (if you're not, then he discusses the arrow of time very thoroughly - it's quite an exhilarating read). While his argumentation largely explains that since pretty much all of physics (with the exception of some negligible effects from the weak force) is time-invariant (in other words the laws of physics do not forbid time to run in reverse, causing the egg to unshatter), there is no apparent reason for entropy to grow.
It might seem logical that a uniform mist of particles that contains impurities has the potential for entropic growth, but that doesn't quite explain why a) the universe couldn't be experiencing decreasing entropy instead (actually the answer to this would be negative gravity and the reversal of all other forces in nature) and b) why the two would be linked (at this time the cyclic universe is no longer considered a viable option due to its rate of expansion, but a cyclic universe would actually have required the reversal of time to achieve the big crunch). A better answer could be to interpret time as a potential of entropic growth, but again there's no proof of this anywhere other than the fact that the properties of forces cause it to go down this route.
Consider another example that stems from the bubble multiverse theory: "infinite time on the outside equals infinite space on the inside" (this is why a bubble universe might seem infinite on the inside, but is finite on the outside). This in turn brings up two problems to which there are currently no solutions: a) (our) mathematics doesn't have a branch that can reliably handle infinities meaning that we cannot explain how something that has no beginning and no end can flow from one to the and b) and logically, if an increase in entropy is a causal property (not a side effect) of time and there's a finite amount of matter/energy (either in the universe or the multiverse), then at one point you'll still reach the Russian doll situation where you need to explain the starting state of the universe/multiverse and will eventually run out of time as entropy reaches a maximum.
This is all bollocks, though as the ugly truth behind it is that time is a quality/dimension/phenomenon/river we simply do not understand.
As a small thinking exercise consider a simulated universe (khm, a computer game) where, at each iteration, you increase all values by a certain (fixed) amount. This amount doesn't even need to be based on time in our universe - it just happens at every iteration. This means that to the characters in the game time would seem to be flowing at a constant rate. The questions to answer are:
1) why are you increasing (not decreasing or randomizing)
all the values at each iteration?
2) would your game character ever hope to understand your nature or your motives?
3) this one is also from Brian Greene: can you yourself imagine a universe without space and time?