Unfortunately people can only speculate and no one else will be able to speculate as good as you (on this topic). The only one who really knows is the person you mention, have you asked them why? Send a polite email asking how you could have done things differently.

I am no expert on Kickstarter, I don't know what it costs etc but option 2 seems like the safer option from a financial loss point of view. Doesn't Kickstarter also hold some of your funds too in case people want their money back? That's further reason to avoid paying people upfront for their work.

I don't think you're doing anything wrong and the no pay if Kickstarter fails also helps lower risk for people. I don't think it's unfair to ask to be paid for the work you are doing but choosing programmer A (you) means less risk to programmer A and more risk for the other guy. Choosing programmer B means the risk is more even for both parties - that obviously comes at the costs of reduced future earnings.

I can only see his choice being one of risk reduction (which you took steps to reduce too). You do offer multiple platforms with Unity but doesn't Unity start to take a cut if your game starts to make a certain amount? Perhaps that's a factor too.

Note: I can never really be too sure why, so below are personal opinions, speculations and projections based on other experiences

Also your post indicates you proposed kickstarter to the artist, you didn't state if he/she feels ok with Kickstarter. So...

To my mind there are two things "the project owner doesn't want kickstarter and also the other programmer shows more belief in the artist's project".

Both of you (programmers) have different models, neither is wrong, but the royalty model is less risky to project owner (assuming the artist doesn't want kickstarter), more affordable to him/her and indicates that the other programmer believes more in the success of the project more than you do. If he doesn't want kickstarter then - What if he doesn't have upfront capital to pay you?

By going for the royalty model your rival is saying "when you earn, I will get paid, and I kind of believe your game will sell anyway so my labour would still be rewarded"

Again you never mentioned if the artist was interested in kickstarter or not. If the artist is not, then Irrespective of whether its a reasonable upfront payment or not, it is independent of affordability

As for your decision not to use your custom engine, - I doubt if this is relevant in his decision, as long as the other programmer gives the assurance it can be done with his custom engine. Don't forget he would also have the capability to use Unity anyway if need be

I can never be really too sure but slightly longer development time for the other programmer may also indicate to him -"patience to work more thoroughly on the project and less desperation".

And all these combinations comes across as a big 'plus' for the other programmer

- I Create a prototype test of the game in 1 week - Try to kickstart it - If kickstart is success

You've not listed any time/work for creating or running the campaign, which means it would've failed.

Even if you were prepared to run a real campaign, a one-week's-work prototype probably doesn't warrant two month's of fundraising work. Put more work into the prototype first :)

maybe the other programmer simply has much more experience that you do. and that fact came to light during his interactions with both you and the other programmer.

if he won't tell you why you didn't get the job, there's no real way to know.

all else being equal , i'd go with the guy who only asks for profit sharing, and no upfront pay.

maybe you were simply under-bid.