I'm thinking your mask is at fault.

How are you creating it? In Photoshop? Painting on each seperate channel?

Try using a completely red mask (255, 0, 0 or FF0000) and see how it looks (alternatively, paint white directly on the red channel and black on the others), you should get a perfect blend between your background texture and whatever your red texture is (make sure your GBA channels are all 0 in your mask).

Then try drawing a circle with soft edges using a brush with yellow (255, 255, 0 or FFFF00) as the colour on top of this (alternatively paint a nice soft white cirle on the green channel of your existing mask).


Edit: Finally add a green soft cirle to the middle of this (0, 255, 0 or 00FF00), so you have a smooth blend for red into yellow into green on your mask. (You can just erase some of the red channel to do this too)

Screens of the outcomes of all three would help.

Those look more like I'd expect.

You have to remember that the GPU will read each individual channel. (With a result in the range 0 to 1).

So, if you paint the texture fully red, it will come out with weights of (1, 0, 0, 0), meaning the output will be fully the texture defined by the red channel (although in your case it's 50-50 with the base texture).

If you paint it bright yellow (FFFF00), it will read it as (1, 1, 0, 0) you then get a 50-50 split between your red channel and green channel textures (in your case it comes out to a 33-33-33) split.

Hopefully that helps you understand how it should work a bit more, play around with it, if it doesn't give the results you expect. You could try drawing what you would like your outcome to be, and perhaps myself or someone else could come up with some shader code.