Seems Corel bought it up, so Corel Painter is the actual name.

For your game i'd create a quick prototype to see if it's fun and challenging. Probably you need additional ideas to make it interesting.

To measure the color error you can just treat a RGB triple as a 3D point and take the distance to the target color. The colorwheel image just makes this unnecessary complicated (2D color wheels always miss one dimension, your image has hue and brightness but no saturation.) I would measure the error in hue, brightness, saturation values and rate hue error higher than brightness, and brightness error higher than saturation. Probably this would be more fair as it's closer to human perception.

I would not stick to CMY as primary colors at all, you can just use any colors.

I will make a quick prototype at some point soon, I wont be able to for a day or two though. I am pretty sure this mechanic can easily be made challenging, possibly too much so, hence the error margin. The fewer colours, animated elements and the more 'dry' space (space lacking wet paint before your own trail) the easier, but the more colours, animated elements and little to no dry space and you're going to have a hard time mixing the colour you want before getting tangled up in your own paint trail. Especially if you make a mess of the colours lighter than your target then it could become impossible to solve without starting over.

The computer would use an appropriate colour space, but users would be presented with the cyan magenta and yellow as primaries and it would be using a subtractive model as in real world paint mixing. A painter of a physical medium should be able to pick up the game and mix colours in a way that's relatively familiar. Painters usually use a colour wheel, though they often for some reason use the inaccurate ryb wheel some use cmy which are more accurate primary colours.

It is essentially just a pain mixing game, where the player needs only knowledge of the colour wheel and is provided some target colour they need to mix. The point it becomes a more complicated challenge is when I forbid them from lifting the brush from the palette, so if they move the brush they paint a trail behind them if their brush is soaked and if they leave their brush in the path of some animated element then even if they don't move the brush then it will become mixed in a potentially undesirable manner.

Essentially it just colour picks from the pixel below, mixes that into the colour on your brush, then paints the pixel it read from with that colour on the brush. so everywhere you drag your brush you adjust the colour on your brush and leave a trail of new colour behind. A trail that can be used for mixing later if needed, or a trail that can become an obstacle if you're not careful.

cyan magenta and yellow make sense for a painter in real world colour mixing so it makes sense to me that I should use that as my colour model. It also seems to make sense to me that you would get rewarded for the correct value range, and similar hue and chroma/saturation etc.

Ok, here are some problems i expect related to colors:

Color perception depends on the surrounding of a focused color, this could irritate the player, he might think he's on track and at some point he may feel frustrated if the error is larger than he thinks and it feels unpredictable or unfair. (All games i know that use color mixing for puzzles have those issues, but i can't name examples.)

Humans can detect an accurate value for hue (we can say if a red tone goes more towards yellow or more towards blue). Brightness and saturation are only relative to another color, but we have no way to 'number' it for a single color. (So you could make early levels easier by putting the focus on only two primary colors).

Many people might have no idea that e.g. magenta and cyan mixes to blue. They are used to their water color set where 6 basic colors are available and they didn't need to know how to mix with only 3 colors. You need to train them, again by focusing on 2 colors initially. (Or provide them full spectrum of 6 basic colors RyGcBm - i don't see a need to display numeric color values to the user, so there is also no need to limit your game to only 3 of them.)