• Create Account

### #ActualL. Spiro

Posted 11 October 2012 - 04:50 PM

Firstly, don’t use magic numbers such as “0.0039215686274509803921568627451”.
Use:
float scale = 1.0 / 255.0;

Secondly are you aware that DitherByChance() returns values from the range of 0-255?
Reading from the texture in the shader gives you a value between 0 and 1.
Then you multiply that by 255.
For example, if the texture gave you 0.52 for red as per your example, you then do the following to it:
double rPercentChance = frac(oldColor.r / scale);
Which evaluates to the following:
double rPercentChance = frac(0.52 / 0.0039215686274509803921568627451);
Which evaluates to:
double rPercentChance = frac(132.59999999999999999999999999993);

Is this what you were expecting?
If so, why would you not just do:
double rPercentChance = frac(oldColor.r * 255.0);
instead? Multiplication is faster than division.

L. Spiro

### #2L. Spiro

Posted 11 October 2012 - 04:50 PM

Firstly, don’t use magic numbers such as “0.0039215686274509803921568627451”.
Use:
float scale = 1.0 / 255.0;

Secondly are you aware that DitherByChance() returns values from the range of 0-255?
Reading from the texture in the shader gives you a value between 0 and 1.
Then you multiply that by 255.
For example, if the texture gave you 0.52 for read as per your example, you then do the following to it:
double rPercentChance = frac(oldColor.r / scale);
Which evaluates to the following:
double rPercentChance = frac(0.52 / 0.0039215686274509803921568627451);
Which evaluates to:
double rPercentChance = frac(132.59999999999999999999999999993);

Is this what you were expecting?
If so, why would you not just do:
double rPercentChance = frac(oldColor.r * 255.0);
instead? Multiplication is faster than division.

L. Spiro

### #1L. Spiro

Posted 11 October 2012 - 04:49 PM

Firstly, don’t use magic numbers such as “0.0039215686274509803921568627451”.
Use:
float scale = 1.0 / 255.0;

Secondly are you aware that DitherByChance() returns values from the range of 0-255?
Reading from the texture in the shader gives you a value between 0 and 1.
Then you multiply that by 255.
For example, if the texture gave you 0.52 for read as per your example, you then do the following to it:
double rPercentChance = frac(oldColor.r / scale);
Which evaluates to the following:
double rPercentChance = frac(0.52 / 0.0039215686274509803921568627451);
Which evaluates to:
double rPercentChance = frac(132.59999999999999999999999999993);

Is this what you were expecting?

L. Spiro

PARTNERS