this should return 10000000 but its retuning 9999997
float banana=7.0;
uint apple=uint(pow(10.0,banana));
if i set banana to a const or just put 7.0 directly in to the pow it works fine
any other number works fine (up to 8 only testing full numbers)
changing banana to an int will not fix it
i am using chrome on windows 10 if that matters
is any one else getting this or know why it happens?
The number 10,000,000 cannot be exactly represented in IEEE 32-bit floating-point.
When you use constant expressions, the compiler can use higher precision computations and store the result more precisely. But when it must use runtime evaluation, it cannot. Each step is limited to the precision of that step. And float precision is limited to float precision.
So the value pow computes is approximate. The fact that you convert it to a type that has enough precision doesn’t matter; you’ve already lost that information.
so then the question is why dose it work if banana is set to 8.0?
What does “work” mean in this context? Are you reading these integers in some kind of data structure? Are you doing a direct test of the integer with whatever you think the value ought to be?
And it’s entirely possible that the value you would get just so happens to be produced by the imprecise float followed by a conversion. This is not a guarantee of getting the wrong answer, it is a lack of a guarantee of getting the right one.
my goal of this was to basically encode data in to a number… think of a 32bit int as an array with 9 places that can store an int form 0-9
i was doing something like value*pow(10.0,pos) to set the values and something similar on the other side to decode and everything works perfectly… untill pos=7.0
i have a work around but just seemed odd so here i am
doing the same with javascript and apparently Delphi will get the expected result
maybe its just me with this error tho… any chance you can see if you can reproduce it?
The correct answer is to stop using a function that deals in numbers that don’t have the precision you need. pow is inappropriate because floats (pow’s return type) simply do not have the precision you need.
And to be honest, you only have 9 possible values, all of which are known at compile-time. So just make a const array of 9 uints and move on with your life.
yep moved on and the code runs infinitely better that before 
(texelFetch is slow af and now i have 32x less of them)
but i still find this behavior strange… as i said it works fine if the value is input as a const
it also works fine if the value is a larger variable
if it was a float error with in pow than i would expect it to be the same output regardless of a const
and i would also expect it to cause an error with a bigger number such as 8.0
if this is something that the gpu just does strangely than an understanding of it might help me avoid issues in the future and if it is a bug than it would be nice if some one else can verify it
thanks for trying to help tho