normals from 'distance' cubemap


I have a cubemap where I store distances from the center of an object and I need to create a normal based on sampling adjacent distances, computing the positions & then the normals.
But I get moire patterns & slightly wrong normals with the method I use.

In a 2D case, ok I would sample 2 or 4 or 8 adjacent samples, the direction of sampling is well defined ( x/y axis in texture space), so everything is ok.

In the cubemap case, how should one select the adjacent samples?
Since the initial sample vector is 3d vector, how could I define the ‘plane’ of sampling? ( say a 3x3 region).

I guess my moire patterns are the result of the continuous change of orientation of the sampling ‘plane’ x-z axes.

I should note that I perform the computations in world space
(because it helps for the cubemap sampling)

Any ideas for a case like this?

Many thanks,