Ok I got it now, so (i,j) is you ellipse parametrization inside the the square.

Conics are very far from me at the moment so i can only help you in a general way.

First I assume you are using a sahder to perform these operations that seems to be done per fragment.

Second, you will have to compute the normal at each sprite (what you have done already) plus a tangent vector which is a vector on the sprite. In you case the tangent vector could be one of the sprite edge. normals and tangent should be normalized and given to your vertex shader in object space.

You will also need the matrix Mw that transforms your vertices from object space to world space (actually, this one contains all translation, rotation, scale operations)

Then you will have to transform your tangent and normals vectors in world space and vertices too. This is important to have all the needed data in the SAME space.

So, say that (i,j,n) are the sprite basis vectors (stored in world space after Mw matrix multiplication - n is the normal vector) and (x,y,z) the world basis vectors.

Your vertices coordinates are in world space after multiplying then by the matrix Mw and you need them in the sprite space to check if then a fragment is in the ellipse or not. So you have to compute the matrix P that transform world coordinates into sprite space coordinates.

This matrix contains in each columns the vectors i,j and n respectively.

then:

sprite_space_coord = inverse§ * world_space_coord.

after that, you use the coordinates sprite_space_coord to check if they are in the ellipse or not.

I hope this is quite understandable, don’t hesitate to ask more questions.