Ray picking IntersectTriangle.

I try to use in Vulkan my pick ray from DX 11
but work not perfect like in DX 11.
Who do ray picking in Vulkan ?
On my video Hit = 1 when we pick 3d plane.
https://www.youtube.com/watch?v=CWI5DHqCrEM&feature=youtu.be

In my program LeftHand matrix.
I using DX10 functions:
g_World IdentityMatrix
D3DXMatrixInverse,inverseV, 0, View
D3DXVec3TransformNormal rayD,rayD,inverseV
D3DXVec3TransformCoord,vert1, Planka2,g_World (i not show for all vert1,vert2,vert3)
D3DXIntersectTri,vert1,vert2,vert3, rayO, rayD, pU, pV, pDistance

I try my metod to transform 3d coords model to 2d screen coords.
This method work fine in Vulkan API.

But sometimes i need use Ray picking.

I’m using ray picking for triangles in OpenGL/Vulkan and it works fine with both. Please add some details. What functions are you using for ray picking in Vulkan?

Read please post #2

[QUOTE=Ronniko;40712]In my program LeftHand matrix.
I using DX10 functions:
g_World IdentityMatrix
D3DXMatrixInverse,inverseV, 0, View
D3DXVec3TransformNormal rayD,rayD,inverseV
D3DXVec3TransformCoord,vert1, Planka2,g_World (i not show for all vert1,vert2,vert3)
D3DXIntersectTri,vert1,vert2,vert3, rayO, rayD, pU, pV, pDistance[/QUOTE]

none of these are actually DX10 functions

I load DX10 dll and using DX10 functions.
I do not see anything strange.
I using function operating with matrix4x4 and vectors and IntersectTriangle.
Thats all.

Those functions come from the D3DX library, which despite the name is not part of Direct3D itself. Those functions all run on the CPU. They have nothing to do with GPU ray tracing or anything.

If you’re looking for a CPU math library or something, there are plenty of those around. Vulkan doesn’t provide that. So it’s not clear what this question has to do with Vulkan.

I don’t see the relevant parts of the calculation there. It’s not even inside a code tag.

Since these are D3D util functions, they may use a wrong z range. Vulkan has 0…1, so it’s important to see how you calculate the worldpos and raydir.

But as Alfonse said this is not related to Vulkan.

And as for Vulkan I’d go with something not related to D3D like glm.

vec.x = (((2.0f * Mousepos.x) / ScreenSzX) - 1.0f) / pMatProj._11;
vec.y = -(((2.0f * Mousepos.y) / ScreenSzY) - 1.0f) / pMatProj._22;
vec.z = 1.0f;

            // Create a view inverse matrix

D3DXMatrixInverse(&m, NULL, &CDirectXRenderer::GetInstance()->Director()->View());

            // Determine our ray's direction

vRayDir.x = vec.x * m._11 + vec.y * m._21 + vec.z * m._31;
vRayDir.y = vec.x * m._12 + vec.y * m._22 + vec.z * m._32;
vRayDir.z = vec.x * m._13 + vec.y * m._23 + vec.z * m._33;

            // Determine our ray's origin

vRayOrig.x = m._41;
vRayOrig.y = m._42;
vRayOrig.z = m._43;

D3DXMatrixIdentity(&worldMat);
//worldMat = aliveActors[0]->GetTrans();
D3DXMatrixInverse(&mInverse, NULL, &worldMat);

D3DXVec3TransformCoord(&vROO, &vRayOrig, &mInverse);
D3DXVec3TransformNormal(&vROD, &vRayDir, &mInverse);
D3DXVec3Normalize(&vROD, &vROD);

I found bug !
With DX10 function D3DXVec3TransformNormal work fine.
Its show in my video https://www.youtube.com/watch?v=lqA-aRppu7A&feature=youtu.be

But if i use my function , i see bug(video from https://forums.khronos.org/showthread.php/13143-Ray-picking-IntersectTriangle?p=40711&viewfull=1#post40711)
[b]Vec3TransformNormal(D3DXVECTOR3 *pout, CONST D3DXVECTOR3 *pv, CONST D3DXMATRIX *pm)
{
pout->x = pm->m[0][0] * pv->x + pm->m[1][0] * pv->y + pm->m[2][0] * pv->z;
pout->y = pm->m[0][1] * pv->x + pm->m[1][1] * pv->y + pm->m[2][1] * pv->z;
pout->z = pm->m[0][2] * pv->x + pm->m[1][2] * pv->y + pm->m[2][2] * pv->z;
return pout;

}[/b]

What wrong in my Vec3TransformNormal ?

Perhaps it has something to do with Direct3D using Row-Major ordering for 2D arrays (eg. matrices) and Vulkan / GLSL does not (GLSL uses Column-Major matrices as far as I know)? You could try replacing this function with an equivalent one where you swap the subscript operators, for example:


pm->m[1][0]

becomes


pm->m[0][1]

I’m afraid I’m not very familiar with Direct3D so there might be some other differences between the APIs.