Adreno 506 vkCreateComputePipelines has returned VK_INCOMPLETE

AndreyOGL_D3D · May 25, 2019, 1:42pm

Hi All, I have problem on Adreno 506.

vkCreateComputePipelines has returned VK_INCOMPLETE. I have no error message from Vulkan debug layers,
but i see an error message from Adreno Driver(Android Studio/logcat):

31569-31637/com.qualcomm.vulkan.compute.d I/Adreno: Shader compilation failed for shaderType: 5

Sample code:

    void VkSample::Test(VkDevice mDevice)
    {
    	const VkDescriptorSetLayoutBinding vkDescriptorSetLayoutBinding[4] = {
    			{0, VK_DESCRIPTOR_TYPE_STORAGE_BUFFER, 1, VK_SHADER_STAGE_COMPUTE_BIT, NULL },
    			{1, VK_DESCRIPTOR_TYPE_STORAGE_BUFFER, 1, VK_SHADER_STAGE_COMPUTE_BIT, NULL },
    			{2, VK_DESCRIPTOR_TYPE_STORAGE_BUFFER, 1, VK_SHADER_STAGE_COMPUTE_BIT, NULL },
    			{3, VK_DESCRIPTOR_TYPE_STORAGE_BUFFER, 1, VK_SHADER_STAGE_COMPUTE_BIT, NULL },
    	};

    	VkDescriptorSetLayoutCreateInfo vkDescriptorSetLayoutCreateInfo = {};
    	vkDescriptorSetLayoutCreateInfo.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_LAYOUT_CREATE_INFO;
    	vkDescriptorSetLayoutCreateInfo.bindingCount = 4;
    	vkDescriptorSetLayoutCreateInfo.pBindings = vkDescriptorSetLayoutBinding;

    	VkDescriptorSetLayout vkDescriptorSetLayout = VK_NULL_HANDLE;
    	VkResult res =  vkCreateDescriptorSetLayout(mDevice, &vkDescriptorSetLayoutCreateInfo, NULL, &vkDescriptorSetLayout);

    	VkPipelineLayout vkPipelineLayout = VK_NULL_HANDLE;

    	VkPipelineLayoutCreateInfo vkPipelineLayoutCreateInfo = {};

    	vkPipelineLayoutCreateInfo.sType = VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO;
    	vkPipelineLayoutCreateInfo.setLayoutCount = 1;
    	vkPipelineLayoutCreateInfo.pSetLayouts = &vkDescriptorSetLayout;

    	res =  vkCreatePipelineLayout(mDevice, &vkPipelineLayoutCreateInfo, NULL, &vkPipelineLayout);

    	VkComputePipelineCreateInfo vkComputePipelineCreateInfo = {};

    	VkPipelineShaderStageCreateInfo& vkPipelineShaderStageCreateInfo = vkComputePipelineCreateInfo.stage;
    	vkPipelineShaderStageCreateInfo.sType = VK_STRUCTURE_TYPE_PIPELINE_SHADER_STAGE_CREATE_INFO;
    	vkPipelineShaderStageCreateInfo.flags = 0;
    	vkPipelineShaderStageCreateInfo.pNext = NULL;
    	vkPipelineShaderStageCreateInfo.stage = VK_SHADER_STAGE_COMPUTE_BIT;
    	vkPipelineShaderStageCreateInfo.module = CreateShaderModuleFromAsset("shaders/spirv.spv");
    	vkPipelineShaderStageCreateInfo.pName = "CullObjects";
    	vkPipelineShaderStageCreateInfo.pSpecializationInfo = NULL;

    	vkComputePipelineCreateInfo.sType = VK_STRUCTURE_TYPE_COMPUTE_PIPELINE_CREATE_INFO;
    	vkComputePipelineCreateInfo.layout = vkPipelineLayout;
    	VkPipeline vkComputePipeline = VK_NULL_HANDLE;

    	res =  vkCreateComputePipelines(mDevice, NULL, 1, &vkComputePipelineCreateInfo, NULL, &vkComputePipeline);
    }

Original Shader was compiled using glslang tool(GitHub - KhronosGroup/glslang: Khronos-reference front end for GLSL/ESSL, partial front end for HLSL, and a SPIR-V generator.)

Simple project(Android Studio) are based on Vulkan-Compute sample from Adreno Vulkan SDK:

See Sample.cpp, line 634, method void VkSample::Test. I attached original shader CSShaderGPUInstanceCulling.hlsl, Spir-v shader:
spirv.spv

I have no problem on Mali G71, G76, AMD(Windows/Linux), nVidia(Windows), Intel(Windows)

What do you think about it ?

Sascha_Willems · May 25, 2019, 5:09pm

From my experience with lower-spec Adreno SOCs this may be a driver bug, so unless the device can be updated with newer drivers you may be out of luck.

But can you check if you see something like “location/component mismatch” in adb after the shader compilation failure?

And are you using spirv-opt to optimize SPIR-V? If so, try without this.

AndreyOGL_D3D · May 25, 2019, 5:45pm

Hi @Sascha_Willems, how should me check something like “location/component mismatch” using adb ?

No, i use code are based on GlslangToSpv function from glslang tool:

	glslang::TShader shader(EShLangCompute);
    	glslang::TProgram program;

        // Enable SPIR-V and Vulkan rules when parsing GLSL/HLSL
        EShMessages messages = static_cast<EShMessages>(EShMsgSpvRules | EShMsgVulkanRules | EShMsgReadHlsl);
        shader.setEntryPoint(entryPoint);
        shader.setStrings(shaderStrings, n);
        shader.setHlslIoMapping(true);

        result.size = 0;
        result.data = NULL;
        if (!shader.parse(&Resources, 100, false, messages)) {
        	Helper::OutInfo(shader, "parse");
        	return false;  // something didn't work
        }

        program.addShader(&shader);

        //
        // Program-level processing...
        //

        if (!program.link(messages)) {
        	Helper::OutInfo(shader, "link");
        	return false;
        }
        static std::vector<unsigned int> spirv;
        spirv.clear();
    	glslang::GlslangToSpv(*program.getIntermediate(stage), spirv);

AndreyOGL_D3D · June 7, 2019, 5:41am

I fixed compute shader issue for Adreno 506. I should use the same compilation options that is used in glslangvalidator.exe. I should fix code used for shader compilation with glslang tool. Anyway Adreno has a bug inside driver.

jgavris · June 13, 2019, 8:10pm

I’m seeing a similar issue with some of our OpenCL shader modules compiled with the latest revision of clspv (https://github.com/google/clspv).

The Adreno GPU’s return VK_INCOMPLETE, while some other phones we have with ARM GPU’s (Samsung Galaxy S10 for example) run these shader modules happily.

AndreyOGL_D3D · June 13, 2019, 8:44pm

Hi @jgavris, I have found problem, Adreno driver cannot compile SPIR-V code for Compute Shader without spirv tools optimization. I compile hlsl code on Android using glslang tool without ENABLE_OPT macro.

jgavris · June 17, 2019, 3:16pm

@AndreyOGL_D3D Thanks for the reply. Your message was a little unclear to me. Did you find it only works with or without spirv-opt?

AndreyOGL_D3D · June 17, 2019, 5:07pm

I use full code of glslang tool without ENABLE_OPT macro in 3DEngine project, so the final spirv code will not have optimizations.

This macro requried SPIRV optimization, so you must add spirv-tool project to glslang:

void GlslangToSpv(const TIntermediate& intermediate, std::vector<unsigned int>& spirv,
                  spv::SpvBuildLogger* logger, SpvOptions* options)
{
....
#if ENABLE_OPT
    // If from HLSL, run spirv-opt to "legalize" the SPIR-V for Vulkan
    // eg. forward and remove memory writes of opaque types.
    bool prelegalization = intermediate.getSource() == EShSourceHlsl;
    if ((intermediate.getSource() == EShSourceHlsl || options->optimizeSize) && !options->disableOptimizer) {
        SpirvToolsLegalize(intermediate, spirv, logger, options);
        prelegalization = false;
    }

    if (options->validate)
        SpirvToolsValidate(intermediate, spirv, logger, prelegalization);

    if (options->disassemble)
        SpirvToolsDisassemble(std::cout, spirv);

#endif

....
}

glslangvalidator.exe already has SPIRV optimizations option. I tried to use glslangvalidator.exe to produce spirv- code, in this case i have no problem!

Note!

Adreno driver can compile spirv code without spirv-optimizations for Vertex/Pixel Shaders.
Adreno driver cannot compile spirv code without spirv-optimizations for Compute Shaders

You can compare the ywo version of shaders:

without spirv-optimizations
with spirrv-optimizations

Do you have any questions ?

jgavris · June 17, 2019, 5:30pm

That’s very helpful, thank you.

I’ve noticed that many of my compute shaders do compile and run fine on Adreno with optimizations though. It’s only one or two difficult ones, and I’ve yet to isolate what makes these ‘special’ compared to the others.

jgavris · August 7, 2019, 4:46pm

@AndreyOGL_D3D Have you tried out Android Q yet? The shader compilation issue I had on Android P has magically gone away. I am also seeing substantial performance increases across the board.

AndreyOGL_D3D · August 7, 2019, 7:06pm

Hi! @jgavris

no, I have no device with Adreno/Android Q.

Good news! So, Qualcomm fixed Vulkan driver problem.
Right now I have another problem with Adreno, i see no 3D scene, only 2D text, but it work very well on Mali…

AndreyOGL_D3D · May 12, 2020, 1:39pm

Hi! @jgavris

Some time ago i have update Android 9 for smartphone with Adreno 506 GPU, I have the same problem with vkCreateComputePipelines, also it still doesn’t work by another reason, issue with vkQueueSubmit for compute task, i’m going to investigate problem.

benn-geomagical · January 4, 2021, 7:45pm

I’ve run into the same issue with Galaxy S8 Adreno 540 for both graphical and compute pipelines. The Adreno driver appears to be allergic to certain spirv optimization passes. The workaround was to compile without optimizations and then use spirv-opt in a separate pass to tune which optimizations are used. I was never able to determine what exact combination of statements in a shader triggered the bug, however. At one point the optimizations --convert-local-access-chains and --eliminate-common-uniform were the poison pills.
As of today it appears to be --ssa-rewrite
spirv-opt --help provides a list of the passes done by -O and -Os. I dumped those into a config file and now use spirv-opt -Oconfig=spirv-opt.cfg … and have eliminated the --ssa-rewrite passes. It is entirely possible that one or more of the other optimzations will be problematic as the shaders change in development and we hit some other combination of instructions that cause the problem to recur, but this is working now.

AndreyOGL_D3D · January 5, 2021, 7:29am

@benn-geomagical Hi, thank you for additional information, for new GPU Anreno 6x I have no issue.

system · October 19, 2021, 2:00pm

This topic was automatically closed 183 days after the last reply. New replies are no longer allowed.