Please help with CL_LINK_PROGRAM_FAILURE error with clLinkProgram of library

[devent]

Hello. I’m trying to create a OpenCL library from a collection of C source files with math. functions in it. The structure is as following:

• opencl_utils.h
• utility.h
• hashing.h
• hashing.c
• noise_gen.h
• noise_gen.c

I can compile and link everything up to hashing.c.

		std::vector<Program> programs;
		auto opencl_utils_h = compileProgram(readFile("src/main/cpp/opencl_utils.h"));
		programs.push_back(opencl_utils_h);
		auto utility_h = compileProgram(readFile("src/main/cpp/utility.h"),
				{opencl_utils_h}, { "opencl_utils.h" });
		programs.push_back(utility_h);
		auto hashing_h = compileProgram(readFile("src/main/cpp/hashing.h"),
				{ opencl_utils_h }, { "opencl_utils.h" });
		programs.push_back(hashing_h);
		auto hashing_c = compileProgram(readFile("src/main/cpp/hashing.c"),
				{ opencl_utils_h, hashing_h }, { "opencl_utils.h", "hashing.h" });
		programs.push_back(hashing_c);

		Program hashing_lib;
		try {
			hashing_lib = linkProgram(programs, "-create-library");
			logger->debug("Successfully linked {}", "hashing_lib");
			libraries.push_back(hashing_lib);
	    } catch (const cl::Error &ex) {
	    	logger->error("Link library {} error {}: {}", "hashing_lib", ex.err(), ex.what());
	    	throw ex;
	    }

The hashing_lib is then a successful linked program/library.

Now I try to link the noise_gen library with the hashing_lib.

	    programs.clear();
		auto noise_gen_h = compileProgram(readFile("src/main/cpp/noise_gen.h"),
				{ opencl_utils_h }, { "opencl_utils.h" });
		programs.push_back(noise_gen_h);
		auto noise_gen_c = compileProgram(readFile("src/main/cpp/noise_gen.c"),
				{ noise_gen_h, opencl_utils_h, hashing_h, utility_h },
				{ "noise_gen.h", "opencl_utils.h", "hashing.h", "utility.h", });
		programs.push_back(noise_gen_c);
		try {
			Program noise_gen_lib = linkProgram(hashing_lib, noise_gen_c, "-create-library");
			logger->debug("Successfully linked {}", "noise_gen_lib");
	    } catch (const cl::Error &ex) {
	    	logger->error("Link library {} error {}: {}", "noise_gen_lib", ex.err(), ex.what());
			 cl_int buildErr = CL_SUCCESS;
			 auto buildInfo = noise_gen_c.getBuildInfo<CL_PROGRAM_BUILD_LOG>(&buildErr);
			 for (auto &pair : buildInfo) {
				 logger->error("Error link {} {}", "noise_gen_lib", std::string(pair.second));
			 }
	    	throw ex;
	    }

This fails with error -17 CL_LINK_PROGRAM_FAILURE. Why? And how can I even get the CL_PROGRAM_BUILD_LOG? The clLinkProgram return a NULL program. Should I use the hashing_lib or the noise_gen_c program? Neither of them returns a log.

The source is here: https://github.com/devent/anl-opencl/blob/feature/Feature%234484/anlopencl/src/test/cpp/opencl_test.cpp#L127
The sources of the headers: https://github.com/devent/anl-opencl/tree/feature/Feature%234484/anlopencl/src/main/cpp

I’m using the C++ bindings for OpenCL but the functions calls are the same. The call is basically:

    cl_program prog = ::clLinkProgram(
        ctx(),
        0,
        NULL,
        options, // "-create-library"
        2,
        programs, // hashing_lib, noise_gen_c
        notifyFptr, // NULL
        data, // NULL
        &error_local);

[devent]

I have now changed my sources and I have concatenated all sources in one single string. OpenCL can successfully compile them together and create a library. The library I can then successfully link to the kernel program.

This is not the best solution I think. How big can the sources string for the clCompileProgram be?

	void createPrograms() {
		std::stringstream ss;
		ss << readFile("src/main/cpp/opencl_utils.h");
		ss << readFile("src/main/cpp/utility.h");
		ss << readFile("src/main/cpp/hashsing.h");
		ss << readFile("src/main/cpp/hashsing.c");
		ss << readFile("src/main/cpp/noise_gen.h");
		ss << readFile("src/main/cpp/noise_gen.c");
		Program p = compileProgram(logger, ss.str());
		logger->debug("Successfully compiled sources.");
		try {
			library = linkProgram(p, "-create-library");
			logger->debug("Successfully linked sources");
	    } catch (const cl::Error &ex) {
	    	logger->error("Link library error {}: {}", ex.err(), ex.what());
			 cl_int buildErr = CL_SUCCESS;
			 auto buildInfo = p.getBuildInfo<CL_PROGRAM_BUILD_LOG>(&buildErr);
			 for (auto &pair : buildInfo) {
				 logger->error("Error link {}", std::string(pair.second));
			 }
	    	throw ex;
	    }
	}

https://github.com/devent/anl-opencl/blob/feature/Feature%234485/anlopencl/src/test/cpp/opencl_test.cpp#L73

[hterrolle]

My testing OpenCL Kernel is 2772 lines.

23 Kernel and 10 functions.

So it is not a size problem i think.

[devent]

My kernel will be about 5000 lines. I’m trying to port the Accidental Noise library (but just the core noise generation functions) to OpenCL.

https://github.com/JTippetts/accidental-noise-library

[waman]

@devent

hashsing

I struggle somewhat to organize the code linked in this thread with the branches and what is the present state of the art so to speak.

For instance I see a typo when the hashing .c and .h files are programmatically included but then again I’d expect a different exception so I assume you have fixed that or use different code meanwhile.

That said it’s hard for me to try and compile what code?

A stab in the dark (very much) is this: Known oneAPI DPC++ compiler issue

but that is just a google result rather than something I would have gotten from the compilation (which I am not able to do for reasons mentioned above)

[devent]

Yes, there was a typo. I fixed it. The current code is in the develop branch and it runs fine now with the clang compiler. I just tested and updated the develop branch. I use Eclipse CDT to run my code, so the project doesn’t contain any Makefile or CMake to build it.

Edit: my original post didn’t had the typo and it still gave me the described error.

Edit2: Eclipse actually generates Makefiles. I just updated the develop branch with those makefiles. But you need to update those because I used some libraries that I have locally.

[devent]

I actually succeeded now in creating a library of my noise-generation code to be used in a kernel. I learned now something about OpenCL Nvidia driver that isn’t obvious from the error messages.

1. my NVIDIA GeForce GTX 1050 doesn’t support the OpenCL intermediate language. So I couldn’t use clCreateProgramWithIL. Version OpenCL 3.0 CUDA 11.4.112 and Profile: FULL_PROFILE but no IL support.
2. I have to include all sources in the library as one big string. Then OpenCL can compile and link them as a library.
3. The header files are just cl_program objects that are not compiled. That means that for each header file you want to add you must just do clCreateProgramWithSource and not clBuildProgram or clCompileProgram.
4. Code can’t be double. That means that you need to make sure that function/definitions in your library are not copied in your final kernel code. I had a random numbers generator code both in my lib and in my kernel included and this doesn’t work.

But finally it works. Now I can compile beforehand my noise generator functions in a OpenCL library and use it in any kernel I like. Sadly the GTX 1050 doesn’t support IL so I have to compile the library every time I start the application.

For curious people my code is here

https://github.com/devent/anl-opencl/blob/develop/anlopencl/src/test/cpp/OpenCLContext.cpp#L252