Latency issue with Open SL


We are developing Android music app that allows to play small samples (like piano as instance).
We are using Open SL for playing, recording and adding run-time pitch effect (custom algorithm).
On our test devices everything is OK: no latency, we touch button, sound play at once, as expected.
But the issue is that some our users have weird latency. We recommend to all of them to clear memory, remove background apps and may be turn off/turn on device. Some of them inform us that it helps (is this good recommendation?). While others still have latency issue. And this is strange. Because we have at least two users with the same device, but one of them has issue, and other one doesn’t have latency (Samsung Galaxy Note 3 n9005)
Could you please help with some advice why there is latency? What to look for in our code or on device to get rid off latency?

Thanks in advance!

      • Updated - - -

I’ve added code of our implementation if it can help: