Sure… Feed your vertices into something like Coldet (Sourceforge) and then use Ray-Intersection collision on that data on the client side.
You may or may not be able to use the same data structure that you draw from to draw either using Arrays from the client side, or even by mapping the VBO to client space for intersection checks. More likely (and I would recommend initially) you’ll need to duplicate the data so you have a collision structure on the client side that works separately from the VBO on the GPU.
Start simple with a slow recursive check on the data for collisions, and then get more technical later by feeding the data into something like an Octree, so you can discard large chunks of data to speed up collisions.
In any case this will be faster than picking and direct drawing, as you are doing now.
Ultimately you should be able to merge the two structures in some way, perhaps “fence” the area it’s stored in so that it’s shared between the GPU and CPU, and then save some memory.