Can you calculate something per instance, not per vertex?

Preamble: do not render quads with instancing. Instancing, on various hardware, works best when the size of the instance is not exceedingly small. Small instances can throw away lots of performance on such hardware.

Also… why would your instanced base quad have 6 distinct vertices (as opposed to 4 vertices where 2 are shared)? If you’re going to use instancing, it would be better to use a triangle strip with each instance being a 4-element strip.

As for the main thrust of your question… no. There is no instance shader, and there is no way to make the VS execute some of its code per-instance rather than per-vertex.

However, you really shouldn’t stress about this. If the computation is “based on the instance’s position”, then that means the primary performance concern is reading the per-instance data, not the computation. And GPUs have to be able to efficiently deal with multiple invocations reading the same memory addresses, so you can assume that having 4 invocations read from that address will be not much slower than having 1 invocation read from that address.

So just accept it and move on.