That’s up for you to decide. Though why would you store double, if you later downcast it to single anyway? Makes only sense if the math on it is particularly evil.
I mean, depends how many numbers. Nobody in the end cares if you store your random 42 floats as double. But it adds up if you have like millions.
On the GPU it is another matter. GPUs are devices that are made for single precission float ops. Double ops may have like 5x slower throughput (depending on architecture).