How could this have been working until now?
We should not call `cudaFree` on uninitialized memory.

We have to free the old pointers before resize. This also prevents
memory leakage in case the vector gets shrinked.

Apparently the previous code relied on the fact that `resize` somehow
initialized the memory to null pointers or that `cudaFree` just ignores
pointers it didn't allocate.