Our solution is reference counting. We can only remove a vertex if no wavelet needs it, that is, when the reference count goes to zero.
The rules are pretty simple: The coarsest vertices start with count 1. Every time we add a wavelet, we increase the count for its vertices. When removing a wavelet, we decrease the reference counts, and if the count goes to zero, we remove that vertex.
Although in retrospect this may seem kind of obvious, it wasn't obvious to us in the beginning. Our first solution was to keep track in which order the wavelets were added, and which wavelet added which vertex. The removal of detail was then done in the reverse order. However, this order dependency would not allow view dependent change of detail, where the detail is added and removed in different order, for example for a slowly rotating object. Reference counting removes that limitation.