Fix client-side removal of devices
Because events may be in-transit when a client removes the device, we need to make this a full roundtrip to the server. Otherwise the client may assume a device is removed, releases all references and then gets the original device added event for that device.
Better to have this as a round-trip instead.