Currently the only way to clear Cached Contacts is to re-provision the device. This cannot be disabled.

More...