About "Removing Invariant With A Cast", it says:

"The ability to cast away invariant-correctness is necessary in some cases where the static typing is incorrect and not fixable, such as when referencing code in a library one cannot change. Casting is, as always, a blunt and effective instrument, and when using it to cast away invariant-correctness, one must assume the responsibility to ensure the invariantness of the data, as the compiler will no longer be able to statically do so."

Given that it is (implicitly) stated that 1) removing invariant is a rare occasion and 2) that it carries a large danger, have you considered making it a two step process:

invariant int* p = ...;
int* q = cast(int*) cast(void*) p;

It would be similar to the process where you statically cast an object by first casting it to void*. When casting by mistake from an invariant the compiler could warn you, but that would still be overridable with the cast to void.