[CMake] Allow user specified compiler flags to take precedence