Of course you can. Its not as easy as software testing, but you certainly can run it through a gamut of prompts intended to provoke it to act in ways that you find unacceptable. I mean there is a whole field studying and applying AI alignment.