The fact that its produced from aluminum refining is irrelevant. Also everything is poisonous in high enough doses.

I understand that calcium fluoride isn't going to harm you because its not soluble, its just like eating some inert rocks but its basically pointless to add to toothpaste.

I'm not worried about fluoride, its very helpful for dental health, I did a bunch of my PhD on this.