Arima reminded me Javert from 'Les Miserables'. Just like Javert, he was in dilemma. Javert couldn't bring himself to arrest Jean Valjean, who had saved his life, but he can't go against the law and not arrest him either because the law was his way of living. If he didn't arrest him, it would be like his life up until that point had been meaningless. That's why he chose to commit suicide as an escape. I believe that was what Arima did. He couldn't bring himself to kill Kaneki since he thinks of him as his son. He couldn't not fight him either though since he's an investigator. I think he was hoping for Kaneki to kill him; that's why he was taunting him in a way. And when that didn't work out, he killed himself.
Awesome reaction by the way!