The minister, former lawyer Freddy Vidovic, had been convicted in 2015 of bribery and aiding an escape attempt by a businessman with ties to a former president jailed for corruption.
Paz recalled that he had promised during his election campaign to close the ministry, which he said had been abused by his left-wing rivals for decades.
“Today I keep my word. The ministry of persecution is over, the ministry of injustice is over,” he said.