In a written response included in the report, the IRS commissioner of the Tax Exempt and Government Entities Division said there was no criminal behavior behind the actions of the agents, but rather inefficient management.
"We believe the front-line career employees that made the decisions acted out of a desire for efficiency and not out of any political and partisan viewpoint," the commissioner wrote.
Obama called the inspector general's findings outrageous and forced Miller's resignation, which takes effect in early June.
Meanwhile, the commissioner of the IRS' Tax Exempt and Government Entities Division also announced his retirement Thursday. Joseph Grant will leave in June, according to an internal IRS memo provided to CNN. Miller also is scheduled to exit then.
On Friday, Miller told the committee that more changes would be coming to the IRS over the controversy, implying that more officials would be forced to leave.
"We're not done yet. We're not," he said. "We now have the Treasury inspector general's report. We now have the sense of the facts. Now is the time for those that remain, including the incoming acting commissioner ... to take those actions."
Obama has appointed Danny Werfel, a White House budget office official who has served in both Democratic and Republican administrations, to succeed Miller through the end of the fiscal year on September 30.
Treasury Secretary Jack Lew met Friday with Werfel and directed him to conduct "a thorough review of the organization in an effort to restore public confidence in the IRS and ensure the organization is providing excellent and unbiased service to the taxpayer," a Treasury official told CNN.
The official said Lew asked Werfel to report to Obama within 30 days on progress made in holding wrongdoers responsible, correcting problems that allowed the targeting to occur and providing a "forward-looking systemic view" of the IRS.
In his opening statement, Camp said the controversy "goes against the very principles of free speech and liberty" on which the nation was founded.
He also said the IRS lied to Congress about the targeting, and cited what he called five violations of taxpayer rights by the agency's practice, including intimidation of conservative groups and leaking of confidential information.
"The reality is this is not a personnel problem," he said, instead calling it the result of an agency being too large and powerful, with the freedom to abuse that power. "Under this administration, the IRS has abused its power to tax and destroyed the faith of the American people" in the tax system.
Levin agreed that the IRS targeting was wrong, and he singled out former and current IRS officials for misconduct.
However, Levin specifically disagreed with Camp that the issue reflects a cultural problem in Obama's administration.
"If this hearing becomes essentially a bootstrap to continue the campaign of 2012 and prepare the campaign of 2014, we will be making a very, very serious mistake," Levin said.
Among the recommendations made by the Treasury inspector general: The IRS must better document reasons why applications are chosen for review, develop a process to track requests for assistance, develop and provide training to employees before each election cycle and immediately resolve outstanding cases.
The report also called on Treasury to develop guidelines to explain social welfare activity -- the primary factor in obtaining tax-exempt status.
Already, the controversy has leaked into the debate over House Republican efforts to repeal Obama's health care reform law. The IRS official in charge of that agency's implementation of the program, Sarah Hall Ingram, once headed the unit under scrutiny in the scandal.
Miller, who appointed Ingram to the position, on Friday described her as an excellent public servant.
Camp told CNN on Thursday that he does not yet know if the scandal rises to the level of criminal conduct. Other Republican leaders have said they want criminal charges in the case.
"But clearly this is serious," he said. "I think the penalties should be serious. I think Infringing on people's constitutional rights is not something we should look (at) as a trifling matter."
Camp promised more hearings to follow, partly to hear from Shulman, who was running the agency when the targeting program went into effect.
Shulman also will testify before the House Oversight Committee on Wednesday, a House GOP aide told CNN. Shulman voluntarily agreed to attend. He is no longer in the government.
Another official at the heart of the scandal, Lois Lerner, told the committee through an attorney that she was in Montreal. Lerner didn't attend the hearing, and Levin's opening statement said she should lose her job as director of the program assessing applications for tax exempt status.