The evacuation of American diplomats, soldiers and even CIA operatives from Yemen is stirring deep concerns that the U.S. is losing a vital foothold in territory that the most notorious Al Qaeda affiliate calls home. 

The White House says Defense Department officials remain on the ground and are coordinating with Yemeni counterparts, but the retreat of most U.S. personnel is seen as a significant setback for what had been a cornerstone of American counterterrorism operations. 

"Bottom line is increased danger to the United States homeland," House Armed Services Committee Chairman Mac Thornberry, R-Texas, told Fox News. 

President Obama last fall, in announcing military action against the Islamic State, cited counterterrorism efforts in Yemen as a model and a success story. 

But Thornberry said there is now "less pressure" on America's chief enemy in that region, Al Qaeda in the Arabian Peninsula, which he described as a "serious" threat. 

"That makes it easier for them to plot and plan against us," he said. 

The State Department confirmed earlier this week it had closed the U.S. Embassy in Yemen and evacuated its staff because of the political crisis and security concerns following the takeover of much of the country by Iran-linked Shiite Houthi rebels. 

But The Washington Post reported that the embassy closure also has forced the CIA to withdraw personnel. Officials told the Post the CIA has pulled "dozens" of operatives and other staffers from the country, including senior officials who were working with Yemen's government against Al Qaeda operatives. 

One former U.S. official called the development "extremely damaging" to the CIA mission there. 

Some CIA personnel reportedly remain in Yemen as the agency tries to maintain its intelligence network. But coordination with Yemeni agents undoubtedly becomes more difficult.   

House intelligence committee Chairman Devin Nunes, R-Calif., predicted the pull-out would "hinder the United States' campaign against al-Qaeda in the Arabian Peninsula," the group which claimed responsibility for the recent Paris terror attack -- and which has attempted attacks inside the U.S. over the past several years. 

Nunes also said the chaos in Yemen should fuel concerns about Iran -- with whom the U.S. and other countries are negotiating regarding its nuclear program -- since the Yemen rebels are Iran-backed. 

Yemen has been in crisis for months, with Houthi rebels besieging the capital and then taking control. 

The Houthis last week dissolved parliament and formally took over after months of clashes. They then placed President Hadi and his Cabinet ministers under house arrest. Hadi and the ministers later resigned in protest. 

On Tuesday, U.S. officials said the embassy closure would not affect counterterrorism operations against Al Qaeda's Yemen branch. 

"The United States remains firmly committed to supporting all Yemenis who continue to work toward a peaceful, prosperous and unified Yemen," State Department spokeswoman Jen Psaki said. "We will explore options for a return to Sanaa when the situation on the ground improves." 

Both the CIA and the military's Joint Special Operations Command run separate drone killing programs in Yemen, though the CIA has conducted the majority of the strikes, U.S. officials have said. 

On Wednesday, White House Press Secretary Josh Earnest also said counterterrorism operations are still ongoing. 

"There continue to be Department of Defense personnel ... on the ground in Yemen that are coordinating with their counterparts in Yemen ... and continuing to carry out the kinds of actions, the counterterrorism actions that are necessary to protect the American people and our interests," he said. 

Earnest stressed that Yemen has long had a weak central government and faced challenges, but the U.S. nevertheless has "succeeded in applying significant pressure to the AQAP leadership that's operating in Yemen." 

The Associated Press contributed to this report.