The military stressed a “significant force” would remain in Gaza.
“This is another stage in the war effort”, Israel Defense Forces (IDF) spokesman Lt Col Peter Lerner told the BBC.
The pull-out is being interpreted as tactical, rather than a sign the war may be moving closer to its end.
Also on Sunday, Israel and Hamas said they had both sent delegations to Cairo to join fresh ceasefire negotiations.
It is six months to the day since Hamas attacked southern Israeli border communities on 7 October, killing 1,200 people and taking more than 250 hostage.
Israel says that of 130 hostages still in Gaza, at least 34 are dead.
More than 33,000 Gazans have been killed in Israel’s offensive in Gaza since then, the Hamas-run health ministry says, the majority of them civilians. Gaza is on the brink of famine, with Oxfam reporting that 300,000 people trapped in the north have lived since January on an average of 245 calories a day.
‘War not over’
Lt Col Lerner said troops would rotate out because the military had completed its mission in Khan Younis.
Khan Younis has been under Israeli bombardment for months, and the city and surrounding area are largely destroyed.
Lt Col Lerner said: “The war is not over. War can only be over when they [hostages] come home and when Hamas is gone.”
“It is a decline in the forces but there are more operations that need to be conducted. Rafah is clearly a stronghold. We need to dismantle Hamas’ capabilities wherever they are.”
US National Security spokesman John Kirby said it appeared to be a “rest and refit” and “not necessarily… indicative of some coming new operation for these troops”.
Israel has long warned of a planned ground offensive in the southern city of Rafah, where more than a million displaced Palestinians are sheltering.
But international pressure for a ceasefire is mounting, and the US – Israel’s closest and most powerful ally – warned earlier this week that its ongoing support for the Gaza war is dependent on “specific, concrete steps” to boost aid and prevent civilian deaths.
Marking six months of the war, Israeli Prime Minister Benjamin Netanyahu said Israel was “one step away from victory” but insisted there would be no ceasefire without the release of Israeli hostages.
“The achievements of the war are considerable: We have eliminated 19 of Hamas’s 24 battalions, including senior commanders,” he said.
Tens of thousands of Israelis rallied against Mr Netanyahu on Saturday, demanding a deal to free the hostages held in Gaza.
Anti-government protesters were joined by hostages’ families.
The rallies in Tel Aviv and other cities came hours after the Israel Defense Forces (IDF) recovered the body of hostage Elad Katzir.
Organisers said the protest in Tel Aviv had drawn 100,000 people, while other counts put attendance at around 45,000.
More demonstrations are expected on Sunday evening.