The IOM said the circumstances of their deaths and their nationalities are still unknown but they believe they died being smuggled through the desert towards the Mediterranean.
The organisation, part of the United Nations, said it was “profoundly shocked” by the discovery.
Libya is investigating, the IOM said.
The grave was found in south-west Libya, it said.
An IOM spokesperson said: “Each report of a missing migrant or a loss of life represents a grieving family searching for answers about their loved ones or acknowledging the tragedy of the loss.
“The cost of inadequate action is evident in the increasing human deaths and the disturbing conditions migrants find themselves in.”
The organisation said the tragedy highlighted the need for a co-ordinated response to smuggling of migrants and legal migration pathways.
Libya is among the main departure points for migrants trying to cross the Mediterranean to enter Europe.
The Geneva-based IOM also called on Libyan authorities and other UN agencies to ensure a dignified recovery, identification and transfer of the remains.
The discovery of the mass grave comes after at least 60 migrants, who set off from Libyan coast, died when a rubber dinghy ran into trouble in the Mediterranean Sea.
The IOM said earlier this month that 2023 was the deadliest year for migrants since records began a decade ago, with at least 8,565 people dying on migration routes worldwide.
The UN agency, which provides humanitarian assistance to vulnerable people in Libya, said the figure was 20% up on the year before.
