The tragedy occurred off Godoria, a locality in the Obock region of northeast Djibouti.
After being alerted by local residents, a team of gendarmerie mobilized on Tuesday afternoon near the reported site of the sinkings and discovered two survivors and the bodies of three women and two men.
The coast guard was also alerted and launched search and rescue operations.
Those operations are still underway with two patrol boats, IOM said.
According to local witnesses the boats capsized roughly 30 minutes after leaving due to overloading and heavy tide.
An IOM team is assisting local officials in the search for survivors.
IOM staff have discovered an 18-year old male survivor, who reportedly boarded the first boat with 130 people on board.
Included in that group were 16 women. This survivor said he did not have information about the second boat.
The five recovered bodies are being transported by the Ministry of Health to a hospital in Obock for examination.
IOM manages a Migrant Response Centre (MCR) in Obock, where over 500 migrants are being assisted ahead of repatriation to their countries of origin under the voluntary return programme.