CANBERRA – Australia’s Prime Minister, Tony Abbott, has defended Peter Dutton after the immigration minister was caught joking about the plight of Pacific Island nations facing rising sea levels.

Dutton has been heavily criticised since making the joke about climate change on Friday.

A boom microphone picked up Dutton chatting with Prime Minister Abbott about the plight of Pacific Island nations facing rising sea levels.

Noting that a meeting in Canberra was running a bit late, Dutton quipped it was running to “Cape York time”, to which Abbott replied: “We had a bit of that up in Port Moresby.”

Dutton then said: “Time doesn’t mean anything when you’re about to have water lapping at your door.”

The president of Kiribati has since called the joke “vulgar” and “quite unbecoming of leadership”.

Labor MP Matt Thistlethwaite called on Dutton to apologise for the remark, saying the “distasteful” comments will do “enormous damage” to Australia’s relationship with its neighbours.

He said Pacific communities have been pleading with the Abbott government to do more on climate change, and Dutton’s comment made it harder to find a solution.

Dutton originally refused to answer questions on the incident but, following criticism from Indigenous and Pacific leaders, on Sunday he said he regretted his remarks.

“Obviously it was a private conversation – I should have realised the mike was there,” Dutton told Sky News. “I didn’t; it was directly behind me. I made a mistake and I apologise to anyone who has taken offence to it. It was a light-hearted discussion with the PM and I didn’t mean any offence to anyone.

“If anyone has taken offence they should accept my apology. I’m disappointed that it allowed for a distraction from what was a very good policy announcement.”

Dutton said the government’s decision to accept 12,000 Syrian refugees and participate in air strikes against Islamic State fighters in Syria “should have been the story for the week and I’m sorry for that”.

Dutton had only just returned from a trip to Geneva to co-ordinate Australia’s response to the migrant crisis in Europe when he made the comments.

Abbott dismissed international criticism of Dutton’s remarks and repeated his earlier comment that Dutton should be remembered this week for agreement to bring 12,000 Syrian refugees to Australia, and not for a “lame joke”.

Tony de Brum, the Marshall Islands Foreign Minister, said he was aghast by Dutton’s remarks.

“We thought there was some resistance to the science of climate change amongst our friends to the south, but we didn’t expect there would be that indifference to a matter of life and death for their neighbours,” de Brum told RTCC, a climate advocacy website.

“It is very insensitive to our grandchildren, who must see these waves and these tides and these winds and cry whenever we have a bad storm.

“Next time there is a king tide and waves are battering my home, I’ll ask Peter Dutton over and we’ll see if he’s still laughing.”

Cape York traditional owner Gerhardt Pearson said the reference to “Cape York time” was aimed at portraying Aboriginal people as “lazy good-for-nothings”.

“We are constantly burdened with the view espoused by the likes of Dutton. It’s one of soft bigotry and low expectations, and it continues to dominate policy responses,” he said in a statement.

“Dutton’s joke using this tired old stereotype reminds me of how white superintendents ran our lives, dressed in their safari jackets and white helmets, pipe in their mouths, hands in their pockets; and how they would look down at my hard-working grandfather, mother or brother, as if they were his slaves.”

The prime minister of Papua New Guinea, Peter O’Neill, said Dutton’s comments were unfortunate and he hoped the attention the issue had received would help highlight the threat faced by many people.

“Rising sea levels is a serious issue affecting thousands of our people around the Pacific,” O’Neill said.

“Communities are under threat and they are losing homes and their food source. People around the Pacific are living in fear with each high tide of storm.

“People are being forced off the land where their families have lived for thousands of years. Connection to the land is very important for Pacific people so having to leave their land is heartbreaking for many people.”

He said Pacific island nations looked to the December climate change conference in Paris to get help from other countries.

Peter Dutton, 46, has been a Liberal member of the Australian House of Representatives representing the Division of Dickson, Queensland since the November 2001 federal election.

Dutton was the Minister for Health and the Minister for Sport from September 2013 until December 2014, when he was appointed Minister for Immigration and Border Protection after a cabinet reshuffle.