The Prime Minister had backed Jamaican foreign minister Kamina Johnson Smith to replace the Labour peer as secretary-general.
But allies rejected Mr Johnson’s move and re-elected Baroness Scotland for a second term during the Commonwealth Heads of Government Meeting (Chogm) na sexta.
UK officials, who had been privately critical of Baroness Scotland, had been confident Ms Johnson Smith would succeed.
Mr Johnson had said his choice would have the “vast experience and support to unite our unique family of nations and seize the opportunities ahead”.
Ms Johnson Smith accepted her defeat, tweetando: “Much love always, I continue to serve, e claro, sincere congratulations to Baroness Scotland.”