Kate Middleton and Prince William marked a fresh start by moving to Forest Lodge, as Adelaide Cottage became "a place of pain," royal biographer Sally Bedell Smith tells PEOPLE.

In moving to their new home, the eight-bedroom, $21 million Georgian mansion Forest Lodge, Prince William, Kate Middleton, and their three kids are leaving behind a difficult season that occurred during their three years living at their prior home, the four-bedroom Adelaide Cottage.

The Prince and Princess of Wales likely realize that their new neighbors aren't necessarily pleased with the new "security" they've installed around the property. But according to Bond, Kate and William won't let any "fury over" their new home prevent them from celebrating the holiday season.

Tensions between the brothers have simmered for years, punctuated by sporadic private contact and competing public appearances. Harry, 41, has lived in the U.S. since what became known as "Megxit," while William has stepped up domestic duties and international diplomacy as he prepares to become king.