Despite already being married and even ​moving in together, Hailey Baldwin and ​Justin Bieber are still planning to hold a wedding ceremony, a source told PEOPLE.

“After being married for a few weeks, they both realized that they actually felt like husband and wife without having a religious ceremony,” the source said.

Social media evidence from the past few months show the couple has definitely felt like newlyweds, despite not having an official ceremony.

“They want to openly celebrate their marriage,” the source continued. “This is why Hailey decided to change her name. She loves being married to Justin. They are still planning a wedding and a religious ceremony.”

The insider also discussed the time the family spent together over Thanksgiving week and Hailey's 22nd birthday.

“They had a small birthday celebration for Hailey with his family and spent the rest of the weekend relaxing in Canada,” the source shared. “Justin loves being in Canada because it’s very peaceful. He can take Hailey out and it’s not a mess with photographers. They are much more left alone. They seem very happy together.” We're happy for the couple and more importantly are excited to see photos for the ceremony, whenever it does happen!