When babies take their first breath, a hole in their heart slams shut. I learned about this in my anatomy class, and I thought it was one of the most interesting things I had ever heard.
An unborn baby's circulation isn't like ours. The oxygen rich blood comes in through the umbilical cord and bypasses the lungs to supply their body with oxygen, and then the de-oxygenated blood leaves their body through the umbilical cord. Since they aren't using their lungs, they don't need all of the different chambers in the heart. There is a hole that connects the right and left atria called the foramen ovale.
This is also what people refer to when they say they have a "hole in their heart." What that means is that the foramen ovale didn't seal completely. There is still a little gap and the blood can pass back and forth between those two chambers, mixing the oxygen rich and oxygen poor blood, making the whole circulatory system less efficient. It can be fixed with a simple surgery, so no worries there.