No one knows how pigeons home, or how swallows migrate, or how green turtles find Ascension Island from thousands of miles away to lay their eggs. These kinds of navigation involve more than following familiar landmarks or orientating in particular compass directions: they are goal directed.
Most research on animal navigation has been done on homing pigeons, and it is generally agreed that they can use familiar landmarks when they are near home, but racing pigeons routinely find their way home from unfamiliar places over distances of 600 miles. They do not do it by remembering the twists and turns of the outward journey and they do not seem to be navigating by the sun or the stars, although on sunny days they may use the sun as a compass. They may use a sense of smell when relatively close to their home, but this cannot explain homing from hundreds of miles away irrespective of wind direction. All this leaves a magnetic sense, and a range of animal species can detect magnetic fields, including termites, bees and migrating birds. There is speculation that such a magnetic sense might depend on quantum effects in the retina if it does not depend on iron-containing crystals in other parts of the animal's body. But even if navigating animals have a compass sense, this cannot by itself explain homing. Imagine that you are taken to an unfamiliar place and given a compass. You would know from the compass where north was, but not where home was. Even if a magnetic sense is exquisitely sensitive and can measure the dip in the earth's magnetic field, not just the north-south direction, this could give information on latitude but not longitude. Moreover the earth's magnetic field is continually changing.
The conventional approach is to say that navigation depends on a combination of different cues working together with a magnetic sense, but this is very vague. By contrast, there is the possibility navigation depends on a sense or principle as yet unrecognised by science and that some relatively simple explanation might be possible. Rupert Sheldrake proposes that animals are linked to their home through a field, which he calls a morphic field, like a kind of invisible elastic band that gives them a sense of direction and a 'pull' towards their destination. He discusses this idea in his book Seven Experiments That Could Change The World, Chapter 2. See also the website Animal Navigation.