How does the bird know its own nest

It's probably a mix of different things. Birds usually have very good eyesight, and can see the area where their nest is built. They also have good directional sense, and can find the general area. If they have babies, and they are awake, the birds can hear them calling. Then when they are very close to the nest, some can smell their own scent on it.

Many birds also use landmarks to find their way around, just as you would.
How do you know your own house? Turn left at the store, up the block on the left, there's the pink flamingo in the front yard.

Birds do it pretty much the same way.