Don't feel cheated! He is a kohen blessing the entire congregation with the bircat kohanim - special Priestly Blessing - that is recited on certain occasions throughout the year, depending on the local custom.
The reason for turning around is so that the members of the congregation will not look at his hands at the time of blessing. According to Jewish tradition the Divine Presence rests on the hands of the kohen at the time of the blessing and it is disrepectful to look at it.
There are many congregations that ensure that the kohen's hands will not be viewed at the time of the blessing without having the people turn around. The kohen covers his hands with a tallit or prayer shawl. Other congregations go a step further and have the people cover their eyes with their own prayer shawls as well.