The Nephilim are mentioned twice in the Torah (Genesis 6, Numbers 13). "Nephilim" means "the fallen ones." They were people of giant stature. It is not clear how large they were, but were large enough to scare other people. The Targum Yerushalmi, which is a Midrash, explains that they were Angels who descended to the Earth. According to this they were called fallen ones because they fell from their heavenly stature. The Ramban explains otherwise, that they were the other children of Adam. Because Adam, the First Man, was the handiwork of G-d, his children were of greater physical stature than their descendants. Because these people were so close to the Creator, they knew that they had no grandfather, they should have been spiritually elevated. Because they didn't live up to their potential they were considered "fallen ones."
