/robots.txt – robots exclusion file
Use page meta tags robots
Password protected web – web server solution
Password protected web – cms solution
Computers and Humans apart