# This file allows all Spiders. # An empty robots.txt or no robots.txt file will accomplist the same thing. User-agent: * Disallow: /~dshields/ # no longer hosted on dawn User-agent: * Disallow: /~bsstreet/ # directory empty, giving 404s User-agent: * Disallow: /~profileplastics/cpg/ # tons of 404s User-agent: * Disallow: /~rsmith/ # nothing in users directory, not modified since 2004.