The Design and Implementation of laffer1

When you have a small site, the worst-case scenario is a spider indexing you that doesn't honor robots.txt.  Yandex is particularly bad about this.  Trying to tune for a large indexing operation becomes the bottleneck. 

It's effectively the load test.