The basis of my argument in favor of four-story buildings rests firmly on the solid foundation laid by Christopher Alexander in A Pattern Language (Pattern 21, Four-Story Limit). Alexander's arguments have more to do with the mental health effects of high buildings than any other cause - people living above about the fourth floor are literally "beyond reach and out of touch." They tend to lose their connectedness with the ground and the wider society.
In fact, I disagree slightly with Alexander on this point - I think buildings of five and even six stories can be fine, although I generally prefer four story buildings. In towns, it seems that lower buildings can be more appropriate, as long as all parts of the town remain within walking distance. Buildings of three or even just two stories can work very well in towns. (Any settlement that consists entirely of free-standing single-story buildings is at best a village, and not within the realm of our consideration here.)
There are two critical issues regarding building height. First, buildings must not be out of scale, which means that a five-story building with low ceilings is probably fine, whereas a four-story building with very high ceilings may overwhelm the street. Second, buildings must not be so high that people in them become disconnected from the ground. While elevators may make it easy for people in upper stories to come and go, the irritation of waiting for an elevator, coupled with the socially uncomfortable spaces that all elevators seem to become, pose a subtle discouragement to going out. (We should not forget that people who have dogs tend to show better mental health than those who don't. The reason may be simple - dog owners get out a couple of times a day to walk the dog; others may tend to stay in more. Being out and about is stimulating and healthy.) In any case, the climb must be low enough that walking up the stairs never becomes a disincentive to going out.
Counting stories is not so simple as it seems. On the following pages, we will see buildings whose height might be given differently by different observers. The use of fractions in counting floors in useful, as when a partial story in the attic of a building might most accurately be counted as half a floor.
Since the horrific attacks of 11 September 2001, anyone inhabiting a skyscraper must feel more strongly the sense of potentially being trapped on an upper floor by some calamity. While people in low buildings have some chance of escape in the event of a fire consuming a lower floor, those in high buildings are not likely to escape. Many people also have a sense of vertigo when they approach the windows of a tall building (this, oddly, is true of the World Trade Center's chief architect). They aren't comfortable places to be, and we should stop building more of them.
City Design Home