I think this is caused by Apple bug that causes the WindowServer process to leak a little memory every time a window is created. The only solution is to make sure that you either log out all users on the machine (which causes the WindowServer process to restart) or restart the machine. We recommend that you try to plan a restart of the machine weekly – that should be often enough to prevent this particular problem for a system getting “average” use. You can check the status of the windowserver by opening a Terminal window and running the command:
ps auxww | grep windowserver
Look at the 5th value; if it’s at or near 3.5 GB, then you need to either log everyone out or just restart the machine. There is a hard limit of 4GB/process. When the process gets into the 3+ GB range, it becomes more likely that it will be unable to allocate additional memory when it needs to.
We know that Apple is aware of this problem (they requested logs from us) and that a side effect of it has affected Pixar, so we’re hopeful that they will release a fix before too long, if not in a 10.5.x release, then hopefully in Snow Leopard.