Linux swapper used to be very aggressive on file cache, evicting it to the point that for the next second you'll need all of these libraries again. That is the main reason of the slowdowns.
Fortunately now we have MGLRU patchset, which "freezes" the active file cache for a desired amount of milliseconds, and in general is much smarter algo.
This may be applicable for desktops, but not for servers.
In a low-memory situation, the admin wants to ssh into the server and fix the problem that led into memory exhaustion in the first place. Whoops, MGLRU freezes the active file cache only, which includes the memory hog, but does not include sshd, bash, PAM, and other files that are normally unused when nobody is logged in, but become essential during an admin intervention. So, de facto, the admin still cannot login, and the server is effectively inaccessible. The only difference is that the production application is still responding, which is not so helpful for restarting it.
Fortunately now we have MGLRU patchset, which "freezes" the active file cache for a desired amount of milliseconds, and in general is much smarter algo.