Here's the output from the vmstat command which I ran on a host that was running 4 Gigablast processes which were spidering and answering queries. Combined, the the processes are only using about 1 gigabyte of the 1.5 gigabytes of physical RAM available. From the output you can see that the disk cache is using the other .5 gigabytes.

So why is it continually swapping the Gigablast processes back and forth? This is the one thing about Linux that totally puzzles me. If you have insight into this issue could you please help me out?

$ cat /proc/version
Linux version 2.4.22 (root@lenny) (gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-81)) #1 Tue Aug 26 22:07:38 MDT 2003

$ vmstat 1
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 2  0  0    120  30180   1736  46696   0   0   424    25  134    77   5   9  86
 3  0  0    120  13908   1776  63192   0   0  8220     0  268   275  15  41  44
 0  1  1    120   5012   1784  71668   0   0  4344  5196  452   337   9  32  59
 0  1  1    120   4756   1784  71796   0   0   128  7824  294    94   0   6  94
 2  0  1    120   3200   1788  72436   0   0  1028  3364  266    91   2   6  92
 0  2  0    140   3628   1800  70796   0   0  5368     0  216   286  41  30  30
 1  0  0    140  23144   1856  51904   0   0  1968    28  339   500  49  16  36
 4  1  0    140  20324   1968  53188   0   0  2152     0  210   368  72   7  21
 6  0  0    140  15940   2020  54992   0   0  1796     0  413   315  85  10   5
 6  3  0    140  12096   2112  56732   0   0  1880     0  468   478  80  19   1
10  0  0    140   8660   2152  58736   0   0  2040     0  232   448  81  19   0
 5  3  0    140   4556   2184  61052   0   0  2340     0  482  1198  35  32  34
 1 18  0    268   4248   2156  60252   0 216  2632   216  358  1497  63  29   8
15  4  0    496   3920   1804  59096   0   0  2844     8  238   824  50  20  30
 2  0  1    496   3652   1828  57412   0   0  2820     0  244   533  84  15   1
 8  0  0    496   3888   1844  60892   0   0   912     0  178   314  88  12   0
 1 18  1    916   3468   1856  60448   0 388  2260  8488  400  1184  39  31  31
 4  0  1    916   3004   1856  60880   0   0   748 10180  453   594  32  22  46
17  1  1   1044   4736   1856  58116   0 292   488 13376  464   379  20  15  65
12  9  0   1044   4216   1860  58112   0   0  2352    32  300   891  66  26   8
 1  0  0   1172   3744   1864  57448   0   8  2252    60  225   310  93   7   0
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 4  0  1   1172   3788   1884  60744   0   0  1936     0  205   477  83  17   0
 5  0  0   1236   3900   1884  57156   0  44  3996   152  473  1076  50  39  11
 5  1  0   1236   3868   1896  57080   0   0  1012     0  195   261  97   3   0
 9  3  1   1236   3316   1896  58052   0   0  4408     0  228   484  72  28   0
 6  0  0   1236   4172   1908  60964   0   0   852     0  163   221  81  19   0
10  0  0   1236   3924   1912  59468   0   0  3364     0  279   751  73  27   0
 4  0  0   1452   4228   1920  61500   0 228  2868   484  308   540  57  43   0
 3 13  0   1580   4392   1912  59852   0   0  2032  4128  313   690  59  37   4
 2  0  1   1580   3044   1468  59396   0 220  2768   224  304   231  88  12   0
 1  0  0   1580   4492   1200  57944   0   0   584     0  151   276  93   7   0
 5  5  0   1580   3920   1196  61264   0   0  1604     0  174   326  76  24   0
17  3  1   1580   3732   1208  59892   0   0  2748   256  253   621  78  22   0
 1  0  0   1580   3656   1208  58952   0   0  1748     0  340   345  93   6   1
 1  0  0   1580   7156   1208  55484   0 184   528   184  165   266  79  21   0
 2  0  0   1580   6136   1224  60136   0   0   536     0  153   161  87  13   0
 2  0  1   1580   3016   1228  61516   0   0  2664     0  197   219  96   4   0
 3  0  0   1696   3952   1220  58928   0   0  2268   384  260   492  72  28   0
 5  0  0   1696   4136   1228  58780   0   0   244     0  132   137  96   4   0
 9  3  1   1952   3748   1232  61220   0  28  1612  4156  315   578  66  34   0
 5  1  0   2080   4156   1228  60052   0  36  1924  4180  453   433  60  40   0
 7  0  1   2080   3752   1232  59024   0   0  2084  4112  239   329  84  16   0
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
12  2  0   2336   3904   1252  62540   0   0   896   136  233   410  73  20   8
 1  9  0   2848   3416   1252  61012   0 692  2824  4760  250   249  82  18   0
 8  5  0   3232   3896   1260  58960   0   0  1928     8  388   769  79  21   0
 1  0  0   3232   4624   1272  58512   0   0   404     0  284   233  90  10   0
 1  1  1   3488   3696   1276  61864   0   0  2104  4132  217   332  72  28   0
 3  4  0   3488   3648   1280  59752   0   0  2880     0  201   301  89  11   0
 3  0  0   3616   3860   1288  62116   0   0  1812     0  207   601  77  23   0
23  0  0   4128   4452   1288  59788   0   0  2888  4140  352   485  69  31   0
 1  0  0   4128   5360   1296  58376   0 608  1272   608  399   447  76  24   0
 9  0  0   4128   3812   1300  62368   0   0  1912     0  206   619  79  21   0
 2  0  0   4768   3788   1304  61072   0 1176 1772  5312  384   606  67  33   0
 2  1  0   5024   5060   1312  62440   0 400  2024   400  188   346  81  19   0
 2  0  0   5024   4052   1328  62484   0 176  1168   176  164   338  89  11   0
 2  0  0   5152   3832   1328  60952   0   0  2820     0  218   473  78  22   0
 1  0  0   5408   4128   1332  59516   0 556  1256  4696  257   375  83  17   0
 4  0  0   5408   4856   1332  62376   0   0  1112     0  312   891  68  32   0
 4  0  0   5920   3948   1332  60380   0  64  3740    64  239   427  81  19   0
23  0  1   6048   3804   1336  62716   0   0  2136  3972  255   824  58  42   0
 2  0  0   6048   3476   1336  63076   0   0  1992   168  423   409  83  17   0
 3  3  0   6176   5748   1336  63900   0   0   704   128  250   444  74  26   0
 5  0  0   6176  15760   1384  52536   0   0   856    64  221   426  59  26  15
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 6  0  0   6176  11576   1384  53564   0   0  1044     0  231   389  79  21   0
 5  0  0   6176  11560   1384  58084   0   0   420     0  164   367  77  23   0
 5  0  0   6176   7784   1388  59052   0   0   972     0  171   327  95   5   0
 3  3  0   6176   5676   1392  59784   0   0   732     0  173   340  98   2   0
 4  3  0   6176   9664   1392  60036   0   0   252     0  195   255  90  10   0
 1  0  0   6560   4236   1404  64348   0   0   344  4132  354   460  71  29   0
 1  0  0   6560  13640   1404  55920   0   0   260    16  297   428  73  27   0
 3  0  0   6560  12892   1404  56208   0   0   288     0  165   262  87  13   0
 1  0  0   6560  13128   1408  56624   0   0   420   128  228   400  90  10   0
 3  0  0   6560  12332   1408  57064   0   0   368     0  208   380  86  14   0
 3  2  0   6560  12232   1420  57196   0   0   212     0  259   218  93   7   0
 1  0  0   6560  12432   1420  57340   0   0   144     0  144   483  76  24   0
 1  0  0   6560  11560   1420  57636   0   0   296     0  209   500  86  14   0
 1  0  0   6560  11744   1420  57740   0   0   104     0  183   426  85  15   0
11  0  0   6560  11308   1420  58092   0   0   352     0  145   259  91   9   0
 6  1  0   6560  10612   1420  58432   0   0   288     0  217   433  88  12   0
 2  1  0   6560   7680   1428  60100   0   0  1588     8  183   234  87   9   4
 2  0  0   6560   3840   1432  61808   0   0  2736     0  177   240  92   8   0
 5  1  0   6560   4456   1436  65052   0   0   532  4096  206   196  83  17   0
 2  2  0   6560   4316   1436  65228   0   0   180     0  127    71  99   1   0
 6  0  1   6944   3740   1440  64516   0 296  2508   296  189   317  87  13   0
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 6  0  0   7576   4828   1440  61168   0 1576  2136  1576  429   555  70  30  0
 4  0  0   7960   3868   1452  64504   0  92  1576  4188  383   245  64  36   0
 2  2  0   8088   4088   1448  63124   0   0  2816   132  217   269  83  15   2
 4  3  0   8400   3904   1456  66540   0   0   952     4  170   322  68  30   2
 2  0  1   8400   2804   1456  64840   0   0  3036     0  301   320  92   8   0
 1  0  0   8656   4672   1456  62612   0   0  1080  4032  217   126  97   3   0
 1  0  0   8784   5540   1456  61596   0   0   372     0  210   219  92   8   0
 6  8  0   8784   4260   1460  66168   0   0   388  7680  252   254  63  37   0
 3  2  0   9040   3372   1460  66812   0 680  2596   680  298   339  89  11   0
 4  0  0   9424   4076   1460  63328   0 792  2100   796  450   607  64  36   0
 3  0  0   9552   3948   1464  67092   0 336  1888   336  269   454  74  26   0
 1  0  0   9552   3856   1464  64816   0   0  2708     0  186   295  82  18   0
 2  1  0  10064   5488   1440  66676   0 120   352  5188  291   362  78  22   0
 2  0  1  10064   3352   1448  67680   0   0  1164     0  172   102  93   7   0
 3  3  0  10576   3800   1452  63484   0 988  3000  5476  420   525  62  38   0
 5  0  0  10576   4984   1456  63272   0   0   400     0  326    91  89  11   0
 3  0  0  10576   4340   1464  67292   0   0   404     0  157   375  77  23   0
 3  0  0  10796   4972   1460  64744   0 512  2288  4736  412   518  66  34   0
 3  2  0  10924   6548   1460  62516   0 188  1968  4328  516   532  59  41   0
 4  0  0  10924   3832   1464  67448   0   0  1588     0  208   230  68  32   0
 4  0  0  11216   3836   1452  64720   0   0  2872   344  187   238  86  14   0
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 2  0  0  11216   4600   1452  64608   0   0    60     0  119    47  96   4   0
 2  0  0  11432   3864   1452  69320   0   0  1544  4280  237   346  77  23   0
16  0  0  11432   3880   1424  66400   0   0  2808     0  185   376  92   8   0
 3  0  0  11432   4252   1392  65568   0   0   768     0  170   351  91   9   0
 3  0  0  11432   4472   1404  69892   0   0   200     0  201   472  79  21   0
 8  1  0  11600   4128   1360  69240   0   0  1688  4112  368   382  76  24   0
 1  0  0  11628   3124   1344  68120   0 132  3068   136  215   298  84  16   0
 6  3  1  11716   3708   1348  70972   0 248   992  3988  160   380  80  20   0
 5  0  0  11972   3896   1296  68756   0 456  2208   848  308   422  78  22   0
23  0  0  12908   4096   1296  66788   0 180  1940  4296  481   365  75  25   0
 6  0  0  12908   4100   1300  69656   0   0  1352     0  228   423  73  26   1
13  0  0  13292   4132   1292  67128   0   0  3124  4512  292   237  76  24   0
 1  0  0  13292   4860   1296  70292   0   0   264     0  176   200  79  21   0
 2  0  0  13804   3884   1288  67616   0   0  2940     8  221   485  85  15   0
 7  0  0  13932   4264   1288  70340   0   0  2036     0  198   436  65  35   0
 4  0  0  13932   3896   1272  69136   0 340  2116  4472  462   361  78  22   0
 1  3  0  14040   3916   1204  68368   0 240  2012   368  268   257  93   7   0
 1  0  0  14040   4128   1204  68088   0   0   744     0  157   272  89  11   0
 2  0  0  14360   4332   1204  70072   0 1380 1808  5472  296   562  61  39   0
 5  0  0  14360   4856   1204  67456   0 104  2268   108  223   226  79  21   0
 2  5  0  14360  16908   1216  59372   0   0   376    64  253   297  73  22   5
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 2  0  0  14360  16936   1216  59460   0   0    88     0  128   121  98   2   0
 4  5  0  14360  15864   1220  60184   0   0   728     0  142   168  93   6   1
 9  5  0  14360   8700   1220  63952   0   0  3768     0  223   255  95   5   0
 1  0  0  14360   7184   1236  69056   0   0   952     0  325   386  54  46   0
 2  1  0  14360   3788   1236  70716   0   0  1788     0  235   159  96   4   0
 3  1  0  14532   3884   1236  68536   0  68  2556  8320  267   316  81  19   0
 8  0  0  14532   3888   1236  72620   0 348  1116   348  242   316  81  19   0
 6  0  0  14576   3716   1236  69252   0  76  3808    76  345   425  76  24   0
 6  0  0  14568   4888   1240  72760   0   0   432     0  330   474  71  29   0
 1  3  0  14648   3716   1200  73024   0  36  2008  4148  249   221  83  17   0
 1  0  0  14744   3884   1196  70032   0   0  2764   128  193   350  79  21   0
 4  0  0  14744   3936   1196  74136   0   0   644     0  142   182  85  15   0
 7  2  0  14744   3904   1184  71176   0   0  1732     0  212   436  79  21   0
 3  3  0  14852   4172   1184  70352   0 128  2984  4168  253   181  86  14   0
15  1  0  14852   3864   1184  73848   0   0  1016     0  209   326  84  16   0
 2  2  0  14972   4076   1172  71600   0 176  2524  4292  566   550  62  38   0
 1  0  0  14972   4200   1076  70808   0   0  1328  4128  285    82  87  13   0
12  2  0  14972   3824   1040  74840   0   0   720     0  154   206  84  16   0
12  5  0  15100   3872   1040  71360  32   0  3476  3624  266   420  77  23   0
 1  0  0  15228   4224   1040  70720   0   0   768  4140  318   225  89  11   0
 5  3  0  15228   4500   1040  73580   0   0  1140     0  186   349  77  23   0
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 4  0  0  15228   3020   1040  72080   0   0  3352     0  209   320  92   8   0
 5  1  0  15356   4084   1044  74776   0 408   728   408  165   226  84  16   0
10  2  0  15356   3768   1032  73216   0 132  2192   132  200   223  91   9   0
 2  1  0  15612   3864   1020  71064   0   0  2424  4276  557   479  67  33   0
 3  0  0  15612   4624   1024  74100   0   0   328     0  276   321  78  22   0
 3  0  0  15740   7672   1032  69808   0   0  1572  4140  270   431  66  34   0
 2  0  0  15740   4196   1036  70196   0   0  2948     0  215   329  75  25   0
 6  3  0  15764   4148   1032  72824   0   0  1640     0  178   369  73  27   0
 4  0  0  15764   4432   1032  70740   0   0  2280  4200  467   595  65  33   2
 1  0  0  15764   5780   1032  68984   0   0   676     0  132   188  83  17   0
 1  0  0  15764   5752   1032  69076   0   0    92     0  124    72 100   0   0
 1  2  1  15764   3688   1032  73300   0  48  2496  4148  211   294  77  23   0
 1  0  0  15764   4248   1032  71004   0 216  2564   220  230   434  81  19   0
 2  1  0  15764   3644   1040  74216   0   0  1512   128  172   198  84  16   0
 7  1  1  15764   3220   1040  71672   0 120  2956   128  375   484  84  11   5
 3  0  1  15764   2864   1040  74592   0  24  2232  4132  330   292  63  37   0
 2  0  0  15764   6332   1040  71552   0   0  2720     0  190   485  92   8   0
10  2  1  15792   3676   1048  74244   0   0  1484  3936  159   281  80  20   0
 2  0  1  15792   3468   1048  72784   0  16  2192  4344  543   353  75  25   0
 5  0  0  15792   4776   1048  74164   0   0   608     0  270   413  68  32   0
 7  0  0  15792  20876   1084  57948   0   0   576    64  163   223  67  16  17
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 2  0  0  15792  21268   1088  58012   0   0   132     0  243   112  95   5   0
 4  5  0  15792  20864   1092  58264   0   0   256     0  138   134  98   2   0
 4  0  0  15792  20436   1092  58760   0   0   496   128  178   297  90  10   0
 1  0  0  15792  20512   1096  58992   0   0   232     0  281   323  87  13   0
 3  0  0  15792  17384   1096  60676   0   0  1680     0  311   266  96   4   0
 4  0  0  15792  10356   1096  63808   0   0  3124     0  363   266  74  26   0
 2  0  0  15792  11172   1096  68080   0   0   164     0  139   135  88  13   0
 2  0  0  15792   8556   1096  69688   0   0  1608     0  171   281  92   8   0
 6  1  0  15876   4188   1096  71260   0  12  3176  4024  270   325  65  35   0
10  0  0  15876   5520   1112  74192   0   0    48     0  126   181  79  21   0
 2  0  1  15988   3608   1112  73096   0   8  3360  4056  429   330  63  37   0
 2  0  0  15988   4188   1112  71532   0   0  1252     0  220   408  88  12   0
 1  3  0  16004   4112   1120  74908   0 100   640  4176  196   174  75  25   0
 5  0  0  16004   3752   1132  71008   0   0  3740     0  211   345  61  39   0
 4  0  0  16004   6120   1136  72744   0   0   336     0  152   229  84  16   0
 2  4  0  16004   3596   1140  72716   0   0  3176     0  337   528  81  19   0
 1  0  0  16080   4336   1140  71036   0 104  1380  4176  270   531  79  21   0
 1  0  0  16080   4244   1140  71136   0   0   100   384  183    95  97   3   0
 6  0  1  27736   3564   1148  72856   0  24  2628  4120  309   566  62  38   0
 3  0  0  27176   3392   1148  72436   0 120  1216   244  251   167  77  23   0
 2  2  0  27144   3436   1156  71484   0   0   976     0  148   180  97   3   0
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 2  0  0  27124   3940   1160  74528   0 1304  956  5360  272   191  51  21  28
 3  0  0  27080   3756   1180  72580   0 260  2552   388  218   429  90  10   0
14  2  1  26808   5096   1188  69868  84 1108 2016  9064  490   567  59  22  19
 4  0  0  25548   5912   1192  73032   0   0   436     0  478   736  50  50   0
 5  0  0  25548   3940   1192  73548   0   0  1560     0  301   385  93   7   0
 1  1  1  25548   3336   1192  71800   8 176  2848   432  258   147  91   9   0
 1  0  0  25544   4124   1200  70480 116  60   836    60  171   200  97   3   0
 4  0  0  25528   3940   1204  74916   0 140   408   140  167   167  81  19   0
 2  0  0  25528   3776   1140  74448  32 408  1004  4500  248   208  92   8   0
14  0  1  25624   3664   1032  71516  96  68  3268    68  227   382  80  20   0
 2  0  0  25052   4908   1036  74744   0  36   388    36  276   318  74  26   0
 2  0  0  25040   3132   1036  75092   4   0  1616     0  211   261  88   8   4
 2  0  1  24924   3788   1024  73144   4  20  2372  4156  380   520  54  32  14
 3  0  0  24924   4036   1032  72980  24  28   824    36  199   278  92   3   5
 3  0  0  24900   4028   1036  75356  28   0  2200     0  172   352  70  23   7
 1  0  0  25512   5416   1020  71668  64   0  2472  4132  714   709  47  36  18
 3  0  0  24960   6588   1020  70436   4  96   132    96  339   406  78  19   3
 4  0  0  24904   6084   1028  74936  24   0   412     0  146   195  81  18   1
 1  3  1  24904   3544   1028  75128  36 240  2784  4356  234   378  87  12   1
 9  1  0  24828   5392   1036  72052  24   0  1652     0  191   391  80  19   1
 4  0  0  24776   4424   1056  75480   0 136  2044   276  233   615  68  32   0
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 7  0  0  24776   3840   1060  74060   4   4  2252  4036  451   354  82  17   1
 6  0  0  24776   3848   1060  73552   4   0   652     0  239   247  89  11   0
 7  0  0  24736   3864   1068  72728  28  88  4320    88  272   633  51  48   1
 1  0  0  23600   9824   1076  67156   4   4    36  4132  271   226  66  34   0
 2  0  0  23596   5676   1080  72924  40   0  1696     0  195   304  70  27   3
 2  0  0  23596   3904   1060  72480  96   0  3076     0  265   407  80  10  10
 2  0  0  23596   5688   1080  75148  24   0   232     0  171   150  86  10   4
 1  3  0  23596   3464   1060  75708  40   0  1472  4520  241   172  54  16  30
18  0  0  23552   4180   1000  72592 164   4  2704  1456  253   513  40  23  37
10  0  0  23552  17880   1032  62400  28   0   176   196  272   371  68  19  13
 5  0  0  23400  17456   1036  62944  76   0   668     0  386   636  78  20   2
 6  3  0  23400  15120   1044  64348   4   0  1416     0  388   461  83  13   4
13  0  0  23400   8936   1044  67536   0   0  3188     0 1300   760  88  12   0
 2  0  0  23400   8320   1044  67896  16   0   372     0  415   422  91   9   0
13  4  1  23400   6152   1048  72608   0   0   568 11980  426   386  80  20   0
 4  3  0  23384   5800   1056  73308   0   0   752  2356  281   289  87  13   0
 6  0  0  23276   4252   1056  71552  44   0  3780     0  298   668  61  31   8
24  0  0  23276   4136   1056  71592  24   0   208     0  176   303  98   2   0
24  0  0  23276   8376   1060  71516 136   0   192     0  192   368  82   7  11
 6  2  0  23272   3932   1064  74144  60   0  2044     0  215   668  69  23   9
 5  1  0  23232   3880    904  71952  16  40  3000    40  249   758  60  22  18
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 6  0  0  23200   7776    916  72736   0  12   364    12  217   516  76  24   0
 1  1  0  23180   4044    924  74280  16   0  3776     0  239   492  64  20  16
 0  2  1  23180   3888    944  73928   0  68   544 12840  419   291   1  40  59
 2  0  0  21424  12764    956  73932  44   0   188  5716  353    61   2   6  92
 0  1  0  21324  15400   1004  75112 2204  0  2328   148  414   407   8  13  79
 0  0  0  14168  42736   1036  75704 380   0  1004     0  182   112   2   2  96
 0  0  0  14168  42736   1036  75704   0   0     0     0  181    85   0   0 100
 0  0  0  14168  42736   1036  75704   0   0     0     0  117     7   0   0 100
Ͳʽ38Ԫid