Subject: give me some SPEED man!!

From: "THE BIG FAT CLOWN, WITH A BIG RED BOOK OF BAAD JOKES." <[email-masked]>
Date: Wed, 6 May 1992 02:00:00 -0700

I should think everyone has seen the message in the News bulletin,
that runs along the lines of server is slow due to heavy use ... work by
Haakon over the summer will improve this.

What is Haakon planning on doing, and could anyone spill the beans on why
the load makes it so slow.  If someone where to set up another MOO, how
would the avoid Mundo-Lag ( ... Dudes ?)

Colin

---

From: Pavel Curtis <[email-masked]>
Date: Wed, 6 May 1992 10:22:33 -0700

What is Haakon planning on doing, and could anyone spill the beans on why
   the load makes it so slow.

There are three major reasons for the lag on LambdaMOO:

1) The server currently does an enormous amount of value-copying, to allow it
   to safely and easily free up storage when it's no longer needed.  The idea
   is that every part of the code makes its own copies of the values it uses,
   so every part can feel free to dispose of its copy when its own, local needs
   are satisfied.

   The solution to this problem is at the top of my agenda for when I get back
   to doing major server hacking this summer.  I will begin using
   reference-counting to determine when it's safe to free values.  This will
   almost entirely eliminate the copying of values.  Since the server currently
   spends a major fraction of its time (around 60% on average, as I recall, and
   much much more in certain nasty cases), this change should have a
   significant impact on server performance.

2) The entire LambdaMOO database is maintained in memory on the server.  For
   LambdaMOO currently, this means about 85 megabytes of virtual memory.  The
   machine on which LambdaMOO runs has only 64 megabytes of physical memory, so
   there is the potential for major thrashing, moving pages from memory to disk
   and back again.  As it turns out, however, this is not a significant problem
   in normal use.  It becomes a major problem only when the server is making a
   checkpoint of the database into a disk file.  To do this, the server process
   clones itself (`forks', in UNIX parlance) so that one clone can write out a
   consistent copy of the DB to disk while the other continues to provide
   service to the users.  Of course, when the server is already using 85
   megabytes and it clones itself, it begins using 170 megabytes; further, one
   of those clones is systematically running through essentially *all* of that
   memory, copying it to disk.  This leads to an incredible amount of thrashing
   and, thus, to incredible lag.

   The solution to this problem is second on my agenda for the summer.  I will
   be making the major changes necessary to move the primary storage of the DB
   from memory to disk, more along the lines of a standard database
   implementation.  This will eliminate the need for checkpointing at all
   (since the DB will already be on disk) and vastly decrease the memory
   requirements of the server (since memory would only be used as a cache of
   disk contents).  This major source of lag will be completely eliminated.  As
   a pleasant side-effect of this change, it will become much more reasonable
   for people to run the LambdaMOO server on small machines, notably PC's.

3) The current LambdaMOO task scheduler is not well-suited to the typical mix
   of forked vs. command tasks.  In particular, it is forced by the current
   specification of the language to give background tasks five seconds or
   30,000 ticks of runtime before aborting them.  That is five seconds during
   which the server must ignore command tasks.  Further, the scheduler is not
   uniformly giving highest priority to command tasks.  In brief, the scheduler
   lacks sufficient flexibility to do a good job.

   The solution to this problem is at least third on my agenda.  I say `at
   least' because it may be that the reference-counting fix for (1) above will
   significantly ameliorate this problem.  With that fix, I am predicting that
   the vast majority of tasks will very quickly run out of ticks, well before
   they could run out of seconds.  Thus, background tasks will hold up the
   scheduler for much smaller amounts of time, so command response will be that
   much better.

   The fix for this problem lies in a cute idea I had that builds on the
   solution to (2) above.  One aspect of the disk-basing implementation is that
   the server will maintain a list of all DB changes that need to be written to
   the disk.  Embedded in that list are markers, pointing out where the
   boundaries between tasks were.  The server must make sure that the state of
   the disk at any given time reflects exactly the state at some inter-task
   boundary; otherwise, if the server or server machine were to crash, the disk
   copy could be left in an inconsistent state.  Call these task-boundary
   markers `commit points'.  In the new system, the built-in function
   `suspend()', if given an argument of 0, would not necessarily suspend the
   task.  Rather, the server would, at its discretion, do one of three things:
     (a) It might do what's currently done; that is, it might save a copy of
	 the task's control stack on a task queue, insert a commit point
	 into the DB log, and let the next scheduled task begin running.
     (b) It might make the copy of the stack and insert the commit point, but
	 let this task continue running rather than moving to another one.
     (c) It might do nothing at all.
   The idea is that suspend(0) would become just a hint to the scheduler that
   this would be an acceptable place to interrupt the task.  If the suspend(0)
   is in the inner loop of the task, being executed many times, the server will
   almost always ignore it, taking option (c) above.  At strategically-chosen
   times, it would instead take option (b).  Finally, when the scheduler
   decided that this task had run for as long as it was advisable to let it
   run, it would either take option (a) the next time suspend(0) was called or
   else simply stop running the task, effectively backing up in time to the
   last commit point.  It would be as if the scheduler had taken option (a) at
   that time.

   I realize that this is all pretty complicated and difficult to grasp, but
   the programmer's view is simple: suspend(0) will mean either nothing or what
   it currently does and, further, suspend(0) is such a cheap operation that
   you can feel free to insert it in every inner loop of your code.  You would
   no longer have to test ticks_left() or seconds_left() (indeed, they may well
   disappear), but instead you'd do the suspend(0) unconditionally.  The
   scheduler would be free to employ standard, proven operating system
   algorithms to provide good interactive response time and reasonable
   background-task throughput.

   There are more details to be worked out in this fix, but this is enough to
   give you an idea of where I'm headed.

I expect all of these fixes to be released by the end of this summer, almost
certainly along with many other improvements to the server and the MOO
programming language.  In particular, I will release a new version of the
server right after fix (1) is working and again right after fix (2) is done, so
both users of LambdaMOO itself as well as the operators of other MOOs should
see performance improvements as soon as possible.

I hope that this message serves to bolster the community's resolve in toughing
out the current lag problems.  Help *is* on the way; I have not abandoned you
and I'm not deaf to your complaints.

	Pavel/Haakon

PS to Clay Luther: I believe that fix (1) will entirely obviate the need for
your new built-in functions listassoc() and listiassoc(); I'm pretty sure that
the whole reason for the slowness is value copying.  This is especially true of
certain large a-lists, like $player.messages...

---

From: Clay Luther <[email-masked]>
Date: Wed, 6 May 1992 15:17:10 -0700

> certain large a-lists, like $player.messages...

Kudos.  Of course, I wouldn't complain if the new mods increased the speed so
much that the need for listassoc/listiassoc was obliviated.  That would be
wonderful, indicating an overall enhancement in the server that listassoc/
listiassoc only barely resolve.

However, I cannot state strongly enough the value of the regexp/regsub functions
that we added to the server.  These are almost too useful to ignore.

Although one could make an argument that regexp/regsub could be implemented in
moo, assuming the server speed increases manifold, I would counter that the
implementation currently residing in the modded server is 1) stable, 2) fast,
and 3) already in there (like ragu).  Why waste it and time trying to
reimplement the services at a higher level in the server.  Besides, regexp/
regsub will benefit everyone using the moo server, not just those who choose to
run a lambda-derived db.

And with my programmers clamoring everyday for me to add this or that mod (now
that they know I don't mind dipping my fingers into those muddy waters), who
knows what really interesting and useful mods we can come up with ;).

So, go go Pavel.  But keep your ear to the ground, all the same.

-- 
Clay W. Luther                  [email-masked]
System Manager                  Center for Network Neuroscience
University of North Texas       PO Box 5218 Denton, Texas 76203
817/565-3896,3472               aka. Malraux@GodNet (Bushashuru!)

---

From: Judy Anderson <[email-masked]>
Date: Wed, 6 May 1992 20:46:39 -0700

You would
   no longer have to test ticks_left() or seconds_left() (indeed, they may well
   disappear), but instead you'd do the suspend(0) unconditionally.  

I would be unhappy if ticks_left(), or some other way of computing
actual ticks used by a process, were to go away.  We really don't have
any other methods of tuning programs.  Yes, I realize that ticks don't
really mean much, but they're all we have...

Judy Anderson yclept yduJ     [email-masked]    'yduJ' rhymes with 'fudge'
   Join the League for Programming Freedom, [email-masked]

---

From: Pavel Curtis <[email-masked]>
Date: Thu, 7 May 1992 08:15:03 -0700

I would be unhappy if ticks_left(), or some other way of computing
   actual ticks used by a process, were to go away.  We really don't have
   any other methods of tuning programs.

Good point.  I had forgotten about their usefulness in performance measurement.

	Pavel