Type "systune -i"
Type "maxlkmem". It should tell you the current value of maxlkmem in pages. By default it's usually 2000.
Type "maxlkmem 100000", where 100000 pages should be more than the amount of RAM on your machine (a page is usually either 4K or 16K). When asked to confirm, type "y".
The change will fail, but when you type "maxlkmem" again, you will see that the number of pages has increased to approximately three quarters of the RAM installed in the computer. Remember this number of pages, call it N.
Type "maxlkmem N", where N is the number recorded above. Again type "y" when asked to confirm. This time the change will succeed.
Type "maxlkmem" just to make sure!
Type "quit" to exit systune.
Check that the new value of maxlkmem has been correctly recorded in /var/sysgen/stune.
Check that nobody is doing anything on your machine, then reboot your machine by typing "/etc/reboot". This will move /unix.install to /unix and start you running on the new kernel.
For further information read the introduction to Section 2 of the manual (type "man 2 intro"). Alternatively, look in the book on "IRIX Admin: System Configuration and Operation", which is part of the Insight Guide (type "insight"). The relevant sections are titled System Performance Tuning and IRIX Kernel Tunable Parameters.