Solaris

Some weeks ago i reported about the availability of the Sun Crash Dump Analysis tool in it´s fifth major release. Ben Rockwood of Cuddletech did a nice overview article about the really useful tool: Solaris Core Analysis, Part 2: Solaris CAT.

Ashlee Vance blogs at his new employer about our FISHworks concept. I really think, there is a common misunderstanding about FISHworks. Most people still doesn´t get the idea behind FISHworks. Ashlee writes, that we are too secret about the stuff and that we didn´t show off something.

At first - Fishwork isn´t a filer. Period. It´s an important part of the filer project. Fishwork is more like a way to build appliances, it´s a way to make appliances out of Solaris.

The review of the ZFS Crypto code has started. You will find more informations at the webpage of the project. Furthermore there is a documentation about the new commands for this feature. It looks really interesting and it would give me a good anwser for customer questions about a crypted filesystem.

In a series of videos David Stewart (an engineering manager at Intel's Open Source Technology Center) describes the work Intel put into Opensolaris:

• Intel and Opensolaris
• Powermanagement
• Performance
• Fault Management
• Virtualisation
• I/O-Accelration
• Dunnington(the upcoming hexacore Xeon)

These short episodes are really worth a view.

I wrote some days ago about the need for an different ZFS block allocator to enable the usage of an ZFS pool for an MAID. After thinking a little bit longer about it, i wrote an change request for it.

The change request "67422636 - Disk activation based allocator for blocks in ZFS" is now in the accepted state. This doesn´t already mean that this CR will make it into Opensolaris, but at least it says it isn´t an obviously dumb idea and the engineering will think about it.

There is a really interesting tool to gather informations from a running process. Let´s assume a fellow admin is in her or his well-deserved vacation and now you have to reboot the new machine, because you have to service the memory. Of course you look into the documentation and you find a process that isn´t covered by the documentation and there doesn´t seem to be an startup script or SMF manifest that starts this application. D'oh ... yet another last-minute pre-vacation change ... shouldn't happen, but it does happen.

Let´s assume you found an process called application in the process list:

bash-3.2# ps -ef | grep "application" | grep -v "grep"
    root 27943 27888   0 23:08:23 pts/1       0:00 /bin/perl ./application --start --debug --foobar=e3421
jmoekamp 27945 27824   0 23:08:28 pts/2       0:00 grep application

You look into the application source code (it´s a perl script, so this isn´t a problem and you see, that the application uses environment variables for config information. Now you have a problem. What are the correct settings of the environment variables? How can you gather the correct information to startup the application again. Solaris can help here with a programm called pargs.

At first let´s check the environment of the application first. It´s important to know, that you just can lookup this informations for processes you own. You need to get root privileges to see this information for other processes. Okay, the pargs command offers the -e switch for this task.

# pargs -e 27943
27943:  /bin/perl ./application --start --debug --foobar=e3421
[...]
envp[3]: DATABASEHOST=foobar.server.de
[...]
envp[6]: USERNAME=snafu
[...]
envp[13]: DATABASEDB=importantdatabase
[...]

There is another interesting switch. You can use pargs with the -l switch to get the full command line that was used to start the script, albeit this isn't that interesting, as you could get similar information with /usr/ucb/ps -auxwwww. But with a known process id pargs is much more comfortable.

# pargs -l 27943
/usr/perl5/5.8.4/bin/perl ./application --start --debug '--foobar=e3421' 

Now you should have all informations to restart the script and -by the way- to update the documentation for the fellow admin.

Once in a while you will see some strange processes with <defunct> instead of a process name. This happens when a child process terminates, but the parent process isn´t interested in the outcome because it didn´t waited for the childs response. Almost all resources of the child process are freed up at the moment with the exception of the entry in the process table. The parent process need it to get the exit code from it´s child, thus you can´t simply delete it on the termination of the child. The remaining process table entry will be delete, when the parent proccess reaps the child process by gathering the exit code. But when the parent forgets to reap the child, it´s undead, it´s defunct. Or to stay in the terminology: You´ve produced a Zombie process.

Let´s create such a process. It´s really easy, we just have to create a long running process forking away a child but we don´t use the wait() system call to gather it´s response at the exit.

bash-3.2$ nohup perl -e "if (fork()>0) {while (1) {sleep 100*100;};};"&

Okay, let´s check for our processes. In the output of ps -ecl the zombie processes are marked with a Z:

bash-3.2$ ps -ecl |grep "Z"
 F S    UID   PID  PPID  CLS PRI     ADDR     SZ    WCHAN TTY         TIME CMD
 0 Z    100 27841 27840    -   0        -      0        - ?           0:00 <defunct>
bash-3.2$

A kill -9 to this process is without effect. Obviously, a zombie will go away when you terminate the parent process, but that isn´t alway an option. How can you get rid of this Zombies? Okay, with Solaris you can reap such processes manually. The preap forces the parent to reap the child by calling wait() system call on the child.

bash-3.2$ preap 27841
27841: exited with status 0

And when you look in the process table again you will see, that the zombie founds it´s peace...

bash-3.2$ ps -ecl |grep "Z"
 F S    UID   PID  PPID  CLS PRI     ADDR     SZ    WCHAN TTY         TIME CMD

Obviously, you should ask yourself, why an application leaves such zombie processes, when the task of reaping them away manually gets a frequent task. Often it´s because of bad programming style.

I had an interesting discussion with a blog reader and customer via icq (oh wonderful new world of business). It started with a question about the J4500 and it ended in the the discussion, if MAID is really a viable way for storing data of if it´s just one of this hypes working their way through the storage business every year.

MAID? What is MAID? MAIDs are Massive Arrays of Idle Disk. The idea is to save power by switching of disks in the case they aren´t in use. You can buy such components from companies like Copan.

I´ve uploaded another early pdf of the tutorial. I´ve added some text and the first to figures. Kerberos is just one of the topics where you should explain only the configuration or you have to explain it almost completly. You can´t explain just a few things and still hope your explanation makes sense.

In the Webshop of Dell you can choose Solaris x86 as an option. This is an example from the Dell PowerEdge R905:


But it costs $231 more than the subscription at Sun

With the integration of PSARC 2007/347 NFS/RDMA Solaris got a new implementation for NFS over RDMA:

This integration brings the NFS/RDMA Solaris Implementation up to datewith the latest IETF Drafts. Specifically, it removes support for the initial NFS/RDMA implementation (version 0) and adds support for rpcrdma [1] and nfsdirect [2] (version 1).

I forgot to post the conclusion and the table of content when i published the JET tutorial some weeks ago. Okay ... here they are. This is a real long tutorial. 16 parts in total - the longest tutorial so far:

• Part 1: Introduction
• Part 2: Basic Jumpstart
• Part 3: Files controlling the installation
• Part 4: Jumpstart FLASH
• Part 5: The Toolkit
• Part 6: Prerequisites
• Part 7: Installing the JET Server
• Part 8: Preparing the first installation
• Part 9: Basic installation of a new system
• Part 10: A polished basic installation
• Part 11: Mirrored boot disks
• Part 12: Automatic hardening with SST
• Part 13: Deep dive to post installation
• Part 14: Using flash archives
• Part 15: Using flash archives for recovery
• Part 16: Conclusion

I hope this tutorial is useful for you!

JET and Jumpstart are incredible powerful tools to ease the installation of your solaris systems. You can install your systems via network, configure and customize them. With tools like Jumpstart FLASH it´s easy to clone systems for large computing cluster.

Even when you have only a few systems, it´s a good practice to install you systems by a Jumpstart server. A bare metal recovery of your system is much easier, you can use a common template to setup your system and beside the hard disk space the jumpstart server needs only resources at the moment of the installation.

I´ve integrated the CacheFS and the JET tutorials in the standard release of the LKSF book, thus now all links refering to the released version of the book should deliver the version 0.28 of the LKSF book (with the date of the 18.08.2008 on the titlepage). You can download it here.

CacheFS is one of this hidden gems in Solaris, it´s even that hidden, that it´s in sustaining mode since 1999. That means, that only bugfixes are made for this features but no new features found their way into this component since that day. In the recent days there was some discussion about the declaration of the End-of-Feature status for CacheFS which will lead to the announcement of the removal of CacheFS. After a few days of discussion the ARC decided in favour of the removal.