General

Buffer Extermination? WTF? Normally i'm seeing wait events like "buffer busy", "log sync" or "db file sequential read" when doing my research in Oracle installation in Top5 events when i'm called because of a situation where the performance is not quite at the level the customer wants. I was sitting in front of the console of an system still using 10g as it's database.

I want to add, that the performance problem had its root somewhere else and was quickly found, however this log wait sparked my interest. A much simpler reason. It was the curiosity afterwards, why there were peaks in the wait event statistics in regular intervals with this wait event i never saw before.

"Buffer exterminate"? WTF ... again. Sounds dangerous. Never saw that before in that list, and than my brain rotated … what the heck is "Buffer Exterminate", i have an idea, something is ringing in my head, but somehow my long-term memory management unit of my brain was unable to stage this information in to current working set. Okay … ask Dr. Google.

Metalink [ID 259137.1] is of great help here. The "buffer exterminate" wait occurs (and can only occur), when the buffer cache is shrunk dynamically and a session wants to access data that is in the granule of the buffer that is chosen by Oracle for removal from the buffer cache. The session wanting the block has to wait until the buffer to be removed has been freed to read it from disk then. You can't simply read the block from disk without waiting, as the block in the granule may represent a new state of the block than the one on the disk an simply reading the one from the disk would yield just outdated data. So you can just wait until the granule has been released.

Before you ask, what a granule is: Oracle doesn't allocate memory in the SGA bytewise, but in so called granules. A granule is 4 MB of memory, when your SGA is up to 1 GB when the instance starts. It's 16 MB when your SGA is larger than 1GB at startup.

In Oracle DB 10g, there is a feature called "Automatic Shared Memory Management". The idea is, that Oracle itself monitors the load and configures the layout of the SGA. I think of automatic means as a very good feature. It's like with manual and automatic gearboxes. Surely, a good driver can accelerate faster with a manual gearbox than with an automatic gearbox, however an automatic gearbox is faster and better than 99% of all drivers. That said, given the existence of behaviour patterns explained by the Dunning-Kruger-effect (h/t to Chris Colomb for hinting me to this interesting phenomenon), 99% of drivers think are part of the 1%. This is especially epidemic in Germany. But back to the issue. It's the same with tuning of systems

You activate the ASMM by setting the parameter SGA_TARGET to a value unequal to zero. Now the system sizes the buffer cache (DB_CACHE_SIZE), shared pool (SHARED_POOL_SIZE), large pool (LARGE_POOL_SIZE) and Java pool (JAVA_POOL_SIZE) automatical within the limit set by SGA_TARGET. If one of the other parameters controling one of the mentioned memory areas is set to a value other than 0, the value is assume as the minimum amount of memory.

Of course: When you have fixed SGA_TARGET and you want to grow one part, another has to shrink. It's obvious that you can't do shrinking simply by throwing the block out of the memory. There may be dirty blocks in that granule(changed blocks that weren't written to disk so far by the database writer to the database file, just to the redo logs).

This works really good and this relieves the admin from investing time to find good values for some of the most important SGA parameters.

However if your database tries to move memory back and forth from one kind of shared memory to another tens of times per hour this is surely not without impact on your database performance. I had such a situation in this case. The system started to move around memory in minute intervals just to move it back a minute or two later. As most automatic systems they will work perfectly within their specification, but you may hit a situation where tries to get most out of a situation with restricted resources, where the SGA is confronted with the situation that all components want more memory and as soon you remove memory from one parts, the other part cries and wants its memory back. That's similar to the argument with your significant other about what's the half of the blanket. Better have two blankets Or to get back to the topic: Have enough SGA ...

How do you find out, how many resizing operations took place? You can look that up by a select statement as described in this blog:

select START_TIME, component, oper_type, oper_mode,status, initial_size/1024/1024 "INITIAL", target_size/1024/1024 "TARGET", FINAL_SIZE/1024/1024 "FINAL", END_TIME from v$sga_resize_ops order by start_time, component;

With this statement you will see the recent history of resizings.

In this case a slight increase (4 gigs) of the target size of the SGA moved the system away from growing and shrinking the buffers back and forth. And not a single "buffer extermination" was seen afterwards and no peaks in the wait time statistics and the number of resizing ops was down to one per hour. And that was more than okay.

Other solutions would be the deactivation of ASMM (by setting SGA_TARGET to zero) and configuring everything manually(doing it the old way) or setting some reasonable minima for the values controlled by ASMM. Important to know: In the amount specified SGA_TARGET is not only the amount of memory for the four parts mentioned before, it's for the complete SGA. So the amount of memory used for other parts of the SGA than those managed by ASMM has to deducted from the SGA_TARGET size. And this reduced amount of SGA is available for the SGA areas managed by ASMM.

The year almost over. As usual at this time i'm finalizing my tax declaration at the moment. But nevertheless i wanted to write a short article about the last year.

The year started quite bad. In March i had a really bad argument with a good friend and we didn't talked a word since that. Albeit my communication habits could have been better in that days, i assume the writing were on the wall before that day in march and it was just a matter of when, not if, when the argument broke loose.

But starting in June the world just gone downhill for me. It started with the events that finally led to the state the blog currently is. That is really a large thorn sitting in the flesh hurting me all the time. However the consequences were inevitable.

The job had two really great satisfying moments for me with larger successful projects, and a number of others were i was able to help others to find problems. That was really and deeply satisfying moments.

Other things weren't that well as well: Just two main ones … in my vacation i was ill at the weekend and needed almost the complete vacation to get really rid of that nasty bug, and then just 2 days in the job again i broke my ankle and had a torn ankle ligament. While it was painful at least my body seems still to have its good regeneration capabilties in regard of injuries. I don't need painkillers any longer and i'm just wearing the aircast orthotic device at the moment. 24 days is not that bad.

On the good side: My house is almost completed … perhaps 3 month of work in the evening left. So i will have more time to other things soon: Like really pursuing new opportunities - career as well as personal - , finding a new hobby in the year one after almost two years of construction work that rules out any other free time things. Like reading through the heap of books that piled up this year, to get rid of several kilos. Finding a new car in February. Thinking of a nice S60 at the moment. Trying not to break my bones again. And perhaps a longer vacation abroad this year.

I wish you all a good transition into the next year and successful 2012.

I wish you all a few quiet days with your loved ones and may all you wishes come true!

Ankle is really hurting. A big thank you to the pharmaceutical industry for providing pain killers. However i would like to concentrate a little bit better again. However things are getting better day by day ...

Just thought, that our perception of the world is heavily influenced by the representations of it ... like a map. Just to demonstrate, a question: Assume you are a pilot. You are flying from Frankfurt to San Francisco. You are a lazy pilot and you are allowed by air traffic control to set the shortest direct course on the autopilot and don't touch it again. What is the initial heading you are setting? Just answer with the first thought! No degrees needed ... just something like west,east,south our southeast.

Damned, now i have a lot of time ... because of a mismatch between the real number of residual steps and the perceived number of residual steps on a stairway. Now i'm a proud owner of a fractured ankle join (the doc is not totally sure if it's a ankle ligament injury as well, next x-ray middle of next week will show it). Can't need that right now ...

The newest revision (the rev valid as of December, 2nd) of the document "Partitioning - Topic: Server/Hardware Partitioning" naming the technologies allowing you just to license a subset of the available processors in a system for a Oracle software has been modified: Oracle VM for SPARC is now explicitly named as a hard partitioning technology.

Damned axe … is it really that hard to believe that an automatic system in Debian makes a dumb decision based on dumb assumptions (classic BIBO - bullshit in , bullshit out) ? It has nothing to do with a butchered up distribution by a hosting provider or an error between the keyboard and the chair. I would consider it a bug if it would deliver a differen result given the input to the system and the ruleset codified in the system.

Per default an init.d script is inserted by the insserv in the init.d scripts in debian. If you have ever wondered about the comments section at the beginning of the init.d scripts in debian … they are the input for insserv to build up the dependency graph.

root@narf /etc/init.d # head ssh
#! /bin/sh

### BEGIN INIT INFO
# Provides:             sshd
# Required-Start:       $remote_fs $syslog
# Required-Stop:        $remote_fs $syslog
# Default-Start:        2 3 4 5
# Default-Stop:
# Short-Description:    OpenBSD Secure Shell server
### END INIT INFO

This information is used to put the init.d script at the right place of the sequence of startup.

There is a number of scripts just dependent on the existence of the local filesystem. This ones are run with the sequence number 01 thus first.

root@narf /etc/rc2.d # ls -l S01*
lrwxrwxrwx 1 root root 20 Oct 25 10:46 S01fancontrol -> ../init.d/fancontrol
lrwxrwxrwx 1 root root 19 Oct 25 10:40 S01mailgraph -> ../init.d/mailgraph
lrwxrwxrwx 1 root root 17 Oct 25 08:58 S01rsyslog -> ../init.d/rsyslog
lrwxrwxrwx 1 root root 19 Oct 25 10:45 S01saslauthd -> ../init.d/saslauthd
lrwxrwxrwx 1 root root 14 Oct 25 10:42 S01sudo -> ../init.d/sudo

One of the services started first is syslog, and as you may have recognized in the head of the ssh init.d script, ssh depends on it. So it gets a higher sequence number and thus it's started later:

root@narf /etc/rc2.d # ls -l *ssh
lrwxrwxrwx 1 root root 13 Oct 25 10:44 S04ssh -> ../init.d/ssh/pre>

But wait, what has happened with 02 and 03? They are used by a single service each:

root@narf /etc/rc2.d # ls -l S02*
lrwxrwxrwx 1 root root 17 Oct 25 10:41 S02apache2 -> ../init.d/apache2
root@narf /etc/rc2.d # ls -l S03*
lrwxrwxrwx 1 root root 17 Oct 25 10:44 S03openvpn -> ../init.d/openvpn

Why do they have this special treatment. It's pretty simple. Both services are flagged as being interactive, as they could ask for some user interaction.

root@narf /etc/rc2.d # grep -i "INTERACTIVE" *
S02apache2:# X-Interactive:     true
S03openvpn:# X-Interactive:     true

In this cases both might probably ask for a password for the key. They have to start early to be sure that nothing can gets and block the tty in order to enable both scripts to show the password dialog. Keep in mind that we are in a state where no getty is running. However they can't earlier, as both rely on syslog. Essentially they are started as soon as possible and before other scripts with the same dependencies. And thats basically a part af the problem.

And this essentially leads to the situation, that apache2 was started before ssh.

Another reader hinted me to the fact that lightttpd starts after ssh. At first this is just of cursory interest at a situation where the webserver in question is apache. At second the dependencies are different. At first the init.d script for lighty has no # X-Interactive: hint and at second there is a condidtional (obey it, when it's installed and configured to run, otherwise ignore it) dependency to fam (file alteration monitor) that the ssh script don't have. And fam relies on portmap. Thus the dependencies are a little bit longer thus it's no wonder that you see lighty starting after ssh.

I won't comment what i'm thinking about such a … a… a… solution ...

With a link to apple.com, too. Directly on the front page ...

The "Steve Jobs" links to:

BTW ... the hungarian folk dancers explain other sort algorithms as well: merge-sort, insert-sort, bubble-sort, shell-sort and select-sort

Recently I was doing some work in regard of tuning systems. There is something i really hate about this topic of computing: Tuning scripts. You find them on google easily and i find them on systems at customers quite often.

Simply said: I hate them. The reasons for it are simple. For example recently I found a system with a networking tuning script dating back into 2003 or so. The problem: It was meant to increase some of the settings. However many of them were already higher in the default config of current Solaris 10 versions, thus the tuning script essentially reduced the parameters and thus reduced the performance.

Futhermore: Tuning is a lot about understanding things. Understanding how things work together. On a systemic, on an architectural level. How an application loads all the rest of components. Just dropping a script downloaded from a website found by Google - into /etc/init.d is not about understanding things.

You have to carefully consider each change from the default about the impacts. You have to check each setting, if the setting hasn’t already overtaken by the years. You have to recheck it it with every major update of your environment. You have to recheck it with each new technology you are using in your system. Network tuning scripts dating back to a time when 100 MB/s were normal and 1 GB/s are fast aren’t necessarily up to the task in a time when 10 GBit/s are fast and Infiniband IPoIB networks deliver even more. You had to turn different knobs in a time, when cpu time was precious. You’ve tuned for minium cpu utilization. CPU isn’t a large factor today, you tune for minimize latency or maximize throughput. You have to know what you want to aim for, because minimum latency and maximum throughput are often mutually exclusive. Do you want an extreme or a target in between.

Just using a script to tune something doesn’t lead you through all the thought to make really good tuning decisions.

There are some basic rules from my point of view:

• For Solaris there is for each major update a document updated called “Oracle Solaris Tunable Parameters Reference Manual”. It’s available at here.Check your tuning script at each update against this document in order to ensure that you settings make sense. The 817-0404 is a great read anyway, as it gives you an condensed overview over the possible official knobs to turn.
• Each tuning script has to be carefully designed for each system for each workload. An /etc/system for a database make perhaps no sense for a webserver or may even reduce the performance and vice versa. That’s the reason why I don’t like tuning scripts as part of jumpstart configurations. An NFS server delivering many small files to many clients may need other tunings than a server delivering few large files to a few clients.
• Understand! Understand how components of your architecture interact. Understand the load. Monitor the load for a while. Tuning is at foremost doing a lot of research before and just at last about putting some lines into some scripts. That’s one of the reason why I like tools like dtrace in general, and the scripts from the dtracetoolkit especially. Those tools are like the professionals tools of a mechanic. They can help you to understand the load. The equivalent of the basic set of screwdrivers: All the *stat tools are helpful, too. You need them to know where you can start with the more advanced tools, that are often too focused.
• Have a benchmark load. Have a mechanism to reproduce the load on the system. The more realistic and the less synthetic the better. So you can quantify the impact of each tuning.
• Just change one knob at a time. Otherwise you will never know what setting really helped you or which setting hurted the performance
• Of course: Experience helps a lot in shorten the research phase and sometimes there are some low-brainer settings. But even when you think “Nah … this is just a webserver. Let’s use script number 7b”, you may be surprised by interesting side effects. So there is such think like a no-brainer in tuning
• Large standard applications have often a set of certain tuning parameters that the vendor suggest as a good baseline: At first - look them up in the tuning guide in order if the ISV tuning guide wasn't overtaken by the defaults in the OS. At second: Check them with the vendors manuals with each release ... had once a performance problem with customer that used the tuning parameters suggested for - let's say - version 2, however there weren't that reasonable with version 5.
• It's a good practice to create two baselines: At first check the service performance with a totally untuned system ... just modify settings you need to get the software up-and-running.Run your benchmark. Configure the system with the vendor suggested parameters. Rerun the benchmark. Now do your own tunings. Rerun the benchmark with each knob you turn. And ask yourself: Is the performance impact worth the deviation from the standard parameters from a plain vanilla installation? With such baselines you can assess the impact and thus the value.

A commentator at hackernews asked how i think about -9. In my opinion: It's widespread use is a similar plaque like the –f switch. And this is pretty easy to explain (I'm simplifying things a bit).

-9 is a shorthand for SIGKILL. When you send a SIGKILL to a process, the process is terminated immediately. You can’t catch this signal, you can’t ignore it. A kill with -9 sends this SIGKILL to a process.

A kill without -9 sends a SIGTERM to process. It terminates the process like SIGKILL. However a process is allowed to catch it in order to execute a signal handler … or just ignores to ignore it. A signal handler is nothing more than a code path that is executed when the process receives a signal.

So when you kill a process with a normal kill you give the process the chance to clean up behind itself, to make files consistent, to roll back changes in the case the process isn’t using some transactional mechanisms when changing data, to delete temporary files … and so on ...

It's a good style to write such signal handlers and in many programming languages it's pretty easy. For example in perl:

$SIG{TERM}=\&signalhandler_TERM; 

sub signalhandler_TERM{ 
     $SIG{TERM} = \&signalhandler_TERM; 
     print "leaving process\n";
     #some code to clean up behind you 
     exit(1); 
}

When you send a -9 to a process you take away this chance from the process. It’s killed instantly … even if it just started to modify your files, fscking up your data in order to put it in a new form, even when you have created dozens of temporary files filling up /tmp. Things like that …

Killing a process with -9 is the last possibility. However I see people using it too often too early. A second after the normal kill is send a pgrep on the process follows. Still there and the sword of -9 is falling down.

When a process doesn’t disappear immediately after sending the SIGTERM, it may be just busy to follow your order of terminating itself and is cleaning up things. When your application is dependent to precious resources at cleaning up (for example IOPS on your rotating rust) the process of cleaning up may take a while.

The implicit question in any process, that doesn't react to a normal kill via SIGTERM is the question why it doesn't react to the signal. Just sending a -9 when a normal kill didn't worked is like "Do not care". Monitoring the process with truss or strace what the heck the process is doing after getting the SIGTERM is a good first step. Perhaps you see some cleanup work and know that you just have to wait a little bit longer. Writing a core dump of the process with gcore is often a good second step to save evidence for future research why the process didn't reacted.

And then … and only then … a kill -9 may be feasible.

In short:

Always try a normal kill first. There is just one execption.
• The exception: You know that the integrity of data is depending on the fact, that the process has no chance to write anything for example by executing a signal handler. Consider it as something like a forced panic on process level. The panic is the admins friend as it ensures that nothing is written to disk as soon as the OS is detecting something worth to trigger a panic. A panic isn't a "insult the admin"-thing … it's about protecting data.
• kill -9 is your emergency stop switch and shouldn't be part of your normal administrative procedures in any way
• Just sending a -9 after a normal kill without thinking about reasons is bad administrative style.

Yesterday i held my deduplication talk at the Froscon 2011. I think it was acceptable and the lecture room 3 was really filled. To be honest: I don't expected such an audience at that time. So a big thank you for all who attended the talk.

My talk started at 10:00 o'clock and thanks to apron parking, a rental car pickup in the feeled middle of nowhere in Düsseldorf and a larger traffic jam near Cologne led to an arrival at 09:58 … with a presentation at 10:00 and the urgent need to visit the restroom before a disaster happens.

I had just a short time at Froscon … i had a date with my house, that still needs work and don't accept when i'm saying "No … not this weekend". And thus i was back in the rental car at 11:45 and was back in Lueneburg shortly after 15:00, ten hours after leaving home. 3.30h for 430km … not that bad for an Opel Corsa.

However there is something i've recognized: I'm really missing standing in front of customers and technically interested people and trying to transfer my enthusiasm for technology. I'm really missing being on the road.

A few days i linked to a paper, that explains why it's hard to really delete all data from an SSD. You will find the link to the paper in that article. In those paper, the authors argued, that the SSD need better mechanisms to securely delete data. A paper from 2010 goes in the opposite direction, argueing that SSD make forensics more difficult due to some tuning tricks used in SSD.

When you look at the paper "Solid State Drives: The Beginning of the End for Current Practice in Digital Forensic Recovery?", the situation gets a little bit more complex. The document looks from the perspective of people, who try to gather data from disks for example for law enforcement.

The basic problem is: To keep performance at a high level, modern SSDs are using garbage collection. This is a good thing, as writing into an already used block needs twice as long as writing in an empty block. So the internal controller of the SSD runs a garbage collection that erases areas before the free areas is needed. This is done, when the SSD is idling. On page 8 it's

The X axis shows time and the Y axis shows the approximate percentage of the drive that has been zeroed out. In all three SSD runs, around 160 seconds from the log-in time (i.e. around 200 seconds from power-on), the SSD begins to wipe the drive. After approximately 300 seconds from log-in, the SSD consistently appears to pause briefly before continuing. 350 seconds after log-in, the SSD’s pre-existing evidence data has been almost entirely wiped.

To comparision: On a quick formated HDD, they were able to recover almost all data.

The problem for the digital forensics is: The disk is doing this after a short time you've powered it up. It just wipe out those blocks it knows they are unused by some clever algorithms. And it gets even worse: When your OS uses the TRIM command, it doesn't have to find out by clever tricks, the OS tells the blocks that can be wiped out and it directly wipe them after the TRIM command when there is some spare on the SSD, as the SSD is doing this on it's own controller. And this is the next problem: You can't do anything top stop besides of powering the device down. Perhaps the only way to get to the data is to use a device like the flash chip reader from the article about secure deletion to circumvent the flash firmware in total. But i don't think a police team can seize the computer and power it down in 300 seconds. However it's still not a secure delete, as they were still able to get fragments from the disk.

The conclusion is really interesting:

It seems possible that the golden age for forensic recovery and analysis of deleted data and deleted metadata may now be ending.

No wonder law enforcement agency are so keen on having trojans in their repository. Analysing SSD looks like a really futile endeavour.