I see it all the time: people using Best Practices and ending up in a big mess afterwards. This time it is the mount option forcedirectio. According to a NetApp best practice for Oracle one should always use forcedirectio for the File Systems that store the Oracle Files. So people migrating to these systems read the white papers and best practices and then run into performance problems. A quick diagnosis shows that it is all related to IO. Ofcourse the NAS is blamed and NetApp gets a bad reputation. It is not only NetApp, it is true for all vendors that advice you to use the forcedirectio.
What does forcedirectio do?
It basically by passes the File System buffer cache and because of that it is using a shorter and faster code path in the OS to get the I/O done. That it is ofcourse what we want, however you are now nolonger using the File System Buffer Cache. Depending on your OS and defaults, a large portion of your internal memory could be used as the FS Buffer Cache. So most DBAs don’t dare to set Oracle Buffer Caches bigger than 2-3 GB and don’t dare to use Raw Devices. So the FS cache is used very heavily. It is not uncommon to see that on Oracle Database uses 2 to 10 more caching in the FS than the Oracle Buffer Cache. I have seen a system that used 20 to 40 times more caching the FS than the Oracle Buffer Cache.
So just imagine what happens if one than bypasses the FS Buffer Cache :) Most I/Os will become physical I/Os. Especially the reads will suffer. If the reads suffer, the end user response time will suffer directly. If end users are unhappy, managers will start to realise that they are important and they will so their face and interest again.
So how can we fix all of this? Easy. Just remember that if you remove the FS caching, you want to cache it some where else! And don’t rely on the NAS or SAN cache. The best and the cheapest place to cache the data, is the Oracle Buffer Cache. This will help to improve your Buffer Cache Hit Ratio again :)
So if you use forcedirectio, have a look at your Oracle Buffer Cache size!