Stuff that I have done through learning by error!

Change is Certain

2011-06-24T07:36:00.000-05:00

Sneak Peak: Ubuntu 10.04 LTS Server with Native zfs here I come!!!

Since my last post about 8 months ago, I have been living with my FreeBSD 8.2-RELEASE/STABLE based NAS server. To get to some level of reliability with it, I stripped it down to the OS and SAMBA and am using it primarily for archive. I have moved my SickBeard/SABNZBD/Twonky services to a much smaller Windows 2003 Server that was already in place as my Active Directory Domain Controller and they have been running much more stable much to the delite of both my wife and myself.

However, I recently returned from a photography trip to Utah and ran into some of my old demons, not daemons, when transferring the 40GB of photos to the server. The memory leak associated with zfs, Samba and sendfile would render the server unconscious pretty quickly. In the end I used rsync, bypassing Samba, to get the photos from my computer on which I process the photos to the server. My original design here called for me working directly from the server copy and using zfs snapshots to provide Oops insurance.

Last week during some of my daily tech scanning, Thank God for Google Reader :), for the first time I began to get a feeling that Native zfs on Linux may be stable enough for me to lean on it. Over the weekend, I backed up my NAS server, got a fresh 8GB CF Card, and loaded Ubuntu 10.04 LTS Server on it. LTS in Ubuntu parlance means Long Term Support. This version of Ubuntu is and will be maintained for some number of years and has the reputation for being very stable. My install was the base Ubuntu install only adding OpenSSH and Samba as added servers.

I then compiled and installed the zfs drivers from Darik Horn's PPA, imported my pools (zpool Ver 15, zfs Ver 3, no compression), and voila, I was in business (more details below on how to do this). Without any tuning whatsoever, the performance test results are as follows:

Native write to zfs
dd if=/dev/zero of=/mytestfile.out bs=1k count=4194304
4194304+0 records in
4194304+0 records out
4294967296 bytes (4.3 GB) copied, 212.516 s, 20.2 MB/s

Native read from zfs
dd if=/dev/zero of=/mytestfile.out bs=64k count=65536
65536+0 records in
65536+0 records out
4294967296 bytes (4.3 GB) copied, 69.888 s, 61.5 MB/s

In short - WOW!! If you look at my earlier entry, you will see that on release code from FreeBSD, not development code, the write performance almosts matches the FreeBSD result of 22 MB/sec and while the read code is only about 2/3rds of the FreeBSD 92 MB/sec it is still plenty fast enough.

But the real surprise was on Samba. Using robocopy to copy the file create above down and back up to my box, I measured the following:

Read: 45 MB/sec
Write: 35 MB/sec

This exceeded the tuned read performance on FreeBSD (39 MB/sec) and matched the tuned write performance (35 MB/sec). But the kicker here, my test on FreeBSD were only with 1 GB files on a box where I have 2 GB of RAM. So there is no doubt a memory aid here. The test above was with a 4 GB file with 2 GB of RAM, so the memory impact on the test is much smaller.

So the short of it is, I now have a box that in early testing is equal or slightly more performant as a NAS server than my previous efforts and doesn't contain the instabilities inherent in a zfs on FreeBSD solution when running on modest hardware (Atom 330 with 2 GB of RAM). I think that the days of FreeBSD's dominance in the open source world as the zfs server of choice are over - flame on FreeBSD Fan bois!

In reality, there is no doubt that FreeBSD is a good if not great OS and that they have contributed greatly to the success of zfs with respect to the masses. But the further reality is that the OS and its zfs implementation don't work equally great across the use cases. IMHO, they work great for small boxes with minimal loads like many home NAS servers and they work well in very large boxes with plenty of RAM for zfs where routine transfers over Samba are not 20 times the size of the RAM in the system. But my use case of a home NAS user with the need to load large volumes of data on a fairly regular basis is not well handled.

Stay tuned for furthers stories of what I hope are future successes!

PS: Here' the quick how to portion

1. Install Ubuntu Server 10.04 LTS from the iso image.
2. Take all defaults except for additional software, select OpenSSH and Samba to be added
3. After install is complete and the server is rebooted, apt-get update
4. apt-get install python-software-properties
5. add-apt-repository ppa:dajhorn/zfs
6. apt-get update
7. apt-get install dkms
8. apt-get install ubuntu-zfs

In theory, step 7 should not be needed as the dependency resolution in apt should take care of this. However, I ran into a bug and this is the easy workaround.

A new day, a new perspective

2010-10-01T09:16:00.003-05:00

OK, I won't butcher another song :), but its amazing how much 24 hours can change ones perspective. I have traced the majority of my performance issues to a bad WD15EADS drive. While the drive isn't reporting any problems, it will not transfer at a speed greater than 10 MB/s. The drive is still under warranty, so I am going to file a warranty repair request with WD today. In the mean time, I have configured my 2 good WD drives in a zfs mirror that is large enough to hold everything that I need to have online for a while.

Thanks to the anonymous commentor on my earlier post who called my attention to the correct sector size for the EADS drives. It is 512 bytes, not 4KB, so I have removed my gnop configuration.

Irrespective of the drive, I have finally been able to confirm that my current motherboard's ICH7 implementation does not support AHCI so I am definitely limited to ATA100 performance. With the bad drive out and restoring with zfs send/receive, I did see all of my drives peak at 100 MB/s from time to time. I can now reliably get in the box transfer of in the 35 - 45 MB/s range. This is more inline with my expectations.

I'm still of the opinion that if you are going to use zfs on FreeBSD 8.1 that 8 GB of RAM is the minimum. While I now have a reasonable solution with 2 GB, it has taken an innordinate amount of my time to get it there. I hope others reading this will be able to learn from my experience and save a lot of time.

Without going into a lot of boring detail, the issue with the RAM is as much a problem with FreeBSD as it is with zfs. If you are doing large transfers (i.e. > 10X RAM), FreeBSD 8.1 will not release its Inactive memory fast enough and the cache will starve off thereby at a minimum impacting performance. The worst part of this is that FreeBSD will panic and shutdown from time to time. And since the FreeBSD support for my current board's ACPI isn't solid, the box locks up and doesn't recover without physical intervention. So from a stability point, it seems to me that you need to have enough RAM to make it unlikely that you will starve off the cache and put the OS in a position to panic. I am hoping, that this will be addressed in 9.0 but I'm not enough of a FreeBSD kernel geek to know.

Given that I am back to a sufficient level of performance and the reliability is good, as long as I don't try to remotely execute big transfers, I am going to hang with my current motherboard. In looking around a bit, I received some good recommendations regarding Intel's I3 as well as their current ATOM chips. I like the I3's; however, the cost is a bit more than I want to go at this time. The current 4 GB limitation on the ATOM keeps me from wanting to make an investment in them at this time. I'll keep my eyes open and when chip makers present a sufficiently interesting motherboard, I'll look at upgrading.

lbe

PS: if you are interested in what my current /boot/loader.conf looks like, here it is. Given the limitations of my current level of ignorance, I think this is about as good as it gets for FreeBSD 8.1 on a dual core Atom 330 configured as a NAS server using zfs.

kern.ipc.shmmax=67108864
kern.ipc.shmall=32768
vm.kmem_size_max="1024M"
vm.kmem_size="1024M"
vfs.zfs.arc_min="256M"
vfs.zfs.arc_max="784M"
vds.zfs.vdev.cache.bshift="16"
vfs.zfs.vdev.cache.zise="10m"
vfs.zfs.vdev.cache.max="16384"
vfs.zfs.vdev.min_pending="4"
vfs.zfs.vdev.max_pending="12"
vfs.zfs.vdev.aggregation_limit="131072"
vfs.zfs.vdev.ramp_rate="2"
vfs.zfs.vdev.time_shift="6"
kern.ipc.maxsockbuf=16777216
kern.ipc.nmbclusters=32768
kern.ipc.somaxconn=32768
kern.maxfiles=950000
net.inet.tcp.delayed_ack=0
net.inet.tcp.inflight.enable=0
net.inet.tcp.path_mtu_discovery=1
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.recvspace=65536
net.inet.tcp.rfc1323=1
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.sendbuf_inc =524288
net.inet.tcp.sendspace=65536
net.inet.udp.maxdgram=57344
net.inet.udp.recvspace=65536
net.local.stream.recvspace=65536
net.local.stream.sendspace=65536
net.inet.tcp.sendbuf_max=16777216
aio_load="YES"
net.inet.tcp.mssdflt=9142

Motherboard Blues

2010-09-28T08:25:00.001-05:00

My apology to Wilbert Harrison who wrote the blues classic Kansas City. Hum along if you know the tune;)

I'm going to Insanity, Insanity here I come
I'm going to Insanity, Insanity here I come
They got a crazy way of serving there
And I'm gonna get me some.

I'll be standing in the corner
In the corner of FreeBSD and hardware support
I'm gonna be standing in the corner
In the corner of FreeBSD and hardware support
With my Insanity server
And a bottle of Insanity RAM.

Well I might take IDE
I might take SATA, but if I have USB
I'm gonna get there just the same
I'm going to Insanity , Insanity here I come
They got a crazy way of serving there
And I'm gonna get me some.

I'm gonna pack my files
Leave at the break of dawn
I'm gonna pack my files
Everydrive will be sleeping
Nobody will know where I've replicated
Cause if I stay in the zpool
I know I'm gonna scrub.
Gotta find a friendly mobo
And that's the reason why,
I'm going to Insanity
Insanity here I come
They got a crazy way of serving there
And I'm gonna get me some

Its amazing where my minds goes sometime. This little ditty started with just the first line and before I knew it I had the whole song done. If you like blues, or don't know what blues are, then check out Roy Clark's version of this great standard. He adds the line Kansas City, I spent a week there one day. I feel much like that with my current learnings: Insanity, I spent a week there one day.

Over the last couple of weeks since my last post, I have been tweaking and testing and overall have been unhappy with where I currently am with my server. In short, I have reached the conclusion that 2 GB of RAM just isn't enough RAM for a zfs file server "if" you want a consistent level of "reasonable" performance (i.e. aggregate file transfer >= 30 MB/sec. My current motherboard only supports 2 GB. So, I am halting my testing with it and am turning my search to finding its replacement.

I have started a thread on Hard Forum, Need MB rec for Low Power, ZFS, FreeBSD, Home NAS, asking for help.

I'll let you know what decision I make!!

lbe

FreeBSD ain’t free, if I value my time and include the cost of confusion!

2010-09-08T07:38:00.001-05:00

There I said it. Flame on to all of you FreeBSD idealogues!!! If you look through the history of this blog, you will see that I have been about addressing my NAS needs for a while, well over a year with FreeNAS running on FreeBSD 7.2 and now with FreeBSD 8.1 rolling my own. If you look at the timestamp of the entries, you will notice some significant gaps.

There are several sayings that could be applied during these gaps. “No news is good news” or “ignorance is bliss” are examples of the positive and the polite; however, to do it justice requires Latin: “non impediti ratione cogitationis.” In English, this translates to “unimpeded by the thought process.” Like most things that I know, I can’t take credit for this phrase. I learned about it from a couple of my idols, Click and Clack the Tappet Brothers of Cartalk.

I now know that I wasn’t even thinking about somethings that could have and should have bothered me. Unfortunately, now that I know about them, and think I am on a path to addressing them, I am now worrying about what else I do not know! Read on ...

What have I learned?

1. If you want to run zfs without worries, run it on some big honking hardware!

2. Even with big honking hardware, you should still be worried. You will still get yours, just later instead of sooner:)

3. Reliably running zfs on FreeBSD on commodity, low power hardware requires the sacrifice of millions of brain cells on a weekly if not daily basis!

4. Always have backups!!

5. Always have backups of your backups!!!

Gripes first -

I won’t lie and say that I always RTFM, but I RTFM much more than most. So when I started out of this venture to build the perfect home NAS server, I spent a lot of time in my recliner with my laptop on my bulging belly, both which can be attested to by my wife, considering not only what choice should I make but also how should I make it. I really thought I had found an appliance approach in FreeNAS with which I would be happy. Me being happy with IT appliances in general is a non sequitur because I always want to add some things that aren’t done just the way I want. But FreeNAS got me a long way to where I wanted to go.

Initially FreeNAS was reliable and seemed to be performant, but I had to go and add sabnzbd to download my wife’s, yes it must be her fault :), favorite TV shows from supernews and add Sick-Beard to handle setting up the downloads. What’s the harm with a couple of little Python programs? It wasn’t that difficult to hack Python 2.6, sabnzbd and Sick-Beard into my embedded version of FreeNAS. For brevity’s sake, I’ll skip the details. Prior to doing this, FreeNAS was running pretty well though I had figured out the FreeBSD 7.2 and my Atom 330 based MB didn’t get along well with hyper-threading enabled. But following this, I started to notice performance problems when synchronizing my digital photography library from my desktop to the server using a robocopy script that overwrote everything every time, my error, not robocopy. But I said, what the hey and ignored it.

Then the server started crashing from time to time and I couldn’t figure out why. I enabled a syslog server, poured through the logs, couldn’t find anything, couldn’t make it crash, ... I was going crazy trying to figure out what was up. I reached a point where I was ready to give up on FreeBSD and go back to my tried true reliable Linux and ignore file system goodness until btrfs is ready for use. But before totally jumping, and having to migrate data, I decide to stick a different CF card into my CF/IDE adapter and install FreeBSD 8.1 RC2. I decided I could forego the FreeNAS GUI if I could get stability and consistent performance. Voila, it appeared that I had it, until ... I tried to a duplicate of about 500 GB of data to an external drive to start keeping in a drawer at my office as an offsite backup of the critical files. When doing this, I started to see big time performance problems and started to be able to crash the server somewhat regularly with a cryptic but key error message about running out of kmem.

In researching this, I learned the real meaning of some of the tuning values that I had in my /boot/loader.conf – vm.kmem_size_min and vm.kmem_size_max as well as vfs.zfs.arc_min and vfs.zfs.arc_max. With a little trial and error, I came up with appropriate settings for my little box and eliminated the crashes; however, the transfer performance would drop off as rsync would run over time to replicate my critical data. Finally, I came across references to problems with Western Digital’s 1.5 TB (WE15EADS) Green drives that I am using.

The drives have a 4KB physical sector but report 512 Bytes to the BIOS. So performance drops off on really big writes because zfs on FreeBSD sends 4KB of data to the drive as 8 separate writes of 512 bytes, which requires the firmware in the drive to increase its work load by an estimated factor of 60 (1st 512 Bytes - write 4KB, 2nd 512 Bytes, read 4K, write 4K, ..., 8th 512 Bytes, read 4K, write 4K -- so 4KB of writes become 4KB write + (4KB read + 4KB write)X(4KB/512Bytes - 1) = 60. The drives built in 32 MB cache helps until it fills and the zfs arc kicks in and then the arc begins to fill. So all in all, no big deal right?

Actually it is a very big deal if you are writing files to zfs that are larger than your arc plus the size of the buffer on the drive. And because of the behavior of the zfs arc cache code on FreeBSD, notice I am not calling it a bug because I don’t know enough to point to where it is, the allocated memory is not made available to be re-used at a rate fast enough to sustain the transfer speed and the throughput drops over time.

You can observe this yourself by executing a copy watching the free memory in top drop while the inactive memory increases. This is further exacerbated by memory in the wired pool not being marked as inactive quickly enough. This “appears” to me to indicate that FreeBSD and/or zfs is too aggressive in grabbing memory for caching relative to the rate at which it releases it. This results in transfer speeds well below those of other operating systems running on the same hardware.

Fortuitously, BSD provides gnop, a drive geometry abstraction layer, to create another lie to offset the lie told to the BIOS (512 Byte sectors instead of 4KB sectors) by the drive. Unfortunately, this layer is not saved as metadata on the drive so it will not persist through a reboot. Fortunately, I found a script on a Japanese web site, nothing on the site except for the script and the tests were in english, that I used to create the gnop geometry entries prior to starting zfs.

This significantly increased my performance; however, over time, the transfer rate would still drop because of the aforementioned memory allocation issue, note I didn’t say problem or bug ;). But, I found another person who created a one line perl command, yes perl to the rescue!!! who needs a stinking snake ;), that tried to allocate an exorbitant amount of memory. This does trigger the FreeBSD memory management to release the memory and the kernel kills the overreaching little process to boot. This result in freeing up the memory for re-uses.

So with the gnop geometry implemented and running the perl one liner in cron, I am able to sustain a whopping 9-10 MB/sec sustained transfer rate that pretty much renders the server unusable while big transfers occur. While this isn’t great, it is much better than getting down to 1-2 MB/sec and crashing!

Fortunately, these big writes don’t occur too often, so most of the time, my little low power box can pump transfer rates on the order of 30 - 40 MB/sec as long as the files don’t exceed about 750 MB, based upon my current tunings. I am currently in the process of implementing L2ARC using a CF card in a CF/IDE adapter and a ZIL using higher write speed CF card (my cheap version of SSDs). I am also going to add a third CF on which to place the cache and log directories for sabnzbd and Sick-Beard as well as the SQLite3 database for mediatomb to insure that these applications remain fairly responsive during periods of heavy SMB or rsync usage. While this sounds expensive, I ordered all of the parts needed from Amazon for slightly under $100. You could use USB flash drives with similar benefits if you want to go even cheaper. But I am not always happy with FreeBSD’s performance with USB media so I decided to go the CF/IDE route.

Recommendations

I’ll put out an update in the next couple of weeks to let you know how things go along with what I think is a prudent methodology for others to use in selecting their hardware, configuring their OS and zfs, and in tuning.

If you feel like building a home server using zfs on low power, relatively wimpy software, take 2 aspirin and lay down until the feeling goes away J! If you can’t resist, here are my top ten tips:

1. Run zfs on 64-bit capable hardware.

2. Hyper-threading may be a problem with FreeBSD kernels and some chipsets. The problems are both performance and stability related. Test, Test, Test!!

3. Put as much memory as you can in the box, 2 GB minimum >4 GB recommended.

4. Be careful choosing disks. Stay away from advanced format disks that don’t honestly report their physical sector size.

5. If you use disk that report a different sector size than physical to the BIOS, use gnop to correct

6. Use raw disks, do NOT partition them

7. Implement an L2ARC and ZIL using flash or SSD.

8. Set your vm.kmem_size max to roughly ½ your total physical memory.

9. Set you vfs.zfs.arc_max to roughly ¾ of the vm.kmem_size.

10. Hit the file system hard, both read and writes, and monitor your vm.kmem_size and vfs.zfs.misc.arcstats.

11. BONUS TIP: Perform tests that match your expected usage pattern so that you aren’t surprised as I was when performing large transfers.

In closing, thanks to sub.mesa for his concise documentation on FreeBSD, zfs and his commentary on WD drives. Thanks to Brendan Greg for an excellent article on zfs L2ARC.

While I have little doubt that zfs on FreeBSD is the most performant reliable copy on write filesystem available today without spending large sums of money, I am not sure how long this will be true. I believe that the FreeBSD release cycle and the conservative nature of its maintainers may actually be working against it users desires in this case.

The fuse-zfs project seems to have exorcised many of its reliability demons and is now more feature rich with its implementation of zfs (pool version 23) on Linux. It is still lacking some on performance, but not by a whole lot.

btrfs appears to be coming along at a pretty fast rate. Though both are owned/maintained by Oracle, btrfs seems to me to have a life going forward even if Oracle totally shuts down its participation; whereas, zfs’ path past the currently released code seems to be dead outside of Solaris.

And while these quandaries exist, Microsoft continues to fairly quietly sell Windows Server 2008 with a very tried and tested file system with robust snapshot and performance capabilities. Flame on if you must ye ideologues of ole. NetApp continues to sell their Filers; and Veritas continues to sell its very expensive solutions.

I personally believe that it is time for the open source world to put its differences behind them, BSD or Linux, ext or ufs, zfs or btrfs, and pick something that can deliver a robust and performant copy on write filesystem with the right features! Both zfs and btrfs have similar delivery goals but go about things somewhat different. In the end, I am an engineer, sigh, I care about what works reliably, what is performant, and what is supportable.

Thanks for sticking with my griping and complaining this far. I promise I’ll be better next post ;)

lbe

Trials and Disappointments of the last year

2010-08-20T10:53:00.000-05:00

It is hard to believe that it has been almost a year since my last update. I assure that this is not because I have ceased making errors or learning from them. With Volker moving to Debian, FreeNAS being frozen with FreeBSD 7.2, at least for now, Oracle's Acquisition of Sun and the resulting demise of Open Solaris, my inability in achieving stability with SABnzbd and Sick-Beard hacked into embedded FreeNAS, I decided in June to hang up FreeNAS.

As you can see in my blog, I have had a lot of interest in getting to the benefits of ZFS. I spent a lot of time investigate the possibility of going to Linux with BTRFS and had overal reached a decision to do that. But, because of my laissez faire nature, I really didn't want to write off and relead a couple of terabytes of data to make the transfer. So I decided to give FreeBSD 8.1, which was in RC at the time, a chance. While I like the allure of appliances like FreeNAS with their simple web interfaces and fences to keep you on the right path, as a long term fiddgle and IT designer, I always feel hemmed in with this, hence my desire to add SABnzbd and Sick-Beard.

While installing FreeBSD 8.1, I came close, very close to abandoning it. It had been years since I worked with it at the system administration level. The differences in paradigms from Linux and the at times lacking documentation were driving me crazy. In making some poor decisions, I got into to multi-day ports compilations on my relatively weak hardware. But in the wee hours of the mornings across a weekend, I ended up with a configuration that is stable and performant. It meets my fileserving needs as well as lets me run a handful of other services on it. My ZFS volumes cleanly imported and have been running well. I have had two kernel panics since having it in, err hm production, both associated with heavy writes while moving some ISO images to the server. I have been tuning vm.kmem.size and vfs.zfs.arc_max values to attempt to get this settled down and think I may be there now.

I'll close this post out with the folowing observaton/recommendation - If you are or have a mind to administer FreeBSD directly and are being pinched by the current state of FreeNAS, give FreeBSD 8.1 a try. It isn't nirvana, but it isn't keeping me awake at night either.

I'll post more later with more configuration information.

Cheers, lbe

zfs de-duplication has broken my heart, or ...

2009-09-21T20:18:00.000-05:00

more accurately, my lack of attention to detail has broken my heart. As I stated in my post on the FreeNAS forum "zfs de-duplication - is it working?" - "I absolutely hate it when reality doesn't match my pre-conceived ideas!!"

I'll blame it on the zfs zealots, surely it wasn't me. Surely these fire-brand wielding zfs prosolytes are to blame for connecting my wants, I mean needs, up to future features and not letting me see that the error of my ways, my desire for block level de-duplication, was/is nothing but vaporware -- at least for now :(

While headway is being made on de-duplication for zfs, it is somewhere between alpha and beta land on Solaris (or OpenSolaris, or !@#$Solaris land, ...) Block level de-duplication is no where near being part of zfs version 6 currently implemented in FreeBSD 7. It isn't event part of version 13 that will be available in the next major FreeBSD release.

I was torn with what to do. I considered going to Microsoft Windows Home Server and getting at least file level de-duplcation or possibly cracking the piggy bank and going to Windows Server 2008. But in the end, after doing some scribbling on the back of a napkin and figuring out that I would be money ahead to buy another drive, re-build my array, and be OK (defined as enough space without block level de-duplication) for another 12 - 18 months, I decided to stay with FreeNAS, for now, until something better comes along, until I get bored, err, I'm rambling again.

Stay tuned until next time, when I discuss the trials and travails of getting some CF to IDE adapters to work!

And since it is NCAA Football season here in the States -- GEAUX TIGERS!!!

Bye for now,

lbe

Let's Tune 'er Up!!

2009-09-02T22:11:00.009-05:00

I've had a little time since my last post to work on the tuning. Please consider this some where between coarse to medium tuning and certainly not fine tuning!

As I stated in a previous post, using the 0.71RC1 straight install and enabling nothing more "large read/write" and "use sendfile" in CIFS, I was able to achieve transfer rates of approximately 17 MB/sec. Performance with ftp was actually lower, ~9-10 MB/sec. In order to tune a system, one must know what the components can do. The components in this case were the raw drives, the zfs partition and samba/cifs. The tests of these components and their results are shown below.

Disk Test
I ran diskinfo from an ssh console against the first drive in my array (all three drives are the same - 1.5 TB WD Caviar Green WD15EADS).

nas01:/# diskinfo -tv ad4
ad4
512 # sectorsize
1500301910016 # mediasize in bytes (1.4T)
2930277168 # mediasize in sectors
2907021 # Cylinders according to firmware.

16 # Heads according to firmware.
63 # Sectors according to firmware.
ad:WD-WCAVY0511783 # Disk ident.

Seek times:
Full stroke: 250 iter in 7.376821 sec = 29.507 msec
Half stroke: 250 iter in 5.211402 sec = 20.846 msec
Quarter stroke: 500 iter in 8.364027 sec = 16.728 msec
Short forward: 400 iter in 3.197826 sec = 7.995 msec
Short backward: 400 iter in 3.506082 sec = 8.765 msec
Seq outer: 2048 iter in 0.774789 sec = 0.378 msec
Seq inner: 2048 iter in 0.571217 sec = 0.279 msec

Transfer rates:
outside: 102400 kbytes in 1.051929 sec = 97345 kbytes/sec
middle: 102400 kbytes in 1.142268 sec = 89646 kbytes/sec
inside: 102400 kbytes in 1.950994 sec = 52486 kbytes/sec

Clearly, the disk performance is not the cause of the bottle necks since the minimum disk transfer was 3.5 time or more faster than my sustained transfer rate.

zfs Partition
I used dd to create a 1GB file using two different block sizes, 1 & 64 KB.

nas01:/mnt/vdev0mgmt# dd if=/dev/zero of=mytestfile.out bs=1K count=1048576
1048576+0 records in
1048576+0 records out
1073741824 bytes transferred in 47.028245 secs (22831850 kbytes/sec)

nas01:/mnt/vdev0mgmt# dd if=/dev/zero of=mytestfile.out bs=64K
count=16384
16384+0 records in
16384+0 records out
1073741824 bytes transferred in 11.612336 secs (92465619 bytes/sec)

Again, both of these cases exceeded my test case though only barely with a 1KB block size. So this is not the culprit.

Network Transfer
I use iperf to test transfer rates in both directions, from server to workstation and vice a versa. The work station used is a Quad Core Intel with 8 GB of RAM. The memory use during the tests never exceeded 6 GB eliminating any workstation disk interaction. I ran iperf with two transaction size, 8 and 64 KB. The results are:

C:\>iperf -l 8K -t 30 -i 2 -c 192.168.2.5
------------------------------------------------------------
Client connecting to 192.168.2.5, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[148] local 192.168.2.207 port 57889 connected with 192.168.2.5 port 5001
[ ID] Interval Transfer Bandwidth
[148] 0.0-30.0 sec 984 MBytes 275 Mbits/sec

C:\>iperf -l 64K -t 30 -i 2 -c 192.168.2.5
------------------------------------------------------------
Client connecting to 192.168.2.5, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[148] local 192.168.2.207 port 57890 connected with 192.168.2.5 port 5001
[ ID] Interval Transfer Bandwidth
[148] 0.0-30.3 sec 1.70 GBytes 482 Mbits/sec

Again, both of these cases exceeded my test case significantly but did show that transaction size has a lot to do with network efficiency.

samba/cifs Test
I used the Microsoft RoboCopy utility to copy a 1GB from and to the NAS servers from my Windows workstation. The results are:

W:\tmp>robocopy . c: temp.file (READ)
Speed : 19140465 Bytes/sec.
Speed : 1095.226 MegaBytes/min.
Ended : Wed Sep 02 17:57:30 2009

This test more or less approximated my original test though it was slightly faster.

Conclusions:

Neither the hard drives, the file system nor the network were major contributors to the bottleneck.
The bottle necks then are "probably" in the kernel (stack, ipc, filesystem and network drivers) and in samba/cifs

Being basically lazy, and definitely not a good scientist (sorry teachers :( ), I surfed the web and found some tried and true tunings for samba/cifs as well as some items that seem to make sense for the kernel. Note, that testing has shown this configuration works for my server. I think the samba/cifs settings will likely help on any server as they have for me over the years across multiple Linux and BSD distributions. The kernel tunings are likely to have a heavy depency on the hardware that I use, namely the 1.6 GHz dual-core Atom 330, the Intel chipset (945GC northbridge and ICH7 southbridge) and Realtek NIC (RTL8111C) built into the MS-9832 motherboard and its 2 GB of RAM. If you hardware is too disimilar, you will "definitely" need to validate values on your own.

Here's what I did to tune my server.

samba/cifs tweaks
I added the following two lines to the auxillary parameters on the services/cifs configuration page.

max xmit = 65535
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_SNDBUF=65535 SO_RCVBUF=65535

I also set the send and receive buffers to 65535 to insure that is what they are.

kernel tweaks
I harvested my kernel tunings from multiple locations with references to their source embedded as remarks below. These additons were made to my /cf/boot/loader.conf file since I am booting from a USB flash drive. I used the advanced file editor in the WebGUI to make these changes since it takes care of mounting the flash drive read write and then resets it to read only.

# http://acs.lbl.gov/TCP-tuning/FreeBSD.html
kern.ipc.shmmax=67108864
kern.ipc.shmall=32768
# http://harryd71.blogspot.com/2008/10/tuning-freenas-zfs.html
vm.kmem_size_max="1024M"
vm.kmem_size="1024M"
vfs.zfs.prefetch_disable=1
# http://wiki.freebsd.org/ZFSTuningGuide
vfs.zfs.arc_max="100M"
# ups spinup time for drive recognition
hw.ata.to=15
# System tuning - Original -> 2097152
kern.ipc.maxsockbuf=16777216
# System tuning
kern.ipc.nmbclusters=32768
# System tuning
ern.ipc.somaxconn=8192
# System tuning
kern.maxfiles=65536
# System tuning
kern.maxfilesperproc=32768
# System tuning
net.inet.tcp.delayed_ack=0
# System tuning
net.inet.tcp.inflight.enable=0
# System tuning
net.inet.tcp.path_mtu_discovery=0
# http://acs.lbl.gov/TCP-tuning/FreeBSD.html
net.inet.tcp.recvbuf_auto=1
# http://acs.lbl.gov/TCP-tuning/FreeBSD.html
net.inet.tcp.recvbuf_inc=16384
# http://acs.lbl.gov/TCP-tuning/FreeBSD.html
net.inet.tcp.recvbuf_max=16777216
# System tuning
net.inet.tcp.recvspace=65536
# http://acs.lbl.gov/TCP-tuning/FreeBSD.html
net.inet.tcp.rfc1323=1
# http://acs.lbl.gov/TCP-tuning/FreeBSD.html
net.inet.tcp.sendbuf_auto=1
# http://acs.lbl.gov/TCP-tuning/FreeBSD.html
net.inet.tcp.sendbuf_inc =8192
# System tuning
net.inet.tcp.sendspace=65536
# System tuning
net.inet.udp.maxdgram=57344
# System tuning
net.inet.udp.recvspace=65536
# System tuning
net.local.stream.recvspace=65536
# System tuning
net.local.stream.sendspace=65536
# http://acs.lbl.gov/TCP-tuning/FreeBSD.html
net.inet.tcp.sendbuf_max=16777216

While I originally had the intent of testing the impact of the individual settings, I quickly grew bored of the reboots and rigor required (hence my reason for choosing an engineering vs. a scientific career :)). I can clearly say, you "must" disable the zfs pre-fetch in order to get read rates up to the levels that I have achieved.

Tuned Results!
My first test were to verify that I achieved my end goal of having read and write rates in excess of 35 MB/sec. My goal was achieved. Yeah!!!

C:\>robocopy w: . temp.file (READ)
Speed : 39179078 Bytes/sec.
Speed : 2241.844 MegaBytes/min.

C:\>robocopy . w: temp.file2 (WRITE)
Speed : 35574390 Bytes/sec.
Speed : 2035.582 MegaBytes/min.

While the previous test is verification of my goal, I wanted to see if the changes to the kernel network configuration changed the base network performance. It did, significantly.

C:\>iperf -l 8k -t 30 -i 2 -c 192.168.2.5
------------------------------------------------------------
Client connecting to 192.168.2.5, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[148] local 192.168.2.207 port 58332 connected with 192.168.2.5 port 5001
[ ID] Interval Transfer Bandwidth
[148] 0.0-30.0 sec 985 MBytes 276 Mbits/sec

C:\>iperf -l 64k -t 30 -i 2 -c 192.168.2.5
------------------------------------------------------------
Client connecting to 192.168.2.5, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[148] local 192.168.2.207 port 58334 connected with 192.168.2.5 port 5001
[ ID] Interval Transfer Bandwidth
[148] 0.0-30.3 sec 1.89 GBytes 537 Mbits/sec

nas01:/# iperf -l 8k -t 30 -i 2 -c 192.168.2.207
------------------------------------------------------------
Client connecting to 192.168.2.207, TCP port 5001
TCP window size: 65.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.2.5 port 55338 connected with 192.168.2.207 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-30.1 sec 2.11 GBytes 601 Mbits/sec

nas01:/# iperf -l 64k -t 30 -i 2 -c 192.168.2.207
------------------------------------------------------------
Client connecting to 192.168.2.207, TCP port 5001
TCP window size: 65.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.2.5 port 49701 connected with 192.168.2.207 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-30.1 sec 2.12 GBytes 604 Mbits/sec

The network tuning increased the small block size read rate from 275 to 601 Mbps and the large block size read rate from 482 to 537 Mbps. I did not capture the screen on my write tests, but the results were approximlately 550 Mbps untuned and are now 601 Mbps for the small block and 604 Mbps for the large block.

As I sleepily look back and what I have done, I realize that I have not tested the NAS server to the point of determining the effect of filling the cache since I have tested with files size of 1 GB and have 2 GB of RAM in the server. That will have to wait for another day.

Again, as I warn earlier, you "must" test to make sure that these tweaks are amenable to your hardware. If you are running 32 bit version of FreeNAS, there are many more kernel tweaks needed. If you are running with less memory, you will need to reduce some of the allocations. If you have a much larger server with many more clients, you will need to increase the allocations and probably want to have a better NIC than my bargain basement Realtek.

Realtek takes a beating in many of the NAS and network centric forums; however, depending upon the usage patternse, the Realtek can be a more than capable GigE NIC as this testing shows. It just bears getting the tuning right!!!

Bye for now, more errors are coming this way!!!

NAS - OS selection

2009-08-26T10:27:00.010-05:00

To ZFS or not to ZFS; To Windows SIS or not to Windows SIS; to use a dedicated NAS appliance OS or not to .... Well you get the point. This has been what I have spent most of my brain cycles, yes all three of them, thinking about for this project.

In general, I want a solution that at a least can automagically de-duplicate files (Windows SIS) but would love block level de-duplication. My primary use for this server will be to serve up files to Windows clients (CIFS/SMB) with some potential for FTP or other solutions. And with a little time, I'll probably put some type of streaming solution in place.

As of now, I have decided to use FreeNAS 0.7rc1 with ZFS support. I believe it is the best combination of ease, performance and flexibility for me. (If something else works better for you, good. I'm happy for you. :)

With the initial install things are working well booting from my SD card. Setup was pretty simple though the FreeNAS documentation for ZFS is pretty non-existant at this time. In short, the procedure is:

Format all of your drives
Create a virtual device (RAIDZ in my case) using all of the drives (whole disks are preferable to slices)
Create a management object (there is probably a better name for this)
Create a single ZFS share for CIFS.

I then enabled Active Directory support and connected the FreeNAS server to my home AD domain. I then created the two directories needed in the ZFS share and created their CIFS Shares and shared them out. Additionally, I enabled the SSH service so that I could log in remotely since this will be a headless system. Voila! Done! All is great!

Well actually, all is good. I ran into some limitations of the existing interoperability between FreeNAS 0.7rc1 (based on FreeBSD 7.2 RELEASE) which contains ZFS Version 6 and Samba used for CIFS. Currently, you cannot administer access control using the Windows tools. This will apparently be addressed with FreeBSD 7.3 which will provide ZFS Version 13 and a new version of CIFS which will support the zfsacl module as a Samba vfs object. For my relatively simple home setup, this is not a major problem. I logged into my FreeNAS server using SSH and modified the permissions using chown and chmod to do what I need and things are working fine now. So while a bit of an aggrevation, this does not impact functionality.

The following are a few brain droppings on other OSes that I considered. Please note, you will not find any religious zeal or bias here as it relates to Windows vs *nix vs Linux vs a Hamilton Beach Toaster. They are all good "tools" and should be used as appropriate. If you aren't intelligent enough to figure out "as appropriate", well then ....

Windows Server 2008 R2 - Given my desire to integrate with my home AD, currently running on a tried and reliable Windows 20083 Server and my current files server, this makes sense. Howevever, I don't want to invest this kind of money, even if I get an MS employee friend to buy a copy for me. SIS works reliably and would certainly meet my needs.

Windows 7/Vista/XP - Honestly, these desktop OSes could be configured to meet my needs but not my want to have de-duplication.

OpenSolaris & eon - These two solutions provide probably the pinnacle from a ZFS perspective; however, it wasn't clear that OpenSolaris provided full support for my hardware and my Solaris skills are not what they used to be. So laziness kept me from pursuing this one too much.

Linux (any thing from Fedora to ClarkConnect to OpenFiler) - there are a wealth of solutions out there based upon Linux. But not having kernel level ZFS support (FUSE is an interesting approach but not for me) or an alternative ready (like BTRFS) makes Linux unattractive to me.

Nexenta (Hybrid Solaris kernel & GNU user space) - This caught my eye and I gave it a spin. You have the originator in ZFS running the kernel and user space commands that I am current on. This sounded great, until I installed NexentaCP 2 and had problems with installing it to my SD card and then problems trying to get the network configured properly. I'm sure that both of these issues could be addressed with some knowledge; however, the documentation for Nexenta was practically non-existant. This made me seriously question if I wanted to go this route.

Hence in the end, my decision to go with FreeNAS. It has ZFS, though still in the early stages. It provides de-duplication. A base install gives me reasonable performance, which I hope to improve with tuning. The WebGUI is pretty well thought out. The only need that I have had for the command line was setting up directory permissions. So my lack of experience with FreeBSD shouldn't be too much of a problem.

Stay tuned for more advice, err stories about errors, to come!

NAS - Hardware Assembled with Error, of course!

2009-08-26T10:15:00.006-05:00

OK, I got all the pieces in and put them together. Voila, they all fit. Kind of. Sort of ....

The Chenbro case is a nice little case; however, I have not figured out how to remove the front cover. The online manual depicts a cover that is slightly different from my physical cover. However, regardless of the manual that says you need to remove the cover to install the motherboard and SD card reader, one does not. You need to release the cover and get it open enough to remove the blank in the slim optical device slot and remove the SD card reader holder, but that is it. There is plenty of play to do so. This gives full access to the mounting points for the mother board.

While installing the motherboard, I realized an oversight, yes an error, that I made. The MS-9832 motherboard contains only one internal USB header block. So it is not possible to plug in both the the up front USB ports as well as the the SD card reader. Furthermore, I am missing a cable to connect the card reader at all. So for now, I connected the two USB ports on the front of the chassis to the internal USB header and have put my SD card in a nifty SD to USB converter. I will use the SD for the OS.

The WD drives easily mounted in the hot swap trays and slid in wihtout a problem.

I am not installing a CD or DVD reader in this unit to keep the overall power consumption down. I have added a PATA CD/DVD reader/writer from LG sitting on top of the case with the cable going through the open side access. I will use this to load the operating system on the SD card and will then remove it and button it up for the long run.

All in all, the hardware came up fine!

NAS - Hardware Requirements

2009-08-18T15:07:00.010-05:00

My hardware requirements are very few:

Low power consumption - +/- 30W full load, <10w>
High Efficiency PSU
Quiet Case - fanless if possible
Minimum of 4 hot swap 3.5" drive bays
3 Gbps SATA II Drives for the storage array
OS to be loaded on solid state device (SD)
GigE Network Interface
USB HID and minimal video support for OS installation

My selected hardware is:

1 - Chenbro ES34069
Mini-ITX Case with 4-Hot Swappable Drive Bays, 120W PSU
$171.85

1 - Chenbro E434440
4-in-1 Card Reader - Card reader - 4 in 1 ( MMC, SD, miniSD )
$13.84

3 - WD WDxxxxx
1.5TB 7200RPM SATA II Hard Disk
$358.50

1 - MSI MS-9832-05S
Industrial IM-945GC Mini-ITX Mainboard with 1.6 GHz dual core Atom 330 processor, 2GB DDR2 667 RAM, on-board 4 x SATA II (3 Gbps), 2 - GigE NICS
$207.02

Total Cost to date - $751.21

I expect to add an 8GB Class 6 SD card (approximately $18) to the card reader upon which I will install the OS once selected.

That's all for now.

lbe

NAS - Getting Started

2009-08-18T10:36:00.001-05:00

Over the years, I have mused about taking the time to share some of the things that I have designed, built and/or implemented to share back with the Internet community from which I have stolen, errr adopted, I meaned learned so much. In the past, I decided not to do so because of all kinds of excuses. I am going to try to change that now!

For the past year, I have been musing in the background about buying or building a NAS server for home. I have made a decision to build and have selected the hardware and will soon finalize the OS decisions. I have searched the "whole" Internet, every little corner, ever bit, maybe even a qubit or two. And remarkably, I have failed to find a solution that meets what I personally want. That isn't to say I haven't found a lot, but just not the whole enchilada. So I have decided to start here.

Stay tuned!

lbe