Recently, I’ve been exploring high availability iSCSI targets and using them as virtual machine storage. I have always been a bit weary of iSCSI performance over gigabit networks due to some not-so-great experiences many years ago. iSCSI technology has progressed quite a bit since then. FreeNAS has an excellent implementation of iSCSI, as well as Nexenta. I wanted to get a good grasp on how well everything was performing, so I decided to run some basic benchmarks.
The virtual machine I’m working with has Ubuntu 15.04 installed, but these commands will work on just about any linux distribution in existence. The hyperviser is running VMware ESXi 6, with this particular virtual machine stored on an iSCSI target served from a FreeNAS virtual machine running on another VMware ESXi hyperviser. The FreeNAS virtual machine has been given PCI Passthrough to a 3ware 9650SE-16ML, connected to 4x1TB Hitachi SATA Hard Drives exported as JBOD (individual drives, no RAID). FreeNAS configured a RAID10 with the four drives, which is where the iSCSI target resides. The network is gigabit, with an Adtran NetVanta 1524ST switch. I have not enabled Jumbo Frames. The theoretical maximum transfer speed on a gigabit network is about 125MB/s. Of course, with the overhead associated with TCP/IP and iSCSI encapsulation, a single link should be a little less. Lets get started.
To test WRITE speed of hard disk using the DD command:
# sync; dd if=/dev/zero of=tempfile bs=1M count=1024; sync 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 10.8496 s, 99.0 MB/s
So, this command writes a bunch of zeros to a file called tempfile, with a size of 1024MB. If you want to use a larger test file, you can change “1024” to a higher number. For instance, changing 1024 to 10000 would write a 10GB temp file. This command only tests the write speed. As you can see, the reported write speed was 99MB/s, not too shabby.
To test READ speed of a hard disk using the dd command:
# sudo /sbin/sysctl -w vm.drop_caches=3 vm.drop_caches = 3 # dd if=tempfile of=/dev/null bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 9.27495 s, 116 MB/s
The first command clears the Linux Cache so we can get the actual read speed of the hard drives. The second command uses DD to read a bunch of null data, specifically 1024MB of null data. Like before, you can adjust 1024 to a higher number to increase the size of the tempfile. As you can see, the results on my test VM was 116MB/s. Slightly higher than the write speeds, which is pretty typical.
Now, if you need to test a different drive, say an external USB drive, or a NFS/Samba/CIFS mount point, you can use this command to specify the device.
# sync; dd if=/dev/zero of=/path/to/mount/tempfile bs=1M count=1024; sync
Simply change /path/to/mount to the mount point you would like to test. Simple as that. When you are finished testing your disk throughput, don’t forget to delete the tempfile! Otherwise, you’ll have a big file full of nothing taking up space.