Skip to content

How to setup a NAS with GlusterFS Striped & Replicated + NFS on Ubuntu 14.04

Gluster is one of the most amazing storage technologies to hit the Open Source world in quite some time. Since it’s been acquired by Red Hat, it’s growth and adaptation has only been accelerated even more. Some refer to it as software defined storage. I agree with that, if you like to use industry lingo. Gluster allows you to create a pool of storage that can span multiple servers / nodes and grow over time. You can create volumes that mirror or replicate data across multiple servers or you can stripe a volume across multiple servers. You can even strip with redundancy, kind of like a RAID 5 but instead of disks, its servers. Very cool stuff.

Gluster has an NFS server built in. It actually works very well too. This makes it compatible with pretty much every hypervisor there is, such as VMware ESXi, Citrix XenServer, KVM, Proxmox, oVirt, and every other virtualization platform there is. I have yet to come across one that doesn’t support NFS. More so, every OS and server variation also supports NFS. This makes Gluster an excellent choice to build a storage infrastructure on. You can start small and grow it over time, or go all in. Either way, this guide will help you get started.

Today I’m working with four Ubuntu 14.04 LTS servers (virtual machines, actually). Each has a second virtual disk attached, 30GB in size, that will be dedicated to Gluster on each host. I’m going to set up a stripe, with redundancy, across these four nodes, which will be presented as a single NFS share available to a hypervisor for virtual machine storage, or a server for file storage. If you plan on following this guide exactly, go ahead and provision four Ubuntu 14.04 virtual machines with an extra 30GB drive attached to each. 1GB of RAM and 1 CPU will be more than sufficient. Lets get started.

Creating a GlusterFS volume striped + replicated across 4 nodes

FYI, look at the bash prompt label to see which server i’m working on. I.E. mike@gfsu1:~$ is the gfsu1 node. If the bash prompt is a #, that means the instruction needs to be performed on all four nodes.

The four Ubuntu Gluster nodes i’m working with are named:

  • gfsu1 (GlusterFS-Ubuntu-1)
  • gfsu2
  • gfsu3
  • gfsu4

So, just keep an eye out for what node I’m working on based on the bash prompt label. First, make sure everything is updated and upgraded.

#  sudo apt-get update
#  sudo apt-get upgrade

Next, make sure software-properties-common is installed.

#  sudo apt-get install software-properties-common

Now we can add the Gluster PPA and install Gluster on our four nodes.

#  sudo add-apt-repository ppa:gluster/glusterfs-3.5
#  sudo apt-get install glusterfs-server

That was easy, right? Next, let’s create a partition on our secondary disks and get them formatted. On each node, go through this process.

#  sudo fdisk /dev/sdb

(select n for new partition, then p for primary, then enter 3 times to accept defaults.  "w" to save and exit)

Now we need to put a file system on our new partition. First we’ll download xfsprogs so we can use xfs filesystem type.

#  sudo apt-get install xfsprogs
#  mkfs.xfs /dev/sdb1

Next, we’ll create a mount point for the new partition called gluster.

#  sudo mkdir /gluster
#  sudo mount /dev/sdb1 /gluster

We now have GlusterFS installed and our second disk mounted. Next we’ll make each of the four Gluster nodes aware of each other. I’m going to pick the second node, gfsu2, to probe the other three nodes, gfsu1, gfsu3, and gfsu4.

mike@gfsu2:~$ sudo gluster peer probe gfsu1
mike@gfsu2:~$ sudo gluster peer probe gfsu3
mike@gfsu2:~$ sudo gluster peer probe gfsu4

Now, if you run the following command from any of the 4 servers, you should get a status showing 3 other peers.

mike@gfsu2:~$ sudo gluster peer status
Number of Peers: 3

Hostname: gfsu3
Port: 24007
Uuid: e23c5e7e-0c0f-428d-a43a-694a3191deba
State: Peer in Cluster (Connected)

Hostname: gfsu1
Port: 24007
Uuid: e9797afa-337f-47f2-8a68-92cc857c69b5
State: Peer in Cluster (Connected)

Hostname: gfsu4
Port: 24007
Uuid: 2578fa22-cc2b-4976-8c06-69fd074b4193
State: Peer in Cluster (Connected)

Screen Shot 2015-06-06 at 6.08.29 PM

If you get an error that says “peer probe: failed: Peer does not support required op-version” then run the peer probe from one of the other servers. I’ve seen this happen on Ubuntu before when adding GlusterFS peers. It’s weird, but if the nodes are probed from another system it doesn’t cause any problems down the road.

Screen Shot 2015-06-06 at 6.06.25 PM

Now that all 4 nodes are seeing each other, we can create our striped + replicated GlusterFS volume.

mike@gfsu2:/$ sudo gluster volume create test-volume stripe 2 replica 2 transport tcp gfsu1:/gluster gfsu2:/gluster gfsu3:/gluster gfsu4:/gluster force

You should see a status of “success” and “please start the volume to access data.

Screen Shot 2015-06-06 at 6.17.07 PM

Let’s start the new volume we created, called “test-volume.”

mike@gfsu2:/$ sudo gluster volume start test-volume

You should see a “success” status again. You can run the gluster volume info command to see what the volume looks like. It should show the number of bricks as 1 x 2 x 2 = 4.

mike@gfsu2:/$ sudo gluster volume info

Screen Shot 2015-06-06 at 6.20.19 PM

Or, you can specify the volume name and make sure the NFS server is running within Gluster.

mike@gfsu2:/$ sudo gluster volume status test-volume

Screen Shot 2015-06-06 at 6.21.40 PM

You can see the “Y” in the “Online” column for each NFS server. You only need to use the NFS server on one of the nodes to access the entire volume. Next, let’s authenticate our subnet access to the NFS server.

mike@gfsu2:/$ sudo gluster volume set test-volume nfs.rpc-auth-allow 192.168.1.*

This command gives the entire 192.168.1.x subnet access to the NFS server within gluster. Modify this to suite your network. You can specify a single address or an entire subnet using * (asterisks) as wildcards.

Screen Shot 2015-06-06 at 6.26.01 PM

Now, you can add the NFS share to your server or hypervisor, using the IP address of any of the nodes. I’m going to add the NFS share to my ESXi server. The mount point is /test-volume which is the name of the gluster volume we created.

Screen Shot 2015-06-06 at 6.26.58 PM
If you run into any problems, feel free to ask for help in the comments below. Thanks!