Skip to main content

GlusterFS + Heketi [Ubuntu 18.04]

This basic setup uses gluster00 and gluster01 as its nodes and gluster02 as its client.

Requirement to this guide : Having an empty / unused partition available for configuration on all bricks. Size does not really matter, but it needs to be the same on all nodes.

Configuring your nodes

Configuring your /etc/hosts file :

## on gluster00 :
127.0.0.1 localhost localhost.localdomain glusterfs00
10.1.1.3 gluster01
10.1.1.4 gluster02

## on gluster01
127.0.0.1 localhost localhost.localdomain glusterfs01
10.1.1.2 gluster00
10.1.1.4 gluster02

## on gluster02
127.0.0.1 localhost localhost.localdomain glusterfs02
10.1.1.2 gluster00
10.1.1.3 gluster01

Installing glusterfs-server on your bricks (data nodes). In this example, on gluster00 and gluster01 :

apt-get update -y && apt-get install glusterfs-server -y

Connect on either node peer with the second host. In this example I'm connected on gluster00 :

gluster peer probe gluster01

Should give you something like this :

Number of Peers: 1

Hostname: gluster01
Uuid: 6474c4e6-2957-4de7-ac88-d670d4eb1320
State: Peer in Cluster (Connected)

Creating your storage volume

Now that you have both of your nodes created and in sync, you will need to create a volume that your clients will be able to use.

Syntax :

gluster volume create $VOL_NAME replica $NUMBER_OF_NODES transport tcp $DOMAIN_NAME1:/path/to/directory $DOMAIN_NAME2.com:/path/to/directory force

## actual syntax in for our example

gluster volume create testvolume replica 2 transport tcp glusterfs00:/gluster-volume glusterfs01:/gluster-volume force

Start the volume you have created :

gluster volume start testvolume

Configuring your client(s) 

apt-get install glusterfs-client

Once completed, you will need to mount the storage that you previously created. First, make sure you have your mount point created :

mkdir /gluster-data

Mount your volume to your newly created mount point :

mount -t glusterfs gluster00:testvolume /gluster-data
Adding / Removing a brick from production

Once your node is ready with the proper packages and updates...
Make sure to edit its /etc/hosts and update every other nodes as well with your new entry :

echo "10.1.1.5 gluster03" >> /etc/hosts

Adding a new brick

Once you've completed the above points, simply connect on a node already part of the cluster :

gluster peer probe gluster03

And connect it to the volumes you want the new node to be connected to :

gluster volume add-brick testvolume replica 3 gluster03:/gluster-volum

Removing a clustered brick
Re-adding a node that has been previously removed

Adding Heketi

Requirement : Already existing GlusterFS install

Base Heketi install

curl -s https://api.github.com/repos/heketi/heketi/releases/latest | grep browser_download_url | grep linux.amd64 | cut -d '"' -f 4 | wget -qi -
yum install wget -y
curl -s https://api.github.com/repos/heketi/heketi/releases/latest | grep browser_download_url | grep linux.amd64 | cut -d '"' -f 4 | wget -qi -
ls
md5sum heketi-*
for i in `ls | grep heketi | grep .tar.gz`; do tar xvf $i; done
ls
sudo cp heketi/{heketi,heketi-cli} /usr/local/bin
heketi --version
heketi-cli --version
sudo groupadd --system heketi
sudo useradd -s /sbin/nologin --system -g heketi heketi
sudo mkdir -p /var/lib/heketi /etc/heketi /var/log/heketi
sudo cp heketi/heketi.json /etc/heketi
sudo vim /etc/heketi/heketi.json

Make sure you replace the "key" values with proper passwords

{
  "_port_comment": "Heketi Server Port Number",
  "port": "8080",

	"_enable_tls_comment": "Enable TLS in Heketi Server",
	"enable_tls": false,

	"_cert_file_comment": "Path to a valid certificate file",
	"cert_file": "",

	"_key_file_comment": "Path to a valid private key file",
	"key_file": "",


  "_use_auth": "Enable JWT authorization. Please enable for deployment",
  "use_auth": false,

  "_jwt": "Private keys for access",
  "jwt": {
    "_admin": "Admin has access to all APIs",
    "admin": {
      "key": "KEY_HERE"
    },
    "_user": "User only has access to /volumes endpoint",
    "user": {
      "key": "KEY_HERE"
    }
  },

  "_backup_db_to_kube_secret": "Backup the heketi database to a Kubernetes secret when running in Kubernetes. Default is off.",
  "backup_db_to_kube_secret": false,

  "_profiling": "Enable go/pprof profiling on the /debug/pprof endpoints.",
  "profiling": false,

  "_glusterfs_comment": "GlusterFS Configuration",
  "glusterfs": {
    "_executor_comment": [
      "Execute plugin. Possible choices: mock, ssh",
      "mock: This setting is used for testing and development.",
      "      It will not send commands to any node.",
      "ssh:  This setting will notify Heketi to ssh to the nodes.",
      "      It will need the values in sshexec to be configured.",
      "kubernetes: Communicate with GlusterFS containers over",
      "            Kubernetes exec api."
    ],
    "executor": "ssh",

    "_sshexec_comment": "SSH username and private key file information",
    "sshexec": {
      "keyfile": "/etc/heketi/heketi_key",
      "user": "root",
      "port": "22",
      "fstab": "/etc/fstab"
    },

    "_db_comment": "Database file name",
    "db": "/var/lib/heketi/heketi.db",

     "_refresh_time_monitor_gluster_nodes": "Refresh time in seconds to monitor Gluster nodes",
    "refresh_time_monitor_gluster_nodes": 120,

    "_start_time_monitor_gluster_nodes": "Start time in seconds to monitor Gluster nodes when the heketi comes up",
    "start_time_monitor_gluster_nodes": 10,

    "_loglevel_comment": [
      "Set log level. Choices are:",
      "  none, critical, error, warning, info, debug",
      "Default is warning"
    ],
    "loglevel" : "debug",

    "_auto_create_block_hosting_volume": "Creates Block Hosting volumes automatically if not found or exsisting volume exhausted",
    "auto_create_block_hosting_volume": true,

    "_block_hosting_volume_size": "New block hosting volume will be created in size mentioned, This is considered only if auto-create is enabled.",
    "block_hosting_volume_size": 500,

    "_block_hosting_volume_options": "New block hosting volume will be created with the following set of options. Removing the group gluster-block option is NOT recommended. Additional options can be added next to it separated by a comma.",
    "block_hosting_volume_options": "group gluster-block",

    "_pre_request_volume_options": "Volume options that will be applied for all volumes created. Can be overridden by volume options in volume create request.",
    "pre_request_volume_options": "",

    "_post_request_volume_options": "Volume options that will be applied for all volumes created. To be used to override volume options in volume create request.",
    "post_request_volume_options": ""
  }
}
Back to top