Cluster
- HaProxy
- DRBD + Pacemaker & Corosync NFS Cluster Centos7
- Keepalived LoadBalacing
- Distributed memcached on 2 Webserver [CentOS7]
- GlusterFS + Heketi [Ubuntu 18.04]
HaProxy
This is not a tutorial of how haproxy works, this is just some notes on a config i did, and some of the options i used that made it stable for what i needed.
In the example bellow you will find a acceptable cipher, how to add a cookie sessions on HA, SSL offloading, xforward's, ha stats, good timeout vaules, and a httpchk.
global
log 127.0.0.1 local0 warning
maxconn 10000
user haproxy
group haproxy
daemon
spread-checks 5
tune.ssl.default-dh-param 2048
ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
defaults
log global
option dontlognull
retries 3
option redispatch
maxconn 10000
mode http
option dontlognull
option httpclose
option httpchk
timeout connect 5000ms
timeout client 150000ms
timeout server 30000ms
timeout check 1000
listen lb_stats
bind {PUBLIC IP}:80
balance roundrobin
server lb1 127.0.0.1:80
stats uri /
stats realm "HAProxy Stats"
stats auth admin:FsoqyNpJAYuD
frontend frontend_{PUBLIC IP}_https
mode tcp
bind {PUBLIC IP}:443 ssl crt /etc/haproxy/ssl/domain.com.pem no-sslv3
reqadd X-Forwarded-Proto:\ https
http-request add-header X-CLIENT-IP %[src]
option forwardfor
default_backend backend_cluster_http_web1_web2
frontend frontend_{PUBLIC IP}_http
bind {PUBLIC IP}:80
reqadd X-Forwarded-Proto:\ https
http-request add-header X-CLIENT-IP %[src]
option forwardfor
default_backend backend_cluster_http_web1_web2
frontend frontend_www_custom
bind {PUBLIC IP}:666
option forwardfor
default_backend backend_cluster_http_web1_web2
backend backend_cluster_http_web1_web2
option httpchk HEAD /
server web1 10.1.2.100:80 weight 1 check cookie web1 inter 1000 rise 5 fall 1
server web2 10.1.2.101:80 weight 1 check cookie web2 inter 1000 rise 5 fall 1
Enable xforward on httpd.conf on the web servers
LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\ " combine
LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"" combined-forwarded
Cookie
It is also possible to use the session cookie provided by the backend server.
backend www
balance roundrobin
mode http
cookie PHPSESSID prefix indirect nocache
server web1 10.1.2.100:80 check cookie web1
server web2 10.1.2.101:80 check cookie web2
In this example we will intercept the PHP session cookie and add / remove the reference of the backend server.
The prefix keyword allows you to reuse an application cookie and prefix the server identifier,
then delete it in the following queries.
Default name of cookies by type of feeder backend:
Java : JSESSIONID
ASP.Net : ASP.NET_SessionId
ASP : ASPSESSIONID
PHP : PHPSESSID
Active/Passive config
backend backend_web1_primary
option httpchk HEAD /
server web1 10.1.2.100:80 check
server web2 10.1.2.101:80 check backup
backend backend_web2_primary
option httpchk HEAD /
server web2 10.1.2.100:80 check
server web1 10.1.2.101:80 check backup
Test config file:
haproxy -c -V -f /etc/haproxy/haproxy.cfg
Hapee Check syntax
/opt/hapee-1.7/sbin/hapee-lb -c
Hapee VRRP
# /etc/hapee-1.7/hapee-vrrp.cfg
vrrp_script chk_hapee {
script "pidof hapee-lb"
interval 2
}
vrrp_instance vrrp_1 {
interface eth0
virtual_router_id 51
priority 101
virtual_ipaddress_excluded {
eth0
eth1
}
track_interface {
eth0 weight -2
eth1 weight -2
}
track_script {
chk_hapee
}
}
vrrp_instance vrrp_2 {
interface eth1
virtual_router_id 51
priority 101
virtual_ipaddress_excluded {
X.X.X.X
}
track_interface {
eth0 weight -2
eth1 weight -2
}
track_script {
chk_hapee
}
}
Doc
https://cbonte.github.io/haproxy-dconv/
DRBD + Pacemaker & Corosync NFS Cluster Centos7
On Both Nodes
Host file
vim /etc/hosts
10.1.2.114 nfs1 nfs1.localdomain.com
10.1.2.115 nfs2 nfs2.localdomain.com
Corosync will not work if you add something like this: 127.0.0.1 nfs1 nfs2.localdomain.com - however you do not need to delete 127.0.0.1 localhost
Firewall
Option 1 Firewalld
systemctl start firewalld
systemctl enable firewalld
firewall-cmd --permanent --add-service=nfs
firewall-cmd --permanent --add-service=rpc-bind
firewall-cmd --permanent --add-service=mountd
firewall-cmd --permanent --add-service=high-availability
On NFS1
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.1.2.115" port port="7789" protocol="tcp" accept'
firewall-cmd --reloadfirewall-cmd --reload
On NFS2
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.1.2.114" port port="7789" protocol="tcp" accept'
firewall-cmd --reloadfirewall-cmd --reload
Disable SELINUX
vim /etc/sysconfig/selinux
SELINUX=disabled
Pacemaker Install
Install PaceMaker and Corosync
yum install -y pacemaker pcs
Authenticate as the hacluster user
echo "H@xorP@assWD" | passwd hacluster --stdin
Start and enable the service
systemctl start pcsd
systemctl enable pcsd
ON NFS1
Test and generate the Corosync configuration
pcs cluster auth nfs1 nfs2 -u hacluster -p H@xorP@assWD
pcs cluster setup --start --name mycluster nfs1 nfs2
ON BOTH NODES
Start the cluster
systemctl start corosync
systemctl enable corosync
pcs cluster start --all
pcs cluster enable --all
Verify Corosync installation
Master should have ID 1 and slave ID 2
corosync-cfgtool -s
ON NFS1
Create a new cluster configuration file
pcs cluster cib mycluster
Disable the Quorum & STONITH policies in your cluster configuration file
pcs -f /root/mycluster property set no-quorum-policy=ignore
pcs -f /root/mycluster property set stonith-enabled=false
Prevent the resource from failing back after recovery as it might increases downtime
pcs -f /root/mycluster resource defaults resource-stickiness=300
LVM partition setup
Both Nodes
Create a empty partition
fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p):(ENTER)
Partition number (1-4, default 1): (ENTER)
First sector (2048-16777215, default 2048): (ENTER)
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-16777215, default 16777215): (ENTER)
Using default value 16777215
Partition 1 of type Linux and of size 8 GiB is setCommand (m for help): w
The partition table has been altered!
Create LVM partition
pvcreate /dev/sdb1
vgcreate vg00 /dev/sdb1
lvcreate -l 95%FREE -n drbd-r0 vg00
View LVM partition after creation
pvdisplay
Look in "/dev/mapper/" find the name of your LVM disk
ls /dev/mapper/
OUTPUT:
control vg00-drbd--r0
**You will use "vg00-drbd--r0" in the "drbd.conf" file in the below steps
DRBD Installation
Install the DRBD package
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum install -y kmod-drbd84 drbd84-utils
modprobe drbd
echo drbd > /etc/modules-load.d/drbd.conf
Edit the DRBD config and add the to hosts it will be connecting to (NFS1 and NFS2)
vim /etc/drbd.conf
Delete all and replace for the following
include "drbd.d/global_common.conf";
include "drbd.d/*.res";global {
usage-count no;
}
resource r0 {
protocol C;
startup {
degr-wfc-timeout 60;
outdated-wfc-timeout 30;
wfc-timeout 20;
}
disk {
on-io-error detach;
}
net {
cram-hmac-alg sha1;
shared-secret "Daveisc00l123313";
}
on nfs1.localdomain.com {
device /dev/drbd0;
disk /dev/mapper/vg00-drbd--r0;
address 10.1.2.114:7789;
meta-disk internal;
}
on nfs2.localdomain.com {
device /dev/drbd0;
disk /dev/mapper/vg00-drbd--r0;
address 10.1.2.115:7789;
meta-disk internal;
}
}
vim /etc/drbd.d/global_common.conf
Delete all and replace for the following
common {
handlers {
}
startup {
}
options {
}
disk {
}
net {
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
}
}
On NFS1
Create the DRBD partition and assign it primary on NFS1
drbdadm create-md r0
drbdadm up r0
drbdadm primary r0 --force
drbdadm -- --overwrite-data-of-peer primary all
drbdadm outdate r0
mkfs.ext4 /dev/drbd0
On NFS2
Configure r0 and start DRBD on NFS2
drbdadm create-md r0
drbdadm up r0
drbdadm secondary all
Pacemaker cluster resources
On NFS1
Add resource r0 to the cluster resource
pcs -f /root/mycluster resource create r0 ocf:linbit:drbd drbd_resource=r0 op monitor interval=10s
Create an additional clone resource r0-clone to allow the resource to run on both nodes at the same time
pcs -f /root/mycluster resource master r0-clone r0 master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
Add DRBD filesystem resource
pcs -f /root/mycluster resource create drbd-fs Filesystem device="/dev/drbd0" directory="/data" fstype="ext4"
Filesystem resource will need to run on the same node as the r0-clone resource, since the pacemaker cluster services that runs on the same node depend on each other we need to assign an infinity score to the constraint:
pcs -f /root/mycluster constraint colocation add drbd-fs with r0-clone INFINITY with-rsc-role=Master
Add the Virtual IP resource
pcs -f /root/mycluster resource create vip1 ocf:heartbeat:IPaddr2 ip=10.1.2.116 cidr_netmask=24 op monitor interval=10s
The VIP needs an active filesystem to be running, so we need to make sure the DRBD resource starts before the VIP
pcs -f /root/mycluster constraint colocation add vip1 with drbd-fs INFINITY
pcs -f /root/mycluster constraint order drbd-fs then vip1
Verify that the created resources are all there
pcs -f /root/mycluster resource show
pcs -f /root/mycluster constraint
And finally commit the changes
pcs cluster cib-push mycluster
On Both Nodes
Installing NFS
Install nfs-utils
yum install nfs-utils -y
Stop all services
systemctl stop nfs-lock && systemctl disable nfs-lock
Setup service
pcs -f /root/mycluster resource create nfsd nfsserver nfs_shared_infodir=/data/nfsinfo
pcs -f /root/mycluster resource create nfsroot exportfs clientspec="10.1.2.0/24" options=rw,sync,no_root_squash directory=/data fsid=0
pcs -f /root/mycluster constraint colocation add nfsd with vip1 INFINITY
pcs -f /root/mycluster constraint colocation add vip1 with nfsroot INFINITY
pcs -f /root/mycluster constraint order vip1 then nfsd
pcs -f /root/mycluster constraint order nfsd then nfsroot
pcs -f /root/mycluster constraint order promote r0-clone then start drbd-fs
pcs resource cleanup
pcs cluster cib-push mycluster
Test failover
pcs resource move drbd-fs nfs2
Other notes on DRBD
To update a resource after a commit
cibadmin --query > tmp.xml
Edit with vi tmp.xml or do a pcs -f tmp.xml %do your thing%
cibadmin --replace --xml-file tmp.xml
Delete a resource
pcs -f /root/mycluster resource delete db
Delete cluster
pcs cluster destroy
Recover a split brain
Secondary node
drbdadm secondary all
drbdadm disconnect all
drbdadm -- --discard-my-data connect all
Primary node
drbdadm primary all
drbdadm disconnect all
drbdadm connect all
On both
drbdadm status
cat /proc/drbd
Keepalived LoadBalacing
LVS Config
## Pool ID
virtual_server <WAN "frontend" IP> 80 {
delay_loop 6
lb_algo sh # source hash
lb_kind NAT
protocol TCP
real_server <LAN "backend" IP Server 1> 80 {
weight 1
TCP_CHECK {
connect_timeout 3
}
}
real_server <LAN "backend" IP Server 2> 80 {
weight 1
TCP_CHECK {
connect_timeout 3
}
}
}
virtual_server <WAN "frontend" IP> 443 {
delay_loop 6
lb_algo sh # source hash
lb_kind NAT
protocol TCP
real_server <LAN "backend" IP Server 1> 443 {
weight 1
TCP_CHECK {
connect_timeout 3
}
}
real_server <LAN "backend" IP Server 2> 443 {
weight 1
TCP_CHECK {
connect_timeout 3
}
}
}
VRRP
vrrp_instance VI_LOCAL {
state MASTER
interface eth1
virtual_router_id 51
priority 101
virtual_ipaddress {
10.X.X.X
}
track_interface {
eth0
eth1
}
}
vrrp_instance VI_PUB {
state MASTER
interface eth0
virtual_router_id 52
priority 101
virtual_ipaddress {
X.X.X.X
}
track_interface {
eth0
eth1
}
}
vrrp_instance VI_PUB2 {
state MASTER
interface eth0
virtual_router_id 53
priority 101
virtual_ipaddress {
X.X.X.X
}
track_interface {
eth0
eth1
}
}
sysctl
# Use ip that are not configured locally (HAProxy + KeepAlived requirements)
net.ipv4.ip_nonlocal_bind = 1
# Enable packet forwarding
net.ipv4.ip_forward=1
# Disables IP source routing
net.ipv4.conf.all.accept_source_route = 0
# Enable IP spoofing protection, turn on source route verification
net.ipv4.conf.all.rp_filter = 1
# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15
# Decrease the time default value for tcp_keepalive_time connection
net.ipv4.tcp_keepalive_time = 1800
# Enable TCP SYN Cookie Protection
net.ipv4.tcp_syncookies = 1
# Enable ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts = 1
# Enable bad error message Protection
net.ipv4.icmp_ignore_bogus_error_responses = 1
# Log Spoofed Packets, Source Routed Packets, Redirect Packets
net.ipv4.conf.all.log_martians = 1
# Increases the size of the socket queue
net.ipv4.tcp_max_syn_backlog = 1024
# Increase the tcp-time-wait buckets pool size
net.ipv4.tcp_max_tw_buckets = 1440000
# Arp
net.ipv4.conf.lo.rp_filter = 0
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
DR
vim /etc/modules
iptable_mangle
xt_multiport
xt_MARK
ip_vs
ip_vs_rr
ip_vs_nq
ip_vs_wlc
${IPTABLES} -t mangle -A PREROUTING -p tcp -d <VIP-WAN>/32 -j MARK --set-mark 0x1
Keepalived
virtual_server fwmark 1 {
delay_loop 10
lb_algo lc
lb_kind DR
protocol TCP
persistence_timeout 28800
real_server <WAN-WEB1> 0 {
weight 1
TCP_CHECK {
connect_port 443
connect_timeout 3
}
}
real_server <WAN-WEB2> 0 {
weight 2
TCP_CHECK {
connect_port 443
connect_timeout 3
}
}
}
Distributed memcached on 2 Webserver [CentOS7]
Install memcached
yum install memcached libmemcached -y
vi /etc/sysconfig/memcached
Change options to listen to the private IP on both web's:
OPTIONS="-l 10.1.1.X -U 0"
Restart memcached
systemctl restart memcached
systemctl enable memcached
Edit php ini
vi /etc/php.ini
session.save_handler = memcache
session.save_path = "tcp://10.1.1.100:11211, tcp://10.1.1.101:11211"
Install php-pecl-memcache
yum -y install php-pecl-memcache
echo "extension=memcache.so" >> /etc/php.d/memcache.ini
systemctl restart httpd
Allow in FW
firewall-cmd --zone=public --permanent --add-port=11211/tcp
firewall-cmd --reload
Check if memcached is running
watch memcached-tool X.X.X.X stats
Create test page:
vim /home/www/domain.com/session.php
Test Page:
<?php
header('Content-Type: text/plain');
session_start();
if(!isset($_SESSION['visit']))
{
echo "Page to test memcache.\n";
$_SESSION['visit'] = 0;
}
else
echo "You have visited this server ".$_SESSION['visit'] . " times. \n";
$_SESSION['visit']++;
echo "Server IP: ".$_SERVER['SERVER_ADDR'] . "\n";
echo "Client IP: ".$_SERVER['REMOTE_ADDR'] . "\n";
print_r($_COOKIE);
?>
GlusterFS + Heketi [Ubuntu 18.04]
Requirement to this guide : Having an empty / unused partition available for configuration on all bricks. Size does not really matter, but it needs to be the same on all nodes.
Configuring your nodes
Configuring your /etc/hosts file :
## on gluster00 :
127.0.0.1 localhost localhost.localdomain glusterfs00
10.1.1.3 gluster01
10.1.1.4 gluster02
## on gluster01
127.0.0.1 localhost localhost.localdomain glusterfs01
10.1.1.2 gluster00
10.1.1.4 gluster02
## on gluster02
127.0.0.1 localhost localhost.localdomain glusterfs02
10.1.1.2 gluster00
10.1.1.3 gluster01
Installing glusterfs-server on your bricks (data nodes). In this example, on gluster00 and gluster01 :
apt update
apt upgrade
apt-get install software-properties-common
add-apt-repository ppa:gluster/glusterfs-7
apt-get install glusterfs-server
Enable/Start GLuster
systemctl enable glusterd
systemctl start glusterd
Connect on either node peer with the second host. In this example I'm connected on gluster00 and allow peer on the other hosts using the hostname :
gluster peer probe gluster01
Should give you something like this :
Number of Peers: 1
Hostname: gluster01
Uuid: 6474c4e6-2957-4de7-ac88-d670d4eb1320
State: Peer in Cluster (Connected)
If you are going to use Heketi skip the volume creation steps
Creating your storage volume
Now that you have both of your nodes created and in sync, you will need to create a volume that your clients will be able to use.
Syntax :
gluster volume create $VOL_NAME replica $NUMBER_OF_NODES transport tcp $DOMAIN_NAME1:/path/to/directory $DOMAIN_NAME2.com:/path/to/directory force
## actual syntax in for our example
gluster volume create testvolume replica 2 transport tcp glusterfs00:/gluster-volume glusterfs01:/gluster-volume force
Start the volume you have created :
gluster volume start testvolume
Configuring your client(s)
apt-get install software-properties-common
add-apt-repository ppa:gluster/glusterfs-7
apt install glusterfs-client
Once completed, you will need to mount the storage that you previously created. First, make sure you have your mount point created :
mkdir /gluster-data
Mount your volume to your newly created mount point :
mount -t glusterfs gluster00:testvolume /gluster-data
Adding / Removing a brick from production
Once your node is ready with the proper packages and updates...
Make sure to edit its /etc/hosts and update every other nodes as well with your new entry :
echo "10.1.1.5 gluster03" >> /etc/hosts
Adding a new brick
Once you've completed the above points, simply connect on a node already part of the cluster :
gluster peer probe gluster03
And connect it to the volumes you want the new node to be connected to :
gluster volume add-brick testvolume replica 3 gluster03:/gluster-volum
Removing a clustered brick
Re-adding a node that has been previously removed
Install Heketi on one of the nodes
Requirement : Already existing GlusterFS install
Download Heketi bin
wget https://github.com/heketi/heketi/releases/download/v9.0.0/heketi-v9.0.0.linux.amd64.tar.gz
tar -zxvf heketi-v9.0.0.linux.amd64.tar.gz
Copy bin
chmod +x heketi/{heketi,heketi-cli}
cp heketi/{heketi,heketi-cli} /usr/local/bin
Check heketi is working
heketi --version
heketi-cli --version
Add a user/group for heketi
groupadd --system heketi
useradd -s /sbin/nologin --system -g heketi heketi
Create dir for heketi
mkdir -p /var/lib/heketi /etc/heketi /var/log/heketi
vim /etc/heketi/heketi.json
Make sure you replace the "key" values with proper passwords
{
"_port_comment": "Heketi Server Port Number",
"port": "8080",
"_enable_tls_comment": "Enable TLS in Heketi Server",
"enable_tls": false,
"_cert_file_comment": "Path to a valid certificate file",
"cert_file": "",
"_key_file_comment": "Path to a valid private key file",
"key_file": "",
"_use_auth": "Enable JWT authorization. Please enable for deployment",
"use_auth": false,
"_jwt": "Private keys for access",
"jwt": {
"_admin": "Admin has access to all APIs",
"admin": {
"key": "KEY_HERE"
},
"_user": "User only has access to /volumes endpoint",
"user": {
"key": "KEY_HERE"
}
},
"_backup_db_to_kube_secret": "Backup the heketi database to a Kubernetes secret when running in Kubernetes. Default is off.",
"backup_db_to_kube_secret": false,
"_profiling": "Enable go/pprof profiling on the /debug/pprof endpoints.",
"profiling": false,
"_glusterfs_comment": "GlusterFS Configuration",
"glusterfs": {
"_executor_comment": [
"Execute plugin. Possible choices: mock, ssh",
"mock: This setting is used for testing and development.",
" It will not send commands to any node.",
"ssh: This setting will notify Heketi to ssh to the nodes.",
" It will need the values in sshexec to be configured.",
"kubernetes: Communicate with GlusterFS containers over",
" Kubernetes exec api."
],
"executor": "ssh",
"_sshexec_comment": "SSH username and private key file information",
"sshexec": {
"keyfile": "/etc/heketi/heketi_key",
"user": "root",
"port": "22",
"fstab": "/etc/fstab"
},
"_db_comment": "Database file name",
"db": "/var/lib/heketi/heketi.db",
"_refresh_time_monitor_gluster_nodes": "Refresh time in seconds to monitor Gluster nodes",
"refresh_time_monitor_gluster_nodes": 120,
"_start_time_monitor_gluster_nodes": "Start time in seconds to monitor Gluster nodes when the heketi comes up",
"start_time_monitor_gluster_nodes": 10,
"_loglevel_comment": [
"Set log level. Choices are:",
" none, critical, error, warning, info, debug",
"Default is warning"
],
"loglevel" : "debug",
"_auto_create_block_hosting_volume": "Creates Block Hosting volumes automatically if not found or exsisting volume exhausted",
"auto_create_block_hosting_volume": true,
"_block_hosting_volume_size": "New block hosting volume will be created in size mentioned, This is considered only if auto-create is enabled.",
"block_hosting_volume_size": 500,
"_block_hosting_volume_options": "New block hosting volume will be created with the following set of options. Removing the group gluster-block option is NOT recommended. Additional options can be added next to it separated by a comma.",
"block_hosting_volume_options": "group gluster-block",
"_pre_request_volume_options": "Volume options that will be applied for all volumes created. Can be overridden by volume options in volume create request.",
"pre_request_volume_options": "",
"_post_request_volume_options": "Volume options that will be applied for all volumes created. To be used to override volume options in volume create request.",
"post_request_volume_options": ""
}
}
Load all Kernel modules that will be required by Heketi.
for i in dm_snapshot dm_mirror dm_thin_pool; do
sudo modprobe $i
done
Create ssh key for the API to connect to the other hosts
ssh-keygen -f /etc/heketi/heketi_key -t rsa -N ''
chown heketi:heketi /etc/heketi/heketi_key*
Send key to all hosts
for i in gluster00 gluster01 gluster02; do
ssh-copy-id -i /etc/heketi/heketi_key.pub root@$i
done
Create a systemd file
vim /etc/systemd/system/heketi.service
[Unit]
Description=Heketi Server
[Service]
Type=simple
WorkingDirectory=/var/lib/heketi
EnvironmentFile=-/etc/heketi/heketi.env
User=heketi
ExecStart=/usr/local/bin/heketi --config=/etc/heketi/heketi.json
Restart=on-failure
StandardOutput=syslog
StandardError=syslog
[Install]
WantedBy=multi-user.target
Reload systemd and enable new heketi service
systemctl daemon-reload
systemctl enable --now heketi
Allow heketi user perms on folders
chown -R heketi:heketi /var/lib/heketi /var/log/heketi /etc/heketi
Create topology
vim /etc/heketi/topology.json
{
"clusters": [
{
"nodes": [
{
"node": {
"hostnames": {
"manage": [
"gluster00"
],
"storage": [
"10.1.1.2"
]
},
"zone": 1
},
"devices": [
"/dev/vdc","/dev/vdd","/dev/vde"
]
}, {
"node": {
"hostnames": {
"manage": [
"gluster01"
],
"storage": [
"10.1.1.3"
]
},
"zone": 1
},
"devices": [
"/dev/vdc","/dev/vdd","/dev/vde"
]
}, {
"node": {
"hostnames": {
"manage": [
"gluster02"
],
"storage": [
"10.1.1.4"
]
},
"zone": 1
},
"devices": [
"/dev/vdc","/dev/vdd","/dev/vde"
]
}
]
}
]
}
Load topology
(note you can make changes and the load it again in the future if you want to add more drives)
heketi-cli topology load --json=/etc/heketi/topology.json
Check connection to other devices work
heketi-cli cluster list
Notes
Mount all volumes
for i in `gluster volume list`
do mkdir -p /etc/borg/gluster_backup/$i && \
mount -t glusterfs 127.0.0.1:$i /mnt/$i
done