Setting up mogileFS

Introduction

This is the first post in a number of posts designed to look at using distributed file systems for use from PHP. MogileFS (https://code.google.com/p/mogilefs/) is the first one I will look at. From initial research the pros of MogileFS are as follows:

  • Easily distributed
  • Simple to setup/administer
  • Different classes of files
  • No single point of failure (although the MySQL database needs to be mirrored)
  • Well tested in production

The cons:

  • Files aren’t really file system accessible (uses HTTP)

Terminology/Components

It took me a little while to grasp the concepts of the mogileFS components, and therefore I thought I would write my own understanding on them here.

  • Database – Usually MySQL, all trackers point to it
  • Tracker – Interface for interacting with mogile
  • Storage Node – Where the files are stored
  • Domains – A ‘namespace’ of where files are kept, think buckets, or ‘root’ folder
  • Keys – A ‘filename’ can be anything, but gives files their names
  • Classes – Class for files, mainly gives retention policy (number of devices or hosts to backup onto)

MySQL Database

You will have one MySQL database and it’s up to you backup/distribute make this high availability. All trackers need to point to this ONE MySQL database.

Tracker (mogilefsd)

You want to have at least 2 trackers, and these point to the MySQL database above. The app ‘talks’ and makes requests to the trackers and they carry out tasks, such as distributing the load, doing the file system checks, and making sure the hosts and devices are online.

Storage Node (mogstored)

The storage node can be separate computers, and are made up of hosts (the program that takes commands, etc), and the ‘devices’ that are stored. You can (and probably should) have multiple devices on each storage node, these can be separate disks, folders, partitions, etc. You do NOT need RAID, and can use any file system you wish for the underlying storage (will use ext4 in these examples). You add hosts, and then devices to these hosts as you go.

The test setup

I am going to create a test system on 2 virtual hosts as follows:

Test layout of the mogile servers
Test layout of the two mogile servers

I will install mogile1 first, add some files, and then bring up mogile2, and hopefully the files should sync between the two systems.

The hosts file on both of the systems are as follows:

127.0.0.1 localhost
10.10.200.80 mogile1
10.10.200.90 mogile2

mogile1 – Install MySQL

This is probably the easiest part of the process.

$ sudo apt-get install mysql-server

Once installed, I then, modified the /etc/mysql/my.cnf and changed the bind-address to 0.0.0.0, so that mogile2 can access later, and then restart mysql:

$ sudo service mysql restart

Next is to create the database we’ll use with the user to access it.

mysql> create database mogilefs;
Query OK, 1 row affected (0.00 sec)

mysql> GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,ALTER ON mogilefs.* TO mogileuser@localhost IDENTIFIED BY 'secret';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,ALTER ON mogilefs.* TO mogileuser@'10.10.200.%' IDENTIFIED BY 'secret';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

And that’s the end of setting up the database from the mysql point of view.

Installing the tracker, and storage nodes

Instead of a large install process, there are PPA’s out there including what you need for debian/ubuntu. The following steps will install the tracker, storage and utility parts of the system:

$ sudo add-apt-repository ppa:saz/mogilefs
$ sudo apt-get update
$ sudo apt-get install mogilefsd mogstored mogilefs-utils

This will install a load of items, but fail during install on mogilefs-util, this can be rectified by:

$ sudo apt-get install -f
$ sudo apt-get upgrade

There is an issue with the version of Syscall.pm (Syscall perl module), that comes with Ubuntu 14.04, the version 0.25 needs to be patched with the file here: Syscallpatch.txt:

$ sudo -i
$ cd /usr/share/perl5/Sys
$ wget http://www.fun-ideas.co.uk/wp-content/uploads/2015/05/Syscallpatch.txt
$ patch Syscall.pm Syscallpatch.txt
$ exit

Setup the default configurations with the following:

$ sudo dpkg-reconfigure mogilefsd
$ sudo dpkg-reconfigure mogstored

I used the defaults for these. It sets up the files in /etc/mogilefs/. Update the /etc/mogilefs/mogilefsd.conf file, and update the db_dsn, db_user, db_password, and listen host/port (so that it’s 0.0.0.0 so that other programs can connect).

Next step is to setup the database tables, using the mysql authentication details from above, we are using one of the utlities to create the database called mogdbsetup.

$ sudo mogdbsetup --dbhost=mogile1 --dbname=mogilefs --dbuser=mogileuser --dbpassword=secret

The database is then created. Start the deamon, and check it’s working:

$ sudo service mogilefsd restart
Restarting mogilefsd: mogilefsd.
$ mogadm check
Checking trackers...
127.0.0.1:7001 ... OK

Checking hosts...
No devices found on tracker(s).

Hopefully you see the same as the above.

Configuring and adding the storage node

Lets create two ‘devices’ to use for storing the files. Normally these would be different partitions/drives, but in this instance for test purposes we’ll just create two folders in /var/mogdata/:

$ sudo mkdir /var/mogdata/dev1
$ sudo mkdir /var/mogdata/dev2
$ sudo chown mogstored /var/mogdata/*

Restart or start the storage deamon:

$ sudo service mogstored restart

Add the host, and devices to the tracker (these commands do not need to be added as root):

$ mogadm --trackers=mogile1:7001 host add mogile1 --ip=10.10.200.80 --port=7500 --status=alive

And then add the two devices we’ve made to the trackers
$ mogadm --trackers=mogile1:7001 device add mogile1 1
$ mogadm --trackers=mogile1:7001 device add mogile1 2
$ mogadm check
Checking trackers...
127.0.0.1:7001 ... OK

Checking hosts...
[ 1] mogile1 ... OK

Checking devices...
host device size(G) used(G) free(G) use% ob state I/O%
---- ------------ ---------- ---------- ---------- ------ ---------- -----
[ 1] dev1 4.063 1.668 2.395 41.06% writeable N/A
[ 1] dev2 4.063 1.668 2.395 41.06% writeable N/A
---- ------------ ---------- ---------- ---------- ------
total: 8.126 3.336 4.789 41.06%

Domains, and adding some files

Domains are used as ‘buckets’ or the top level way of accessing files. You can create as many domains as you like. So lets add a couple of domains:

$ mogadm domain add testing
$ mogadm domain add toast
$ mogadm domain list
domain class mindevcount replpolicy hashtype
-------------------- -------------------- ------------- ------------ -------
testing default 2 MultipleHosts() NONE

toast default 2 MultipleHosts() NONE

And add some classes:

$ mogadm class add testing myclass --mindevcount=3 --replpolicy="MultipleHosts(2)"
$ mogadm class add toast myclass --mindevcount=3 --replpolicy="MultipleHosts(2)"

Let’s upload a couple of files:

$ echo "Hello world" | mogupload --trackers=mogile1:7001 --domain=testing --class="myclass" --key="/helloworld" --file="-"
$ mogupload --trackers=mogile1:7001 --domain=testing --class="myclass" --key="/mysqlconfig" --file="/etc/mysql/my.cnf"
$ $ moglistkeys --trackers=mogile1:7001 --domain=testing
/helloworld
/mysqlconfig

You can get stats as follows:

$ mogstats --db_dsn="DBI:mysql:mogilefs:host=mogile1" --db_user="mogileuser" --db_pass="secret" --verbose --stats="all"

Setting up mogile2

Install

$ sudo add-apt-repository ppa:saz/mogilefs
$ sudo apt-get update
$ sudo apt-get install mogilefsd mogstored mogilefs-utils
$ sudo apt-get install -f
$ sudo -i
$ cd /usr/share/perl5/Sys
$ wget http://www.fun-ideas.co.uk/wp-content/uploads/2015/05/Syscallpatch.txt
$ patch Syscall.pm Syscallpatch.txt
$ exit
$ sudo dpkg-reconfigure mogilefsd
$ sudo dpkg-reconfigure mogstored

Modified the /etc/mogilefs/mogilefsd.conf file, and update the db_dsn, db_user, db_password, and listen host/port (so that it’s 0.0.0.0 so that other programs can connect).

The database is then created. Start the deamon, and check it’s working:

$ sudo service mogilefsd restart
Restarting mogilefsd: mogilefsd.

Create dev3, and dev4 files in /var/mogdata and change the ownership mogstored. And add the host and devices to the trackers:

$ mogadm --trackers=mogile1:7001,mogile2:7001 host add mogile2 --ip=10.10.200.90 --port=7500 --status=alive

And then add the two devices we've made to the trackers
$ mogadm --trackers=mogile1:7001,mogile2:7001 device add mogile2 3
$ mogadm --trackers=mogile1:7001,mogile2:7001 device add mogile2 4
$ mogadm check
Checking trackers...
127.0.0.1:7001 ... OK

Checking hosts...
[ 1] mogile1 ... OK
[ 2] mogile2 ... OK

Checking devices...
host device size(G) used(G) free(G) use% ob state I/O%
---- ------------ ---------- ---------- ---------- ------ ---------- -----
[ 1] dev1 4.063 1.669 2.394 41.08% writeable N/A
[ 1] dev2 4.063 1.669 2.394 41.08% writeable N/A
[ 2] dev3 4.063 1.585 2.478 39.01% writeable N/A
[ 2] dev4 4.063 1.585 2.478 39.01% writeable N/A
---- ------------ ---------- ---------- ---------- ------
total: 16.251 6.507 9.744 40.04%

Check that files are being added to the dev folders, and being synced between the devices.

Extra config

Instead of having to write --trackers=mogile1:7001,mogile2:7001 on all the commands, you can create a file in /etc/mogilefs/mogilefs.conf (or ~/.mogilefs.conf) containing the following line:

trackers=mogile1:7001,mogile2:7001

You can also add a domain=testing or whatever for the default domain.

Final words

This is an initial setup. Please note that all the mogilefs deamons are not security protected in anyway, and therefore should be firewalled properly from the hosts that shouldn't be able to access the files. Also the MySQL database should be replicated and backed up properly (possibly even using some of the domains).

This was a detailed installation guide, and should get you up and running. I will be creating a few more guides once I have had a play with PHP accessing the files.

References

https://code.google.com/p/mogilefs/wiki/InstallOnUbuntu
https://code.google.com/p/mogilefs/wiki/InstallHowTo
https://github.com/bradfitz/sys-syscall/issues/5

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Mounting partitions contained in an LVM

Introduction

When creating a virtual machine with an LVM partition, it will create partitions within that LVM partition. Mounting those ‘sub’ partitions requires an extra step to access them. I used this when I was cloning virtual machines so that I could change the hostname of the copied host before booting it up.

A tool called kpartx is your friend here. It allows the system to see partitions within a LVM partition, so they can be mounted. In this guide we will be accessing partitions created as part of my mogilefs playground virtual machines. Below are the partitions that we are trying to access:

$ sudo fdisk -l /dev/vhosts/mogile2

Disk /dev/vhosts/mogile2: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders, total 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000b30be

Device Boot Start End Blocks Id System
/dev/vhosts/mogile2p1 * 2048 9437183 4717568 83 Linux
/dev/vhosts/mogile2p2 9439230 10483711 522241 5 Extended
/dev/vhosts/mogile2p5 9439232 10483711 522240 82 Linux swap / Solaris

NOTE: Make sure that the virtual machine is NOT running before mounting these partitions

So let’s try and mount one of these partitions (/dev/vhosts/mogile2p1):


$ sudo mount /dev/vhosts/mogile2p1 /mnt/temp
mount: special device /dev/vhosts/mogile2p1 does not exist

So the following will help map these partitions to the host OS so they can be mounted. If you haven’t got kpartx installed, then on debian/ubuntu systems install with sudo apt-get install kpartx

kpartx guide

To allow the host system to see the partitions for mounting, we run the following:

$ sudo kpartx -a /dev/vhosts/mogile2

We can then mount with

$ sudo mount /dev/mapper/vhosts-mogile2p1 /mnt/temp

And then use it as we normally would. In this case, I used it to edit the /mnt/temp/etc/hostname and /mnt/temp/etc/hosts file

Finally un mount it:
$ sudo umount /mnt/temp

And if you are done with it, and cleaned up, then remove it from the partition tables:

$ sudo kpartx -d /dev/vhosts/mogile2

Reference(s):

https://www.normation.com/en/blog/2011/04/07/mounting-partitions-stored-in-a-logical-volume-or-a-disk-image/

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Cloning a KVM Virtual Machine

Introduction

This is a guide on copying the base 14.04 Ubuntu image to another machine to use. Hope fully all that is needed it to create the new file system, and change the hostname.

Creating the file system

Firstly, I create the file system the machine would use, on the existing LVM partitions, make this the same size as the base 1404 (can resize it later if needs be):

$ sudo lvcreate -L 5G -n mogile1 vhosts
Logical volume "mogile1" created

Copying the base1404 image to mogile1

Make sure that the base 1404 server isn’t running (not listed in virsh list), and then copy the drive over. (Block size of 64k, seems a good bet for copying)
$ sudo dd bs=64k if=/dev/vhosts/base-1404 of=/dev/vhosts/mogile1

Creating the virtual host

Next to make the copy. I copied the /etc/libvirt/qemu/base-1404.xml to /tmp/mogile1.xml and changed the following lines:

<name>mogile1</name>
<uuid>ade28df31-dbca-a8b6-af90-1a435s0d0057</uuid>
<source dev='/dev/vhosts/mogile1'/> (within <disk type=’block’> section)
<mac address='52:54:00:62:3d:9a'/>

Make sure that the uuid is only HEX digits and letters (0-9 a-f).You can change the ram or CPUs, etc as needed. (uuidgen is a good command for making a new UUID)

Once you have created the xml you can use the following commands to create, and then start the domain:

$ virsh define mogile1.xml
Domain mogile1 defined from mogile1.xml
$ virsh dumpxml mogile1

You can then start it up with:
$ virsh start mogile1

Remember to change the hostname in /etc/hostname and /etc/hosts after reboot, and all will be good. Although you could do this on the copy of the hard drive before you boot it up. See the how to mount partitions guide for information on this

Reference: http://www.greenhills.co.uk/2013/03/24/cloning-vms-with-kvm.html

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Creating a base 14.04 Ubuntu using LVM and KVM

Creating a new virtual machine

Introduction

I wanted to create a base install of Ubuntu so that I could keep all the security patches up to date, and then create copies of the base image for other tests, and playing around.

The guide was made when installing a basic Ubuntu 14.04 server image on a KVM hypervisor. All the commands were run as root (or via sudo).

Creating the file system

Firstly, I create the file system the machine would use, on the existing LVM partitions:

$ sudo lvcreate -L 5G -n base-1404 vhosts
Logical volume "base-1404" created

Just using a small volume for the base install (5G). To see the current volumes that exist use lvdisplay to list them all.:

Creating the virtual host

Next to create the virtual host, this is relatively simple to get started:

$ sudo virt-install -n base-1404 -r 512 --disk path=/dev/vhosts/base-1404 -c /home/software/os/linuxisos/ubuntu-14.04.2-server-amd64.iso --os-type linux --os-variant ubuntutrusty --accelerate --network=bridge:br0 --hvm --vnc --noautoconsole

Starting install…
Creating domain… | 0 B 00:04
Domain installation still in progress. You can reconnect to the console to complete the installation process.

The options give 512MB ram, the location to the hard drive to use, a location to a cdrom drive to use (or image) for install. Type of OS, and the variant. Whether to use acceleration, what network to use, and to start with a vnc console, to install it with.

Note: To get a list of os variants available, use:

$ sudo virt-install --os-variant list

Connection to the vhost over VNC

To connect the VNC console to actually carry out the installation, have a look at what is being listened to on the network:

Listing all VNC terminals for the KVM virtual hosts
Listing all VNC terminals for the KVM virtual hosts

The last one should be the one to connect to, so lets tunnel from desktop machine to there, and try and connect:

$ ssh sb1 -L5905:localhost:5905

And then connect using your VNC software:

Conecting to the new virtual machine using remmina and VNC
Conecting to the new virtual machine using remmina and VNC

And away you go:

Ubuntu install page over KVM
Ubuntu install page over KVM

If you need the network address, look in the folder: /etc/libvirt/qemu for the xml configuration file (in this case base-1404.xml), and look for the tag. Or xmldump from the virsh.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Welcome

Welcome to my new blog. Over the next few weeks and months, I will be adding content with regards to distributed file systems for PHP projects, starting with experiments with MogileFS, and GlusterFS.

Also will probably start with KVM virtual machine guides.

Thanks for visiting.

 

Facebooktwittergoogle_plusredditpinterestlinkedinmail