New volume group and LVM for snapshots


To backup a large MySQL database on a regular basis, I am going to use a method of taking snapshots of the database files using LVM in linux. The volume I will be using to create the logical volume is a RAID 1 (mirrored) array, as I created in the mdadm creating a new array guide.

Creating the volume group with lvm

The array we are using as the basis of the volume group is /dev/md0. Firstly we need to initialise the RAID array (you can use raw block devices or other partitions) for use as a physical volume. This is simple:

$ pvcreate /dev/md0
  Physical volume "/dev/md0" successfully created

The volume can now be used to make a volume group (I am calling this volume group vgssd, to state that it’s the SSD group):

$ vgcreate vgssd /dev/md0
  Volume group "vgssd" successfully created

Show some information about the newly created volume group

$ vgdisplay vgssd
  --- Volume group ---
  VG Name               vgssd
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               439.87 GiB
  PE Size               4.00 MiB
  Total PE              112607
  Alloc PE / Size       0 / 0   
  Free  PE / Size       112607 / 439.87 GiB
  VG UUID               ae5CoA-J0v4-eo0j-GyJd-DGvl-t7bF-zQd1wV

This shows we’ve got around the 440GB free for creating logical volumes on this device. We won’t be using it all, as we need to allow room for snapshots.

Creating the logical volume with lvm

It’s up to you to decide how to create and use your partitions. Remember that using LVM allows you to resize (extend and shrink) your volumes, as well as add more physical hard drives to the system. We have created a block device that now is mirrored over 2 SSD harddrives. I will be setting up a 250GB partition for use with mysql data, and allowing loads of space for snapshots, and extending it going forward.

$ lvcreate -L250G -n mysql_data vgssd
  Logical volume "mysql_data" created

I am going to mount this at: /var/mysql_data, and format it to be an ext4 filesystem, and make sure that access times are not recorded on mounting in fstab (noatime)

$ mkdir /var/mysql_data
$ mkfs.ext4 /dev/vgssd/mysql_data
mke2fs 1.42.9 (4-Feb-2014)
Discarding device blocks: done                            
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
16384000 inodes, 65536000 blocks
3276800 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
2000 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

And the line added to /etc/fstab is as follows:

/dev/vgssd/mysql_data /var/mysql_data ext4 defaults,noatime 0 0

So that shows how to use lvm to create a partition that can be easily extended, backed up, etc.


mdadm creating a new array


This is a short guide on using mdadm, and other command line tools to create a RAID 1 mirrored array from 2 harddrives in Linux (Ubuntu 14.04)


Firstly, lets get a list of the harddrives in the system:

$ lsblk
sda                       8:0    0 447.1G  0 disk 
sdb                       8:16   0 447.1G  0 disk 
sdc                       8:32   0   1.8T  0 disk 
├─sdc1                    8:33   0   512M  0 part /boot
└─sdc2                    8:34   0   1.8T  0 part 

We’re going to use sda, and sdb to create the array of devices. I am going to be creating a partition that nearly fills the harddrives (440G), but not completely. This allows us to rebuild from other drives in the future that may not be exactly the same size.

Creating the partitions

Note: We are deleting and modifying partitions here, please make sure you have backed up data, and that you are using the correct drive numbers

$ fdisk /dev/sda
Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 
Using default value 1
First sector (2048-937703087, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-937703087, default 937703087): +440G

Command (m for help): t
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): p
Disk /dev/sda: 480.1 GB, 480103981056 bytes
255 heads, 63 sectors/track, 58369 cylinders, total 937703088 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000071b7

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048   922748927   461373440   fd  Linux raid autodetect
Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Do the same on /dev/sdb, and afterwards, check the partitions are OK:

$ lsblk
sda                       8:0    0 447.1G  0 disk 
└─sda1                    8:1    0   440G  0 part 
sdb                       8:16   0 447.1G  0 disk 
└─sdb1                    8:17   0   440G  0 part 
sdc                       8:32   0   1.8T  0 disk 

OK we have the two partitions. Make sure there no superblocks set on these two drives (i.e. make sure they’re clean):

$ mdadm --zero-superblock /dev/sda /dev/sdb
mdadm: Unrecognised md component device - /dev/sda
mdadm: Unrecognised md component device - /dev/sdb

This shows that these were clean drives, but it’s worth doing. Next we’re going to create the array itself

$ mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
Continue creating array? y       
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

Check the state of the array:

$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sdb1[1] sda1[0]
      461242176 blocks super 1.2 [2/2] [UU]
      [>....................]  resync =  0.1% (893248/461242176) finish=34.3min speed=223312K/sec

Add the information from the mdadm --detail --scan to the /etc/mdadm/mdadm.conf file, so quicker boots, etc and the device is found. Remove the name field from the output of the mdadm command.

ARRAY /dev/md0 metadata=1.2 name=hogg:0 UUID=23529206:b91dd393:10786167:71e5ce60

Also you need to update the boot image with the following:

$ update-initramfs -u

And that’s it for creating a mirrored RAID 1 array. We’re not building file systems on this device, as I will be using it as the basis of a LVM with snapshots. In a follow up guide here, I’ll be creating a logical volume on top of this raid array.


Git – Creating an empty repository for development


This is a simple guide on how to create an empty git repository for development purposes, as well as creating a remote repository to push/pull to and from.

I am assuming you have a remote server, with SSH usersname and keys created for this guide. I find that using SVN is a good way to get going and hosting remote git repositories, and this is the way I previously used SVN for source code management.

Files already on local machine

If you have files already on a local machine the following code will turn this into a local repository:

$ git init 

It’s as simple as that. Then use git add and git commit to commit the files to the master branch.

Creating the remote git repository

It’s all well and good to have a local git repository, but next we need to have somewhere central to manage this. I use SSH to talk to the remote git server. To create a remote SSH into that box, and by default I store my git repositories in /var/git/. So firstly create a new folder for the repository, and then initiate a ‘bare’ repository.

$ mkdir /var/git/repo_name/
$ cd /var/git/repo_name/
$ git init --bare

That will create an empty repository in that location. We can now add the remote repository to the local branch, and push the contents to it, and test a checkout/clone elsewhere. So on the local box in the repository folder, carry out the following commands to add a remote, and push the new branch to it:

$ git remote add origin gunja@worker1:/var/git/repo_name
$ git remote
$ git push origin --all

That will push all the local bits to the remote server. To check that it worked, you just now need to move to a new folder, and try and clone it:

 $ git clone gunja@worker1:/var/git/repo_name