Thursday, 25 September 2014

Cloning your Raspberry Pi SD card using dd and linux

Why clone?

Having a clone of your raspberry Pi SD card is very useful and alows you to really experiemnt with your raspberry Pi with the full knowledge that you can go back to a known good state.

What do i end up with?

What you end up with is an exact copy of your raspberry Pi SD card in a file, which you can then copy and keep for a rainy day, give to your friends, or make an exact copy of your system for use on a second raspberry Pi.

What you will need
1) your raspberry Pi SD card.
2) a PC or laptop running linux with an SD card reader (this procedure cannot be done on the raspberry Pi itself)

You will need to be the root user to make the image, so su to root first.

$ su
or on ubuntu based systems
$ sudo -i

Next plug your SD card into your PC and look for the device name that was assigned to the card. It will look something like /dev/sdX or /dev/mmcblkX This is important as we will be using this identifier for making the image.

One way of getting this information is by looking at the messages log file.

# tail -f /var/log/messages

when you plug your card in or take it out, the device name is logged in this file and you can see the messages skipping by.

Sep 24 15:48:00 p1155-awdeb kernel: [111145.426105] sd 6:0:0:0:
     [sdb] 7710720 512-byte logical blocks: (3.94 GB/3.67 GiB)
Sep 24 15:48:00 p1155-awdeb kernel: [111145.433046]  
     sdb: sdb1 sdb2 < sdb5 sdb6 >
Sep 24 15:48:01 p1155-awdeb kernel: [111145.818914] EXT4-fs 
     (sdb6): mounted filesystem with ordered data mode.
Sep 24 15:48:01 p1155-awdeb kernel: [111145.961317] 
     FAT-fs (sdb5): utf8 is not a recommended IO charset 
     for FAT filesystems, filesystem will be case sensitive!

In this case the sdcard was given a device name of /dev/sdb

Most desktops will automount any partitions it finds so next we will want to make sure that any partitions found on the SD card are not mounted before taking the image.

Using mount and lsblk commands we can see that /dev/sdb5 and /dev/sdb6 are mounted.

# mount
/dev/sdb6 on /media/0eb36e9e-40f5-47f4-a751-4e197c0dd7c8
/dev/sdb5 on /media/boot type vfat
# lsblk
sdb     8:16 1 3.7G  0 disk 
├─sdb1  8:17 1 1.2G  0 part 
├─sdb2  8:18 1   1K  0 part 
├─sdb5  8:21 1  56M  0 part /media/boot
└─sdb6  8:22 1 2.4G  0 part /media/0eb36e9e-40f5-47f4-a751-4e197c0dd7c8

Now unmount any mounted partitions for your SD card. Any other partitions can be left mounted, we are only concerned with partitions on the SD card.

# umount /dev/sdb6
# umount /dev/sdb5

(Tip: If you use the eject button in the GUI, your dev device names will disappear from your system, so then we can't use them to make our image. Using umount at the command line, like in the above example, keeps them there for us to use with the dd command later on)

Run the lsblk and mount commands again and check that they are now unmounted.

Now lets make the Image. We will be using dd to make an exact bit copy of the SD card. replace /dev/sdX with the name given to your SD card.

# dd if=/dev/sdX of=/root/myimagefile.dd bs=64K conv=sync,noerror

so in the above example where the sdcard is identified as /dev/sdb the command would be.

# dd if=/dev/sdb of=/root/myimagefile.dd bs=64K conv=sync,noerror

This will take a while, but wait for it and you will end up with an image file which is an exact copy of your SD card.

Monday, 25 August 2014

Generating random strings with tr and urandom in Bash

I recently needed a script to generate a random string in a bash script for naming multiple PCs in a desktop deployment of about 200 Linux PCs. I wanted each PC to have a unique name, this is a job for a bash script.

A google search turned up this site immediately and is where I got what i needed to get started.

From this site, I liked a combination of tr, urandom and head to generate the strings because it turns out that you can use tr to generate a whole load of different strings, alphanumeric strings, and even hexadecimal numbers!

So down to business, I wanted an letters only, 6 characters long, easy

  tr -dc "[:alpha:]" < /dev/urandom | head -c 6

If you put this into a bash prompt, you do not get a new line at the end so an echo at the end will fix that.

  tr -dc "[:alpha:]" < /dev/urandom | head -c 6;echo

This looks like what I want and now that I had played around with this on the command line, it's now ready to go into a bash script:

  #!/bin/bash

  STRING=`tr -dc "[:alpha:]" < /dev/urandom | head -c 6`
  echo $STRING

Note the ` back-ticks around what we were testing so that we can assign it to the STRING variable. The echo at the end is no longer needed too. Here's the script running.

  waltersa@p1155-awdeb:~$ ./genstring.sh 
  wjVlxh
  waltersa@p1155-awdeb:~$

Getting a random hex number would be easy too, simply change [:alpha:] to [:xdigit:] Here's a list of some of the possibilities, from the tr man page.... have fun.

       [:alnum:]
              all letters and digits

       [:alpha:]
              all letters

       [:digit:]
              all digits

       [:lower:]
              all lower case letters

       [:print:]
              all printable characters, including space

       [:punct:]
              all punctuation characters

       [:upper:]
              all upper case letters

       [:xdigit:]
              all hexadecimal digits

Wednesday, 9 July 2014

Using SCCM reporting to get the GUID of PCs

I recently needed to get the GUID and PC name of every single PC in our organisation. I used the reporting feature in SCCM and put together a very simple SQL query. Here's the query, in case it is useful to someone else.
select 
    v_R_System.Netbios_Name0, v_R_System.SMBIOS_GUID0
from
    v_R_System
This returns the PC name and GUID of every single machine in the SCCM database.

Wednesday, 2 July 2014

Using dd and ssh to transfer LVM volumes over the network

Important: This article assumes that the volume that you are transferring is not mounted and is not in use. If you destroy your servers, or lose any data when following this article, it's entirely your fault. Make sure you understand the commands before executing them.

We have two Xen hosting servers that host HVM virtual machines with a mix of operating systems, backed by local LVM volumes. Using ssh and dd here's how we transferred an exact bit copy of an LVM volume from one server to another.

These commands will only work for volumes that are not in use and that are not mounted anywhere. Please don't use these commands on mounted volumes.

On the source server, dcamnode02, lvs shows the current disks available, we wanted to move the svn_root and svn_swap partitions to another server.
root@dcamnode02:~# lvs
  LV                VG   Attr   LSize   Origin Snap%  Move Log
  svn_root          vg01 -wi-a-  10.00g
  svn_swap          vg01 -wi-a-   4.00g

First, on the destination server, dcamnode01, we created new lvm volumes of the same size as the ones that we would be transferring.
root@dcamnode01:~# lvcreate -L 10G -n svn_root vg01
  Logical volume "svn_root" created
root@dcamnode01:~# lvcreate -L 4G -n svn_swap vg01
  Logical volume "svn_swap" created

Then on the source server, we used dd to copy the source lvm volumes to newly created lvm volumes on the the destination server.
root@dcamnode02:~# dd if=/dev/vg01/svn_root bs=8M conv=sync,noerror \
   | ssh -c blowfish root@10.108.150.11 "dd of=/dev/vg01/svn_root bs=8M"
root@10.108.150.11's password: 
163840+0 records in
163840+0 records out
10737418240 bytes (11 GB) copied, 199.229 s, 53.9 MB/s
156+345822 records in
156+345822 records out
10737418240 bytes (11 GB) copied, 193.94 s, 55.4 MB/s
root@dcamnode02:~#
When you use dd and do not specify an output file using 'of=' then the output goes to standard output (the screen). Here we use a pipe to take standard output and pipe it to an ssh command. Ssh then then executes a remote command, 'dd' on the destination server, which is where we give the output file name. The result is that the dd image gets transferred over ssh to the remote server.

After the volume was transferred, an md5sum of the source and destination volumes can be performed to make sure the transfer happened correctly.
root@dcamnode02:~# md5sum /dev/vg01/svn_root 
0a310a5aaf3a1b7159c5f68e31525e27  /dev/vg01/svn_root
root@dcamnode02:~#

root@dcamnode01:~# md5sum /dev/vg01/svn_root 
0a310a5aaf3a1b7159c5f68e31525e27  /dev/vg01/svn_root
root@dcamnode01:~#
Both md5sums match. All is good.

Monday, 23 June 2014

Editing files using bash scripts and sed

Sometimes you need to edit a setting in a conf file from a bash script.
Here's a little script that installs ntp, and edits configuration files using the sed command.

First this script removes any previous ntp servers from /etc/ntp.conf
    #remove ntp server lines
    sed -i".bak" '/^server/d' /etc/ntp.conf
the -i".bak" creates a backup of the file before changing it.
the caret ^ symbol means the start of the line. So ^server means any lines starting with "server"
the d at the end means delete the line

with all these lines removed we can add the ntp server line back in. We use >> to append the new "server" line to the end of the conf file.
    #add the local ntp server
    echo "server ntp.mydomain.com iburst" >> /etc/ntp.conf
Another way to edit files can be like a search and replace.
    sed -i 's/^UTC$/LOCAL/' /etc/adjtime
so we search for ^UTC$
^ means the start of a line, $ means the end of a line
^UTC$ matches lines that contain UTC and nothing else
We then specify some text to replace UTC with, in this case it's LOCAL.

Here's the complete script.

Wednesday, 18 June 2014

Installing netbeans 8.0 on debian wheezy

We provide over 200 debian wheezy desktops in student labs and are currently getting another desktop image ready for the next academic year. Here's the steps we took to install netbeans 8.0 along with a script to add some post install tweaks. These desktops are multi-user, running the Mate desktop. To do a multi-user install of netbeans, you need to run the installer as root.
  1. log in as a normal user.
  2. download the netbeans installer from https://netbeans.org/
  3. su to root in a terminal.
  4. run the netbeans installer from the command line at the root prompt.
  5. click the customize button to install glassfish and tomcat.
  6. change the install path for each package to /opt/<name of package>
Once done, we needed to perform some post-install tweaks.
  • We needed to fix permissions to the installed files and directories.
  • The netbeans menuitem did not appear and so that needed to be added.
  • And the third tweak was a workaround with the mouse focus not matching the mouse pointer and disappearing menus as described in this bug
To make things easy, here's a bash script to do the above tweaks.

We run the Mate desktop environment and so this script sets the DESKTOP_SESSION variable to mate for the netbeans process. If you use gnome3 then DESKTOP_SESSION=gnome-shell is what you want. Edit the script accordingly. This environment variable is used to get around the disappearing menus/mouse pointer bug.

Thursday, 5 June 2014

Modifying Dependencies in a deb

I recently needed to install evoluspencil onto debian wheezy, but the deb install fails because of unmet dependencies. The deb is looking for firefox, but the debian project renames firefox to iceweasel and so the dependency cannot be met. One solution is to modify the evoluspencil debian package to allow iceweasel to satisfy the dependency. Here's a video of the steps, which are laid out in comment #5 of this bug report.
mkdir pencil
dpkg-deb -x evoluspencil_2.0.5_all.deb pencil/
dpkg-deb --control evoluspencil_2.0.5_all.deb pencil/DEBIAN
nano pencil/DEBIAN/control  # edit -> Depends: firefox|iceweasel
dpkg -b pencil/ evoluspencil_2.0.5_debian.deb
sudo dpkg -i evoluspencil_2.0.5_debian.deb