Thursday, 10 December 2015

Purging old backup files in Linux

Here's a quick script for cleaning out old backup files. We organise our backups into directories by the amount of time you want to keep the backups for. So when writing your backup script, you do not need to think about purging old backup files, this is done automatically for you. This cleanup script goes through each of the backup directories and cleans them out when the timestamp on backup files exceed the given time period.

This has since been replaced by a proper backup system, but it still remains for devices that cannot have the backup client installed, for example a config dump from a network device where it runs a small proprietary o/s

We saved this script in /usr/local/bin/purge-drbackups.sh
Here's the script in full:


#!/bin/bash

# exit if a variable is not initialised
set -u

###########################
## variables for editing
BASE_BACKUP_DIR=/srv/backup
KEEPFOR_EVER=${BASE_BACKUP_DIR}/keepfor_ever
KEEPFOR_ONEMONTH=${BASE_BACKUP_DIR}/keepfor_onemonth
KEEPFOR_THREEMONTHS=${BASE_BACKUP_DIR}/keepfor_threemonths
KEEPFOR_TWOYEARS=${BASE_BACKUP_DIR}/keepfor_twoyears
###########################

#keepfor ever
printf "== Processing keepfor ever\n"
printf "No Action needed in directory ${KEEPFOR_EVER}\n"

#keepfor onemonth
printf "== Processing onemonth dir\n"
/usr/bin/find ${KEEPFOR_ONEMONTH} -type f -mtime +31 -exec rm -vf {} \;

#keepfor threemonths
printf "== Processing threemonts dir\n"
/usr/bin/find ${KEEPFOR_THREEMONTHS} -type f -mtime +92 -exec rm -vf {} \;

#keepfor twoyears
printf "== Processing twoyears dir\n"
/usr/bin/find ${KEEPFOR_TWOYEARS} -type f -mtime +730 -exec rm -vf {} \;

Some notes on the stript

  • set -u is included at the start of the script to make the script exit if it encounters an unset variable. I mention this because we are using the rm command and could potentially delete files starting at the root directory if an unset variable is used as part of the path passed to the rm command.
  • The second point i want to make is to think about how long you want to keep backups for, you will notice that we had one directory where we only keep files for a month. This is very short I would not recommend keeping critical files for only one month. Instead, keep them for as long as possible, typically measured in years, not months. Do some research on the subject, there's books and websites that cover this topic in detail.
  • The aim of this script was to stop the server hard disk from filling up, and to keep it like that automatically. Adjust it for your own needs as required.

I would recommend having this script run once daily with cron. On debian systems you create a script in /etc/cron.daily/

Here's the contents of ours, stored in /etc/cron.daily/daily-scripts

#!/bin/bash

/usr/local/bin/purge-drbackups.sh

One final note, don't forget to make both the script and the cron executable with the chmod +x command. Thats it!

Friday, 2 October 2015

Install Arch on a Raspberry Pi with MATE Desktop

These instrutions show you how to get GNU/Linux with the MATE Desktop installed on a Raspberry Pi using Arch Linux Arm. Arch Linux Arm is a distribution of Linux for ARM computers that can be installed as an alternative to raspbian, which usually comes pre-installed on a raspberry Pi SD Cards. This is a lighweight, up to date and rolling distribution, read on if you want to try it out.

To get started, navigate to the install instructions for your Pi on the Arch Linux Arm website. These instructions tell you how to set up partitions, install the base operating system and where to download the base Arch Linux image from. These instructions assume that you have access to a linux desktop or laptop computer and are all linux commands.

http://archlinuxarm.org/

For Raspberry Pi 1, B and model B+ models, follow the the ARMv6 instructions. If you have a Raspberry Pi 2, follow the ARMv7 ones.

After completing the install instructions put the SD Card into your Pi, boot up and log in as root, you will find the root password at the end of the install instructions.

As root, perform a system wide update with the command:

  # pacman -Syu
Once done, install mate desktop and the lightdm window manager:
  # pacman -S xorg xf86-video-fbdev mate mate-extra lightdm lightdm-gtk-greeter

and configure lightdm to autostart on bootup, this will make the Pi autostart a graphical user interface:

  # systemctl enable lightdm

Finally change the default passwords for the alarm and root users. As root and issue the following commands from a command prompt

  # passwd alarm
  # passwd root

An optional extra configuration would be to enable autologin. You can get instructions for doing this from the arch wiki here.

You now have a bleeding edge rolling distribution on your raspberry Pi. For further reading and information on using Arch, the arch linux wiki is an excellent source of information

https://wiki.archlinux.org/

Wednesday, 8 April 2015

Network booting and Imaging with Clonezilla and PXELINUX

If you want to quickly look at the PXE menu file you can do so here

Ok theres a lot of stuff about PXE out there, I thought I'd do an overview of our current PXE setup. We use PXELINUX to boot from the network and provide network boot images. We mainly use this setup for deploying PC images using clonezilla.

We deploy about 250 PCs in the space of a couple of days every year and so we had a few goals for this PXE boot setup.

  • It had to integrate with the current network setup in our orgainisation.
  • It had to be easy to use.
  • It had to be as zero touch as possible.
  • It had to be reliable.
To meet these requirements, we went with clonezilla live edition and made this bootable over the network. We could then pass preseeded answers to all questions that are asked by the live edition, in fact multiple menuitems with preseeded answers were created for each task. This reduced errors when using the system and made imaging accessible and easy to use.

As mentioned, we use clonezilla boot options heavily to answer a lot of questions that would get the same answers on each boot. The clonezilla guys are very helpful and you can basically pre-answer all of the questions that get asked, including what image to pull down! This saves a lot of typing.

To find out what answers you need to pass at boot time, the general idea is to burn a clonezilla ISO to CD and do the tasks that you need to do manually. At the end, the clonezilla CD will summarise the chosen options for you, just take a note of these and include them in your network boot menu. Here's a link to the clonezilla documentation on the subject

Clonezilla needs storage space on the network, we used a separate storage server with access over SSH for this. This means that all of our PC images are password protected and can be in separate user accounts for separate departments or people. I won't cover the setup of the storage server since it's just a plain old SSH server.

Here's the tutorial that I got a lot of my PXE setup information from, our setup differs where we use nginx instead of apache, but largely it's the same.

The overall system is made up of a Debian GNU/Linux server which serves up the PXE boot images over TFTP and network OS filesystems over HTTP. A second server acts as storage for clonezilla PC images that are to be deployed. This storage server can be a windows server, ssh server, NFS.

When PCs boot up, they get the address of the PXE server and the name of the file to boot via dhcp. This dhcp configuration snippet was given to the network admins to add to the organisation's DHCP configuration. We asked for the network guys to allow specific subnets to get this configuration, as PXE booting was not going to the whole network.
    ##### PXE-specific configuration directives...
    allow booting;
    allow bootp;
    next-server 10.108.150.38;
    filename "pxelinux.0";

We used tftpd-hpa on our Debian server to serve up the TFTP PXE files. Here's our current /etc/default/tftpd-hpa

    # /etc/default/tftpd-hpa

    TFTP_USERNAME="tftp"
    TFTP_DIRECTORY="/srv/tftp"
    TFTP_ADDRESS="0.0.0.0:69"
    TFTP_OPTIONS="--secure --ipv4"

Nginx was configured to serve the same directory over http. This allows the larger squashfs files to be downloaded over http which is much faster. Here's a sample /etc/nginx/sites-available/pxe

    server {
        listen   80;
        server_name pxe;
        root   /srv/tftp;
    }
Then enable this config by linking to the file from sites-enabled and restart nginx. (If the default config is in here, remove it)
    cd /etc/nginx/sites-enabled/
    ln -s ../sites-available/pxe
    /etc/init.d/nginx restart

Next copy some pxelinux files into the tftp directory. On the debian server, install syslinux then (look at step 5 here) copy the pxelinux files that get installed. To be honest you will find different tutorials all recommending a different set of files to copy, some more some less. It all depends on the features you use in your PXE menus. Here's the ones I use:

    apt-get install syslinux
    cp /usr/lib/syslinux/chain.c32 /srv/tftp
    cp /usr/lib/syslinux/ifcpu64.c32 /srv/tftp
    cp /usr/lib/syslinux/mboot.c32 /srv/tftp
    cp /usr/lib/syslinux/memdisk /srv/tftp
    cp /usr/lib/syslinux/menu.c32 /srv/tftp
    cp /usr/lib/syslinux/pxechain.com /srv/tftp
    cp /usr/lib/syslinux/chain.c32 /srv/tftp
    cp /usr/lib/syslinux/pxechain.com /srv/tftp
    cp /usr/lib/syslinux/vesamenu.c32 /srv/tftp

Ok so now we are nearly ready to serve the PXE boot images over tftp. Next we need to create the PXE menu and add some network enabled operating systems (e.g. clonezilla live network boot).

Available in our network boot menus are: a memory error checker memtest86+, clonezilla, a debian live LXDE desktop environment, gparted and the System Rescue CD. Finding out exactly where to download these images can be troublesome, so we'll go through these. Generally you are looking to download a zip which is named closely to the a corresponding iso file. The zip file will contain the network boot version of the iso file.

memtest86+ - On the downloads page, download the pre-compiled bootable binary

Clonezilla - From the projects front page, it's in Downloads -> stable releases -> Select CPU architecure "i686-pae", and file type "zip". You may want a different CPU Architecture, read the notes on this page.

Debian Live LXDE - from live.debian.net front page, Under user, Download releases, stable, amd64, webboot. Look for the latest version of the desktop you want. then download three files, ending with vmlinuz, initrd.img, squashfs. These are the kernel, the initial ram filesystem, and the live filesystem respectively.
Here's the download location:
http://cdimage.debian.org/debian-cd/current-live/amd64/webboot/

gparted - Download the gparted live ZIP file from here
http://sourceforge.net/projects/gparted/files/gparted-live-stable/

System Rescue CD - All the files you need are on the ISO file. The files you want to copy from the CD are:

   isolinux/rescue32
   isolinux/rescue64
   isolinux/initram.igz
   sysrcd.md5
   sysrcd.dat

Once I had downloaded all of these files, I made an images directory under /srv/tftp/ and copied the various images into a directory hierarchy. I'm just going to do a tree command on the filesystem and you can work out what goes where. Here it is And here is just the directories

Finally on to creating the PXE boot menu itself. Create a directory under /usr/tftp called pxelinux.cfg in here create a file called default. This file contains all of the menu items and options. Again I'll just post our complete working menu file so that you can take and compare to your own config files.

    mkdir /svr/tftp/pxelinux.cfg
    touch /svr/tftp/pxelinux.cfg/default

Here's a link to our PXE menu file. Some menu items have a password associated with them (which is blah), these are generated with the sha1pass tool. Also you can optionally hide the menu completely by uncommenting two lines near the top of the file the lines starting with MENU SHIFTKEY and NOESCAPE.

There's a lot of information here and a lot that can go wrong with your setup. If you feel that some aspect needs more explanation, comment and I can do a post that specifically covers that area. Anyway I hope this helps someone out there.

Monday, 23 February 2015

Finding rogue DHCP servers with a Raspberry Pi

You can download the dhcp-server-sniffer perl script here.

The main purpose of this script is to find rogue DHCP servers in a subnet and get their IP and MAC address. If you are looking to capture a complete dhcp converation, this is not the tool for you, dhcpdump is a nice tool for that job.

I decided to put the DHCP server sniffer script onto a Raspberry Pi so that we could monitor all traffic on a particular subnet that is used for teaching server administration. Sometimes a rogue DHCP server accidently gets run on this subnet.

Getting the script to run on the Pi was a matter of copying the script onto the Pi and then installing some CPAN libraries

apt-get install libnetpacket-perl libnet-pcap-perl libnet-dhcp-perl

One i was happy that is was running correctly, I left the Pi in a comms cabinet and patched it into the subnet that I wanted to monitor. It is currently sniffing DHCP traffic for the a subnet in that cabinet. I didn't bother attaching a keyboard, mouse or monitor to it so access is over the network via ssh only.

I did encounter a small problem, when I put the Pi into the comms cabinet and plugged it into the network, it grabbed a DHCP address. There's no monitor on the Pi now, so how do I find out the IP address of the pi so that I can ssh to it?!

One way is to use nmap and go through the entire network to see which devices are listening to the ssh port.

So from some other Linux desktop machine you could run the following nmap command:

waltersa@awdeb:~$ nmap -p22 10.108.5.0/24 -oG - | grep open
Host: 10.108.5.2 () Ports: 22/open/tcp//ssh///
Host: 10.108.5.3 () Ports: 22/open/tcp//ssh///
Host: 10.108.5.129 () Ports: 22/open/tcp//ssh///
waltersa@awdeb:~$

Here we can see that three devices answered, and I happen to know that the first two are the routers for the local subnet, so our pi has to be the third one.

A second, more foolproof way is to use a combination of fping and arp. Before putting the Pi into comms cabinet, make a note of it's MAC address, the one on my Pi was b8:27:eb:53:90:e5.

Then from any Linux machine on the local subnet, ping all machines in the subnet with fping. 'apt-get install fping' if you need to.

fping -g 10.108.5.0/24

Pinging every machine in the subnet causes your PC to do arp lookups for each device on the subnet, these are MAC address and IP address pairings. To see this information do an arp -an and look for the MAC address of our Pi.

labadmin@specialistlab:~$ arp -an
? (10.108.5.158) at 00:23:ae:6a:f9:67 [ether] on eth0
? (10.108.5.3) at 00:04:96:41:45:c0 [ether] on eth0
? (10.108.5.196) at 00:18:8b:19:38:61 [ether] on eth0
? (10.108.5.2) at 00:04:96:43:9f:40 [ether] on eth0
? (10.108.5.129) at b8:27:eb:53:90:e5 [ether] on eth0
? (10.108.5.57) at 00:18:8b:19:45:46 [ether] on eth0
? (10.108.5.103) at 00:24:7e:0e:2e:8e [ether] on eth0
? (10.108.5.1) at 00:00:5e:00:01:01 [ether] on eth0
? (10.108.5.132) at 00:23:ae:6a:b2:de [ether] on eth0
? (10.108.5.125) at 00:18:8b:1a:3b:4d [ether] on eth0
labadmin@specialistlab:~$

Here we can see that our Pi, b8:27:eb:53:90:e5, has an IP address of 10.108.5.129.

Now that we have all the information we need to ssh the Pi, I'll talk about the DHCP sniffer script.

This script captures packets in a switched networking environment so it will not capture a complete DHCP conversation. It will only capture those parts that are broadcasts. This is good enough for our purposes as a rogue DHCP server IP address, which is what we want to find, is included in the DHCP payload in these ethernet broadcasts.

Lets talk about the output of the script. Run it with sudo.

pi@raspberrypi ~ $ sudo ./dhcp-server-sniffer.pl 
Timestamp         | Message Type  | DHCP Server IP  | Ether MAC   
20150220 10:43:21 | DHCP Ack      | 10.100.32.12    | 0004964145c0
20150220 10:43:21 | DHCP Ack      | 10.100.32.12    | 000496439f40
20150220 10:53:49 | DHCP Nak      | 10.100.32.12    | 0004964145c0
20150220 10:53:49 | DHCP Nak      | 10.100.32.12    | 000496439f40
20150220 10:53:50 | DHCP Offer    | 10.100.32.12    | 0004964145c0
20150220 10:53:50 | DHCP Request  | 10.100.32.12   
20150220 10:53:50 | DHCP Ack      | 10.100.32.12    | 0004964145c0
20150220 10:53:50 | DHCP Ack      | 10.100.32.12    | 000496439f40

Timestamp
This is the current local time of the computer that is doing the sniffing, namely our Pi's time, not the time from the DHCP server or the client.

Message Type
This field is taken from the DHCP packet payload, this is the type of DHCP packet being captured. You will probably see DHCP Ack, DHCP Offer, DHCP Request and DHCP Nak type packets. This wikipedia page isn't bad for looking up more information about this.

DHCP Server IP
The IP address displayed read from the DHCP Payload data itself and is the IP address of the DHCP server that is talking to the client PC. Is not taken from the IP layer of the packet.

Ether MAC
Ok this is where it gets a little tricky, the ether MAC is taken from the ethernet part of the packet. So it may or may not be the MAC address of a DHCP server. In the above output, the MAC address that is displayed are the MAC address of the two routers, which act as DHCP relay agent. So these are not the MAC addresses of our DHCP servers. The question would be why display them here at all? Well the answer is, for a rogue DHCP server, the MAC address displayed WILL be the MAC address of the rogue DHCP server. The reason for this is that this conversation is not being relayed by the subnet routers. This MAC address is very usefull for giving to the network administrator to do lookups against the switches mac address table, or CAM table in order to pinpoint the rogue to a particular switchport.

Well I think thats it. I'm going to leave that Raspberry Pi hidden in that comms cabinet waiting for when a rogue DHCP server appears again and starts to cause havoc.

You can download the dhcp-server-sniffer perl script here.

Friday, 6 February 2015

DHCP Server Sniffer Perl Script

I needed something to detect rogue DHCP servers on our network so I created a small perl script that listens to DHCP broadcasts and prints the IP address of the DHCP server along with the originating MAC address. This is useful in an information gathering situation where you need to confirm that there is in fact a rogue DHCP server present on your LAN.

You can download the dhcp-server-sniffer perl script here.

This perl script uses libpcap to sniff the local subnet for DHCP broadcast packets that contain a DHCP server IP address. In particular we want to capture REQUEST packets sent by the client.

A typical DHCP session looks like the following:

Client does a DISCOVER (broadcast)
Server sends and OFFER (can be broadcast or unicast)
Client sends a REQUEST (this is a broadcast)
Server sends an ACK

When a client sends one of these REQUEST packets, they also include the IP address of a DHCP server they are sending to. These are ethernet broadcast packets and so are ideal for our packet sniffer. Once captured, these packets are then processed and the IP address of the DHCP server is printed to the screen. If the packet being processed happens to be a reply from the server, the originating MAC address is also printed. This way we can easily see the IP addresses of any hosts that are acting as DHCP servers, be them legitimate or rogue.

You will need to run the script as root, and can optionally pass the interface name to listen on to the script at the command line.

This script was developed and tested on Debian GNU/Linux using Perl 5 and should run on any Linux based system without much problem.

Some output from a sample run.
root@p1155-awdeb:~# ./dhcp-server-sniffer.pl
DHCP Offer detected | DHCP Server IP: 10.100.32.5 | src MAC: 0004964145c0
DHCP Request detected | DHCP Server IP: 10.100.32.5
DHCP Ack detected | DHCP Server IP: 10.100.32.5 | src MAC: 0004964145c0
DHCP Release detected | DHCP Server IP: 10.100.32.5
DHCP Offer detected | DHCP Server IP: 10.100.32.5 | src MAC: 000496439f40
DHCP Request detected | DHCP Server IP: 10.100.32.5
DHCP Ack detected | DHCP Server IP: 10.100.32.5 | src MAC: 0004964145c0
DHCP Ack detected | DHCP Server IP: 10.100.32.5 | src MAC: 000496439f40

The CPAN Libaries needed to run this script:

http://search.cpan.org/dist/Net-Pcap/
http://search.cpan.org/dist/NetPacket/
http://search.cpan.org/dist/Net-DHCP/

To install these CPAN libraries on debian based systems:

  apt-get install libnetpacket-perl libnet-pcap-perl libnet-dhcp-perl

Reference material:

http://tools.ietf.org/html/rfc2131
http://www.tcpdump.org/

Monday, 5 January 2015

Writing a Raspberry Pi dd image file to an SD card

This is essentially a follow up to the post Cloning your Raspberry Pi SD card using dd and linux, where we made a clone of a Raspberry Pi SD card and saved it to a file.

In this post we will be taking this image file and writing it back out to an SD card, which we can then use to boot a Raspberry Pi.

What you will need
1) A file containing an image of the SD card (you should have made this earlier)
2) A Raspberry Pi SD card. (Warning: This card will be completely overwritten by this process)
3) A PC or laptop running linux with an SD card reader, this procedure cannot be done on the Paspberry Pi.

On the PC/laptop you will need to be the root user to make the image, so su to root first.

$ su
or on ubuntu based systems
$ sudo -i

Next plug your SD card into your PC and look for the device name that was assigned to the card. It will look something like /dev/sdX or /dev/mmcblkX This is important as we will be using this identifier for writing the image to the SD card.

One way of getting this information is by looking at the messages log file.

# tail -f /var/log/messages

when you plug your card in or take it out, the device name is logged in this file and you can see the messages skipping by.

Sep 24 15:48:00 p1155-awdeb kernel: [111145.426105] sd 6:0:0:0:
     [sdX] 7710720 512-byte logical blocks: (3.94 GB/3.67 GiB)
Sep 24 15:48:00 p1155-awdeb kernel: [111145.433046]  
     sdX: sdX1 sdX2 < sdX5 sdX6 >
Sep 24 15:48:01 p1155-awdeb kernel: [111145.818914] EXT4-fs 
     (sdX6): mounted filesystem with ordered data mode.
Sep 24 15:48:01 p1155-awdeb kernel: [111145.961317] 
     FAT-fs (sdX5): utf8 is not a recommended IO charset 
     for FAT filesystems, filesystem will be case sensitive!

Note: I have changed the real SD card dev name to /dev/sdX in this sample output. Just make a note of the /dev name that the SD card was given on your system.

Most desktops will automount any partitions it finds so next we will want to make sure that any partitions found on the SD card are not mounted.

Using mount and lsblk commands we can see that /dev/sdX5 and /dev/sdX6 are mounted.

# mount
/dev/sdX6 on /media/0eb36e9e-40f5-47f4-a751-4e197c0dd7c8
/dev/sdX5 on /media/boot type vfat
# lsblk
sdX     8:16 1 3.7G  0 disk 
├─sdX1  8:17 1 1.2G  0 part 
├─sdX2  8:18 1   1K  0 part 
├─sdX5  8:21 1  56M  0 part /media/boot
└─sdX6  8:22 1 2.4G  0 part /media/0eb36e9e-40f5-47f4-a751-4e197c0dd7c8

Now unmount any mounted partitions for your SD card. We are only concerned with partitions on the SD card.

# umount /dev/sdX6
# umount /dev/sdX5

(Tip: If you use the eject button in the GUI, your dev device names will disappear from your system, so then we can't use them to write our image. Using umount at the command line, like in the above example, keeps them there for us to use with the dd command later on)

Run the lsblk and mount commands again to be sure that the partitions are now unmounted.

Now we are ready to write the dd image file to the SD card. We will be using dd to write the image back to the SD card. In the line below, replace /dev/sdX with the name given to your SD card.

Data Loss Warning: Make sure you have correctly identified the device name of the SD card before running the command below, if you get this wrong and use the wrong /dev device name, you could overwrite one of your PC's hard disks. Any data loss is entirely your fault

# dd if=/root/myimagefile.dd of=/dev/sdX bs=64K

This will take a while, when it is finished remove the SD card from your PC. The SD card is now ready to be plugged into a Raspberry Pi and should boot up as normal

A final Note: If you have an image of a smaller SD card and want to put it onto a larger card, e.g. you made an image of a 4GB SD card and want to write it to an 8GB or 16GB card, this is fine. Copy the image to the SD card as described above, but as an extra last step expand the SD card image so that you can access the full size of the larger SD card. In the above example, the extended partition /dev/sdX2 and then the logical partition /dev/sdX6 would need to be expanded. Gparted is a nice tool to do this, so

# apt-get install gparted