What happens when you build a multi-terabyte RAID server? You fill it to the brim with steam(y?)ing crap.
Every file and piece of media I download ends up in a single directory on my server. Programs, music, movies–let’s not be coy–all dumped into /spacepants/media/Now_Playing/ and shared with NFS and CIFS/SMB over your gigabit ethernet. You likely know your server or share as something else, but if you’re reading this, you’re going to become intimate with spacepants. Here, let me introduce you.
$ uname -a
Linux spacepants 2.6.18-jen35-default #1 SMP Tue Oct 3 01:27:41 CEST 2006 i686 athlon i386 GNU/Linux
$ more /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4] [linear]
md0 : active raid5 sdd4[0] sdc4[5] sda4[4] sdb4[3] sde4[2] sdf4[1]
1535411840 blocks level 5, 128k chunk, algorithm 2 [6/6] [UUUUUU]
md4 : active raid0 sdd2[0] sde2[2] sdf2[1]
4409472 blocks 64k chunks
md5 : active raid5 sdd3[0] sde3[2] sdf3[1]
7823360 blocks level 5, 128k chunk, algorithm 2 [3/3] [UUU]
md1 : active raid1 sdb1[0] sde1[2](S) sdf1[3](S) sdd1[4](S) sdc1[1]
104320 blocks [2/2] [UU]
md3 : active raid1 sdb3[0]
3911744 blocks [2/1] [U_]
md2 : active raid1 sdb2[0]
1469824 blocks [2/1] [U_]
unused devices:
I’m retarded. Let me fix that.
$ sudo /sbin/mdadm --add /dev/md3 /dev/sda3
mdadm: added /dev/sda3
$ sudo /sbin/mdadm --add /dev/md2 /dev/sda2 /dev/sdc2
mdadm: added /dev/sda2
mdadm: added /dev/sdc2
.. :P
$ df -H
Filesystem Size Used Avail Use% Mounted on
/dev/md3 4.1G 3.7G 379M 91% /
udev 265M 291k 264M 1% /dev
/dev/md1 104M 25M 74M 26% /boot
/dev/md5 8.0G 5.5G 2.6G 69% /home
/dev/mapper/spacepants-misc2
2.2G 787M 1.4G 37% /tmp
/dev/mapper/spacepants-usr
6.5G 4.2G 2.4G 65% /usr
/dev/mapper/spacepants-misc1
2.2G 2.0G 181M 92% /var
/dev/mapper/spacepants-sunita
6.5G 297M 6.2G 5% /spacepants/sunita
/dev/mapper/spacepants-home
6.5G 2.5G 4.0G 39% /spacepants/unknown
/dev/mapper/spacepants-work
13G 3.3G 9.7G 26% /spacepants/work
/dev/mapper/spacepants-media
1.6T 1.6T 20G 99% /spacepants/media
/dev/md4 4.5G 1.7G 2.9G 37% /spacepants/speedy
//192.168.1.12/mythtv
3.3G 3.2G 90M 98% /home/media/mythtv-remote
Right. It’s a work in progress, but that’s spacepants. Seven 320GB drives in a big jfs/reiserfs/LVM/RAID0/RAID1/RAID5 cacophony. But back to organizing–which, if you haven’t gathered already, is an everyday struggle for me.
We’re going to start from end. Because this is my latest addition and because I feel like it. This is a quick script that I smashed together earlier today and placed lovingly in my cron.hourly. It scans your Incoming/ folder (/spacepants/media/MythVideo/_Now Playing_/ in this case) and links recent folders and files for easy browsing–easy browsing is a big WAF bonus, which is what this is really all about. Isn’t it?
#!/bin/bash
# Move into our _Today_ folder because we want our links to be relative.
cd /spacepants/media/MythVideo/_Today_/
# Clear out the existing links.
rm /spacepants/media/MythVideo/_Today_/*
# Find all top-level folders that were created in the last 48 hours,
# ignoring any folder that starts with _* and links to here (_Today_).
find -P ../_Now Playing_/ -maxdepth 1 -mindepth 1 -wholename '../_Now Playing_/_*' -prune -o -type d -mtime -2 -exec ln -s '{}' . ;
# Does the same as above, but only for top-level files 5k or larger
# -- and ignore .* this time around.
find -P ../_Now Playing_/ -maxdepth 1 -mindepth 1 -wholename '../_Now Playing_/.*' -prune -o -type f -mtime -2 -size +5k -exec ln -s '{}' . ;
# Rename all links, replacing underscores (_) with spaces.
# (.. again with the WAF.)
for file in * ; do mv "$file" "`echo $file | sed 's/_/ /g'`" ; done
# Move on to the weekly archive -- note -mtime -8.
cd /spacepants/media/MythVideo/_This Week_/
rm /spacepants/media/MythVideo/_This Week_/*
find -P ../_Now Playing_/ -maxdepth 1 -mindepth 1 -wholename '../_Now Playing_/_*' -prune -o -type d -mtime -8 -exec ln -s '{}' . ;
find -P ../_Now Playing_/ -maxdepth 1 -mindepth 1 -wholename '../_Now Playing_/.*' -prune -o -type f -mtime -8 -size +5k -exec ln -s '{}' . ;
for file in * ; do mv "$file" "`echo $file | sed 's/_/ /g'`" ; done
# The monthly archive -- -mtime -32
cd /spacepants/media/MythVideo/_This Month_/
rm /spacepants/media/MythVideo/_This Month_/*
find -P ../_Now Playing_/ -maxdepth 1 -mindepth 1 -wholename '../_Now Playing_/_*' -prune -o -type d -mtime -32 -exec ln -s '{}' . ;
find -P ../_Now Playing_/ -maxdepth 1 -mindepth 1 -wholename '../_Now Playing_/.*' -prune -o -type f -mtime -32 -size +5k -exec ln -s '{}' . ;
for file in * ; do mv "$file" "`echo $file | sed 's/_/ /g'`" ; done
I saved mine as /etc/cron.hourly/updatemedia.sh — but do with it what you will. Tell the boss and procrastinate the process of actually organizing that media one more day. :P