IT Conversations Premium Editions and wget

The new premium editions of the IT Conversations shows are great, but I couldn't download them using wget, e.g.:

$ wget http://www.conversationsnetwork.org/audio/premium/username|email/ITC.Rails-TimBray-2007.05.19.mp3

--14:27:31--  http://www.conversationsnetwork.org/audio/premium/username
           => `username'
Resolving www.conversationsnetwork.org... 74.53.160.146
Connecting to www.conversationsnetwork.org|74.53.160.146|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
14:27:32 ERROR 404: Not Found.

The problem was caused by the | character separating the username and email address in the personalised URL. "URL encoding" this to "%7C" fixed the problem.

Ambient Email Notifier (some code)

Will asked about the code I was using in my ambient email notifier. The full code is a bit difficult to figure out because I've got it tied to a system tray icon thingi, which can go in another post another day, but here are some relevant bits.

First, you can get the number of emails with a particular label in gmail with the following Python code:

import feedparser
def msgCount(uid, pwd, filter):
    inbox = feedparser.parse("https://%s:%s@gmail.google.com/gmail/feed/atom%s" % (uid, pwd, filter))
    return len(inbox["entries"])

uid is your gmail address without the @gmail.com bit, filter is "" for the inbox, and "/label/" to get messages tagged with a particular label.

So, after calling msgCount a few times, for different labels, I compute the colour of the RGB LED:

colour = (inbox > 0 and 1 or 0) + (news > 0 and 2 or 0) + (work > 0 and 4 or 0)

This is sent over the serial port to the picaxe which decodes bit 0 for blue, 1 for green and 2 for red.

def triggerAmbient(colour):
    com = serial.Serial("COM3", 2400, timeout=0.25)
    for attempt in range(0,10):
        com.write("%c" % (colour) )
    com.close()

It tries to send a few times in case the picaxe doesn't get it the first time. The code on the picaxe just listens for a byte on the serial input and outputs the lowest 4 bits to the output pins:

main:
 serout 0, n2400, ("Ok")
 serin 3, n2400, b0
 gosub nibble3
 b0 = b0 & 7
 serout 0, n2400, (#b0)
 goto main
 
nibble3:
 if bit2 = 1 then
  high 1
 else
  low 1
 endif
 if bit1 = 1 then
  high 2
 else
  low 2
 endif
 if bit0 = 1 then
  high 4
 else
  low 4
 endif
 return 

This means the python script has control over the colour and you can test that it's working by simply opening up a terminal on COM3 and typing away (A is 01000001 in ASCII, meaning pin 1 is switched on, B is 01000010 so pin 2 is on, etc.)

Update: I've detailed the changes I made to get this working under Linux.

abcde — A Better CD Encoder

ABCDE is a nice CD ripper/encoder utility for Linux.

It's all nice and automatic, and I managed to configure it to suit my mp3 management style pretty quickly. My .abcde.conf file:

OUTPUTDIR=/home/tom/Desktop
OUTPUTTYPE=mp3
LAMEOPTS='-h -b 224'
ACTIONS=default,playlist
OUTPUTFORMAT='${ARTISTFILE}/${ARTISTFILE} - ${TRACKFILE}'
VAOUTPUTFORMAT='Various/${ARTISTFILE} - ${TRACKFILE}'
PLAYLISTFORMAT='${ARTISTFILE}/${ARTISTFILE} - ${ALBUMFILE}.m3u'
VAPLAYLISTFORMAT='Various/Various - ${ALBUMFILE}.m3u'

That sets it up to rip to mp3s at 224kbs and write out a playlist file. The files are saved onto the desktop to test before I move them to my mp3 folder in the "Artist/Artist - Track.mp3" name format that I prefer.

mungefilename () {
 echo "$@" | sed s,:,\ --\ ,g | sed s,/,\ --\ ,g | tr \* + | tr -d \"\?\[:cntrl:\]
}

I also changed the filename processing so that colons and forward slashes are converted to double dashes (my scripts use single dashes to split the artist from the track title.) And I don't convert spaces to underscores and I leave single quotes alone.

MAXPROCS=2

Runs an encoder process on each core so it goes super fast.

I run it from an icon on the desktop with the following command line:

gnome-terminal -x abcde

Update: I just discovered the "gnome-terminal -x" is unnecessary, there's an option to launch an application in a terminal window automatically...

Backup Log Visualisation

I'm having a bit of trouble figuring out how to describe this. It's a page that shows the current status and history of my backup strategy, and hopefully it's fairly obvious what's going on.

Each time the rsync process runs, the results are logged to a text file. A python routine parses the log file to determine the time the rsync was run, the number of files copied etc.

A little calendar indicates if a copy was made in each time slot of the day, this gives an interesting indication of when my computer is switched on. A histogram at the top shows the all time counts, the calendar only shows the last 30 days.

I'm using the Google Chart API to graph the number of files, total size, number of files copied and size of files copied.

The results of sbackup are determined by looking at the dated backup files and their sizes and graphed on a calendar. The size of the incremental and full backups are plotted on separate scales.

I'm also running a dump from MySQL once a day.

Podcast and Audiobook Queue Visualisation

Some screenshots of a little web app I created to visualise my podcast and audio book listening queue:

This shows files currently on my mp3 player. The width of each file represents it's size, the "window width" being 128mb.

Files that will be copied during the next "sync" are indicated with a red dot. Files are copied according to priority, to fill the player and to have only 3 audio book chapters at any time on the player unless there is nothing else to listen to.

Audio book chapters that have been copied/listened to already are highlighted, as are those that have been downloaded but not yet queued.

The displayed title is taken from the id3 tags if possible, otherwise the filename is used. Dates and redundant "book titles" are removed from the title if possible to maximise the amount displayed, and there is a "hover panel" to display other details.

There's also a pretty calendar to provide a gauge of the age of podcasts in different categories.

Photos by Day of Year

One last "heat-map", by day of the year:

	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec	Total
1	174	390	1	21	3	9	13	51	13	10	16	72	773
2	7	332	20	3	13	19	8	19	49	83	20	15	588
3	37	271	13	9	35	7	93	43	115	39	48	40	750
4	6	86	12	19	4	24	30	186	133	38	144	34	716
5	70	99	2	12	4	5	45	49	70	12	74	52	494
6	73	88		5	57	55	29	93	84	166	120	17	787
7	169	102	2	55	11	83	30	23	123	39	34	49	720
8	203	206	3	70		20	58	22	88	39	45	20	774
9	130	88	7	21	4	89	47	15	51	106	12	78	648
10	65	154	9	17	5	51	51	25	12	146	12	27	574
11	30	381	6	46	2	100	58	42	42	134	48	1	890
12	23	199	20	38	13	35	90	38	29	102	48	3	638
13	36	190	19	23	3	21	115	59	15	49	2	19	551
14	50	263	23	10	39	7	104	159	70	17	53	23	818
15	57	197	3	9	14		32	263	21	92	24		712
16	37	199	16	39	10	57	10	188	101	60	45	42	804
17	66	238	23	20	12	22	80	63	69	16	15	2	626
18	109	223	15	43	12	70	96	54	53	12	5	9	701
19	163	239	87	117	17	20	20	110	101	14	108	13	1009
20	164	31	52	97	68	31	64	18	105	5	18	37	690
21	200		32	73	19	21	28	72	37	22	5	10	519
22	194	16	1	43	1	63	24	21	112	36	7	20	538
23	287	7	33	80	9	27	80	18	60	5	32	104	742
24	176	108	3	4	8	16	47	106	49	17	8	187	729
25	308	31	15	95	28	34	94	159	56	5	46	333	1204
26	239	4	15	4	23	50	46	70	88	41	3	90	673
27	307	24	3	43	15	32	7	67	65	51	41	71	726
28	239	11	4	43	10	6	26	14	49	41	10	13	466
29	166		20	7	11	11	32	3	6	24	32	11	323
30	278		4	36	22	10	27		54	77	44	104	656
31	336		7		13		17	83		32		192	680
Total	4399	4177	470	1102	485	995	1501	2133	1920	1530	1119	1688	21519

Interesting that there are still some "holes" in there after 8 or so years.

Photos by Time of Day

Another "heat-map", photos by time of day for each day of the week:

	12am				1				2				3				4				5				6				7				8				9				10				11				12pm				1				2				3				4				5				6				7				8				9				10				11				Total
Mon																																																																																																	2449
Tue																																																																																																	2828
Wed																																																																																																	2774
Thu																																																																																																	2658
Fri																																																																																																	2545
Sat																																																																																																	4395
Sun																																																																																																	3870
Total	222				138				34				12				8				18				26				328				360				617				1223				1912				1938				1826				2049				2094				1984				1978				1248				962				1376				550				389				227				21519

(Hover for a tooltip indicating the actual number of photos in each 15 minute segment.)

Photo Calendar Heatmap

A "heat map" of the number of photos I've taken over the years:

Year	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec	Total
2000										274	337	62	673
2001	237	193	242	100	103	323	972	1335	1214	1053	475	347	6594
2002	270	310	202	125	70	113	19	191	120	18	84	102	1624
2003	1427	810	26	377	45	90	45	45	106	175	39	97	3282
2004	80	38	21	94	43	71	46	138	110	38	211	257	1147
2005	190	80	24	53	58	80	95	55	81	52	43	392	1203
2006	1975	2697	141	330	178	171	48	384	98	15	102	372	6511
2007	298	248	62	129	23	154	246	466	178	179	172	127	2282
2008	165												165
Total	4642	4376	718	1208	520	1002	1471	2614	1907	1804	1463	1777	23502

Green represents the average number of photos, yellow the maximum.

Different events can clearly be seen, some of note:

• June—October 2001, Travelling in Europe
• January—February 2003, Travelling in the USA
• January—February 2006, Roadtrip around Victoria

I'm going to have to find something to do during May...

Cover-art Wallpaper under Ubuntu

This is an update to the previous post to create a cover art "stack" for Windows XP wallpaper to generate and update the wallpaper in Ubuntu.

Not much had to change, except the setWallpaper function:

def setWallpaper( bmp ):
    os.system( "gconftool-2 -t string -s /desktop/gnome/background/picture_filename %s" % bmp.replace(" ","\\ ") )

Ubuntu File System Setup

The following is a rough summary of how I've setup my Ubuntu file system, including some links to resources I found useful.

I have 3 hard-drives, a 500gb drive for the system and "non-critical" files, and two 300gb drives to hold "valuable" data. "Valuable" data includes documents, photos, source-code, music and some videos, less valuable "non-critical" data includes recorded TV programs, movies and so on.

On my Windows XP system I was using a RAID-1 setup to automatically mirror my data drive to protect against hard-drive failure. Using a RAID-1 array keeps everything nice and simple, but under Ubuntu I decided that the tools were more powerful and I would be able to "manually" mirror the data to gain a little more control. In particular, while I want a copy of the data in case of hard-drive failure, a slightly delayed backup in case of catastrophic user error would also be very valuable.

So, each drive is mounted separately, and the 500gb drive is partitioned to give some space for the system and the majority for "stuff" (videos, backups etc.)

I highly recommend partitioning the drives before/during installation, as while it was possible to move everything around, it was time consuming and fiddly. I used a Gparted LiveCD to split my main drive to get a separate /stuff partition, but this took forever (about 15 hours, and my system isn't slow...). I used the GParted LiveCD because this tutorial on installing WinXP on a Linux system suggested that the version of GParted on the Ubuntu LiveCD wouldn't work properly.

This guide on moving /home to a separate partition was helpful, though I think I just used cp -a to copy the files, rather than the complicated find | cpio command..

Once I had it all up and mounted, I created a script to run rsync to mirror the drives:

#!/bin/bash
date >> /home/tom/backup_mirror_home.log
rsync --archive --delete --stats /home/ /homeMIRROR/ 2>&1 >> /home/tom/backup_mirror_home.log

This was added in as a cron job to run every 3 hours, which should be often enough (sudo crontab -e):

# m h  dom mon dow   command
0 */3 * * * /home/tom/backup_mirror_home.sh

The rsync process took a while to copy the 200gb or so across the first time, but is very fast (probably less than a minute) when just synchronizing the existing files as most data on the drive is photos and music that don't change very often.

A useful trick to keep an eye on the progress of the copy operations is watch df. df shows the current usage on the various disks, and watch will run it every 2 seconds so you can watch the percentages change slowly and be convinced something is actually happening.

Once all that was done, I setup sbackup to backup to the /stuff drive and an external USB drive.

My actual process was a little more complicated than described above, as the 300gb drives already contained NTFS partitions with the critical data on them and I wanted to move it over to ext3 while at no point having less than 2 copies of the data. The steps were roughly as follows:

1. Mount ntfs drive (/dev/sdb), copy data to 500gb drive (/dev/sda)
2. Format /dev/sdb as ext3, copy data from /dev/sda to /dev/sdb.
3. Move /home to /dev/sdb, and reorganise it a bit (rename/move folders/files about.)
4. Format /dev/sdc as ext3, mirror from /dev/sdb