Scrapbook Photo Backups

Wed, 07 Jan 2009 15:41:23 GMT

While I don't really believe all the "LJ is going to close" drama brought about by yesterday's sudden layoffs (I do feel for those laid off with such lack of notice or severance, though - my thoughts go out to you all!) (if you have no idea what I'm talking about, check out this post, which lays it out quite well), I did realize it would definitely be prudent to back up my LJ and Scrapbook files, which I have been quite remiss in doing to date. LJ entry backup is quite well documented at these links:

However, Scrapbook backup is poorly documented (if at all, really). The best resource I've come across is this entry in lj_dev, which has solutions that basically don't work. So I took it upon myself to fix this situation. :)

The basic problem with the Perl client is that it was never updated to deal with LJ's split-authentication system after there were some security problems that prompted changes a long while back. Thanks to the wonders of the WWW::Mechanize Perl module, I believe I've fixed that client so that it works perfectly fine, and so, I present to you:

Tim's updated version of fotoup.pl
MD5 checksum: abeedb22e0937ab0f4d9da62347993e3
SHA1 checksum: 188f0506ed8be5dbb4b7ed833dfa6a2599bffeba

I will try to keep this entry up-to-date with the latest checksums if I make any further changes - if you download a version with any other checksums, it may not be the most up-to-date. This client is intended for people who are somewhat technical, and have Perl installed on their systems. You'll probably also need to be able to install Perl modules. I apologize, I don't have time to write up detailed instructions on any of that right now. Once you have those bits done, though, all you need to do is create a ".fotoup.conf" file with your configuration in it. The minimum configuration looks like:

server: pics.livejournal.com
username: your LJ username
password: your LJ password
backupdir: a directory on your system that you want to back up to

That file should be in your UNIX home directory. If you're on Windows, I'm not sure on the correct location, sorry. Now just run "./fotoup.pl --backup", and wait. You should get output that looks something like this:

$ ./fotoup.pl --backup
To upload: 0 from data, 0 from receipt

Fetching export.xml from server...
Total pictures: 3563
Already backed up: 0
Pictures to backup: 3563
Bytes to fetch over network: 4889620910
  Fetching image 1/3563 ...  0.1%
  Fetching image 2/3563 ...  0.1%
  Fetching image 3/3563 ...  0.2%
  Fetching image 4/3563 ...  0.3%
  Fetching image 5/3563 ...  0.3%
...

Sometimes you will see an "MD5 of downloaded file doesn't match, retrying." error - don't panic, the server just probably had a hiccup - the script will automatically retry once before failing fatally. If it does fail again, you can re-start and you won't lose all the files you already downloaded - just run the exact same command again. It'll then look something like this:

$ ./fotoup.pl --backup
To upload: 0 from data, 0 from receipt

Fetching export.xml from server...
Total pictures: 3563
Already backed up: 5
Pictures to backup: 3558
Bytes to fetch over network: 4874557302
  Fetching image 1/3558 ...  0.1%
  Fetching image 2/3558 ...  0.1%
  Fetching image 3/3558 ...  0.2%
  Fetching image 4/3558 ...  0.2%
  Fetching image 5/3558 ...  0.2%
...

Note the "Already backed up" number. I added the re-try functionality because I was seeing this error quite often, hopefully it won't occur twice in a row very often, so you shouldn't have to restart repeatedly, but you might want to keep an eye on things anyway, and re-start if necessary.

I'm sorry these instructions aren't more non-geek-friendly. Please feel free to play around and write up better instructions if you'd like. If you want to link to the updated version of the client, please consider linking to this entry so ensure that the MD5/SHA1 checksums are included as well. Feel free to put links to your instructions in the comments here as well, so that others can also benefit from them. Thanks!

Edit 2009-01-08 08:37 EST: Updated fotoup.pl on the server, new MD5/SHA1 sums: abeedb22e0937ab0f4d9da62347993e3/188f0506ed8be5dbb4b7ed833dfa6a2599bffeba New version should fix Windows problems with MD5, still looking at the Windows 404 loops.

Krellish Musings

Scrapbook Photo Backups