Roving Thoughts archives

My current photo processing workflow (as of June 2010)

In Thom Hogan's June 14th update (now here), he wrote:

Tonight's homework: document your workflow. Really. Write it down. Include everything that happens from pressing the shutter release to looking at the final image (wherever it may have ended up, e.g. on a wall, on Facebook, etc.).

I have some spare time today for once, since I already wrote today's techblog entry, so I feel like tackling this one just because.

To start with, a note. My workflow is strongly influenced by two things. First, I'm a Linux user, which means a limited choices for software and tools (and a bunch of scripting, because I'm comfortable with that). Second, it's strongly oriented around my Project 365 work, with an inevitable time-based focus on how I organize and approach things.

So:

• the camera puts the picture in the default Nikon directory structure on my 4GB SD card. I have my camera set to the defaults, where it just numbers images sequentially and only resets the numbering when it rolls over every 10,000 images.

  A bit of negative workflow: I've learned the hard way that I can't tell a good picture from a bad one from just looking at the camera LCD (and it goes both ways; good pictures have looked bad on the LCD, and bad pictures have looked great). So I almost never delete pictures in the camera and generally it has to be completely and obviously a bad picture before I will.

  (The common causes are accidentally taken pictures or pictures where it is clear that the exposure is nowhere near where I want it.)

• when I want to pull things off the camera, I use a script to rsync the entire card to my current master directory. This happens every day, generally only once.

  (Note that when I say 'the entire card', I really mean it; the master directory is an exact image of the card's directory layout.)

  I don't reformat the card when I do this. Instead, pictures stay on the card until the nominal remaining capacity drops below somewhere in the 90 to 50 images range (I typically take around 50 pictures a day, so this gives me at least a day's margin on card space). At that point I move the current master directory to my archival area, start a new one, and immediately reformat the card. Master directories are numbered sequentially as d90-pool-NN; I'm going to have to go three digits soon.

  (This is why I have to use rsync; I need something that will not re-copy already copied images.)

  My iron rule on card reformatting is that I must have run the card sync script a second time immediately before I reformat, and it must have reported nothing synced. This is designed to avoid accidentally reformatting a card with un-transferred photos.

  Yes, this does mean that I have every photograph I've ever taken (and not immediately deleted in the camera). Disk space is cheap at the low-ish rate that I photograph.

  (I am not claiming that I have a useful archive of every photo I've taken, because it's not. But if I really want to find something, at least it hasn't been deleted so it's possible.)

• at the end of each day I use an exiftool-based script to copy all of the day's photographs to a temporary staging directory. Usually this happens at the same time as I'm pulling them off the camera.

  (This is also the point where I pop the camera battery out and drop it in the battery charger. Also, I clear the staging directory of the previous day's pictures before running the script. This is not scripted, because I don't script things that delete pictures.)

• I use Bibble 5 on the staging directory in a multi-pass approach to decide on my selects and completely process them (all in the staging directory). At the end of this I have Bibble 5 write the 'developed' JPEGs to a subdirectory and I go through them with xli to make sure that I'm happy with them; if not, I process them some more until I am.

  The actual details of how I work in Bibble 5 are far too long (and variable) to go into this entry, which is already long enough.

• I use a script to copy all of the bits of the final selects from the staging area to my Flickr archive area (which lives inside my general photo archive area). This is broadly organized by day (and by month and year once each is finished and I archive it). By 'all of the bits' I mean the original raw file from the camera, the Bibble data file about the edits I did, and the final generated JPEG.

  (The script picks out what to copy based on what pictures have generated JPEGs.)

  If I had to use chromatic aberration correction, I use the GIMP to trim off the last few pixels on the sides of the picture if they need it, because the current version of Bibble 5 corrupts the very edge of the picture in this case. (If I have cropped an edge in it doesn't need this, so I don't just hit every CA-processed image with an ImageMagick script or the like.)

  (In theory I could crop the image by those few pixels in Bibble 5. In practice, Bibble 5 on Linux is currently unusably slow when cropping in magnified view. So I get to use the GIMP.)

• I upload the JPEGs to my Flickr using Flickr's basic uploader page in Firefox. After this finishes, I blank out the default filename-based titles that Flickr has given the pictures and add tags for appropriate things if they're missing.

  (Then I agonize over what to chose as my Project 365 photo, except on days when the choice is completely obvious or I only had one thing that was worth uploading to start with.)

On some days, I'm selecting images for more than just my Flickr uploads; the most common case is that I am also selecting for TBN's website. In these cases I generally repeat the last three stages for each separate reason, sometimes entirely independently and sometimes interleaved (where as I look at each image, I decide both if it's good for Flickr and if it's good for TBN).

(Note that I have two completely separate photo archive areas, one for the master directories, and one general photo archive area for all of the pictures that I've selected for various things. The second area has subdirectories for the thing, like flickr and tbn and family, and then generally date-based within each reason. If I had a higher volume of pictures, I would probably want to be more organized and consistent about my directory structures.)

As a Linux user, my strong impression is that Bibble 5 is about my only good choice for processing anything more than a few photographs in raw format. There are some free programs that will process individual raw format pictures, generally not really very well or fast, but I haven't found one that does a decent and acceptably fast job at browsing through them so I can make my selects.

(At this point I am nowhere near willing to either give up Linux or to get a second computer just to do photo processing.)

I could simplify a bunch of this workflow if I could bring myself to trust Bibble 5's 'catalog' asset management features. I would probably use multiple catalogs, with one for my master archive and then one for each reason I pick out photographs (a Flickr one, a TBN one, etc), and switch to formatting the card every time I copied the pictures off it (even though this makes me nervous; leaving the pictures on the card is vaguely reassuring just in case something disastrous happens on the computer). However, this would mean giving up the principle that nothing except my own scripts gets to go anywhere near the master archives.

(I'm a sysadmin. No, I don't trust your program.)

With a more complicated copying scheme I could change my master archives over to a date-based directory structure while still not reformatting cards immediately. I would have to rsync to a staging area, then hard-link the files into their final destinations (chosen based on their EXIF dates); anything that was already hardlinked wouldn't need to be looked at a second time, which would make it reasonably efficient.

Sidebar: looking back at the history of this

A lot of the dance around my master directories is because when I started out, I was planning to burn each master directory to DVD when it was 'done' as a backup archive; this is also why I got a 4GB SD card, because it went well with wanting roughly DVD-sized chunks of work. I never actually implemented this plan; my backups are instead just rsync'd to an external USB drive every so often.

(Don't panic, my machine has mirrored drives to start with.)

It's interesting and a bit depressing to see how pervasively this never implemented backup plan has shaped the rest of my workflow.

Back when I was using Bibble 4, my theoretical workflow was to use the staging area only to make my selects, then have Bibble 4 copy the selects to the per-day P365 archive area, re-point Bibble 4 to it, and process them there. This never entirely worked; every so often I would have to do most of the processing before deciding whether something was a select or not, and every so often I would get pulled into processing an image before pausing to copy it.

When the Bibble 5 beta came out, it forced my hand by not supporting directory based file copying (it could only copy files around inside its asset-management catalogs). If I was copying the files outside of Bibble 5 anyways, it was much easier to do all of the processing in one directory instead of theoretically splitting it across two separate ones.