My current photo processing workflow (as of June 2010)

In Thom Hogan's June 14th update (now here), he wrote:

Tonight's homework: document your workflow. Really. Write it down. Include everything that happens from pressing the shutter release to looking at the final image (wherever it may have ended up, e.g. on a wall, on Facebook, etc.).

I have some spare time today for once, since I already wrote today's techblog entry, so I feel like tackling this one just because.

To start with, a note. My workflow is strongly influenced by two things. First, I'm a Linux user, which means a limited choices for software and tools (and a bunch of scripting, because I'm comfortable with that). Second, it's strongly oriented around my Project 365 work, with an inevitable time-based focus on how I organize and approach things.

So:

• the camera puts the picture in the default Nikon directory structure on my 4GB SD card. I have my camera set to the defaults, where it just numbers images sequentially and only resets the numbering when it rolls over every 10,000 images.

  A bit of negative workflow: I've learned the hard way that I can't tell a good picture from a bad one from just looking at the camera LCD (and it goes both ways; good pictures have looked bad on the LCD, and bad pictures have looked great). So I almost never delete pictures in the camera and generally it has to be completely and obviously a bad picture before I will.

  (The common causes are accidentally taken pictures or pictures where it is clear that the exposure is nowhere near where I want it.)

• when I want to pull things off the camera, I use a script to rsync the entire card to my current master directory. This happens every day, generally only once.

  (Note that when I say 'the entire card', I really mean it; the master directory is an exact image of the card's directory layout.)

  I don't reformat the card when I do this. Instead, pictures stay on the card until the nominal remaining capacity drops below somewhere in the 90 to 50 images range (I typically take around 50 pictures a day, so this gives me at least a day's margin on card space). At that point I move the current master directory to my archival area, start a new one, and immediately reformat the card. Master directories are numbered sequentially as d90-pool-NN; I'm going to have to go three digits soon.

  (This is why I have to use rsync; I need something that will not re-copy already copied images.)

  My iron rule on card reformatting is that I must have run the card sync script a second time immediately before I reformat, and it must have reported nothing synced. This is designed to avoid accidentally reformatting a card with un-transferred photos.

  Yes, this does mean that I have every photograph I've ever taken (and not immediately deleted in the camera). Disk space is cheap at the low-ish rate that I photograph.

  (I am not claiming that I have a useful archive of every photo I've taken, because it's not. But if I really want to find something, at least it hasn't been deleted so it's possible.)

• at the end of each day I use an exiftool-based script to copy all of the day's photographs to a temporary staging directory. Usually this happens at the same time as I'm pulling them off the camera.

  (This is also the point where I pop the camera battery out and drop it in the battery charger. Also, I clear the staging directory of the previous day's pictures before running the script. This is not scripted, because I don't script things that delete pictures.)

• I use Bibble 5 on the staging directory in a multi-pass approach to decide on my selects and completely process them (all in the staging directory). At the end of this I have Bibble 5 write the 'developed' JPEGs to a subdirectory and I go through them with xli to make sure that I'm happy with them; if not, I process them some more until I am.

  The actual details of how I work in Bibble 5 are far too long (and variable) to go into this entry, which is already long enough.

• I use a script to copy all of the bits of the final selects from the staging area to my Flickr archive area (which lives inside my general photo archive area). This is broadly organized by day (and by month and year once each is finished and I archive it). By 'all of the bits' I mean the original raw file from the camera, the Bibble data file about the edits I did, and the final generated JPEG.

  (The script picks out what to copy based on what pictures have generated JPEGs.)

  If I had to use chromatic aberration correction, I use the GIMP to trim off the last few pixels on the sides of the picture if they need it, because the current version of Bibble 5 corrupts the very edge of the picture in this case. (If I have cropped an edge in it doesn't need this, so I don't just hit every CA-processed image with an ImageMagick script or the like.)

  (In theory I could crop the image by those few pixels in Bibble 5. In practice, Bibble 5 on Linux is currently unusably slow when cropping in magnified view. So I get to use the GIMP.)

• I upload the JPEGs to my Flickr using Flickr's basic uploader page in Firefox. After this finishes, I blank out the default filename-based titles that Flickr has given the pictures and add tags for appropriate things if they're missing.

  (Then I agonize over what to chose as my Project 365 photo, except on days when the choice is completely obvious or I only had one thing that was worth uploading to start with.)

On some days, I'm selecting images for more than just my Flickr uploads; the most common case is that I am also selecting for TBN's website. In these cases I generally repeat the last three stages for each separate reason, sometimes entirely independently and sometimes interleaved (where as I look at each image, I decide both if it's good for Flickr and if it's good for TBN).

(Note that I have two completely separate photo archive areas, one for the master directories, and one general photo archive area for all of the pictures that I've selected for various things. The second area has subdirectories for the thing, like flickr and tbn and family, and then generally date-based within each reason. If I had a higher volume of pictures, I would probably want to be more organized and consistent about my directory structures.)

As a Linux user, my strong impression is that Bibble 5 is about my only good choice for processing anything more than a few photographs in raw format. There are some free programs that will process individual raw format pictures, generally not really very well or fast, but I haven't found one that does a decent and acceptably fast job at browsing through them so I can make my selects.

(At this point I am nowhere near willing to either give up Linux or to get a second computer just to do photo processing.)

I could simplify a bunch of this workflow if I could bring myself to trust Bibble 5's 'catalog' asset management features. I would probably use multiple catalogs, with one for my master archive and then one for each reason I pick out photographs (a Flickr one, a TBN one, etc), and switch to formatting the card every time I copied the pictures off it (even though this makes me nervous; leaving the pictures on the card is vaguely reassuring just in case something disastrous happens on the computer). However, this would mean giving up the principle that nothing except my own scripts gets to go anywhere near the master archives.

(I'm a sysadmin. No, I don't trust your program.)

With a more complicated copying scheme I could change my master archives over to a date-based directory structure while still not reformatting cards immediately. I would have to rsync to a staging area, then hard-link the files into their final destinations (chosen based on their EXIF dates); anything that was already hardlinked wouldn't need to be looked at a second time, which would make it reasonably efficient.

Sidebar: looking back at the history of this

A lot of the dance around my master directories is because when I started out, I was planning to burn each master directory to DVD when it was 'done' as a backup archive; this is also why I got a 4GB SD card, because it went well with wanting roughly DVD-sized chunks of work. I never actually implemented this plan; my backups are instead just rsync'd to an external USB drive every so often.

(Don't panic, my machine has mirrored drives to start with.)

It's interesting and a bit depressing to see how pervasively this never implemented backup plan has shaped the rest of my workflow.

Back when I was using Bibble 4, my theoretical workflow was to use the staging area only to make my selects, then have Bibble 4 copy the selects to the per-day P365 archive area, re-point Bibble 4 to it, and process them there. This never entirely worked; every so often I would have to do most of the processing before deciding whether something was a select or not, and every so often I would get pulled into processing an image before pausing to copy it.

When the Bibble 5 beta came out, it forced my hand by not supporting directory based file copying (it could only copy files around inside its asset-management catalogs). If I was copying the files outside of Bibble 5 anyways, it was much easier to do all of the processing in one directory instead of theoretically splitting it across two separate ones.

The Nikon DSLR trick with Auto ISO and Manual mode

Nikon DSLRs have a reasonably smart automatic ISO mode, where you set your minimum shutter speed and maximum ISO and when the camera has hit the minimum shutter speed it starts raising the ISO. They are also famous (or infamous in some quarters) for not turning off Auto ISO if you go into Manual mode, contrary to what you might expect.

(What happens in this semi-Manual mode is that the camera works out its idea of the correct exposure and then attempts to get there purely by changing the ISO.)

I actually sort of like this, because it enables a trick: it essentially turns Manual mode into a combined Aperture+Shutter priority mode, and in turn what this does is give you a convenient way to vary auto ISO's minimum shutter speed as conditions change:

• if I am shooting braced or with better support than expected, I can switch to M and drop the shutter speed down to lower the ISO.
• if I switch from one end of a zoom to the other I can either drop or raise the shutter speed as necessary (depending on how I set my minimum shutter speed).
• if I am suddenly taking pictures of action or something else where I want a fast shutter speed, I can increase the shutter speed without moving from my preferred (or necessary) aperture.

(Life would be somewhat simpler if Auto ISO also let us pick a minimum aperture; even though I can shoot a 50mm f/1.8 wide open, I often don't want to and I'd rather raise the ISO a bit and be at, say, f/2.8.)

Using Manual mode this way means that you really want to be able to control exposure compensation, and in turn this probably makes this trick unusable on bodies with only a single control wheel (where you lose access to exposure compensation in M mode).

The one thing that I really have to remember when doing this is to pay attention to the ISO and to the exposure meter, because the camera can overexpose if you push it. Generally if I'm doing this I want the ISO to always be above base ISO; the ISO going to base ISO when I'm at a comfortable shutter speed is a sign that I should switch to another exposure mode, because M mode probably isn't getting me anything useful.

The problem with taking pictures of people bicycling

(Well, one of them.)

There are three shutter speeds for taking pictures of moving bicycles:

• too slow: the bicyclist and bicycle are motion blurred (useful only for artistic effect).

• too fast: everything is frozen motionless and the bicyclist looks like they are doing an unnatural track stand.

• just right: the rider and bike are sharp but there is still visible motion blur in the wheels (and perhaps the rider's feet), so the bike still looks like it is actually in motion.

The 'just right' shutter speed is a narrow zone and varies quite significantly depending on how fast the rider is going. (And it goes quite high; I believe I've seen visible motion blur in shots at 1/250th, and the bicyclist wasn't going particularly fast.)

So far my best results have come from cheating, in the form of panning with the bicyclist at 'lower' shutter speeds (lower being relative here). But this has its own problems; it's okay for shots of just the bicyclist, but it's not good for 'rider in context' shots, since the context is blurred.

(And you have to carefully match your panning speed to the bicyclist's speed in order to keep them sharp, which I am not yet all that good at.)

Where to find my Flickr photostream

Someday I will probably put together a 'stalker's guide to Chris' page to have all of this sort of information in one spot, but in the mean time here is my Flickr photostream. My major use of it is for Project 365, which I am on my second year of.

Disclaimer: contents contain bicycles.

(It has no snazzy name-based URL because, as I may have mentioned, I am bad at coming up with names and Flickr doesn't let you change your mind so your first choice had better be the right one. It's much easier not to choose than to choose badly.)