Category: Behind the Scenes

When faced with a problem for which the solution cannot be reached with the resources on-hand, there are two options: outsource the job or acquire the necessary resources to do it yourself. Ah . . . the makings of today’s (long) blog post!

There aren’t too many places “out there” that can scan vast quantities of slides. One company I know that does is a vendor I met a few years ago at a professional conference. I called them to see what services they could provide for scanning 200,000 slides. We talked through the project in the largest and broadest sense. They kindly agreed to investigate what such an undertaking might cost in a non-binding estimate. We sought price estimates based upon two parameters: scans measuring 3,000 and 4,000 pixels “on the long side.” When archivists talk about the number of pixels on the long side, we are addressing the quality of the scan necessary to met a certain standard. So we need to digress for a bit to explain those standards.

Both the 3,000 and 4,000 figures appear in the United States National Archives and Records Administration (NARA) report, Technical Guidelines for Digitizing Archival Masters for Electronic Access: Creation of Production Master Files—Raster Images. We turned to this document a few years ago when trying to determine the level of quality we wanted to attain for the scans we provide to researchers who request reproductions from original material in the photographic archives. So before we proceed, a sidebar is needed in order to explain two important terms in NARA’s mouthful of a title: “production masters” and “raster images.”

At the risk of over-simplification—more details on this topic can be found readily on the Web—a raster image is one that has been created by scanning an analog image from side to side, top to bottom and then representing the image as rows of dots. With computer monitors, those dots are pixels (“picture elements”) and the number of those pixels per line relates to image resolution: the more pixels that represent the image, the higher its resolution.

“Production master” is a term used by NARA to describe raster image files “used for the creation of additional derivative files for distribution and/or display via a monitor and for reproduction purposes via hardcopy output at a range of sizes using a variety of printing devices.” In short, files meant to be used for access and reproduction. The important inherent distinction this definition makes is that the files are not “preservation masters.” Preservation masters are image files serving as surrogates meant to replace the original item.

Warning: math ahead!
The NARA guidelines for 35mm recommends 4,000 pixels across the longest dimension of the negative or slide, with 3,000 pixels as its “Alternative Minimum.” A 35mm negative is approximately 1 7/16 inches long, so 4,000 pixels divided by 1.4375 inches equals 2783 pixels per inch (ppi); lowered to 3,000 pixels, the ppi drops to 2,087. So using round figures, a production master from a 35mm negative or slide should have a resolution of 2,800 ppi, but no less than 2,100 ppi. By comparison, the Nikon Coolscan, which scans at 4,000 pixels per inch, produces scans in the neighborhood of 5,750 pixels on the long dimension.

But why is 4,000 pixels the benchmark? Well, I’ll spare you a double dose of math and say that 2,800 ppi resolution produces a good quality 8 x 10-inch inkjet print. The 8 x 10-inch photographic print was the de facto standard used by publishers before the digital imaging age, and NARA designed their guidelines to meet that equivalency.

(End of side bar).

The vendor estimated that their throughput at the 3,000-pixels specification, based upon prior experience, would be 125 scans per hour. That would be 5,000 slides a week or 200,000 slides in 40 weeks. Now that is a workable time frame! The cost, however, was not workable despite having significant funding available. Their charge was very reasonable per item, but the cost still came in at six figures—not too surprising, in retrospect, when you have 200,000 items. In summary, outsourcing a slide scanning project may be better suited to collections that are sizable but not nearly with the magnitude of Morton, or to institutions that do not have a viable, in fact burgeoning, digital library operation in place such as the UNC Library.

A question emerged in these deliberations. What equipment does that vendor have on-hand that enables it to achieve that level of throughput? They utilize Kodak’s Professional HR 500 film scanners paired with Halse slide carriers. Kodak made three industrial-level scanners for the professional photo lab marketplace. The HR 500 and its successor the HR 500 Plus handle roll film up to the 120 format (2 1/4 inches wide). Their sibling, the HR Universal, handles both roll films and sheet film up to 4 x 5 inches. The Halse carrier is designed to load slides into the scanner using the common Kodak Carousel slide tray that holds 80 slides.

The catch? (Isn’t there always something?!?!) Kodak discontinued manufacture of the scanners in October 2005 because, in their words, “with the decline in film capture, Pro Labs, in general, now have sufficient high-resolution productive scanning capacity.” In other words, professional photographers who used to shoot slides and color transparencies and then have them scanned by pro labs are now shooting all digital. Photographic archives, where many of those slides and negatives have or will end up, do not constitute a large enough market to warrant their continued production.

So, now what?

Some Web searching fortunately led to an alternative source—there is a market for used professional machines and a broker who focuses on finding new homes for unwanted HR scanners. One buyer was Yale University’s ITS Media Services unit. An email to Joseph Szaszfai, manager of Photographic and Digital Imaging Services, and an ensuing phone conversation yielded very positive information with kudos such as, “There’s nothing equal to it.” Using two scanners, Yale generates 80 30MB files per hour per machine. At Yale’s production rate, 200,000 slides could be completed in 62.5 weeks. Again, impressive throughput. So impressive that we are looking to get one!

There may be no way around it: describing, in a meaningful way, my experiments with batch scanning for mass digitization does not lend itself to short blog posts. One of the audiences we hope to reach with the Morton blog is practicing archivists, especially those exploring the realm of mass digitization. If you are not an archivist, or someone interested in scanning large quantities of material, I will not blame you if you stop reading here!

The second installation of this series on scanning slides for mass digitization discussed the issues with film scanners that produce high quality scans but at unacceptably slow speeds. Using the Nikon Coolscan film scanner was one of two approaches I explored with equipment on-hand. The second method investigated, the focus of this post, was the possible use of a good quality flatbed scanner—an Epson Expression 10000XL Photo model running Silverfast Ai scanning software, which has a batch scanning mode.

Using a flatbed scanner addresses the need to scan more than five slides at a time, a serious limiting factor for the Nikon Coolscan. (To be fair, the Coolscan was not designed for volume scanning.) The hope for the flatbed scanner was that in batch mode we might be able to produce image files that would be adequate for online access and basic reproduction requests at a higher rate of throughput. The 10000XL comes with two slide holders that, in total, hold thirty slides; bypassing the holders and laying the slides directly on the flatbed expands that number to forty-eight. Note that laying slides on a scanner’s flatbed only works if the scanner has an adjustable focusing mechanism, because not all slide mounts are the same thickness and the film does not rest on the surface of the flatbed. A fixed-focus scanner, therefore, may produce out-of-focus scans.

My test sample was a varied batch of slides from the 1950s and 1960s that has two sets of handwritten numbers on the mounts and were likely from slide presentations. This group represented a variety of subject matter, lighting conditions, film emulsions, and exposure latitudes (slightly overexposed to slightly underexposed).

I tried two methods: scanning each slide individually using the Silverfast batch mode, and scanning multiple slides as one large file. The scan above is a single image file of forty-eight slides laid out on the flatbed. The scan that follows, made from a different set of slides, uses Epson’s slide holders. (The scan below has two slide openings that are unfilled. That’s because I loaded the slides in numerical order, and two slides were missing.)

Silverfast’s batch mode requires the operator to draw a selection rectangle around each image, as seen in the detailed screen shot below. Though tedious, an advantage of selecting and then scanning individual frames is that an automatic or manual tonal and/or color adjustment can be applied to each slide. A disadvantage is that an automatic or manual tonal and/or color adjustment can be applied to each slide. Does that—being both an advantage and a disadvantage— sound like doublethink? Well, the added step to gain better image quality extends the amount of time it takes to set up each slide. A straight, unadjusted scan from the preview may be faster, but it’s usually not acceptable “as is.” Some tweaking of the preview scan—”preprocessing”—is required to get the final scan close to where you want it. You just can’t labor over it!

Note in the illustration above (made from a different set of slides) the dotted rectangular selection boxes. Note, too, the odd color cast that is the result of the film emulsion’s color dyes shifting over time. This is one reason why you need to make adjustments to raw scans, even if they are very close approximations of the original slides. Here’s an adjusted version, corrected after scanning (“post-processing”), of one of the images above:

One nice aspect about individual scans, however, is that once you click on the “Scan” button you can walk away from the scanner for forty-five minutes and do something else . . . like eat a hearty lunch.

Scanning multiple slides as one image file presents several dilemmas:

At what resolution do you scan the batch? I chose an equivalent of what the Nikon Coolscan would generate—4,000 pixels per inch. That creates a very large file, around 1.25 gigabytes for an 8-bit scan, for forty-eight slides. We experimented with downsizing the file to a presentable JPEG for Web display. Adequate, but it’s another layer of work that drags out the process.

To reduce that workload, several steps could be set up as an “action” in Photoshop where the software records individual steps as a programmed script that can be applied to subsequent batches, even whole directories of image files. You just need to be meticulous to the point that the scans are made so they will look the way you want them to look afterward.

Scanning a batch of slides laid out on a flatbed as one large file produces a lot of dead, black space where the slide mounts do not allow the light through. Using the Epson slide holders creates even more dead space, only it is a mixture of black and white dead space (see illustrations above). Using Photoshop, we experimented with cutting out the images and pasting them into new Photoshop documents. That’s better looking, easier to make tonal adjustments for individual items (see the hungry pirates), and better suited to item-level descriptive indexing, but again, more production work that defeats any gains made by batch scanning. And you have to remember to flatten the file or the resultant file size is quite a bit larger.

What if a batch to be scanned could be limited to a common subject and thus be described as a whole? That would work only so long as the batch has some underlying commonality. The disparate group of slides I scanned did not lend itself to group description. In addition to the pirates at Hatteras, I did scan a batch of slides from an Azalea Festival parade as a second sample. But each grouping had a different number of items. Small groupings compared to larger ones could potentially lead to sloppy online presentation.

While I have a bunch of facts and figures related to scan times and file sizes using both methods, the biggest dilemma of all is an inherent flaw in all but the top-of-the-line flatbed scanners: the lack of edge-to-edge sharpness. In the end, quantitative statistics are moot because of that flaw.

Most scanners have a strip running through the center of the scanning area, its “sweet spot,” that produces its best quality scan. Once you move outside of that sweet spot, image quality drops off—extremely so in the corners. On the Epson 10000XL in our shop, the sweet spot seems to be in the center, falling off gradually to its worst in the corners. Scans of slides outside the zone have a soft, blended pastel look reminiscent of looking through a shower door with textured glass.

Here is a full scan, from the upper right corner of the batch of scans in the opening illustration. The scan below is a 100 percent crop that I then reduced to 440 pixels wide:

The result is perfectly satisfactory for online display at 440 pixels wide. But the devil is in the details! Here are two 100% crops, not sharpened, displayed at 440 pixels wide:


See the flat, distorted appearance of people’s faces? Note the lack of detail in the camera and tripod? If our only intended use of the scans was online display, this batch approach would suffice; for reproduction, however, they are clearly unsatisfactory.

Batting 0-for-2, it was time to plow through the Internet for a better solution. We think we’ve found one . . . the subject of the next post.

It’s been a while since I wrote the introductory post “200,000 slides” on how we might approach the digitization of Hugh Morton’s slides. There’s been a lot happening on this front, but doing that work and then the holidays have delayed my next installment on the topic. I’ll try to keep the posts short and frequent over the upcoming days.

The close of my initial post ended with a set of questions, the last of which was, “And the most challenging and most difficult question, . . . how do you scan 200,000 slides?”

Well, how do you scan 200,000 slides?

There are at least two points to explore in order to answer that question: on what machine and to what resolution? I experimented with two approaches available to us with the equipment on-hand: a dedicated film scanner and a flatbed scanner. Today’s post focuses on the film scanner.

The best quality film scanner available to us is the Nikon Coolscan 9000 ED, which produces scans at 4,000 pixels per inch. We scan at sixteen bits using the eight-pass setting to produce a high quality scan. That resolution easily produces 20×24-inch high-quality inkjet prints on our printer, an Epson Stylus Pro 7800, at 360 dots per inch. (Did you just find yourself saying, “Huh?” If so, Scantips.com is one of many Web sites that can help you!)

Each scan takes about ninety seconds, with a short interval between scans. The scanner can handle only five slides per batch, however, so here’s the disappointing math:

200,000 slides x 90 seconds = 18,000,000 seconds or 5,000 hours, which is 625 eight-hour work days—well over three years.

And that’s just raw scan time without counting the time it takes to set up each slide’s appearance using the scanner’s software, which is considerable. Simply stated, a Nikon Coolscan is not designed to work in such a high-volume environment. High-end film scanners, such as a Hasselblad Flextight, do have batch loaders that handle up to 50 slides, but the added expense of that machine buys you higher resolution—8,000 pixels per inch—not speed. With voluminous slide collections, speed is of the essence!
Link to Part 3

What would do you do with 200,000 slides?

While trying to figure this out for myself, and posterity, I keep hearing two Paul Simon lyrics running in my head:

Kodachrome
They give us those nice bright colors
They give us the greens of summers
Makes you think all the world’s a sunny day

and

Slip sliding away, slip sliding away
You know the nearer your destination
The more you’re slip sliding away

No, I’m not going to jump off a cliff, although I did do that once—and then a bridge later that afternoon—in New Zealand during my younger years. Instead I’ve taken my adventuresome spirit to the computer keyboard and scanner. I’ve already written about six pages in a strategic plan and several longish emails about my exploration of this topic—and there’s likely more words to come. And I’ve sat in front of flatbed and film scanners running tests, because we are trying to bring 21st century technology into the solution to make processing the Morton collection more efficient. But when I try to think the process through completely, the solution seems to get more difficult. On one level—a bird’s eye view—the idea is very basic; as I start to apply more details to the picture—as I drop in elevation to reveal more nuances in the landscape—the technological solution seems to slip away.

During my afternoon break a few days ago I realized that, if I could explain the issues to those of you reading the blog, I could perhaps break this complex topic into simpler parts. So that’s what I’m going to try to do over the course of several blog entries. (I’m afraid one extremely long entry would keep you from ever wanting to return!) So let’s start from the sky and work our way down to ground level.

The ultimate goal of processing these slides is to make them accessible to users, and these days that means online via the World Wide Web in addition to the physical objects. To make the slides accessible, I first needed to sort and organize them because, initially, there was “neither rhyme nor reason” to the way we found the slides. Mostly stored in their original boxes, there were also many loose slides or small batches wrapped in paper, rubber bands, or in odd boxes. And several presentations are still in Kodak Slide Carousels or even the older Cavalcade Slide Trays. Because most of the slides, by far, have a date stamped on the mount by the photographic lab that processed the film, I sorted the original slide boxes by year, one year per shelf in the stacks. Needless to say the boxes stretch from floor to ceiling—and in any other direction I could put them. With that completed, I then started sorting the loose slides by year, then film type (usually Ektachrome and Kodachrome), then by month and, within month by the red or black ink used for the date stamp. I learned I could do this surprisingly quickly, faster than trying to sort scanned images on a screen, so long as I didn’t look at the images. As soon as I pause to look at an image, the sort stalls in the water.

You may be asking two questions: why sort by film type first, and why sort to the level of specificity of red or black date stamps? Well, for the first question there are two reasons. First, whatever scanner we use will likely have different technical settings called “profiles” for different photographic emulsions. If we batch scan a set of slides that are all Ektachrome or all Kodachrome, then the post-scan tonal corrections will be more consistent and more easily regulated. The second reason has to do with reuniting the loose slides with those still in the original box—what I’ve been calling a slide’s “mother box.” If I look at the contents of a box and see that its contents are Ektachrome slides, I can go right to that shorter stack and have fewer slides to examine to see if there any slides that can be returned to their mother box. The specificity of the red and black date stamps follows this same logic: if I open a box and all the slides within it are Kodachromes stamped “May 65” in red ink, then I’ve got far fewer slides through which I’ll have to search. I can make these distinctions much more quickly during the initial sort than I can during a subsequent search.

The traditional way photographers sort slides is on a light box or a light table. Taking a magnifier called a loupe, you look at the images and make selections based upon certain criteria like proper exposure and focus, composition, and people’s eyes being open. Archivists may do the same, calling into play more criteria, such as repetition. Photographers generally work a subject from different angles and with different lenses, and bracket their exposures when they can. This means, for example, that Hugh Morton could have shot an entire roll of film just of a single flower. How many photographs of that single flower would we need to keep? How many would a researcher want to examine? Would an archivist in a history collection keep as many as an archivist in a botany collection?

If I were to look at each slide on a light box for five seconds in order to assess them, here’s the math: 200,000 x 5 = 1,000,000 seconds.
One million seconds! That’s equivalent to about 175 eight-hour work days. (I wonder how hungry or bored I’d be by lunchtime on the first day? Or how my back would feel hunched over a light box all day?) And those five seconds do not include the time to lay the slides on the light box in a readable fashion nor the time to gather them and put them back in the original box. So in round figures, it could take one year to look at all the slides—so long as I did nothing else on those days other than eat lunch.

But what if I digitally scanned each slide without selecting in advance which ones should be scanned? Could I preview and edit images faster (and spare my back) on a large computer screen? From a labor savings standpoint, would scanning everything be more cost effective than the staff time necessary to review each slide and make a decision? What if I threw away the scans I didn’t want to keep because that was faster and cheaper than evaluating and discarding the unneeded slides? Or what if we made all the scans available on the Internet and let researchers decide what was important or useful to their needs? And the most challenging and most difficult question, . . . how do you scan 200,000 slides?

My next post on the topic will explore some of these questions. Who knows? Maybe someday I’ll be able to write a song about all this!

I know you will be relieved to hear that I am no longer suffering from the processing anxiety attacks I referred to in my first post. I’m still anxious, but it’s more like continuous fretfulness than bursts of panic.
I am now in the midst of my second “pass” through the Morton negatives. The first pass involved going through the whole lot of them (estimated at 33,000 items—strips, rolls, or individual frames) and sorting roughly by subject (e.g., Grandfather Mountain, Basketball, Hang Gliding, Nature). In my second pass, I’m refining those initial sorts into smaller categories (Grandfather Mountain—Events; Grandfather Mountain—Grounds and Facilities, etc.), getting all loose material into envelopes, and describing the resulting groupings in a spreadsheet. This, I must say, is a pretty satisfying stage—I’m bringing like materials together and starting to build descriptive tools that can be used to navigate the collection. Order is beginning to emerge!
Of course, most of my excitement is squelched when I remember that I’m less than one-third of the way through the second pass, and there’s still a third and probably fourth pass to be completed. Not to mention those pesky prints, transparencies, slides and films that still await attention. Take it easy, Hull. One pass at a time . . .

These views of our work areas give some idea of what it is like to process a collection as large, varied, and disorderly as the Morton photos. Since I began working on the collection two months ago, I have had regular moments of crisis during which I become nearly paralyzed by all the challenges associated with and possible approaches to this project. How do you impose order on chaos, while respecting what few pockets of order do exist? How do you decide what to digitize, and when? How do you balance the needs and interests of the many people who will use this collection with the preservation needs of the material itself?
I usually manage to calm myself with a few simple mantras:
1) In essence, all I am doing is taking one huge pile of stuff, sorting it into smaller piles, sorting those piles into smaller piles, then sorting those piles, and on and on into infinity (OK, I exaggerate) – and, finally, describing the piles. That doesn’t sound so bad . . . right?
2) By documenting carefully and making smart use of descriptive tools, we can ultimately provide access to the collection in a variety of ways – through a traditional archival finding aid, through digital images, by subject, by date, by format, etc. – offering lots of options. Then, once the collection is available, we can listen to what actual users have to say and incorporate their suggestions.

3) I will never get bored at this job. Hugh Morton crammed more into a given month than most of us do in a lifetime, e.g., saving lighthouses, fighting air pollution, attending countless sporting and political events, hanging out with celebrities, bears, and cougars, and still finding time to enjoy the really important stuff – his family and friends. I’m lucky to have this opportunity to learn from and get to know him through the images he created.