Photoshop, Paint Shop Pro, Gimp), while others are made by automated scripts (IrfanView, ImageMagick, jhead...). Most of the command line scripts I describe below are for Linux but also for Windows with the appropriate tools (Cygwin).
Text and pictures © 2006-2024 Guillaume Dargaud
Last updated on 2021/11/05
"The illiterate of the future will be the person ignorant of the use of the camera as well as the pen." — Anonymous.
Images are taken by pressing a trigger on a camera and then we tend to think it's done. But it's actually where the main work starts. It used to be simple: take a first look at the slides and make 3 piles: the straight to garbage pile, the archive (back in the box, never to be seen again), and those to put in trays for display. Digital photography changed all that.
In the way I want to work with digital images, I've decided on one basic absolute rule: "information pertaining to the images are saved in the images", meaning the 'what, who, where' must be in some extra field of the file and not in the filesystem or in some obscure database program. The idea being that if you move or rename the files or change management system, the information stays with the corresponding image. The problem is to ensure that this information is readily accessible to the user and maintained by programs transforming, using or displaying the images.
To further complicate things, I have images from several main sources, what I will call here the 'source image', which gets corrected for defects and saved into an 'archive image' and then often deleted. This archive image gets transformed into a half resolution jpeg version for quick distribution, a medium jpeg for web viewing and a tiny thumbnail version for web catalogs. Some of those transformations are made by myself and a graphic program (such as Photoshop, Paint Shop Pro, Gimp), while others are made by automated scripts (IrfanView, ImageMagick, jhead...). Most of the command line scripts I describe below are for Linux but also for Windows with the appropriate tools (Cygwin).
Let's start with understanding the basic file formats used in the archival and distribution of digital images. Only about 4 are useful for the average photographer out of the hundreds that exist.
| File type | PS, PSP | TIF | PNG | RAW, DNG, RAF | JPEG |
|---|---|---|---|---|---|
| Origin | Graphic programs | Scanners | Graphic programs | Digital cameras | Digital cameras |
| Main use | Archival, intermediate work copy | Archival, professional printing, intermediate | Archival | Digital negative, archival | Web display, distribution |
| 16 bit support | Yes | Yes | Yes, but unsupported by most programs | Yes | No |
| Compression | Lossless [*] | Lossy (much smaller size), some programs allow for lossless operations | |||
| EXIF [*] | Yes, but only from the creating program | Yes, but support is lacking in most program | Yes, but only from the creating program | Yes | |
| Typical image manipulation programs | Photoshop, Paint Shop Pro, Gimp... | Camera specific program, Adobe Lightroom | IrfanView, Web browser | ||
| Embedded ICC profiles | Yes | Yes, but unsupported by most programs | Yes | Yes, but unsupported by most browsers. | |
| Overall compatibility | Poor | Good, but some progs accept only limited compression methods out of the optional LZW/ |
Good, but frequent lack of support for some options (16 bits, ICC profiles, EXIF, Comments...) | Poor (those are new formats, still in progress and heavily dependent on hardware) | Good, but frequent lack of support for some options (EXIF, ICC profiles) |
| Archival Use | Yes, but produces very large files and only the same prog can open it | Yes | Somewhat (lack of support for ICC and EXIF inhibits this) | Yes, but needs the exact same program to extract a consistent image | Not advised |
| Web Use | No | Possible, but overkill | No | Yes | |
So, seen from the files themselves, a simplified workflow can be: Scanner —> 16-bit TIF temporary file —> Photoshop —> 8-bit TIF archive —> Conversion script —> Large, medium and thumbnail JPEG files.
Another solution is: Digital camera —> source DNG file —> Adobe Lightroom —> 8-bit TIF archive —> Conversion script —> Large, medium and thumbnail JPEG files.
See below for three detailed workflow examples.
There are many other similar programs. This is only a list of things I use daily or that I've used in the past.
Purchase Photoshop here.Purchase Paint Shop Pro here.Manual here.| From scanner | From digital negative (DNG) | From digital camera (JPEG) | |
|---|---|---|---|
| Preliminary stage | Scan the film after careful dust-off, saving as 16-bit TIF files. During the scan pay special attention to white and black point as well as color density and contrast. Perform batch scans only if all images are very similar. | Copy images from camera. Run them through LightRoom (orientation, color correction, histogram...) and save the results as TIF files. |
Copy images from camera. Rotate images losslessly using IrfanView in Thumbnail view. Delete everything that sucks (using the basic thumbnail view from your file manager or running quickly through the heap with IrfanView and the [Delete] key) |
| Advanced high density step (optional) | If the film shows very high contrast, perform several scans with pushed [Input][Lock Exposure][RGB Exposure] in ViewScan. Merge them using Photomatix, save as 16-bit TIF. | If the scene shows very high contrast, you can take several shots changing the exposure settings. Make sure the camera is well fixed on a tripod or you'll need to align the images with PTgui first, using the [Create Panorama][Layers][Individual Layers Only] final file creation option. | |
| Retouch | Open image in your graphic program (PS, PSP, Gimp...), check histogram, apply filters on copied layers such as [Clarify] or [One Step Photo Fix]. Mix the layers according to taste (30% transparency, overlay, possibly delete some parts of some layers...). Merge all. Straighten image. Crop image. Convert to 8 bit per channel. Clean specks or scrapes with clone brush, makeover tool or scratch remover. Save as TIF. | None necessary if you have been careful in Lightroom. | Try to not perform anything that can affect the data bits of the image as it'll drop the quality by increasing the compression artifacts (very visible on the side of smooth areas). But sometimes straightening, crop or defect repair are necessary. In that case save under a slightly different name and keep the original. |
| Organization | Rename files manually using appropriate date and keywords (YYYY-Subject-Keywords) | Rename the TIF files automatically according to EXIF date, if it's still available in the file, otherwise manually. | Rename them automatically according to EXIF date. Add appropriate keywords to image name. |
| Information fields | Open images individually or in batches using an EXIF editor such as ExifFarm. Set appropriate information: Author, Copyright, Comment, Keywords, Location... | ||
| Archival | Move images files to appropriate folders (for instance named according to date YYYYMMDD-Theme-Keywords). Try to avoid spaces in both directory names and file names, as it complicates the writing of scripts. This is your archive copy. Back it up. In most cases you'll want to delete the preliminary files or move them away so as not to confuse the various versions. | ||
| Everyday copy | Run a script to convert from TIF to JPEG, if possible converting the color profile to sRGB, and maybe resampling the resolution if the number of pixels if above 10 million. | Just copy the files, possibly halving the resolution if the number of pixels is above 10 million. | |
| Thumbnail generation | Run a script to resample the images to a fixed size, keeping the aspect ratio, discarding ICC info. Keep EXIF info only if it is used by your server (for instance a PHP application able to add web comments from the EXIF content). | ||
| Web page generation | Either rely on the server to do all the work: Apache can generate pages with links to files, php can generate pages of thumbnails with links to the original files, Gallery2 can do much more. Or run a script to generate static pages from the available images. See script below. | ||
You will find here several scripts I use for everyday image transformation. They all run in shell scripts, meaning they will run in Linux, OSX and a Cygwin bash window under Windows. In windows you can save them in a .sh file but you cannot run them directly with a double click on the file icon. You need to open a bash window in the appropriate directory and invoke ./scriptname.sh
If you really want to run them directly with a double click from Windows Explorer, create each Script.sh and create a twin file called Script.bat containing the following three lines:
SET PATH="/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin" c:\cygwin\bin\bash Script.sh PAUSEThen double click on the .bat file instead of the .sh file.
Those scripts are highly customized for my needs, so you'll probably need to modify them for your own use, but they provide a start.
After you transfer files from your digital camera to your PC, run this script from the directory where you saved the files. Read through the comments to see what it does.
#! /bin/sh # Transfer files from camera and rename them according to the date taken. # This scripts operates only on the files found in the current directory, # so files moved under subdirectories are not touched. ############################################################################### # Optional: Find files without added keywords in their names, left from previous runs echo "Nameless files:" find -iregex ".*/[0-9]*[-_][0-9]*.jpg" ############################################################################### # Copy files from digital camera (remember to reformat the card afterwards) # Note: camera should be in 'Mass Storage' USB mode and connected # Path is different on each model of camera cp /cygdrive/d/camera/DCOM/* . ############################################################################### # Rename according to exif date/time (requires jhead.exe) # Original names are lost (but who cares) # Note that this ignores DNG/AVI/HTM files which your camera may produce as well. jhead -nf"%Y%m%d-%H%M%S" *.jpg *.JPG ############################################################################### # Create dated directories and move corresponding files to them # Ignores DNG/AVI/HTM files ls -1 *.jpg | sed -e "s%\(.*\)-.*%mkdir -pv \1; mv -uv \0 \1/\0%" >/tmp/$$ . /tmp/$$ ############################################################################### # Optional: Convert file extensions to lowercase #find . -name \*.AVI -o -name \*.DNG -o -name \*.JPG | sed -e "s/\(.*\)\.AVI/mv -i \"\0\" \"\1.avi_\"/" -e "s/\(.*\)\.DNG/mv -i \"\0\" \"\1.dng_\"/" -e "s/\(.*\)\.JPG/mv -i \"\0\" \"\1.jpg_\"/" >/tmp/$$.sh #. /tmp/$$.sh #find . -name \*.avi_ -o -name \*.dng_ -o -name \*.jpg_ | sed -e "s/\(.*\)\.avi_/mv -i \"\0\" \"\1.avi\"/" -e "s/\(.*\)\.dng_/mv -i \"\0\" \"\1.dng\"/" -e "s/\(.*\)\.jpg_/mv -i \"\0\" \"\1.jpg\"/" >/tmp/$$.sh #. /tmp/$$.sh
#! /bin/sh
# PrepareImages.sh
# This script needs GenerateHtml.sh in the same directory and ImageMagick on the system.
#
# Purpose: It takes large TIF/PNG images found in ./Large/ and converts them to jpg inplace
# Then it creates half resolution versions in ./Medium/ and thumbnails in ./Small/
# Then it creates simple web pages with the thumbnails and links to the medium images.
# The idea is that the current directory is served on the web.
#
# Use: under the directory where those two scripts are located, create a directory ./Large
# Copy (not move) inside all the images you want to publish (TIF, PNG, JPG...), possibly inside subdirectories.
# Run the script without parameter. You can then delete images left in ./Large/ (optional)
# Next time, you just add other images and run the script again.
# Images are never removed from ./Medium/ and ./Small/
###############################################################################
# Removes spaces found in file or directory names.
# Note: possible name collision here.
# Note: you might want to run those two lines more than once (I usually have no more than one space per filename)
echo
echo "### No spaces in file names"
find Large -name "* *" |
sed -e "s/\(.*\) \(.*\)/mv -iv \"\0\" \"\1_\2\"/" >/tmp/$$.sh
. /tmp/$$.sh
###############################################################################
# Optional: Delete intermediate pano files and other unwanted files
# You might want to customize this to your usual temp files
echo
echo "### Deleting intermediate pano files and non-graphic files"
find Large \( -regex ".*P[0-9]*_*.[jJ][pP][gG]" -o -regex ".*P[0-9]*_*.[tT][iI][fF]" -o -regex ".*[0-9][0-9][0-9][0-9].[jJ][pP][gG]" -o -regex ".*[0-9][0-9][0-9][0-9].[tT][iI][fF]" -o -iname \*.pts \) -exec rm -v {} \;
###############################################################################
# Convert tif to jpg, keeping same resolution.
# Note: possible name collision here. No overwrite if this is the case.
find Large \( -iname \*.tif -o -iname \*.png \) -exec echo echo {} \; -exec echo convert {} jpg:{}.jpg \; |
sed -e "s/\.tif\.jpg/.jpg/" -e "s/\.png\.jpg/.jpg/" > /tmp/$$
echo
echo "### Converting tif & png to jpg"
chmod -R -w Large # Avoids overwrite of already existing jpg files
. /tmp/$$ # Execute tif to jpg conversion
chmod -R u+w Large # Reinstates write permission
###############################################################################
# Delete the now transformed tif and png files
echo
echo "### Deleting tif and png files"
find Large \( -iname \*.tif -o -iname \*.png \) -exec rm -v {} \;
###############################################################################
# Copy files before conversion - Do not overwrite already existing files
# This may take a while if there are many new files
echo
echo "### Creating Medium files (reduce by 50%)"
find Medium -iname \*.jpg >/tmp/Exclude
find Large -iname \*.jpg -exec echo mkdir {} \; -exec echo echo {} \; -exec echo convert -resize 50x50% -sharpen 1 {} jpg:{} \; |
sed -e "s%echo Large%echo Medium%" -e "s%jpg:Large%jpg:Medium%" -e "s%mkdir Large/\(.*\)/.*%mkdir -p Medium/\1%" |
grep -vf /tmp/Exclude >/tmp/$$
. /tmp/$$
echo
echo "### Creating Thumbnail files (reduce to fit within 300x300 pixels)"
find Small -iname \*.jpg >/tmp/Exclude
find Large -iname \*.jpg -exec echo mkdir {} \; -exec echo echo {} \; -exec echo convert -thumbnail 300x300 -sharpen 1 {} jpg:{} \; |
sed -e "s%echo Large%echo Small%" -e "s%jpg:Large%jpg:Small%" -e "s%mkdir Large/\(.*\)/.*%mkdir -p Small/\1%" |
grep -vf /tmp/Exclude >/tmp/$$
. /tmp/$$
###############################################################################
# Convert AdobeRGB profiles to sRGB, if found
# We leave the Large images 'as is', we ignore the thumbnails (the profiles have been striped) and convert only the medium images
echo
echo "### Color profile conversion"
find ./Medium -iname \*.jpg -print -exec exif -m {} \; |
egrep -i "\.jpg|Color" |
tr "\n" "#" |
sed -e 's%#./%\n%g' |
grep "#Color Space" |
grep -v sRGB |
# This line uses ImageMagick
# sed -e "s/#Color Space.*//" -e "s%.*%convert -verbose -profile /cygdrive/c/WINDOWS/system32/spool/drivers/color/AdobeRGB1998.icc \0 -profile /cygdrive/c/WINDOWS/system32/spool/drivers/color/sRGB.icm /tmp/$$.jpg \&\& mv -f /tmp/$$.jpg \0%" > /tmp/$$
# This line uses LCMS. Both work, so use either one
sed -e "s/#Color Space.*//" -e "s%.*%jpegicc -n -q90 -i/cygdrive/c/WINDOWS/system32/spool/drivers/color/AdobeRGB1998.icc \0 /tmp/$$.jpg \&\& mv -f /tmp/$$.jpg \0%" > /tmp/$$
. /tmp/$$
###############################################################################
# Set file date to Exif date, if present
echo
echo "### Setting exif date if available"
jhead -ft **/*.jpg
###############################################################################
# Create basic html pages - Needs the GenerateHtml.sh file in the same directory
echo
echo "### Creating html files"
find Small -mindepth 2 -type d -exec GenerateHtml.sh {} \;
# You should add a basic redirect from here to ./Small/index.html
###############################################################################
# Set proper permissions
echo
echo "### Setting permissions"
chmod -R 755 Large # Needs access to avi files
chmod -R 755 Medium
chmod -R 755 Small
###############################################################################
# Look for remaining useless files
echo
echo "### Files that are not delt with:"
find Large -iname "*.psp*"
echo
echo "### Now go back to ./Large/, delete the remaining unwanted files and run this script again (it'll be much quicker the 2nd time)."
#! /bin/sh # GenerateHtml.sh # Purpose: this script is called by PrepareImage.sh to generate index.html files in every directory containing thumbnails. # The generated page is very basic and contains: # - links to possible avi files located in ./Large # - IMG thumbnails, with links pointing to the ./Medium/ version of the same image # It assumes the server is Apache and it uses some of the icons provided by a default install of Apache. cd $1 Dest=$(echo $1 | sed -e "s%Small/%../../../Medium/%") DestL=$(echo $1 | sed -e "s%Small/%../../../Large/%") echo "<HTML><HEAD><TITLE>Photos $1</TITLE><LINK REL=stylesheet HREF="http://www.gdargaud.net/StyleGD.css" TYPE="text/css"></HEAD>" >index.html echo "<BODY><H1>Photos $1</H1>" >>index.html echo "<P><A HREF='..'><IMG SRC='/icons/back.gif'> [ Up one directory... ]</A></P>" >>index.html ls -1 $DestL/*.avi | sed -e "s%.*/%%" -e "s%.*%<P><A HREF='$DestL/\0'><IMG SRC='/icons/movie.gif'> [ Video ]</A></P>%" >>index.html ls -1 *.jpg | sed -e "s%.*%<A HREF='$Dest/\0'><IMG SRC='\0' ALT='\0'></A>%" >>index.html echo "<P><A HREF="$Dest"><IMG SRC='/icons/folder.gif'> [ $Dest ]</A> just for the list of larger images.</P>" >>index.html echo "</BODY></HTML>" >>index.html
#! /bin/sh
# FindWrongProfile.sh
# Find jpg files with a color profile explicitly set to something different than sRGB
# Jpeg files on a website should be in default sRGB format. This is not crucial.
# Note that 'tr' is an optional install in cygwin
find -iname \*.jpg -print -exec exif -m {} \; |
egrep -i "\.jpg|Color" |
tr "\n" "#" |
sed -e 's%#./%\n%g' |
grep "#Color Space" |
grep -v sRGB |
sed -e "s/#Color Space//"
#! /bin/sh
# ConvertProfiles.sh
# Find all jpg files with a color profile explicitly set to something different than sRGB and converts it to sRGB, assuming the initial profile is AdobeRGB
# Uses jpegicc which is part of LCMS
# Make sure you have a backup before running this !!!
find -iname \*.jpg -print -exec exif -m {} \; |
egrep -i "\.jpg|Color" |
tr "\n" "#" |
sed -e 's%#./%\n%g' |
grep "#Color Space" |
grep -v sRGB |
# This line uses ImageMagick
# sed -e "s/#Color Space.*//" -e "s%.*%convert -verbose -profile /cygdrive/c/WINDOWS/system32/spool/drivers/color/AdobeRGB1998.icc \0 -profile /cygdrive/c/WINDOWS/system32/spool/drivers/color/sRGB.icm /tmp/$$.jpg \&\& mv -f /tmp/$$.jpg \0%" > /tmp/$$
# This line uses LCMS. Both work, so use either one
sed -e "s/#Color Space.*//" -e "s%.*%jpegicc -v -n -q90 -i/cygdrive/c/WINDOWS/system32/spool/drivers/color/AdobeRGB1998.icc \0 /tmp/$$.jpg \&\& mv -f /tmp/$$.jpg \0%" > /tmp/$$
. /tmp/$$ # executes the generated code
Let me say that again: those scripts rename, move and transform image files, so make sure you test or use them only on backup copies of your files.
