Blog

Editing large image collections at home and on the road

Many pho­tog­ra­phers use Light­room on mul­ti­ple com­put­ers, often cer­tain­ly in the com­bi­na­tion lap­top / sta­tion­ary com­put­er. I cur­rent­ly use Light­room reg­u­lar­ly even on 3 com­put­ers: 2 desk­top PCs at two loca­tions and a lap­top when I’m on the road or on vaca­tion. So the ques­tion aris­es, how to man­age the exist­ing images in the library most effec­tive­ly under these con­di­tions. In most cas­es, the com­plete actu­al data­base is prob­a­bly on the sin­gle sta­tion­ary PC and the lap­top is used sep­a­rate­ly on the road with its own data­base for new shots. This means that there is no access to the old­er images on the lap­top while on the road.

I have now searched and found a solu­tion that allows me to use Light­room on all my devices with all the images in my data­base at full capacity.

My situation

Since I was a teenag­er, pho­tog­ra­phy has been an intense hob­by for me. In the begin­ning with clas­sic film, which I also devel­oped myself in my own col­or lab. Since 2003 I also pho­to­graph dig­i­tal­ly, since 2005 exclu­sive­ly. I use Light­room for edit­ing and orga­niz­ing my pic­tures since ver­sion 1.0, which was released in 2007. Cur­rent­ly I edit my images in the Light­room pho­to sub­scrip­tion* with Light­room Clas­sic and var­i­ous plugins.

Fur­ther­more, I have been in the sit­u­a­tion for some time that I reg­u­lar­ly have to com­mute between two loca­tions due to my job. My Light­room image data­base, which has grown over many years and cur­rent­ly con­tains more than 226,000 images, is now over 8TB in size. The cat­a­log file alone amounts to about 5GB, and I still want to be able to view and edit all my images flu­id­ly at two loca­tions and on the road on my lap­top at any time.

I’m also admit­ted­ly a com­put­er nerd and like to dig­i­tal­ly con­nect every­thing pos­si­ble. There­fore, I have built up a net­worked work­ing envi­ron­ment with Light­room Clas­sic in the pho­to sub­scrip­tion, two Win­dows work­sta­tions, two QNAP NAS* stor­ages and a high-per­for­mance trav­el lap­top when I’m on the road. Auto­mat­i­cal­ly run­ning syn­chro­niza­tion scripts reg­u­lar­ly keep the data between the sta­tion­ary NAS stor­age in both places of liv­ing syn­chro­nized. Back at one of my homes, I syn­chro­nize my lap­top via LAN con­nec­tion with one of my NAS stor­ages as need­ed. This con­cept has worked very well and com­fort­ably for me for years now.

No cloud solution

There­fore, some time ago I was look­ing for ways to have all my pic­tures acces­si­ble at any time and any place.

The solu­tion of stor­ing images in a cloud, which may seem obvi­ous to many peo­ple, was not an option for me for two reasons:

1. The sheer size of the image data avail­able: 8TB in a cloud is not effec­tive to han­dle. Cor­re­spond­ing­ly large amounts of stor­age are very expen­sive and dif­fi­cult to han­dle even with the fastest inter­net con­nec­tion. In addi­tion, I always want to have access with the lap­top on the road, which would be too expen­sive and / or too slow via cel­lu­lar access or pub­lic wifi.
   
2. In addi­tion, pho­tos are very per­son­al doc­u­ments that I do not want to give out of my hand, also for rea­sons of pri­va­cy and data pro­tec­tion. And my trust in the data pro­tec­tion of clouds is very limited.

More than 226,000 images and 8TB always avail­able with­out cloud? That’s not possible!

Yes, it is!

My approach

How­ev­er, in the mean­time I had to set up a very com­plex and also expen­sive sys­tem for this pur­pose, which I would like to describe in the fol­low­ing. First of all, the prob­lem in detail:

Size and speed

First of all, 8TB is a mas­sive amount of data. Just copy­ing this amount of data from one hard dri­ve to anoth­er takes hours. SSD dri­ves in these dimen­sions are cur­rent­ly still very expen­sive. Although there are large hard disks (cur­rent­ly over 20TB in size), they are indi­vid­u­al­ly lim­it­ed in speed (usu­al­ly max. 100-200M­B/s), which sig­nif­i­cant­ly slows down image pro­cess­ing of the large image files of cur­rent cameras.

I now store all my image files on NAS stor­age in RAID 6 arrays. The data access is sig­nif­i­cant­ly accel­er­at­ed due to the par­al­lel stor­age of the files on mul­ti­ple disks. I also use a RAID 1 SSD array as write-back cache in each of my QNAP 1635 AX* NAS. Since the NAS is con­nect­ed via 10GBit Eth­er­net in my home net­work, I achieve trans­fer rates of about 500 MB/s.

Connecting my sites

As men­tioned above, I reg­u­lar­ly com­mute between two loca­tions that are 200km apart. At both loca­tions, I have a VDSL Inter­net con­nec­tion with up to 250/40 Mbps. Both are per­ma­nent­ly con­nect­ed to each oth­er via a Fritz!Box 7590* via the built-in VPN-con­nec­tion. In addi­tion, I have a QNAP 1635 AX* NAS run­ning in iden­ti­cal con­fig­u­ra­tion at each location.

After syn­chro­niz­ing the two QNAPs via Eth­er­net for the first time (which took sev­er­al days), I have now set up a dai­ly syn­chro­niza­tion of the two via the inte­grat­ed QNAP HBS 3 Hybrid Back­up Sync. How­ev­er, due to the VDSL con­nec­tion, the speed of the syn­chro­niza­tion is lim­it­ed to the 40 MBit/s of the uplinks in both direc­tions, which rough­ly cor­re­sponds to a data trans­fer rate of 4.3 MB/s. Thus, a cur­rent high-res­o­lu­tion RAW image from my Canon EOS R5 takes 10-15 sec­onds for the trans­fer. For illus­tra­tion: to trans­fer all my images, I would need about 24 days!

Syn­chro­niza­tion only takes 1-2 hours on aver­age in nor­mal dai­ly oper­a­tion. Depend­ing on the vol­ume of new or edit­ed images, it can of course take longer. The two NAS com­pare all files dai­ly in the morn­ing (besides my pho­tos also my doc­u­ments and movies). New or mod­i­fied files are then auto­mat­i­cal­ly copied to the oth­er device.

After an edit in Light­room, this is of course reg­u­lar­ly the cat­a­log file (cur­rent­ly 5GB for me, which alone takes about 20 min­utes to trans­fer) as well as the new or mod­i­fied image files.

I have my Light­room Clas­sic con­fig­ured to write all changes to both the inter­nal data­base and sep­a­rate­ly to XMP, JPEG, TIFF, PNG, and PSD files to pro­vide some inter­nal redun­dan­cy of edit­ing steps:

My rea­son­ing for this approach is that in the event of a defect in the large cat­a­log file, my edits can still be reconstructed.

With the RAW files in the cam­era-spe­cif­ic for­mats (e.g. .ARW, .CR2 and .CR3), for­tu­nate­ly only the small XMP side­car files are changed dur­ing the edit­ing in Light­room. These take up only a few KB in size and can there­fore be trans­ferred very quick­ly. Oth­er for­mats such as .DNG, .JPG, .TIFF and .HEIC must unfor­tu­nate­ly always be trans­ferred com­plete­ly, which takes cor­re­spond­ing­ly longer.

There­fore, I would very much appre­ci­ate if Adobe Light­room Clas­sic could offer the option to avoid chang­ing the files in these for­mats as well, and instead write the changes to an asso­ci­at­ed XMP file. This would great­ly reduce the amount of data for my syncs and also reduce the risk of data cor­rup­tion of my orig­i­nal files, which would oth­er­wise have to be rewrit­ten over and over again.

Impor­tant tip: I exclude the Light­room pre­view files (*Previews.lrdata) from the syn­chro­niza­tion script, as these are always cre­at­ed dur­ing edit­ing and would unnec­es­sar­i­ly pro­long the synchronization.

Data security

Anoth­er side effect of my approach: I always have a com­plete back­up of all my impor­tant data in two dif­fer­ent loca­tions. Even in the event of a fire or theft, my data would not be lost.

Image processing at home

At each loca­tion I use a Win­dows 10 work­sta­tion for image pro­cess­ing. Each of these has an SSD dri­ve of at least 1TB in size with the dri­ve let­ter D:, on which the Light­room cat­a­log file as well as the cache, the pre­views and the Smart Pre­views are locat­ed in the fold­er “\Light­room”. The cor­re­spond­ing fold­er cur­rent­ly occu­pies about 400GB on each of my machines.

The image files them­selves are always on the dri­ve with the let­ter Y: on my Win­dows com­put­ers. This is redi­rect­ed to the NAS share on each of my PCs (via “NET USE Y: \NASx\Photos” under Windows).

When edit­ing the images, the Light­room data­base is locat­ed on the inter­nal SSD dri­ve D: of the respec­tive work­sta­tion for per­for­mance rea­sons. By the way, Light­room does not allow the cre­ation of a data­base on a net­work stor­age (although this would also work with some not rec­om­mend­able tricks). This is because Light­room uses a SQLite data­base for the cat­a­log, which is not multi­user capa­ble. But I don’t give up the hope that Adobe will change this in the future.

To keep my data con­sis­tent, it is there­fore nec­es­sary to syn­chro­nize the cor­re­spond­ing direc­to­ry with the NAS before and after image edit­ing. For this pur­pose I use a syn­chro­niza­tion script with the Win­dows pro­gram SyncBack, which auto­mat­i­cal­ly syn­chro­nizes all rel­e­vant files of the fold­er “D:\Lightroom” sim­i­lar to the solu­tion with HBS 3 on the QNAP NAS stor­age. I cur­rent­ly run this man­u­al­ly before and after each ses­sion with Light­room, but it could also be run auto­mat­i­cal­ly each time the com­put­er boots and shuts down.

With this process, I can eas­i­ly edit all my images at any time at either of my two homes.

And on the road - 8TB images on the laptop?

There are indeed already 8TB SSDs for lap­tops in m2 for­mat avail­able, but they are still very expen­sive. I cur­rent­ly have one 500GB and one 2TB PCIe SSD installed in my small 14-inch trav­el lap­top Leno­vo Idea­pad 5-14*. I assigned the dri­ve let­ter D: to the 2TB dri­ve. In addi­tion, I put a fold­er “D:\Photos” on D:, which I have shared and assigned with “NET USE Y: \Laptop\Photos” to dri­ve let­ter Y:. 

The Light­room data­base is locat­ed in the fold­er D:\Lightroom just like on the sta­tion­ary PCs, and Light­room itself can also find the images on the Y: dri­ve with­out any problems.

Smart Previews

In spite of the SSD’s lim­it­ed size of only 2TB, how­ev­er, I can still access all my images. This works mar­velous­ly thanks to the inge­nious Smart Pre­views that Light­room can use when needed.

On the lap­top, I keep only the most recent image files in my pho­to direc­to­ry on dri­ve Y: for rea­sons of space. On the road or on vaca­tion, I always import them from the mem­o­ry card of my cam­era into Light­room on a dai­ly basis. In order to have access to the oth­er images as well, I have pre­vi­ous­ly cre­at­ed Smart Pre­views of all my images in Light­room at home. With my 266,000 images, the asso­ci­at­ed files take up about 230GB of space in the direc­to­ry “Cat­a­log name Smart Previews.lrdata”.

Since I have copied them com­plete­ly to my lap­top, this allows me to open all my images in Light­room Clas­sic at least in pre­view qual­i­ty and edit them with only few restric­tions. The Smart Pre­view files in Light­room are only resized RAW files in DNG for­mat and have a small size of about 1MB each. They have a res­o­lu­tion of about 4 megapix­els in 3:2 for­mat with a length of the long image bor­der of 2560 pix­els each. Thus their res­o­lu­tion is suf­fi­cient for a full screen dis­play in Full-HD on a lap­top with­out any prob­lems. And it is also suf­fi­cient for image pre­sen­ta­tion on the Inter­net and even for high-qual­i­ty print­ing in a for­mat up to 13x18cm.

By the way, a small box in the upper right cor­ner of the grid view of an image indi­cates that you are only edit­ing a Smart Pre­view of that image:

And Light­room also indi­cates in the his­togram in the upper-right cor­ner of the Light­room win­dow (in the low­er-left cor­ner) that you’re work­ing with a Smart Preview:

So I always have my com­plete Light­room cat­a­log and all my images with me on my small lap­top as well. When I come back after a long vaca­tion with my new pho­tos already edit­ed and key­word­ed dur­ing the trip, they can already be cor­rect­ly added to my com­plete cat­a­log. I can then sim­ply syn­chro­nize the Lap­top with my NAS at home and am imme­di­ate­ly up to date there as well.

If the space on the SSD runs out on the way, it is suf­fi­cient to sim­ply delete old­er RAW files from the pho­to library on dri­ve Y:. The images delet­ed from it can nev­er­the­less still be viewed, edit­ed and export­ed in Smart Pre­view qual­i­ty afterwards.

Laptop synchronization

For the nec­es­sary syn­chro­niza­tion lap­top <-> NAS I also use a syn­chro­niza­tion script with the Win­dows pro­gram SyncBack, which copies all rel­e­vant files auto­mat­i­cal­ly sim­i­lar to the syn­chro­niza­tion between my sta­tion­ary com­put­ers and NAS stor­age described above.

The spe­cial fea­ture of this script is, how­ev­er, that on the one hand all new­ly cre­at­ed and changed files on the lap­top are copied to the NAS. On the oth­er hand, how­ev­er, only the changed files that were pre­vi­ous­ly already present on the lap­top are copied from the NAS to the lap­top. So I avoid that the old­er files, which I have inten­tion­al­ly delet­ed on the lap­top for space rea­sons, are copied back again and again.

In this way, the num­ber of orig­i­nal RAW files on the lap­top remains unchanged dur­ing syn­chro­niza­tion. If I want to post-process old­er files on the go, I can man­u­al­ly copy the rel­e­vant fold­er with the RAW files to the lap­top beforehand.

I have been work­ing with this sys­tem for sev­er­al years now and don’t want to miss this comfort.

Pitfalls of my system

Since the syn­chrni­sa­tion of my datasets occurs not con­tin­u­ous­ly and the SQLite data­base used in Light­room is not mul­ti-user capa­ble, you always have to be very care­ful that the edit­ing of the images between syn­chro­niza­tions is only done on one sin­gle com­put­er. Oth­er­wise data incon­sis­ten­cies can occur.

You can then see this in Light­room by the fact that the inter­nal data­base and the image file have dif­fer­ent edit­ing sta­tus­es. To find such con­flicts, you can dis­play the meta­da­ta sta­tus in Light­room in the grid view (key “G”):

In the above exam­ple, there is a dis­crep­an­cy between the meta­da­ta sta­tus of the file and the data­base in 4 images. How­ev­er, this is not a big prob­lem either. The cor­re­spond­ing images can be select­ed with­in the library fil­ter and then, as desired, either the edit­ing sta­tus of the file can be writ­ten to the data­base (Library Mod­ule, Menu Metadata|Read Meta­da­ta from File) or that of the data­base can be writ­ten to the file (Library Mod­ule, Menu Metadata|Save Meta­da­ta to File).

There­fore, this is also not crit­i­cal. And the most impor­tant thing: In nei­ther case does this cause any dam­age to the actu­al image file.

Since I can nat­u­ral­ly only be at one place at a time, this prob­lem does not occur with my sta­tion­ary com­put­ers. How­ev­er, when I switch from my lap­top to a desk­top com­put­er for image edit­ing or vice ver­sa, it is impor­tant to run an up-to-date syn­chro­niza­tion beforehand.

I have been doing well with this process for many years. How do oth­ers solve the prob­lem, or maybe it’s not a prob­lem at all?

I would appre­ci­ate com­ments and per­haps fur­ther sug­ges­tions for optimization.

*= Affil­i­ate Link