Editing large image collections at home and on the road
Many photographers use Lightroom on multiple computers, often certainly in the combination laptop / stationary computer. I currently use Lightroom regularly even on 3 computers: 2 desktop PCs at two locations and a laptop when I’m on the road or on vacation. So the question arises, how to manage the existing images in the library most effectively under these conditions. In most cases, the complete actual database is probably on the single stationary PC and the laptop is used separately on the road with its own database for new shots. This means that there is no access to the older images on the laptop while on the road.
I have now searched and found a solution that allows me to use Lightroom on all my devices with all the images in my database at full capacity.
Since I was a teenager, photography has been an intense hobby for me. In the beginning with classic film, which I also developed myself in my own color lab. Since 2003 I also photograph digitally, since 2005 exclusively. I use Lightroom for editing and organizing my pictures since version 1.0, which was released in 2007. Currently I edit my images in the Lightroom photo subscription* with Lightroom Classic and various plugins.
Furthermore, I have been in the situation for some time that I regularly have to commute between two locations due to my job. My Lightroom image database, which has grown over many years and currently contains more than 226,000 images, is now over 8TB in size. The catalog file alone amounts to about 5GB, and I still want to be able to view and edit all my images fluidly at two locations and on the road on my laptop at any time.
I’m also admittedly a computer nerd and like to digitally connect everything possible. Therefore, I have built up a networked working environment with Lightroom Classic in the photo subscription, two Windows workstations, two QNAP NAS* storages and a high-performance travel laptop when I’m on the road. Automatically running synchronization scripts regularly keep the data between the stationary NAS storage in both places of living synchronized. Back at one of my homes, I synchronize my laptop via LAN connection with one of my NAS storages as needed. This concept has worked very well and comfortably for me for years now.
No cloud solution
Therefore, some time ago I was looking for ways to have all my pictures accessible at any time and any place.
The solution of storing images in a cloud, which may seem obvious to many people, was not an option for me for two reasons:
- The sheer size of the image data available: 8TB in a cloud is not effective to handle. Correspondingly large amounts of storage are very expensive and difficult to handle even with the fastest internet connection. In addition, I always want to have access with the laptop on the road, which would be too expensive and / or too slow via cellular access or public wifi.
- In addition, photos are very personal documents that I do not want to give out of my hand, also for reasons of privacy and data protection. And my trust in the data protection of clouds is very limited.
More than 226,000 images and 8TB always available without cloud? That’s not possible!
Yes, it is!
However, in the meantime I had to set up a very complex and also expensive system for this purpose, which I would like to describe in the following. First of all, the problem in detail:
Size and speed
First of all, 8TB is a massive amount of data. Just copying this amount of data from one hard drive to another takes hours. SSD drives in these dimensions are currently still very expensive. Although there are large hard disks (currently over 20TB in size), they are individually limited in speed (usually max. 100-200MB/s), which significantly slows down image processing of the large image files of current cameras.
I now store all my image files on NAS storage in RAID 6 arrays. The data access is significantly accelerated due to the parallel storage of the files on multiple disks. I also use a RAID 1 SSD array as write-back cache in each of my QNAP 1635 AX* NAS. Since the NAS is connected via 10GBit Ethernet in my home network, I achieve transfer rates of about 500 MB/s.
Connecting my sites
As mentioned above, I regularly commute between two locations that are 200km apart. At both locations, I have a VDSL Internet connection with up to 250/40 Mbps. Both are permanently connected to each other via a Fritz!Box 7590* via the built-in VPN-connection. In addition, I have a QNAP 1635 AX* NAS running in identical configuration at each location.
After synchronizing the two QNAPs via Ethernet for the first time (which took several days), I have now set up a daily synchronization of the two via the integrated QNAP HBS 3 Hybrid Backup Sync. However, due to the VDSL connection, the speed of the synchronization is limited to the 40 MBit/s of the uplinks in both directions, which roughly corresponds to a data transfer rate of 4.3 MB/s. Thus, a current high-resolution RAW image from my Canon EOS R5 takes 10-15 seconds for the transfer. For illustration: to transfer all my images, I would need about 24 days!
Synchronization only takes 1-2 hours on average in normal daily operation. Depending on the volume of new or edited images, it can of course take longer. The two NAS compare all files daily in the morning (besides my photos also my documents and movies). New or modified files are then automatically copied to the other device.
After an edit in Lightroom, this is of course regularly the catalog file (currently 5GB for me, which alone takes about 20 minutes to transfer) as well as the new or modified image files.
I have my Lightroom Classic configured to write all changes to both the internal database and separately to XMP, JPEG, TIFF, PNG, and PSD files to provide some internal redundancy of editing steps:
My reasoning for this approach is that in the event of a defect in the large catalog file, my edits can still be reconstructed.
With the RAW files in the camera-specific formats (e.g. .ARW, .CR2 and .CR3), fortunately only the small XMP sidecar files are changed during the editing in Lightroom. These take up only a few KB in size and can therefore be transferred very quickly. Other formats such as .DNG, .JPG, .TIFF and .HEIC must unfortunately always be transferred completely, which takes correspondingly longer.
Therefore, I would very much appreciate if Adobe Lightroom Classic could offer the option to avoid changing the files in these formats as well, and instead write the changes to an associated XMP file. This would greatly reduce the amount of data for my syncs and also reduce the risk of data corruption of my original files, which would otherwise have to be rewritten over and over again.
Another side effect of my approach: I always have a complete backup of all my important data in two different locations. Even in the event of a fire or theft, my data would not be lost.
Image processing at home
At each location I use a Windows 10 workstation for image processing. Each of these has an SSD drive of at least 1TB in size with the drive letter D:, on which the Lightroom catalog file as well as the cache, the previews and the Smart Previews are located in the folder “\Lightroom”. The corresponding folder currently occupies about 400GB on each of my machines.
The image files themselves are always on the drive with the letter Y: on my Windows computers. This is redirected to the NAS share on each of my PCs (via “NET USE Y: \NASx\Photos” under Windows).
When editing the images, the Lightroom database is located on the internal SSD drive D: of the respective workstation for performance reasons. By the way, Lightroom does not allow the creation of a database on a network storage (although this would also work with some not recommendable tricks). This is because Lightroom uses a SQLite database for the catalog, which is not multiuser capable. But I don’t give up the hope that Adobe will change this in the future.
To keep my data consistent, it is therefore necessary to synchronize the corresponding directory with the NAS before and after image editing. For this purpose I use a synchronization script with the Windows program SyncBack, which automatically synchronizes all relevant files of the folder “D:\Lightroom” similar to the solution with HBS 3 on the QNAP NAS storage. I currently run this manually before and after each session with Lightroom, but it could also be run automatically each time the computer boots and shuts down.
With this process, I can easily edit all my images at any time at either of my two homes.
And on the road - 8TB images on the laptop?
There are indeed already 8TB SSDs for laptops in m2 format available, but they are still very expensive. I currently have one 500GB and one 2TB PCIe SSD installed in my small 14-inch travel laptop Lenovo Ideapad 5-14*. I assigned the drive letter D: to the 2TB drive. In addition, I put a folder “D:\Photos” on D:, which I have shared and assigned with “NET USE Y: \Laptop\Photos” to drive letter Y:.
The Lightroom database is located in the folder D:\Lightroom just like on the stationary PCs, and Lightroom itself can also find the images on the Y: drive without any problems.
In spite of the SSD’s limited size of only 2TB, however, I can still access all my images. This works marvelously thanks to the ingenious Smart Previews that Lightroom can use when needed.
On the laptop, I keep only the most recent image files in my photo directory on drive Y: for reasons of space. On the road or on vacation, I always import them from the memory card of my camera into Lightroom on a daily basis. In order to have access to the other images as well, I have previously created Smart Previews of all my images in Lightroom at home. With my 266,000 images, the associated files take up about 230GB of space in the directory “Catalog name Smart Previews.lrdata”.
Since I have copied them completely to my laptop, this allows me to open all my images in Lightroom Classic at least in preview quality and edit them with only few restrictions. The Smart Preview files in Lightroom are only resized RAW files in DNG format and have a small size of about 1MB each. They have a resolution of about 4 megapixels in 3:2 format with a length of the long image border of 2560 pixels each. Thus their resolution is sufficient for a full screen display in Full-HD on a laptop without any problems. And it is also sufficient for image presentation on the Internet and even for high-quality printing in a format up to 13x18cm.
By the way, a small box in the upper right corner of the grid view of an image indicates that you are only editing a Smart Preview of that image:
And Lightroom also indicates in the histogram in the upper-right corner of the Lightroom window (in the lower-left corner) that you’re working with a Smart Preview:
So I always have my complete Lightroom catalog and all my images with me on my small laptop as well. When I come back after a long vacation with my new photos already edited and keyworded during the trip, they can already be correctly added to my complete catalog. I can then simply synchronize the Laptop with my NAS at home and am immediately up to date there as well.
If the space on the SSD runs out on the way, it is sufficient to simply delete older RAW files from the photo library on drive Y:. The images deleted from it can nevertheless still be viewed, edited and exported in Smart Preview quality afterwards.
For the necessary synchronization laptop <-> NAS I also use a synchronization script with the Windows program SyncBack, which copies all relevant files automatically similar to the synchronization between my stationary computers and NAS storage described above.
The special feature of this script is, however, that on the one hand all newly created and changed files on the laptop are copied to the NAS. On the other hand, however, only the changed files that were previously already present on the laptop are copied from the NAS to the laptop. So I avoid that the older files, which I have intentionally deleted on the laptop for space reasons, are copied back again and again.
In this way, the number of original RAW files on the laptop remains unchanged during synchronization. If I want to post-process older files on the go, I can manually copy the relevant folder with the RAW files to the laptop beforehand.
I have been working with this system for several years now and don’t want to miss this comfort.
Pitfalls of my system
Since the synchrnisation of my datasets occurs not continuously and the SQLite database used in Lightroom is not multi-user capable, you always have to be very careful that the editing of the images between synchronizations is only done on one single computer. Otherwise data inconsistencies can occur.
You can then see this in Lightroom by the fact that the internal database and the image file have different editing statuses. To find such conflicts, you can display the metadata status in Lightroom in the grid view (key “G”):
In the above example, there is a discrepancy between the metadata status of the file and the database in 4 images. However, this is not a big problem either. The corresponding images can be selected within the library filter and then, as desired, either the editing status of the file can be written to the database (Library Module, Menu Metadata|Read Metadata from File) or that of the database can be written to the file (Library Module, Menu Metadata|Save Metadata to File).
Therefore, this is also not critical. And the most important thing: In neither case does this cause any damage to the actual image file.
Since I can naturally only be at one place at a time, this problem does not occur with my stationary computers. However, when I switch from my laptop to a desktop computer for image editing or vice versa, it is important to run an up-to-date synchronization beforehand.
I have been doing well with this process for many years. How do others solve the problem, or maybe it’s not a problem at all?
I would appreciate comments and perhaps further suggestions for optimization.
*= Affiliate Link