Photo Storage, Backup, and Workflow

Diving photographers and photographers in general take a lot of photos and videos and have different backup and long-term storage needs than the average person. This is especially true when you start recording 4K video.

I have lots of friends that have tragically lost photos over time, it’s usually because of the following reasons:

  • Laptop died.
  • External USB drive was dropped and doesn’t work anymore.
  • Ran out of space and had to delete files.
  • Accidentally deleted files.
  • Phone dropped in the ocean.

I’m also amazed how many professional and semi-professional photographers simply do not take care of their photos and have accidents where they lose years of their digital life.

But I’m an IT nerd and my life revolves around saving data and keeping computers from ruining my life (I know, it’s a cynical view of things).

So here is how I manage files, backups, and disks….

Equipment

Macbook Air mid-2015, 500GB SSD. The laptop that goes everywhere with me.

2TB Samsung T5 solid-state USB external drive. Portable backup, it also goes everywhere with me.

Synology DS 1815+ 8-bay Network Attached Storage (NAS) and 8x8TB drives (NAS-specific Seagate Ironwolf drives) in a Synology Hybrid RAID with 2 redundant drives, and formatted for BTRFS. The big beast in my spare bedroom with 42TB of useable storage. I realize this is a huge setup for most of you, but a nice 2-drive with SHR/RAID 1 or a 4-drive unit with SHR/RAID 5 works swimmingly for the average home user.

1 external 8TB USB drive for backing up the NAS.

USB hub with USB gigabyte ethernet adapter for Macbook, Cat 5e/6 ethernet cables, and a couple gigabit switches. I do have a 24-port managed gigabit switch for the NAS so that it can bond 4 ethernet ports into one super-fast network port.

Custom-built monster Linux desktop with M2 drive for root and 2x8TB hard drives in RAID1 for /home.

Second Macbook Air for my wife.

Considerations

All my cameras use SD cards. I have a reader built into the 2 Macbooks. I have a USB reader for my desktop. At some point in the future I’ll upgrade Macbook and have to figure out how to play this game with USB-C connectors.

I need a travel backup. SSD are the only USB/external drives that I have seen survive a busy travel schedule. Conventional spinning hard drives don’t take abuse and they’re heavy. I’ve killed even the “durable” hard drives.  I used to use a Samsung T3 250GB SSD but ran out of room because my Macbook has 500GB of storage. So I upgraded to the T5 2TB.

Time Machine Settings

No recent backup to NAS because I’ve been on the road since August 9th. I turned off “Back up Automatically” to make Time Machine Editor work.

MacOS uses a low priority for Time Machine backups so that it doesn’t slow down programs that are you are using. With photos and especially video, it takes a long time to make a backup. You can set the priority for Time Machine so that it works faster but possibly makes your computer seem slower and use an application to schedule backups for nighttime. This way, you get fast backups but only when you’re not using your Mac. Obviously, then you need to keep your Mac on overnight. =)

Time Machine Editor

Time Machine Editor Configuration. Only run at night when I’n not using my Mac.

At home, I need to share files across several computers. Network drives help with this. As a side benefit, it takes data off of my laptop or desktop hard drive and puts it somewhere more durable.

Synology NAS supports something called Synology Hybrid RAID (SHR). This allows you to upgrade the size of your storage by replacing the hard drives with larger drives (also as drives get cheaper over time). However, you have to do this one at a time and then allow the data to be synced to the new drive. For 8 drives, this could take a couple of weeks to fully replace all the drives. My NAS can also accept an expander unit of 5 drives.

Time Machine can backup to a NAS very easily. However, it’s slow over wifi. A USB to gigabyte ethernet dongle helps you a ton, and I have one of these at home permanently connected to a USB hub for my laptop.

You need some kind of naming scheme to let you find photos. I use “YYYY.MM Description” as folder names, so “2019.04 Malapascua” and “2019.07 Folly Cove” exist on my NAS.

My NAS has the Hyper Backup software that can backup to USB or another NAS. Think of it as Time Machine for NAS. I need this because when I transfer photos to NAS, the NAS becomes the single backup for everything. With the drive redundancy, I’m mostly OK, I’m making backups to protect against user error (oops, I deleted everything) and any kind of home disaster like fire, flood, etc. When I had 2 homes, I had a second NAS and would backup from NAS to NAS over a home-to-home (H2H) VPN. I like this approach but I only have one home nowadays. I looked at Synology C2 or Amazon Glacier as potentials for backing up my NAS and just set up Synology C2 while I was writing this.

My NAS also backs up photos and video from my phone and makes them available to any kind of laptop/desktop on the local network. It also makes photos on the NAS available to my phone via DS Photo software, so if I’m on the road and need to retrieve something from NAS I can.

Accessing NAS files via DS Photo.

I backup my Linux desktop and personal webserver using Synology Active Backup.

Workflow

Shoot and Transfer. If I’m traveling, I take the SD card out of my camera, insert it into the Macbook’s reader, and transfer it into a new folder named “YYYY.MM Description” under “Pictures”. Yes, I’m gangster and put videos in this directory too even though I have a “Movies” directory. If it’s a long trip or if I have different cameras to store images from (for a long trip I might have my G9, TG5, my wife’s TG4, a Paralenz, a GoPro, a drone, and even a 360° camera), I’ll also use subdirectories with “DD Camera Name”, so something like “2019.04 Malapascua/08 G9”.

Backup! Immediately after I transfer photos to my laptop, I plug in my Samsung T5 and do a backup with Time Machine. I won’t format my SD card until this backup is complete because until then I would have photos on only one device: my laptop hard drive.

Speaking of formatting SD cards, I do a full format before every dive to avoid any kind of filesystem corruption. But of course, after that first backup.

I will edit photos on my laptop during the trip. I have recently switched from Lightroom to darktable because it works on Mac and Linux. More importantly, it writes changes to a file called a “sidecar” in the same directory as the original so that my changes stay with the file across computers. You just have to remember to also transfer across the .xmp files when you transfer photos.

I plug in my T5 every other night on the road or so and let Time Machine backup to it while I sleep. That way my edits and other things on the laptop get backed up too. Night backups while you sleep is an awesome thing.

When I get home, I connect my laptop to the USB hub with ethernet adapter. I remove all of my pictures from darktable. Don’t worry, the changes are still on hard drive in the sidecar file. Then I copy the directory with all of my trip pictures and videos to a “photos” directory on the NAS. I transfer the entire directory so that my “photos” directory has some kind of order inside of it. I check that all the files were transferred to NAS by counting the number and loading the last handful of files to make sure they can be accessed via NAS.

photos drive

My nicely-organized photos directory.

I plug my T5 drive into the laptop and leave it there while my laptop is at home. That way, the laptop backs up to both the NAS and the T5 at night. I think Time Machine backs up first to the drive with the oldest backup, so overnight with both available it alternates backup drives.

If I dont need to use my laptop right away, I run Time Machine as soon as I transfer files to NAS to backup to either NAS or T5. It’s better if I let the laptop run overnight so that it does a backup to both locations. After I have verified that I have backups to both locations (usually the next day), I can safely delete the photos and videos from my laptop hard drive.

I can access photos and videos on the NAS from any computer as a network drive. On my Macbook, I mount the directory as a Windows share. On my Linux desktop, I mount via NFS with cache. In darktable, you can create a local copy of files that you are working on and then the local copy will be deleted when you remove the file from darktable. This saves you a lot of headache where NAS connections time out.

For using Lightroom on a network drive, you have 2 options:

  • Load the files into your Lightroom catalog when they’re on the laptop. That way, it’s faster reading off the local drive. Then you can do one of two things: #1 use Lightroom to transfer the directory to NAS; #2 transfer the files to NAS then have Lightroom “find lost files” on the NAS.
  • Transfer the files to NAS and then load them into the Lightroom catalog via the network share. This is slower but then you don’t have to worry about getting Lightroom to find them on NAS later.

I have a USB drive connected to the NAS to make a backup of the unique files that live on the NAS. Ie, photos and videos. I don’t make a backup of backups on the NAS like in the Time Machine directory… I already have 2 backups plus the original. One huge tip on Hyperbackup is that you need to unclick the “sleep USB drive after backup is complete” option because otherwise you won’t be able to make recurring backups to the USB unless you unplug/replug it.

20190826_1333446412836257628397925.jpg

USB drives can get diver gear tags, too. =) =) =)

See you Underwater!!

–Mike