nikonians

Even though we ARE Nikon lovers,we are NOT affiliated with Nikon Corp. in any way.

| |

Sign up Login
Home Forums Articles Galleries News Workshops Shop Recommended
members
All members Wiki Contests Vouchers Apps Newsletter THE NIKONIAN™ Magazines Podcasts Fundraising
Go to a  "printer friendly" view of this message which allow an easy print Print Go to the page which allows you to send this topic link and a message to a friend Email this topic to a friend
Forums Lobby MASTER YOUR TOOLS Post-Processing & Workflow (Public) Nikon & Nikonians Software (Public) topic #9083
View in linear mode

Subject: "RE: Fork in the Road" Previous topic | Next topic
nrothschild Neil is an expert in several areas, including camera support   US  Registered since 25th Jul 2004 Thu 16-Aug-12 08:48 PM
10916 posts Share on Facebook Share on Twitter Share on Linkedin    Click to send email to this author Click to send private message to this authorClick to view this author's profile
#10. "RE: Fork in the Road"
In response to In response to 9



Hi Eric,

I probably didn't even scratch the surface of what you are going to do . If you need any help or advice, let me know.

My sense, having followed the iMatch forums and poked around in other catalog app support forums, is that apparently there aren't that many people that attempt to (or need to) catalog 100K+ images. Most people seem to be managing more like 20,000 or so.

I often question what I'm doing (like every time I do a big ingest into iMatch), but I'm in so deep it would take man years to actually make all those culling decisions and in the best case I don't think I could get down below 100K.

Anyway, it is difficult to find good practical advice on how to manage large image catalogs and a lot of what you may read may be written by those in the 20,000 league. Keep that in mind.

Depending on your species spread (birding) and some other things, you could easily have a thousand hours of work, all told, cataloging your images. Minimum several hundred I think.

It would take between 50 and 150 HOURS of machine time to ingest 100,000 raw and tif files (JPGs are faster) into iMatch on my old dual core laptop. I haven't done any ingest performance tests on my new CoreI7 laptop (note to self!).

During that time iMatch is well behaved and other things can be done on the machine but iMatch doesn't allow anything else to be done on it during that time.

You would likely not do it all at once of course, but no matter how you slice it up its a lot of machine time. Enough machine time that my ideal catalog app would be multi-user, even me by myself, the other user being batch intensive processes. I could keep a batch user busy for a couple of months, 24/7, just running certain iMatch scripts that are very machine intensive. Anything you have to do 100,000 times can be problematic.

I think LR is quite a bit faster in that regard. I recently eval'd LR 4.1 and was impressed with the ingest performance. It is harder to benchmark LR because it does most of the work in a background process but I monitored the CPU usage to try to figure that out. I didn't take my LR database past about 20K images before I ran out of time. Actually some problems I saw (in the app, not the performance) made me go back to iMatch and spend time perfecting my versioning scripts, strangely enough

My suggestion out of all this is to make an expendable copy of your complete image collection, preferably on a dedicated hard drive, BEFORE you start downloading and installing evals. I assume it is in the 1TB range, or perhaps much higher if you have a lot of TIFs and PSD's.

You do NOT want to test a cataloging app against your "live" image repository because in many or most cases the app can or will be updating at least any XMP files you have, and if you use PM you probably have those sidecars. At the time I last eval'd LR I didn't have 3TB to spare for that.

Unless you are the kind of guy that spends a couple evenings reading manuals before trying a new app, you might be updating those files and not even know it!

I would then try to import as much of the collection as you can in the eval time you have, watching to see how the general performance holds up. That includes the ingest itself as well as basic queries over the catalog.

In most or all cases the performance is related to the average image file size (in addition to hardware issues, including any networking required). PSD's are really tough, then NEFs, and JPGs, especially web size, the easiest. That is why you need to do this yourself rather than relying on anecdotal reports.

In your case you want to look carefully at your D800 raw files, and surely the PSD and TIFs that come out of that. Having spilt much blood over this, starting with the D2h (which was actually the worst, oddly enough), the idea of wildlife volumes of D800 files scares me . (seriously).

The D800 is Parkinson's Law at work. Actually, it's Parkinson's run amok!!!!

You should also think about an exit strategy before you get into this, at least in a general way. It is impossible to think through an exit strategy without knowing where the exit leads but you only want to do this once.

Mainly, any extensive time you put into cataloging needs some sort of interface to get it back out if things don't work. Otherwise you are buried deep with your catalog app vendor, deeper than you might want to be.

This is very different than our other apps although the LR/CNX2 decision was the crucial one that few of us thought about up front (assuming we had a lot of options at the time). In that regard LR is the more problematic decision, in a certain sense, because there are more solutions that can deal with a Nikon edited NEF than the one and only one solution that will ever deal with 100,000+ LR XMP edit files. This is another thing that gave me pause for switching to LR but they are all evil in that way.

I call LR "The Grand Seduction"

Such a simple solution to everything but I don't think there is any way out so you really have to love Adobe to get into serious volumes with that.

If you like PM as much as I do (I just love it!) you then need to think about how you will browse your catalog and edit and output working images long after the main shoot/edit sessions. This is where my iMatch workflow has always fallen apart.

iMatch has some good solutions for that, and in principle I should not use PM except for pre-catalog initial culling and rating work. But iMatch does not support soft crops and that alone always tempts me back to PM when I am doing more than a solitary image here or there. PM is just so easy and intuitive. You know all about that.

I solved part of the problem with my versioning scripts- how to get those new derivative files (created outside the cataloger) cataloged without slogging through them all by hand.

People solve that different ways. Some people only keyword their originals, for example, but I find that original un-cropped wildlife images are not useful alone. In most cases the bird part is just too small (I shoot mostly small songbirds now). So I want all the images keyworded and rated. It is my only viable path to sanity.

The part I never solved is how to crop (within iMatch) and as a wildlife shooter virtually everything gets seriously cropped. On the other hand, once the images are in a catalog it is very tricky, if not impossible, to go back to something like PhotoMechanic because that is "cheating on the cataloger", which really demands to be in control of all that.

That last part is the most critical part and the hardest to figure out up front, and easy to ignore until you are buried pretty deep in your cataloging effort. Don't ask me how I know .

And I did not find a solution to that in LR either. In fact, I found far less flexibility simply because I could not script my way out of the problem, as I am doing with iMatch.

_________________________________
Neil


my Nikonians gallery.

  

Alert Print | Reply | Quote

This is a hot, active topic!Fork in the Road [View all] , dohearne , Thu 02-Aug-12 12:15 AM
 
Subject Author Message Date ID
Reply message RE: Fork in the Road macfish
02nd Aug 2012
1
Reply message RE: Fork in the Road ericbowles Moderator Awarded for his in-depth knowledge and high level skills in various areas, especially Landscape and Wildlife Photoghraphy Writer Ribbon awarded for for his article contributions to the community
03rd Aug 2012
2
Reply message RE: Fork in the Road dohearne
05th Aug 2012
3
Reply message RE: Fork in the Road ericbowles Moderator Awarded for his in-depth knowledge and high level skills in various areas, especially Landscape and Wildlife Photoghraphy Writer Ribbon awarded for for his article contributions to the community
05th Aug 2012
4
Reply message RE: Fork in the Road dohearne
07th Aug 2012
7
Reply message RE: Fork in the Road jacsr Silver Member
06th Aug 2012
5
     Reply message RE: Fork in the Road dohearne
07th Aug 2012
6
Reply message RE: Fork in the Road nrothschild Neil is an expert in several areas, including camera support
16th Aug 2012
8
     Reply message RE: Fork in the Road ericbowles Moderator Awarded for his in-depth knowledge and high level skills in various areas, especially Landscape and Wildlife Photoghraphy Writer Ribbon awarded for for his article contributions to the community
16th Aug 2012
9
          Reply message RE: Fork in the Road nrothschild Neil is an expert in several areas, including camera support
16th Aug 2012
10
               Reply message RE: Fork in the Road PAStime Silver Member
19th Aug 2012
15
                    Reply message RE: Fork in the Road dohearne
19th Aug 2012
16
                    Reply message RE: Fork in the Road PAStime Silver Member
19th Aug 2012
17
                    Reply message RE: Fork in the Road robsb Platinum Member Fellow Ribbon awarded for his expertise in CNX2 and his always amicable and continuous efforts to help members Laureate Ribbon awarded for winning in the Best of Nikonians 2013 images Photo Contest Donor Ribbon awarded for his enthusiastic and repeated support to the Fundraising Campaign 2014
21st Aug 2012
20
                         Reply message RE: Fork in the Road JonK Moderator Awarded for his high level skills and in-depth knowledge in various areas, such as Wildlife, Landscape and Stage Photography Donor Ribbon awarded for his generous support to the Fundraising Campaign 2014
21st Aug 2012
21
                         Reply message RE: Fork in the Road PAStime Silver Member
21st Aug 2012
22
Reply message RE: Fork in the Road dohearne
17th Aug 2012
11
Reply message RE: Fork in the Road JonK Moderator Awarded for his high level skills and in-depth knowledge in various areas, such as Wildlife, Landscape and Stage Photography Donor Ribbon awarded for his generous support to the Fundraising Campaign 2014
18th Aug 2012
12
Reply message RE: Fork in the Road dohearne
19th Aug 2012
13
     Reply message RE: Fork in the Road JonK Moderator Awarded for his high level skills and in-depth knowledge in various areas, such as Wildlife, Landscape and Stage Photography Donor Ribbon awarded for his generous support to the Fundraising Campaign 2014
19th Aug 2012
14
          Reply message RE: Fork in the Road cliddell Silver Member
20th Aug 2012
18
Reply message RE: Fork in the Road pencho
21st Aug 2012
19

Forums Lobby MASTER YOUR TOOLS Post-Processing & Workflow (Public) Nikon & Nikonians Software (Public) topic #9083 Previous topic | Next topic