When Adobe introduced DNG I didn't think much of it at the time. I've been working with computers since before the IBM XT and in that time I've seen many file formats come and go.
I also had serious concerns about going out of my way to be beholden to Adobe for the very structure of my raw images. And at the time, DNG was an open standard used by exactly one vendor, so to me it looked like a proprietary format hiding behind a facade of openness.
In the meantime, DNG seems to be becoming more accepted. And now Nikon has done what I previously thought was unthinkable- to abandon an architecture with which its installed user base has collectively millions if not billions of hours of editing effort. The idea of Nikon someday abandoning the current NEF format is only one step away from what they have done.
So now, in terms of how I view future proofing my image data, my ideas of greater and lesser evils in this regard are changing dramatically. Those, like me, that initially dissed DNG as being "dangerous" to long term raw image preservation might need to rethink things.
There is a general school of thought (not shared by all, of course) that what Nikon did with Capture and View, directly editing the raw file, is "bad", "dangerous", nonstandard, and etc. Not just here, but, for example, Thom Hogan has been blogging that point recently in his discussions of the demise of Capture.
And yet, here we have Adobe creating the DNG standard, which does EXACTLY the same thing. Why is it ok for Adobe but not Nikon?
(I don't follow Thom Hogan enough to know how he feels about DNG, and certainly it is not mandatory to store edits and updated previews in DNG, but still...????)
Personally I use a 3rd party catalog app (iMatch). And my workflow with iMatch is, of course, very Capture-centric. I will probably continue to use IiMatch for various reasons...
And I am pondering the idea that if I use Lightroom (or Bridge/PS), convert Adobe rendered raw (NEF) to DNG, and keep the DNG updated with metadata and up to date edited previews, then I have a far more consistent workflow between my old (capture) world and my new world. It may solve a lot of problems for me.
One of those problems is what to do about future edits in LR to old shoots originally edited in Capture. If I were to re-edit an old image previously worked in Capture, then I would probably use Capture; why reinvent the wheel there?
But if I were to edit an image never before edited in Capture, but shot in the "capture era", I think I would want to do it in LR or whatever raw app I chose to use. And as time goes on and the terminal version of CNX2 becomes more and more stale, this will be more compelling.
So if I continue to store images in original NEF, I have a peculiar problem of figuring out how any particular raw image was edited. I would end up with original raw shoot folders with mixed editing.
If I were to do all my LR editing against DNG files then I would have an unambiguous marker- the image file extension (format).
Because of the increased risk of file corruption when raw files are edited directly, and because of general concerns about long term data corruption, I have long had a practice of setting aside a secondary copy of the original NEF files, never to be touched, and stored offline.
If I were to convert to DNG I would continue to keep that secondary copy of my *NEF* files, for the same reasons, and a side benefit would be that all my raw images would be in two different formats.
As I start to ponder all this I thought I would throw this out for discussion.
Good read Neil, Your knowledge surpasses my ability for complete comprehension but I get your general idea, (I learn as much as I can but my stamina is waning in these later years!)
When I first looked at a change last year DNG did cross my mind, after many google days had passed, I still could not get an affirmative answer, some said DNG could go as well, some said it would be around forever
As we would all agree, TIFF files are just too large to maintain
The general trend around the globe would have to be a move to LR, some, like me a move to Aperture, I tried both but found Aperture more intuitive for my old brain and I like the cataloguing set up as well
With the above 2 options, (LR and Aperture), you can retain that original NEF, I think keeping that original file structure separate and backed up is the way to go. This is not a big deal as it is a choice for an automatic process when importing images
Im sure the big boys in the software world are going to always support Canon and Nikon raw file structure, and as for Nikon NEF, just because Nikon lost some editing options due to a bad business decision, I'm sure our NEF is going to be with us for a long time
Even though I backup the NEF files on import, my edits never touch the original file and I can have as many versions as I like, the downside being that I can not export as any other file except, PSD, TIFF, PNG, and JPEG
IMHO, it is in the software professional business interest to be compatible with the major camera companies
FWIW, most plug-ins I have work with TIFF and I use PS as a plug-in as well, all works a treat
If you do not edit your NEFs directly then my ideas wouldn't help you.
Just to be clear, I'm not trying to predict the demise of the NEF. Just saying that if I had to place a bet on the longer term survivor - NEF vs DNG - I wouldn't have a clear answer now.
Nikon has made it clear that "user continuity issues" are of no concern to them. The only remaining issue is how they see their own interests, and obviously they have compelling personal reasons to provide continuity at least to the out of camera nef format.
Adobe has similar compelling interests, and at this point I don't think Adobe has ever backed away from a standard they created. So it's an interesting horse race I wouldn't want to call, looking out 10 or 20 years.
About every month or two someone says something to make me rethink this also (as you just did), and each time I come around to the same conclusion -- "not yet".
I have three basic reasons -
1) I can always produce DNG later, automatically, with a touch of a button from Lightroom. Might take a fair amount of computer time, but so what -- my computer doesn't charge by the hour. If/when I need to, it's easy.
2) I would still feel obligated to keep the original NEF. I don't think I trust the DNG to keep it for me (well, to be more precise, if there's a reason to keep the NEF because I don't completely trust the DNG, then I don't trust it to also contain the NEF). And if I'm keeping the NEF, then see (1).
3) Writing updates into the DNG, in addition to (slightly) increased risk, means the big file is "touched" and so my incremental backup will grab it again each time. Writing to a catalog or to XMP (tiny, tiny files) means that these incremental edits are much more efficient to back up. Consider if you (as I have frequently done) change hundreds or thousands of images' keywords (see it's not just edits). My nightly backup hiccups with a few meg. Someone with DNG's might back up 500 gigabytes just from one keyword change across large numbers of images. As we all use catalogs and tie to web sites and search engines more and more, this kind of retroactive metadata change I think becomes more common.
I just haven't seen the win yet. Perhaps the strongest argument is that it contains the XMP internally and so it is less likely to get separated, but really I don't see how I'm all that likely to separate it anyway. And besides, I have WAV files as side cars also, I don't think it will embed WAV, will it?
Now if I would have to go manually convert NEF's to apply the edits and produce DNG's, and was building up this huge backlog of eventual manual work - ok, I get it. But so long as it's a few mouse clicks to do later - procrastination is my answer. As I always expect to know more later than I do today, and can make better decisions later.
Regarding your #1 and #2, my idea was to take advantage of the single package containing the edits and updated JPG preview. Simply to maintain a similar workflow to what I have now. I was not suggesting to go to DNG to future proof; it might be a bit early for that .
I would still keep the original NEF, for reasons you and I mentioned.
Regarding #3... I too have thought about the idea of flagging uber gigabytes of raw files when all that has to happen is a change and backup of a tiny XMP file. That is my biggest hangup too.
However, I brought this idea up because a lot of people here are doing the same thing with their NEF files, and if that includes mass updates in order to write keywords and other metadata from whatever app they use, they are having that problem now.
In my case, I now have a somewhat complicated setup between PM, CNX2 and iMatch. I let PM update the raw files with metadata, but I also create an XMP sidecar. PM gives read priority to the XMP.
CNX2 reads the metadata that PM puts into the NEF file. I *DO NOT* use CNX2 to update metadata because it is incompatible with iMatch as I have it configured...
iMatch writes XMP but I do not allow iMatch to update the raw file. Partly because of the problem you mention, but also because the iMatch developer is not 100% confident in the idea of updating raw files (a complex subject - it surely works but I took a very conservative approach to his cautions).
Because PM gives priority to the XMP, PM and iMatch interoperate. CNX2 is basically read-only.
If I had to do it over I probably would have had PM write only XMP, not the NEF, and totally give up on any compatibility with CNX2, but it took me quite a while to totally understand what was going on there.
So whatever I do in the future either has to work with the above or I have to "migrate" things and change some settings in a way that makes the past and the future work together. I'm still mulling all that over.
I also ran into a very strange LR issue (I think it is strange). When I looked carefully at how LR, PM and iMatch all dealt with DNG, I found that when writing back metadata and previews to DNG, LR does *not* update the DNG file's OS modified timestamp. LR does update XMP file timestamps after doing metadata writebacks to NEF files. So that behavior is inconsistent and probably not good from most user's perspectives.
That means that standard backup apps that backup changes only may not recognize changes, and therefore metadata could be lost. Might depend if the app looks at the file size, and then if any particular metadata change results in a change to the file size. A change from a 1 to 2 star rating most likely would not, but keywords probably would, and so on.
I ran into another issue with LR, where LR does *not* appear to automatically save metadata back to DNG files, where it does with NEF files (to the XMP file, with the appropriate preference setting). If that is the case that would be a show stopper for me, simply because I don't want to have to burden myself with an extra step to save metadata every time I touch an image file in LR.
So yes, there are issues. I just thought I would bring this up for a discussion and mention that, warts and all, if someone absolutely wants to stick with a CNX2 type architecture that, in principle, it exists.
The main problem with the LR architecture, assuming you have outside browser type apps that need to accurately portray the raw images (reflecting edits) is that it is really required to maintain up to date "buddy" JPG files, usually in the same folder as the NEF files. I have that issue with iMatch.
Unless one is very good at remembering to selectively rebuild those buddy JPGs, then the raw files will not reflect up to date edits. One way around that is the shotgun approach, where after working a few images in a folder I would just rebuild all the raw images in that folder. Just to make sure I didn't miss any.
And that will be a likely approach for me. And the result is still a substantial amount of unnecessary backup volume. Not as bad as the DNGs, maybe, depending on how it's done, but in some cases could result in even more data moving through the backup.
>I also ran into a very strange LR issue (I think it is >strange). When I looked carefully at how LR, PM and iMatch all >dealt with DNG, I found that when writing back metadata and >previews to DNG, LR does *not* update the DNG file's OS >modified timestamp. LR does update XMP file timestamps after >doing metadata writebacks to NEF files. So that behavior is >inconsistent and probably not good from most user's >perspectives.
I'm not seeing that, and I tested it before my initial post, to make sure I understood how LR did the DNG file updates. Every time I changed something and hit Control-S (write XMP, since I have it off by default) I observed the modification date change in explorer. Modification date, not other dates (Windows in newer versions has something called "date" and I'm not completely sure what it even represents. Here's an example. These two files (shown in explorer) are how Windows 8.1 displays the NEF (line 2) and the converted DNG (it got a different title due to my export settings). I exported it at 1:13pm, then converted to B&W and saved that change with Control-S, and you can see the modification date at 1:14pm.
Shows something else interesting, and maybe I'm missing a codec, the DNG file doesn't show a date created inside of Windows (it clearly does in something like Photoshop). I also don't get the preview thumbnail in Windows explorer (and I did save a large JPG with the DNG).
#6. "RE: Rethinking DNG" In response to Reply # 5 Wed 05-Mar-14 06:42 PM by nrothschild
My error. I had some confusion over inconsistent date column headers in Explorer. At least one of my apps (pretends? to) rewrites the file and Win got changed to read the ambiguous date... so LR is correctly posting the date (modified).
Edit: the same confusion over the auto-write-back. That works fine with DNG.
I get Date Taken and a thumbnail. I have Fast Picture Viewer WIC codecs installed.
I have stayed away from DNG as I always saw it as an unnecessary step and I never bought into the idea that DNG was more future proof than the NEF. I am however using a DNG format with my Nokia 1020 phone because that is the RAW format the phone provides. Photoshop will natively open ACR when it sees a DNG file. You can also use the stand alone tool that Adobe provides to create DNGs from NEFs. s I remember you wind up with a bigger file though.
>And yet, here we have Adobe creating the DNG standard, >which does EXACTLY the same thing. Why is it ok for Adobe but >not Nikon?
It is OK for Nikon and Adobe both. It would continue to be OK for Nikon except that they have decided not to do it any more.
Or, if it's not OK for Adobe and Nikon to make updates to the NEF, why is is OK for Photoshop to make updates directly to a PSD file? Why is it OK for Microsoft Word to make updates directly to a .DOC(X) file? If it's not OK to touch the NEF source file, it's not OK to touch ANY source file, and all edits should be done in sidecars!
Note that I'm not suggesting anyone should ignore the need for backup copies of original files. Only that RAW camera files are no different in this regard than any other file.
Like you, I don't see any inherent benefit to DNG in the near term. Now, your idea of using NEF vs DNG as a marker to detect which editing program was used is an interesting one -- for you. Doesn't apply to me, though.
In terms of the end result, all of Adobe's tools will treat the NEF RAW file in the same manner it treats the DNG RAW file. That being the case, I struggle to see the benefit of DNG conversion. I'm confident enough in my multilevel backup strategies to think that I'm unlikely to lose edit information in any case.
#9. "RE: Rethinking DNG" In response to Reply # 0 Wed 05-Mar-14 11:18 PM by JonK
Colorado Springs, US
Not going into all the advantages or disadvantages (which have been talked about many times), but a key difference between Nikon's implementation of the NEF format and Adobe's implementation of the DNG is that the latter is actually documented and was submitted as a standard to ISO. NEFs have remained undocumented. Although DNG's are supported in many supporting programs, they aren't supported by that many raw converters. One of the reasons is simple: you don't need to buy upgrades that support new cameras if you go with a DNG format. You can simply keep using your older software unless you want a feature in the newer version. That's difficult for raw converter companies who are dependent on upgrades for their revenue stream.
Personally, I'd make the decision based on features or compatibility with raw converters you might want to use rather than archival issues. I tend to use DNG primarily when I need to ship off raw files and want them at reduced sizes. You can do some pretty dramatic things with size reduction given the current state of DNGs, but that doesn't mean you need to use them for everyday processing.
#10. "DNG and ACR?" In response to Reply # 0 Wed 05-Mar-14 11:47 PM by Jim Pearce
I've been thinking along similar lines, although my cataloguing will remain manual. So, I open the NEF in ACR and save it as DNG. ACR with the adjustment brush can do most of what CNX2 could do (Plus cloning!), and non-destructively as well. So, I now have an original NEF and a developed DNG (with no sidecars). If I need to go to Nik I can open the DNG in Photoshop as a smart object, allowing me to use Nik non-destructively as well (Advantage Adobe). Can I save this as a DNG as well (I'm not sure, but TIFF would be okay.)?
The only real downside I see is paying for LR and Photoshop CC when I don't need LR and have minimal need of Photoshop. Sure $10 per month is fine, but that won't last.
So you shoot a NEF, covert to DNG, edit and save non destructively as a PSD (which, if it has smart objects is huge). At a minimum you need to have 3 to use Color Efex Pro non-destructively outside or CNX.
It's madness to have to create so many files just to edit your images.
Furthermore, and I don't understand this, I created a image just the way I liked it is Photoshop CC using a SmartObject layer. Aside from the file size (650mb), the image degraded on export as a TIF or Max Quality JPG. What would cause that? The image was less sharp by a very noticable amount.
#15. "RE: Rethinking DNG" In response to Reply # 0
Adobe CC changes a few things. In a short time almost nothing you do in Photoshop will be destructive. It's down to just a few things. They have added a lot in the last 6 months. Right now nothing in Light Room is destructive. You can still make side car files if you choose. Seems silly to me now that you don't have to but you can.
So if you want to be conservative shoot NEF raw (no reason not to) Cannon still has proprietary raw too. So do the new raw players as well - who cares? Archive the NEF file in case they get valuable and forget about them until then. Import to Lightroom into DNG and stay non destructive workflow until you export product to ship or print. Go back and do something different and sell that too! To repurpose your own images isn't something bad.
You may need to have intermediate saves as a few functions are not non destructive (not many). There is a penalty in that non destructive gets to be huge. Storage is so darn cheap now it's not an issue! BluRay double layer disks hold 50GB. DNG is a good delivery but so is TIF. You sure can't sell NEF files!
The other issue is with Adobe CC everything serious with images is now integrated. So, you want to integrate video or need to do CMYK press ready delivery or just deliver images for illustration to companies that just want an image for something. Staart with common tools and adapt to delivering what people want to buy!
There is more to photos than just prints for the wall or albums. Adobe is where the images end up that sell for money! Nikon won't ever be there nor will Cannon. People that buy images are pros using Adobe products! You can even deliver electronic copies to home users too. People want digital images!
We all need to go beyond selling hard copy prints to people! There is more money doing more! Tools are getting better and the hardware to do it is now amazingly cheap!
There has never been a period in history when Photography existed that it was not leading technology. There has never been a better time when one photographer can't get in on this action than there is now! You have the ability to have more tools on one desktop computer than ever before! The darkroom is now in the light and online!
#17. "RE: Rethinking DNG" In response to Reply # 15
I don't buy it!
Storage may be cheap, but a 2TB drive will only store a little over 3,000 or those monsters that PS created when I tried it. That's just a couple events. And ridiculous.
Actually, this is a terrible time to be a photographer trying to make a living doing anything with images (still or video). Almost everybody has the ability to take good images with easily available equipment and techniques that are almost automatic. People are more interested in a photo that they took, and think is good, than something they would acquire from some one else. Every genre and technique is oversubscribed.
Those top level pros that are still at it are dinosaurs, the last of their breed. They are still able to make it out of loyalty, habit and that fact the the generation that relied on pros is still around. Stock is no longer lucrative or even remunerative.
Many of them probably make more money doing workshops, presentations and selling training to other photographers than they make from selling images.
Photography will always be a great hobby, but the ability to make a living at it is diminishing daily.