I was catching up on some recent posts and came across first this and then this thread on lossless compression.
I didn't understand what the lossless algorithm did initially and consequently have been storing my raw images using uncompressed storage (the second post explains lossless compression in understandable terms). since the advantage of lossless compression is nearly 2:1 for an average image in terms of storage space with no loss of information and with perhaps a small penalty for time when opening in post, i am planning on switching to lossless compression.
I'm curious on what choices others have made for their storage and am curious about what others perceive as the pros and cons? and, finally, can i compress an uncompressed image after the fact using Nikon's algorithm?
Sun 15-Sep-13 07:55 PM | edited Sun 15-Sep-13 07:57 PM by sevtcard
so, this was my reaction too initially...though, from my old programming days (very long ago - think punch cards), i knew that bit compression results in no loss of data, but was not sure what 'lossless' in this case meant - the thread i linked to (reply #2), in my original post, above, explains it nicely...so, at least as i understand it, there is no data loss....
Gerold, How about posting a couple of pictures shot one after another, as quickly as possible, of the same subject, with no changes in aperture, shutter speed, ISO, or focus, from a tripod, using mirror up and a cable release, one compressed with lossless compression and one with no compression. Blow them up as far as you can and point out the difference. I hate to say it, but I doubt you or anybody else can do that.
Sorry Charles but you lost me completely. (That's pretty lossy.) Please go back and re-read my post. I did software engineering for thirty years and I guarantee I'm not confusing lossless with jpeg compression. Yes, jpeg is lossy, sometimes very lossy. Nikon's lossless compression is lossless.
Oh, by the way, why do you think you need to upload 70 meg files? A very small crop can do the job.
it seems we're getting a bit heated - I've always viewed Nikonians as a source for education and a place to ask questions safely....as the OP, i was just looking for some interesting perspectives on the topic, which certainly have been presented...in his defense, i think russ was actually saying that there is no difference in compression strategies (lossless vs uncompresssed)...i am reasonably sure at this point that i was wrongheaded in my original assumptions about the compression strategy i was using and now have changed my thinking based upon some reasonable input from other Nikonians and based upon my own (remote) past experience.....i hope others will not be intimidated from posting their perspectives on what appears to be a misunderstood and apparently controversial topic...
Lossless compresssed, 14 bit RAW, 100% of the time. My thinking is simple. Capture as much information from the sensor as possible, and don't ever throw any of it away, unless I am throwing away the whole image.
so, if my wife could see my metaphorical photographic cellar, she'd probably cringe, since it tends to look like my real one....i never throw anything away...it's always painful for me if i happen to be somewhere and i'm running out of memory storage and i have to delete even the least likely image that might ever see the light of day in order to make room for a potentially good image....so, now with the switch to 14 bit lossless compression, that will moment will be about twice as long in coming....
I am curious if you have looked at this link? it explains the process nicely....when i was in medical school, we had to memorize enormous amounts of generally unconnected data...i would use this exact technique to help me reconstruct the information i needed - i would generate a table of all of the information with salient row and column headers, memorize the headers (and i would use mnemonics to help me remember the row and column headings - another form of hopefully lossless data compression), each cell would get a 0 or 1...i would then generate a number for each row based upon the 0's and 1's (bits). i'd then translate the binary number into decimal and memorize the decimal numbers (one number for each row)....during the test, i'd regenerate the row and column headers, decompress the decimal number into bits (0's and 1's) and fill in the table...so, hundreds and hundreds of points of information were compressed into maybe 50 or so ...with no loss of data and with good marks on the exams....and, yes, despite this shortcut, i'm still a pretty good clinician...
That is a good question....i don't know what their motivation is (maybe processing time?), but 'lossless' is not a claim, it is a well known computer algorithm. Here is a post from MotoMannequin, from May 8th, 2012, which explains the process nicely:
"....PSA: Uncompressed NEF is for those of the mindsent who would try to comare image quality of 2 different brand CF cards (yes these people exist). There actually is a free lunch with lossless compression - the result is exactly the same data as the uncompressed file, just as when you ZIP then unZIP a file, the unzipped one is the same as the original. Therefore, always take advantage of lossless compression.
Lossless compression takes advantage of repeating patterns in data. The simplest and most easy to understand is run-length encoding (RLE). Imagine I'm trying to compress a B&W image, using "B" for a black pixel and "W" for a white pixel. I have a row that looks like this:
...meaning 12 Ws, followed by 1 B, followed by 12 Ws, etc. This compressed 67 bytes down to 18 without any loss of data. I can fully recover the original string from the data in the compressed string. Of course, some data doesnt' compress down so well. The string:
Which is the same length of the original. Data compression algorithms talk about the amount of "entropy" in the data which involves the amount of repeating patterns and how far the data can be compressed. When you try to zip a jpeg you will find it doesn't compress much at all. Even though jpeg runs a different compression algorithm than zip, once the data is compressed is has very low entropy, and zip can't find much to compress. What ZIP does is greatly more clever and complicated than RLE, but you get the idea.
Lossless means you can 100% recover the original data. The only theoretical downside would be additional processing time to do the compression, but it turns out that writing the bytes to flash takes longer than compressing them, so you have a net gain in processing time when using losless.
I guess the other possible downside is that a single byte corrupted in a compressed file might render the entire file unusable, while a corrupted byte in an uncompressed file might only corrupt one pixel. In pracitce, this is a vanishingly rare issue, and one I'm personally not concerned about.
>>... I just don't believe the lossless compressed claim. > >Neither do I. If there's no difference at all (apart from file >size) why would Nikon bother keeping the uncompressed option?
Why do we still have cameras that shoot film? Or that use a pentaprism? Why do people still own record players??
Lossless means lossless. And it works, and it has been used for dozens of years in all types of data transfer. Do you even want to KNOW how much lossless data compression is happening on your internet connection right now?
I understand the concept may be foreign to those not well versed in computer data structures, but when something is using lossless compression, it really is lossless.
And despite the previous posts allusion to data corruption, this is what checksums are for. It's what allows for self healing correction to take place
>>... I just don't believe the lossless compressed claim. > >Neither do I. If there's no difference at all (apart from file >size) why would Nikon bother keeping the uncompressed option? > They include both so that they do not have to listen to some of their customers whine about file difference that cannot be proved.
The choice allows each of us to simplify our lives. Since you don't believe in lossless compression you can have it your way and I simplify my life because I can store significantly more lossless compressed information than I could otherwise - having it my way.
Giving us the choice saves Nikon the hassles of dealing with either one us!
For myself - 14bit lossless for photos where I want the best quality now and possibly in the future, 12bit Compressed for photos for routine events where I know I will not be trying to pull detail from the shadows and probably will not keep the RAW files for more than to years.
Sun 15-Sep-13 09:51 PM | edited Sun 15-Sep-13 09:59 PM by jrp
Lossless compressed, 14-bit. That's what I do. When I started shooting digital I tried uncompressed too and could not see any difference in the final output with lossless compressed, except in file size
do you know if i can compress the images i have already stored using uncompressed mode into a compressed format after the fact? also, i just changed the compression on my camera, but it is still reporting the same potential capacity...any ideas on why that might be the case? thanks
Sun 15-Sep-13 10:03 PM | edited Sun 15-Sep-13 10:04 PM by gpoole
You can compress previously stored images with Capture NX2. Load the image and them do a Save as. You can select loveless compression as during the save as. You should be able to do this in a batch process one folder at a time.
All Nikon DSLRs underestimate the card capacity when using compression. My and many other users experience shows that we get 50 to 70% more images than the original remaining capacity. After you shoot a large number of images you will see that the remaining capacity has decreased by much less that the images you have taken.
Gary in SE Michigan, USA. Nikonians membership - My most important photographic investment, after the camera. D4, D810, D300 (720nm IR conversion), D90, F6, FM3a (black), FM2n (chrome) YashicaMat 124, Graflex Speed Graphic 4x5 My Nikonians Gallery & Our Chapter Gallery
Sun 15-Sep-13 10:04 PM | edited Sun 15-Sep-13 10:06 PM by avisys
>do you know if i can compress the images i have already >stored using uncompressed mode into a compressed format after >the fact? also, i just changed the compression on my camera, >but it is still reporting the same potential capacity...any >ideas on why that might be the case?
I ~suspect~ you'd now have to compress those files with a ZIP, or similar, compression facility, and then extract them from the archive as you need them. But if they are just sitting there, mostly using up space, then that might be a good strategy if you have only occasional need to extract one or a few.
Can't help with the camera-posted capacity --- never has made a lot of sense.
Ah! I see, above, that Capture NX2 will do the job. Great, if it has a good interface and process.
Sun 15-Sep-13 10:11 PM | edited Sun 15-Sep-13 11:00 PM by jrp
I am not aware of Nikon software to convert uncompressed NEFs into lossless compressed NEFs, except for Capture NX2, where -as Gary mentioned- a dialog pop-up window comes up for you to select the format.
It is a one-at-a-time thing. Maybe that can be batched.
As for the images capacity from the card, that is always a very safe (conservative) estimation, not an accurate calculation reading. Attachment#1 (jpg file)
if you save to a different file location or name using NX2, you can change to lossless compression...i just tried with 420 files by saving them via NX2 into a new directory (i first selected them in the NX2 browser, then hit 'save as'- files went from 75k on average to about 55k on average, nearly 30% reduction in size on average...memory's a lot cheaper than it used to be, but it's still finite, so this is helpful, since i've got about 3500 images uncompressed. the compression does not seem to be as good as when i do it in camera (averaging about 40k when compressed with lossless compression) - not sure why that is, would be interested in any insight on that front
thanks all for your responses...i know the camera calculation of available memory is conservative, but didn't expect it to be 1:1 between uncompressed and lossless....it'll be interesting to see how it goes....thanks Gary, i'm going to give a try with nx2 to see how that works (ps, great freudian typo: 'loveless' and not 'lossless').
>do you know if i can compress the images i have already >stored using uncompressed mode into a compressed format after >the fact? also, i just changed the compression on my camera, >but it is still reporting the same potential capacity...any >ideas on why that might be the case? >thanks > >Mark >www.broadwallphotography.com
Waaaay back when, in the dark ages of computing, when a 10 MB hard drive cost several hundred dollars, there were programs that turned an entire drive into zip files. Files, including executable files, were decompressed on the fly. The original zip program was a lossless compression algorithm, just like the modern Nikon lossless compression.
"Beauty is in the eye of the beholder and it may be necessary from time to time to give a stupid or misinformed beholder a black eye." Miss Piggy
Sun 15-Sep-13 11:14 PM | edited Sun 15-Sep-13 11:16 PM by avisys
>Waaaay back when, in the dark ages of computing, when a 10 MB >hard drive cost several hundred dollars, there were programs >that turned an entire drive into zip files. Files, including >executable files, were decompressed on the fly. The original >zip program was a lossless compression algorithm, just like >the modern Nikon lossless compression.
Oh, yes. In the mid-80s, I purchased a 20MB hard drive to attach to an IBM PC. It was the size of a bread box. (Don't ask me to explain how big a bread box is). It cost me $600.00 at Fry's in San Jose.
The last hard drive I bought was a 2 tera-byte drive, the size of a split pack of cards. Let's see . . . that's 10,000 times the capacity of my original drive. The equivalent cost would be $6 million. Not to mention cost per byte per cubic inch, so we're probably talking $360 million.
Nevertheless, regarding ZIP and lossless data compression, there have been minor tweaks to the ZIP (actually PKZIP) algorithm, but in terms of compression ratio vs. compute overhead, there have hardly been any really significant improvements over the years. I'd bet the Nikon compression scheme is virtually identical to PKZIP.
When I got my 800e and went through the books and charts on initial set-up all said set to Lossless compressed, 14-bit. I did. If there is any reason, even a little one, to change that please let me know as I only have a degree in physics.
>When I got my 800e and went through the books and charts on >initial set-up all said set to Lossless compressed, 14-bit. I >did. If there is any reason, even a little one, to change >that please let me know as I only have a degree in physics.
Lossless, is lossless, is lossless. The only difference vs. uncompressed is a little compute time . . . oh, and a hell of a lot smaller file.
I simply avoid the question by using the most RAW format they offer me. I agree with the other poster who said he just uses all the information from the sensor. Maximum bits, no compression for me! IainD
>I simply avoid the question by using the most RAW format they >offer me. I agree with the other poster who said he just uses >all the information from the sensor. Maximum bits, no >compression for me! >IainD
You would get "maximum bits" with lossless compressed.
if you still refuse to believe that a raw image can be compressed without loss of data, try it yourself with WinZip or some other compression application.
I just compressed a random NEF file from my image folders and the resulting .zipx file was one third smaller.
In that particular image the compression ratio was exceptionally good but in general, unless the compression algorithm is written having additional information of the source files, results can be less optimal. The file can get even larger.
A previous post in this thread clearly explained how Nikon's lossless compression formula works. No, I do not have a degree in math, computer science, or physics unless I were to count the structural engineering courses I took as "fun electives" 30+ years ago back at CalPoly. Up to now, I've set both cameras to uncompressed photos, but I'll be changing to lossless compressed.
I also use lossless compressed and I understand that its basis is for compression of identical pixels (but don't quote me on this). For instance, if you have a number of identical pixels (say exact blue sky data, or end-limit white, or end-limit black -- not that any pixel exact), those data will be mapped/compressed as single data values. Thus while the individual bits may disappear, they will be re-expanded (when uncompressed) exactly as originally recorded. This is not the case with JPG or any other compressed file type.