Software tricks that compensate for hardware

If you hear about software compensating for hardware be it in imaging, networking, security, etc. etc. take it very seriously. 

Got to witness it firsthand over the weekend. We were at a friends place and one of our friends who works at Apple, whips out an iPhone8 and says "let me show you the portrait mode on this one". She takes a photo of me and with a single swipe on the icons in the bottom she was able to adjust the lighting on the portrait, AFTER the photo was taken. The background was also blacked out instantly and precisely.

It is interesting to note that Apps have figured out how to identify your face and put bunny ears on them, glasses and hats on your image and move them dynamically as you move your face. That is already 2016! as pointed out by my other friends. So the face and body recognition has come a long way. The speed and precision with which it is being done is what is amazing and scary! 

It takes me (and I do this with practiced ease) 3-4 minutes to cut an outline and create portait shots by blacking or whiting out the background. I use the smart cut tool in Adobe Photoshop CS5 and do a layer by cut followed by a solid layer and reordering the layers. 

The key issue is that the smart cut is only as precise as I can do it manually. My rush and crude job in the picture below shows that I did not get the boundaries right on my head line or right ear, which is because the background color blends with skin color or fuzzy boundaries are hard to delineate. The option is to gradually increase opacity between the layers. It can be done and it will take another few mintutes and can be automated.

How do you adjust the lighting post picture?! Think the iPhone8 is taking a page out of the Lytro book and doing multiple exposure shots (kind of like HDR but by adjusting focus) or is adding image sensor tricks to deliberately over expose or underexpose a group of pixels after it identifies the face prior to taking the pictures and is then able to play with this in the post processing. Dont know what kind of AI went into this, but whatever it is, very impressive!

Note that when I do smart cut and add a black layer, there is no lighting effect. I can also recreate part of the effect by creating a custom Vignette option. But I cannot change the local lighting on my face. It will always match the original. 

The "smart cut" being done with software is what is incredible. This is going to put portrait studio kits out of business. 

The SLR camera and lenses are temporarily safe. It still takes a good zoom to capture pictures of moving subjects against a plain backdrop and CS5 to adjust it. Maybe when you import the photos into an iPhone8 those photographs will be edited with a button click. Photoshop options and customized macros are going to be folded into the iPhone. You dont need those anymore soon..

One idea that I had was this.. Build a housing for an iPhone where the phone's camera focuses on an internal white board inside the box which replaces the plane of the full frame sensor in an SLR. You directly put your 2000$ lens on to this small box and the phone connects to an EF lens, telescope, binocular etc. etc. 

In the mean time photography enthusiasts like me will still lug around a heavy bag... 

Technology is moving very fast. Faster than a common person with respect to that technology field is improving. I am in hardware and see a scary progress in software. A software person is probably seeing the same thing in hardware space. The collective improvement is something neither one can anticipate or expect. . . it beats the average expectation by a wide margin!  The happy go lucky average users (all of us) don't see the larger picture of where this is going!