Security Photos and Open CV

“Hey Sherwin, can you pull our employee ID photos from CCURE and standardize the images so we can upload it to Workday? If you can do it, you have until the end of next week to get this done. Thanks!”

I haven’t worked with image processing at all so the idea of a project involving that was really exciting. I agreed and started going through the discovery phase. Here are a few things that I found out.

  1. The photo quality ranged from 0.9MP to 12MP
  2. Orientation varied from Horizontal and Portrait
  3. The photo composition, where the faces are located, is not consistent
  4. There were 1500+ (1539 to be exact) photos to process
  5. The process needed to be designed with automation in mind

The Challenge

The greatest challenge with this exercise is determining the proper padding after a face has been found. What does this mean? When OpenCV detects a face, the output is similar to image below.

Face Detected

For a more detailed explanation on how to use Haar Cascades and OpenCV, navigate to this link.

def findFace(img):
    grayImg = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    face = faceCascade.detectMultiScale(
        grayImg,
        scaleFactor=1.3,
        minNeighbors=5,
        minSize=(30,30),
        flags = cv2.CASCADE_SCALE_IMAGE)
    if face==():
        x = 0
        y = 0
        if imgWidth<imgHeight:
            x2 = imgWidth
            y2 = imgWidth
            w = imgWidth
            h = imgWidth
        else:
            x2 = imgHeight
            y2 = imgHeight
            w = imgHeight
            h = imgHeight
        imageCoords = (x,y,x2,y2,w,h,0)
        return imageCoords
    else:
        for (x,y,w,h) in face:
            x2 = x + w
            y2 = y + h
        imageCoords = (x,y,x2,y2,w,h,1)
        return imageCoords

As you can see the output is not ideal, especially for a corporate profile picture. There simply is just too much…face. We need to get the rest of the face and some of the head. To start, we have to determine where on the image the face has been detected. OpenCV returns four (4) values x,y,w,h. The x,y values are coordinates while w,h are the detected faces’ width and height. We can calculate the rest of the coordinates that will bound the face by simply adding the width and height to the starting coordinates like x2 = x + w and y2 = y + h.

Finding the Face

Before we can determine how much padding to add, we need to list out the values that we know so far

|Calculated|Variable|Value|Calculated|Variable|Value|Calculated|Variable|Value| |:——–|:————|——-:|:——–|:——-|——|:——–:——–|——:| |No|imageWidth|504|No|h1|150|Yes|x2|459| |No|imageHeight|622|Yes|h2|45|Yes|y2|551| |No|faceWidth|309|No|v1|242|Yes|padding|45| |No|faceHeight|309|Yes|v2|71||||

Coordinates

  • h1 = x
  • v1 = y
  • x2 = x1 + faceWidth
  • y2 = y1 + faceHeight
  • h2 = imageWidth - x2
  • v2 = imageHeight - y2

Using all these values we can now find face’s relative location within the image by comparing both horizontal and vertical values.

if h1 <= h2:
    hRelPos = h1/h2
    hFlag = 'L'
else:
    hRelPos = (h2/h1)
    hFlag = 'R'
if v1 <= v2:
    vRelPos = v1/v2
    vFlag = 'U'
else:
    vRelPos = v2/v1
    vFlag = 'B'
if (hRelPos >=.80 and hRelPos <=1) and (vRelPos >=.80 and vRelPos <=1):
    vFlag = 'C'
    hFlag = 'C'

Once the relative position is identified, we take the flags defined and identify the quadrant and set the padding accordingly. For example, if the face is located on the Bottom-Right hand corner (like in our example) of the image, we check if the face is closer to the right-most portion of the image or closer to the bottom, and we take the lowest value and set that as the padding.

if hFlag=='C' and vFlag=='C':
    padding = int(faceHeight*padThreshold)
if vFlag=='U' and hFlag=='L':
    if h1 > v1:
        padding = v1
    else:
        padding = h1
    if padding/faceHeight > padThreshold:
        padding = int(faceHeight*padThreshold)
if vFlag=='U' and hFlag=='R':
    if h1 > v1:
        padding = v1
    else:
        padding = h2
    if padding/faceHeight > padThreshold:
        padding = int(faceHeight*padThreshold)
if vFlag=='B' and hFlag=='L':
    if h1 > v2:
        padding = v2
    else:
        padding = h1
    if padding/faceHeight > padThreshold:
        padding = int(faceHeight*padThreshold)
if vFlag=='B' and hFlag=='R':
    if h2 > v2:
        padding = v2
    else:
        padding = h2
    if padding/faceHeight > padThreshold:
        padding = int(faceHeight*padThreshold)

Command-Line Arguments

One of the last things we have to do to prepare this solution for automation is to make sure that this script can be run from the command-line. Based on the requirements, we need four (4) arguments.

  1. Image Source
  2. Image Destination
  3. Avatar Size
  4. Run Mode (PREVIEW or WRITE)

We set the available arguments and descriptions.

for opt, arg in opts:
    if opt == '-h':
        print ('\navatar.py -s [source] -d [destination] -a [avatar size] -m [run mode]\n'
                +'\nDescription of options and arguments\n\n'
                +'-s\t: source directory where the images are located\n'
                +'-d\t: destination directory where the processed images will be written to\n'
                +'-a\t: avatar size in pixels\n'
                +'-m\t: run mode PREVIEW (preview images), WRITE (writes images to destination)')
        sys.exit()
    elif opt in ('-s', '--src'):
        src = arg
    elif opt in ('-d', '--dest'):
        dest = arg
    elif opt in ('-a', '--avatar'):
        avatarSize = arg
    elif opt in ('-m', '--mode'):
        runMode = arg
    else:
        assert False, "Unhandled options"

We also make sure that if no value is assigned, we set default values.

  1. If the source is empty we set the source to the current working directory
  2. If the destination is empty we set the destination to the current working directory
  3. If avatar size is empty, the default size is 200px
  4. If the run mode is empty, it defaults to PREVIEW (displays image output on screen)
if src=='':
    src = os.getcwd()

if dest=='':
    dest = os.getcwd()

if runMode=='':
    runMode = 'PREVIEW'

if os.path.exists(src) and os.path.exists(dest):
    imgList = getImgList(src)
else:
    print ('Invalid source/destination directory.')
    sys.exit()

Below is a sample screenshot with the script avatar.py running with parameters. Screenshot

Automation

Finally, we are now ready for automation. My company uses Microsoft System Orchestrator 2012, but I am sure that there are other alternatives for File Watcher/Monitor folder processes. The setup is simple, Monitor Folder, is set to watch D:\FTP-Inbound\CCURE\ for any files.

Orchestrator

Then we set a 30 second delay before the Photo ID Image Processing Script is run. The delay was set to ensure that all images are completely transferred before processing. If the image processor fails or succeeds, my team is then notified of the failure so that we can address it.

Conclusion

I had no prior exposure to Python and OpenCV but the process of working through the problem then designing and developing a solution was definitely fun. The process is now productionized and run in conjunction with our weekly Workday Photo Integrations.

The Full Code

import cv2
import os
import sys
import getopt
import datetime

###################################
# Photo ID Auto Crop
# Sherwin Rubio
# @sherwinrubio
# Version 1.0
####################################

try:
    # Load Haar Cascade Classifiers
    faceCascade = cv2.CascadeClassifier('cascade\haarcascade_frontalface_alt2.xml')

    # Function to get image list from the source directory
    def getImgList(imgPath, imgExt=['jpg', 'bmp', 'png', 'gif', 'jpeg']):
        imagesArray = [fn for fn in os.listdir(imgPath)
        if any(fn.endswith(ext) for ext in imgExt)]
        return imagesArray

    # Finding a face in the image.  This image will return only one (1) face
    def findFace(img):
        grayImg = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
        face = faceCascade.detectMultiScale(
            grayImg,
            scaleFactor=1.3,
            minNeighbors=5,
            minSize=(30,30),
            flags = cv2.CASCADE_SCALE_IMAGE)

        # Logs images of which a face was not detected
        if face==():
            with open('logs\\facesnotfound'+fileStamp+'.log', 'a') as facesnotfound:
                facesnotfound.write('\''+imgName+'\''+'\n')
            x = 0
            y = 0
            if imgWidth<imgHeight:
                x2 = imgWidth
                y2 = imgWidth
                w = imgWidth
                h = imgWidth
            else:
                x2 = imgHeight
                y2 = imgHeight
                w = imgHeight
                h = imgHeight
            imageCoords = (x,y,x2,y2,w,h,0)
            return imageCoords
        else:
            for (x,y,w,h) in face:
                x2 = x + w
                y2 = y + h
            imageCoords = (x,y,x2,y2,w,h,1)
            with open('logs\\processed'+fileStamp+'.log', 'a') as processLog:
                processLog.write('\''+imgName+'\''+'\n')
            return imageCoords

    def definePadding():
            coords = findFace(img)
            h1,v1,x2,y2,fW,fH,foundFace = coords
            h2 = imgWidth-x2
            v2 = imgHeight-y2
            padThreshold = .15
            padding = 0

            # Determining the Primary Quadrant the Face is located
            if foundFace==1:
                if h1 <= h2:
                    hRelPos = h1/h2
                    hFlag = 'L'
                else:
                    hRelPos = (h2/h1)
                    hFlag = 'R'
                if v1 <= v2:
                    vRelPos = v1/v2
                    vFlag = 'U'
                else:
                    vRelPos = v2/v1
                    vFlag = 'B'
                if (hRelPos >=.80 and hRelPos <=1) and (vRelPos >=.80 and vRelPos <=1):
                    vFlag = 'C'
                    hFlag = 'C'

                # Assign the Pad Value depending on Face location 
                # and its proximity to the Source Image Borders
                if vFlag=='U':
                    padding = v1
                    if padding > h1:
                        padding = h1
                    if padding/fH > padThreshold:
                        padding = int(fH*padThreshold)
                elif vFlag=='B':
                    padding = v2
                    if padding > h2:
                        padding = h2
                    if padding/fH > padThreshold:
                        padding = int(fH*padThreshold)
                if hFlag=='L':
                    padding = h1
                    if padding > v1:
                        padding = v1
                    if padding/fH > padThreshold:
                        padding = int(fH*padThreshold)
                elif hFlag=='R':
                    padding = h2
                    if padding > v1:
                        padding = v1
                    if padding/fH > padThreshold:
                        padding = int(fH*padThreshold)
                if hFlag=='C' and vFlag=='C':
                    padding = int(fH*padThreshold)
                if vFlag=='U' and hFlag=='L':
                    if h1 > v1:
                        padding = v1
                    else:
                        padding = h1
                    if padding/fH > padThreshold:
                        padding = int(fH*padThreshold)
                if vFlag=='U' and hFlag=='R':
                    if h1 > v1:
                        padding = v1
                    else:
                        padding = h2
                    if padding/fH > padThreshold:
                        padding = int(fH*padThreshold)
                if vFlag=='B' and hFlag=='L':
                    if h1 > v2:
                        padding = v2
                    else:
                        padding = h1
                    if padding/fH > padThreshold:
                        padding = int(fH*padThreshold)
                if vFlag=='B' and hFlag=='R':
                    if h2 > v2:
                        padding = v2
                    else:
                        padding = h2
                    if padding/fH > padThreshold:
                        padding = int(fH*padThreshold)
                return (padding,v1-padding,y2+padding,h1-padding,x2+padding,foundFace)

            if foundFace==0:
                return (padding,v1,y2,h1,x2,foundFace)

    def createAvatar(paddingCoords,dest,avatarSize,mode):
        padding,y,y2,x,x2,foundFace = paddingCoords
        avatarSize = int(avatarSize)
        print ("Processing "+imgName+'to'+imgName[:6]+'.png')
        avatar = img[y:y2, x:x2]
        avatar = cv2.resize(avatar,(avatarSize, avatarSize))
        if mode=="preview" or mode=="PREVIEW":
            cv2.imshow('Avatar '+imgName,avatar)
            cv2.waitKey(0)
            cv2.destroyAllWindows()
        elif mode=="write" or mode=="WRITE":
            cv2.imwrite(dest+'\\'+imgName[:6]+'.png',avatar)

    src = ''
    dest = ''
    avatarSize = 200
    runMode = ''

    try:
        opts, args = getopt.getopt(sys.argv[1:],'hs:d:aⓜ️',['src=','dest=','avatar=','mode='])
    except getopt.GetoptError:
        print ('\navatar.py -s [source] -d [destination] -a [avatar size] -m [run mode]\n'
               +'\nDescription of options and arguments\n\n'
               +'-s\t: source directory where the images are located\n'
               +'-d\t: destination directory where the processed images will be written to\n'
               +'-a\t: avatar size in pixels\n'
               +'-m\t: run mode PREVIEW (preview images), WRITE (writes images to destination)')
        sys.exit(2)

    for opt, arg in opts:
        if opt == '-h':
            print ('\navatar.py -s [source] -d [destination] -a [avatar size] -m [run mode]\n'
                   +'\nDescription of options and arguments\n\n'
                   +'-s\t: source directory where the images are located\n'
                   +'-d\t: destination directory where the processed images will be written to\n'
                   +'-a\t: avatar size in pixels\n'
                   +'-m\t: run mode PREVIEW (preview images), WRITE (writes images to destination)')
            sys.exit()
        elif opt in ('-s', '--src'):
            src = arg
        elif opt in ('-d', '--dest'):
            dest = arg
        elif opt in ('-a', '--avatar'):
            avatarSize = arg
        elif opt in ('-m', '--mode'):
            runMode = arg
        else:
            assert False, "Unhandled options"

    if src=='':
        src = os.getcwd()

    if dest=='':
        dest = os.getcwd()

    if runMode=='':
        runMode = 'PREVIEW'

    if os.path.exists(src) and os.path.exists(dest):
        imgList = getImgList(src)
    else:
        print ('Invalid source/destination directory.')
        sys.exit()

    print ('------\n')
    print ('Processing Images from:',src)
    print ('Image Output Directory:',dest)
    print ('Avatar Size:',avatarSize)
    print ('Run Mode:',runMode)
    print ('------\n')

    if len(imgList)==0:
        print ('No images in '+src+' to process.')
    else:
        fileStamp = '{}'.format(datetime.datetime.now().strftime('%Y%m%d%H%M%S'))
        with open('logs\\processed'+fileStamp+'.log', 'a') as processLog:
            processLog.write('------\n'
                             +'Start:'+'{}'.format(datetime.datetime.now().strftime('%x %X')+'\n')
                             +'Processing Images From: '+src+'\n'
                             +'Image Output Directory: '+dest+'\n'
                             +'Avatar Size: '+str(avatarSize)+'\n'
                             +'Run Mode: '+runMode+'\n'
                             +'------\n')
        for imgName in imgList:
            img = cv2.imread(src+'\\'+imgName)
            imgHeight,imgWidth,imgChannel = img.shape
            createAvatar(definePadding(),dest,avatarSize,runMode)

        with open('logs\\processed'+fileStamp+'.log', 'a') as processLog:
            processLog.write('------\n'
                             +'End:'+'{}'.format(datetime.datetime.now().strftime('%x %X'))
                             +'  ['+format(len(imgList),"")+'] image(s) processed.\n'
                             +'------\n')
except KeyboardInterrupt:
    print ('\nImage Processing Interrupted')
    try:
        sys.exit(0)
    except SystemExit:
        os._exit(0)

Avatar
Sherwin Rubio
Business Intelligence Architect

My research interests include Business Intelligence and Data Supply Chain Architecture

Next