Cropping to faces

Sometimes you read a question on the internet and wonder “Why isn’t that built in functionality? It can’t be too hard to do.” And occasionally when you see one of those you decide to jump onto it and write it yourself. That was the rabbit hole I fell down tonight. It all started with this question on Stack Overflow. Android has built in face detection on images. Why can’t you automatically crop to them? And for some reason it piqued my interest enough to write some code.

First off, I wanted to set my goals for the program. I didn’t really care about actually cropping an image- that’s trivial. What was interesting was figuring out which part of the image to crop to. The actual cropping has been done hundreds of times by everyone including me. If I wanted to write a crop function later I will. For right now, I just want to be able to pass in an image with a single face and get back a rectangle surrounding that face.

My first couple of test programs were very simple- I wrote a classic image picker (find the code elsewhere, image picking on Android has been answered hundreds of times) and tried to use the FaceDetector class to recognize some faces in a selfie and in some old vacation photos. The face detector itself wasn’t too hard to use. I ended up with something like this:

        FaceDetector fd = new FaceDetector(bmp.getWidth(), bmp.getHeight(), 1);
        Face faces[] = new Face[1];
        int numFaces = fd.findFaces(bmp, faces);
        if(numFaces == 0){
        	return null;
        }
                        
        Face face = faces[0];
        face.getMidPoint(current);
        float eyeDistance = face.eyesDistance();

With this code running under the debugger so I could inspect the results. That… didn’t work so well. No matter what image I used it failed. After much googling I found the issue- the android facial recognition software will only work with 1 specific format of images. So, my function needs to be able to convert the format first. Alrighty then.

	Bitmap bmp = inBmp.copy(Bitmap.Config.RGB_565, true);
        FaceDetector fd = new FaceDetector(bmp.getWidth(), bmp.getHeight(), 1);
        Face faces[] = new Face[1];
        int numFaces = fd.findFaces(bmp, faces);
        bmp.recycle();
        if(numFaces == 0){
        	return null;
        }
        
        Face face = faces[0];
        PointF point = new PointF();
        face.getMidPoint(point);
        float eyeDistance = face.eyesDistance();

Notice the recycle call to make sure we don’t keep extra resources around. Now that we have the format right, it works. At least on some images. My beautiful selfie still failed, but apparently the face recognition software loves my vacation pictures of mom. Which is why I’m an internationally famous criminal and she’s not. Now we get to the interesting part- we have a face, but it doesn’t give us a lot more. All it really gives us is the location of the eyes (or rather, the midpoint between them) and the distance between the eyes. It also gives us the angle of the head, but that’s deeper than I want to get into right now. So how do we figure out the size of the head?

Here’s where a good programmer needs to be a trivia expert. Back in grade school, we were taught about the number phi- the golden mean. For me this meant a 2 week period of my teacher droning on about something that obviously fascinated her, but which she didn’t understand. It was taught as a mystical number that underlies everything in nature. Most of that is garbage and coincidence, but one thing did stick in my memory- various parts of the human face are supposed to be roughly in proportion to each other. Two of these things are the distance between your eyes and the size of your head. If we were to use this fact we can take that distance, scale it up, add a bit of a fudge factor (we’d need it for hair anyway) and we should be able to get the right size. After a bit of googling on proportions and playing with numbers, I found that a ratio of 4* the eye distance in x and 5 in y was just about right, although feel free to play with the numbers yourself. Here’s the code:

	public static Rect getSingleFaceRect(Bitmap inBmp){
		Bitmap bmp = inBmp.copy(Bitmap.Config.RGB_565, true);
        FaceDetector fd = new FaceDetector(bmp.getWidth(), bmp.getHeight(), 1);
        Face faces[] = new Face[1];
        int numFaces = fd.findFaces(bmp, faces);
        bmp.recycle();
        if(numFaces == 0){
        	return null;
        }
        
        Face face = faces[0];
        PointF point = new PointF();
        face.getMidPoint(point);
        float eyeDistance = face.eyesDistance();
        return new Rect((int)(point.x-2*eyeDistance),  (int)(point.y-2.5*eyeDistance), 
        		(int)(point.x+2*eyeDistance),  (int)(point.y+2.5*eyeDistance));        	
	}

Having gotten this, I wanted to increase the scope- what if there are multiple faces? Well, we’d need to keep track of which face is farthest in each of the 4 directions and make a rect big enough to hold all of them. It’s a fairly trivial addition to the code above. And that’s about all I need. A future improvement may be to take into account the angle of the head to keep the face centered, but for now I’m happy.

Full code:

package com.gabesechan.android.reusable.image;

import android.graphics.Bitmap;
import android.graphics.PointF;
import android.graphics.Rect;
import android.media.FaceDetector;
import android.media.FaceDetector.Face;

public class FaceDetection {

	public static Rect getSingleFaceRect(Bitmap inBmp){
		Bitmap bmp = inBmp.copy(Bitmap.Config.RGB_565, true);
        FaceDetector fd = new FaceDetector(bmp.getWidth(), bmp.getHeight(), 1);
        Face faces[] = new Face[1];
        int numFaces = fd.findFaces(bmp, faces);
        bmp.recycle();
        if(numFaces == 0){
        	return null;
        }
        
        Face face = faces[0];
        PointF point = new PointF();
        face.getMidPoint(point);
        float eyeDistance = face.eyesDistance();
        return new Rect((int)(point.x-2*eyeDistance),  (int)(point.y-2.5*eyeDistance), 
        		(int)(point.x+2*eyeDistance),  (int)(point.y+2.5*eyeDistance));        	
	}
	
	public static Rect getMultipleFaceRect(Bitmap inBmp, int maxFaces){
		Bitmap bmp = inBmp.copy(Bitmap.Config.RGB_565, true);
        FaceDetector fd = new FaceDetector(bmp.getWidth(), bmp.getHeight(), maxFaces);
        Face faces[] = new Face[maxFaces];
        int numFaces = fd.findFaces(bmp, faces);
        bmp.recycle();
        if(numFaces == 0){
        	return null;
        }
        
        PointF leftMost = new PointF(999999, 999999);
        PointF rightMost = new PointF(0, 0);
        PointF topMost = new PointF(999999, 999999);
        PointF bottomMost = new PointF(0, 0);
        
        float leftDistance = 0;
        float rightDistance = 0;
        float topDistance = 0;
        float bottomDistance = 0;
                
        PointF current = new PointF();
        for(int i=0; i< numFaces; i++){
	        Face face = faces[0];
	        face.getMidPoint(current);
	        float eyeDistance = face.eyesDistance();
	        if(current.x < leftMost.x){
	        	leftMost = current;
	        	leftDistance = eyeDistance;
	        }
	        if(current.x > rightMost.x){
	        	rightMost = current;
	        	rightDistance = eyeDistance;
	        }
	        if(current.y < topMost.y){
	        	topMost = current;
	        	topDistance = eyeDistance;
	        }
	        if(current.y > bottomMost.y){
	        	bottomMost = current;
	        	bottomDistance = eyeDistance;
	        }
        }
        return new Rect((int)(leftMost.x-2*leftDistance),  (int)(topMost.y-2.5*topDistance), 
        		(int)(rightMost.x+2*rightDistance),  (int)(bottomMost.y+2.5*bottomDistance));        	
	}

}

This Post Has Been Viewed 1,966 Times

Leave a Reply