Facial Landmark Detection

By | October 18, 2015
Facial Feature Detection

Facial landmark detection using Dlib (left) and CLM-framework (right).

Who sees the human face correctly: the photographer, the mirror, or the painter? — Pablo Picasso

If Picasso was alive today, he would have definitely added one more professional to that list — a computer vision engineer!

As computer vision engineers and researchers we have been trying to understand the human face since the very early days. The most obvious application of facial analysis is Face Recognition. But to be able to identify a person in an image we first need to find where in the image a face is located. Therefore, face detection — locating a face in an image and returning a bounding rectangle / square that contains the face — was a hot research area. In 2001, Paul Viola and Michael Jones pretty much nailed the problem with their seminal paper titled “Rapid Object Detection using a Boosted Cascade of Simple Features.” In the early days of OpenCV and to some extent even now, the killer application of OpenCV was a good implementation of the Viola and Jones face detector.

Once you have a bounding box around the face, the obvious research problem is to see if you can find the location of different facial features ( e.g. corners of the eyes, eyebrows, and the mouth, the tip of the nose etc ) accurately. Facial feature detection is also referred to as “facial landmark detection”, “facial keypoint detection” and “face alignment” in the literature, and you can use those keywords in Google for finding additional material on the topic.

Applications of Facial Keypoint Detection

There are several interesting applications of keypoint detection in human faces. A few of them are listed below.

Facial feature detection improves face recognition

Facial landmarks can be used to align facial images to a mean face shape, so that after alignment the location of facial landmarks in all images is approximately the same. Intuitively it makes sense that facial recognition algorithms trained with aligned images would perform much better, and this intuition has been confirmed by many research papers.

Head pose estimation

Once you know a few landmark points, you can also estimate the pose of the head. In other words you can figure out how the head is oriented in space, or where the person is looking. E.g. CLM-Framework described in this post also returns the head pose.

Face Morphing

Facial landmarks can be used to align faces that can then be morphed to produce in-between images. An example is shown in Figure 1.

Bush Schwarzenegger Morph

Figure 1. A morph between President Bush and the The Governator!

Click here for my followup post on Face Morphing.

Virtual Makeover

At my company ( TAAZ.com ) we had written our own facial landmark detector. The detected landmarks were used to the calculate contours of the mouth, eyes etc. to render makeup virtually. An example is shown in Figure 2.

Landmark detection for virtual makeover.

Figure 2. Landmark detection for virtual makeover.

Face Replacement

If you have facial feature points estimated on two faces, you can align one face to the other, and then seamlessly clone one face onto the other. You can also do something goofy like this

https://auduno.github.io/clmtrackr/examples/facesubstitution.html

Click here to read my followup post on Face Swap

In a previous post, we showed how to use facial features to predict facial attractiveness.

Clearly, the ability to detect facial features in images and videos open up possibilities of a ton of interesting applications. Let us now get our hands dirty with some tools that will allow us to do this.

Facial Feature Detection & Tracking Libraries

There has been a flurry of activity in this area in the last 5 years. Part of the reason for this activity is the availability of large annotated datasets like LFPW and Helen. I have listed a bunch of papers in the next section. However, I do not recommend implementing these papers from scratch because now we have access to high quality open source implementations of some of these papers.

In the video below, you can see two of the libraries, Dlib and CLM-framework in action.

Dlib ( C++ / Python )

Dlib is a collection of miscellaneous algorithms in Machine Learning, Computer Vision, Image Processing, and Linear Algebra. Most of the library is just header files that you can include in your C++ application. Oh you prefer python ? No problem, it has a python API as well.

I personally like Dlib more than any other facial feature detection & tracking library because the code is very clean, well documented, the license permits use in commercial applications, the algorithm they have chosen to implement is very fast and accurate, and you can easily integrate the library in your C++ project by just including the header files.

How to compile Dlib ?

  1. Download a copy from github
    git clone https://github.com/davisking/dlib.git
    
  2. Build examples ( OSX / Linux )
    cd dlib/examples
    mkdir build
    cd build
    cmake .. 
    cmake --build . --config Release
    

    These examples are a great way to start using dlib. Make a copy of an example cpp file, modify it, modify examples/CMakeLists.txt and compile again using the instructions above. Easy!

  3. Compile dlib python module
     
    cd dlib/python_examples 
    ./compile_dlib_python_module.bat
    
  4. Set PYTHONPATH environment variable
    # Put the following line in .bashrc or .profile
    export PYTHONPATH=/path/to/dlib/python_examples:$PYTHONPATH
    
  5. Test python module
     python -c "import dlib"
    

If the above line does not give an error, you are all set.

In case you run into any compilation issues, there are additional instructions at Dlib.net

How to run Dlib’s facial landmark detector ?

After you have built the examples, to run the facial landmark detector using a webcam, do the following.

 
cd examples/build/
#Download the face landmark model 
wget http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2
tar xvjf shape_predictor_68_face_landmarks.dat.bz2
./webcam_face_pose_ex

If you want to run it on a single image, you can try

 
./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat faces/*.jpg

CLM-Framework (C++)

CLM-framework, also known as the Cambridge Face Tracker, is a C++ library for facial keypoint detection and head pose estimation. You can see how well it works in the included video. Compiling this library on OSX was bit of a challenge but it was not too bad. The library depends on OpenCV 3 and requires X11.

There are two important ways in which Dlib beats CLM-Framework. First, DLib is much faster than CLM-Framework. Second, Dlib’s license allows you to use it in commercial applications. If I had to pick, I would use Dlib. Interestingly, CLM-Framework depends on Dlib!

How to compile CLM-Framework ?

Compiling CLM-Framework was a bit involved for OSX. For windows and linux there are detailed instructions here. For compiling version 1.3.0 on OSX, I used the instructions for linux but made the following changes.

Most of the dependencies were installed using brew.

In file CMakeLists.txt ( red text was replaced with green text ).

find_package( OpenCV 3.0 REQUIRED )

find_package( OpenCV 3.0 REQUIRED HINTS /path/to/opencv )
INCLUDE_DIRECTORIES(${OpenCV_INCLUDE_DIRS})
INCLUDE_DIRECTORIES(/opt/X11/include)

In file exe/SimpleCLM/SimpleCLM.cpp

writerFace = VideoWriter(tracked_videos_output[f_n], CV_FOURCC(‘D’,’I’,’V’,’X’), 30, captured_image.size(), true);
writerFace = VideoWriter(tracked_videos_output[f_n], CV_FOURCC(‘M’,’P’,’4′,’V’), 15, captured_image.size(), true);

How to run CLM-Framework’s Facial Feature Detector ?

After compiling CLM-Framework, the executables are in the bin directory. For the webcam demo shown in this post, you can use

bin/SimpleCLM

Face++ ( FacePlusPlus ) : Web API

One of the best implementations of facial landmark detection is by FacePlusPlus. They won the 300 Faces In-the-Wild Landmark Detection Challenge, 2013. They provide an easy to use API. The problem is that you need to upload an image to their servers and that raises a lot of privacy concerns. But if privacy is not an issue, Face++ is very good option. You can see a demo at

http://www.faceplusplus.com/demo-landmark/

Facial Feature Detection Research

Many different approaches have been used to solve this problem and it is difficult to summarize them in a blog post. I am simply linking to some important papers ( with major bias toward recent work ) for people who want to investigate more.

  1. Active Appearance Model (AAM) by T. Cootes, G. Edwards and C. J. Taylor. [1998]
  2. Face Alignment through Subspace Constrained Mean-Shifts by Jason M. Saragih, Simon Lucey and Jeffrey F. Cohn. [2009]
  3. Localizing Parts of Faces Using a Consensus of Exemplars by Peter N. Belhumeur, David W. Jacobs, David J. Kriegman, Neeraj Kumar [ 2011 ]
  4. Face Alignment by Explicit Shape Regression by Xudong Cao Yichen Wei Fang Wen Jian Sun [2012]
  5. Supervised Descent Method and Its Applications to Face Alignment by Xuehan Xiong and Fernando De la Torre [2013]
  6. Constrained Local Neural Fields for robust facial landmark detection in the wild by Tadas Baltrusaitis, Peter Robinson, and Louis-Philippe Morency. [2013]
  7. Extensive Facial Landmark Localization with Coarse-to-fine Convolutional Network Cascade by Erjin Zhou, Haoqiang Fan, Zhimin Cao, Yuning Jiang and Qi Yin. [2013]
  8. Face alignment at 3000 fps via regressing local binary features by S Ren, X Cao, Y Wei, J Sun. [2014]
  9. Facial Landmark Detection by Deep Multi-task Learning by Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. [2014]
  10. One Millisecond Face Alignment with an Ensemble of Regression Trees by Vahid Kazemi and Josephine Sullivan. [2014]

Subscribe

If you liked this article, please subscribe to our newsletter and receive a free
Computer Vision Resource guide. In our newsletter we share OpenCV tutorials and examples written in C++/Python, and Computer Vision and Machine Learning algorithms and news.

Subscribe Now

  • Nice write-up!
    I actually explored a large number of open-source facial landmark detectors for a project, and found the CLM framework to outperform everything else (in terms of both speed and accuracy). We eventually used it in our project: http://www.technologyreview.com/news/541866/this-car-knows-your-next-misstep-before-you-make-it/

    Some other interesting ones:

    http://www.ics.uci.edu/~xzhu/face/
    Very slow (~10 seconds an image after hyper threading on a 8-core CPU), but very accurate when it comes to high pose variations

    https://github.com/patrikhuber/superviseddescent
    Nicely written C++ code, though not very robust

    http://www.vision.caltech.edu/xpburgos/ICCV13/
    Specially designed for handling occlusions, but slow on account being written in MATLAB.

  • mark allard

    Great Article

    Everything installed on Ubuntu14.04.3 without a hitch. the example web application here: https://auduno.github.io/clmtrackr/examples/facesubstitution.html worked well with the web cam on my laptop howver the example: ./webcam_face_pose_ex did not, it produced:

    ~/dlib/examples/build$ ./webcam_face_pose_ex

    Error detected at line 36.

    Error detected in file /home/mark/dlib/dlib/../dlib/opencv/cv_image.h.

    Error detected in function dlib::cv_image::cv_image(cv::Mat) [with pixel_type = dlib::bgr_pixel].

    Failing expression was img.depth() == cv::DataType<typename pixel_traits::basic_pixel_type>::depth && img.channels() == pixel_traits::num.

    The pixel type you gave doesn’t match pixel used by the open cv Mat object.

    img.depth(): 0

    img.cv::DataType<typename pixel_traits::basic_pixel_type>::depth: 0

    img.channels(): 1

    img.pixel_traits::num: 3

    The following worked well with the image set from the lfw:

    ./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat faces/*.jpg

    again, thanks for posting, much enjoyed
    Mark

    • dlib uses OpenCV for the webcam demo. Are you sure your webcam works with OpenCV ?

  • Nilanjan Sahu

    plz plz plz …. write a post on aligning one face to another.i learnt alot from your post. i ‘m abhiginner in open cv want implement this thing in my face reco. program

  • Hank White

    Hi,I want to build examples on mac using comment line “cmake –build .”but I got an error like this:

    /Users/xxh/dlib/dlib/gui_widgets/nativefont.h:29:10: fatal error:
    ‘X11/Xlocale.h’ file not found

    I don’t know how to fix it.

  • Sun Steven

    Hi Satya, not only face morphing, how about virtual makeover? Could you share about that in following post?

    • Unfortunately, virtual makeover is something I cannot write about because my company would not like me to reveal our secret sauce :). It took years of fine tuning.

      • Sun Steven

        Oh, yes sorry fot that. I’m a CV engineer and want to make a makeover application at leisure time. So could you introduce some papers about this topic? or just some can-use but simple projects? Thank you!

  • pipi

    Hi Satya, why not write a post on aligning one face to another 🙂 How to operate on the hair?

  • sarmad mueen

    Hi Satya, how can I detect lip movements ?I’m beginner and struggling
    to understand it

    • Download dlib, you can track the landmarks. Compare the location of lip landmarks between two frames ( or maybe frames that are 10 frames apart ). This should tell you if the lip is moving.

  • tzatter

    you didn’t list up ofxFaceTracker https://github.com/kylemcdonald/ofxFaceTracker
    what a difference with this

    • ofxFaceTracker is based FaceTracker. I did try it out ( see my comment below ) and it did not give results anywhere close to the two approaches I have mentioned in this post.

  • Hamid Kavianathar

    excuse me, I want to do a face swapping. I detect the face with Viola Jones method, nut I don’t know how I should change it. could you please help me? I need a robust algorithm.

    • Face swapping requires many steps.
      1. Use the landmark detector used in this post to detect landmarks in two faces.
      2. Use delaunay triangulation to obtain a triangulation of the points.
      3. Warp corresponding triangles from one face onto the other.
      4. Seamlessly clone the warped face into the new image.
      At some point I will write a post describing these steps in detail.

      • Hamid Kavianathar

        thanks for your reply. excuse me, could you please help me more? what their functions in opencv?

  • Diego Moreira

    It can use the library “dlib” along with the opencv library?

    There is a tutorial installation of dlib library with visual studio 2012?

  • ricardo

    Hello, i just looked into your post which is very interesting by the way, i am trying to compile the “dlib” library, i downloaded but when i try to run it, the example of face landmark detection it says that i do not have JPEG support, but i defined it on my source, i included the path to dlibexternallibjpeg on the include directory and also added all the files of that folder to my project together with the source.cpp file on the dlib folder, all of this on visual studio.
    do you know maybe what i am doing wrong?
    i wait for your answer. Regards from Mexico
    Ricardo

  • Nigel Lima

    Hi, thanks for the article, it was really helpful.
    I got to run
    the Dlib library and the project suggested in the article. Everything
    went well. However, there is a huge difference between the frame rate
    showed in your video and the example I ran on my computer. Is there any
    config related to that or is it a matter of hardware?

    Thanks in advance

    • First make sure you are NOT compiling with debug turned on. ( see here http://dlib.net/faq.html#Whyisdlibslow ). Second, most of the time is spent in displaying the lines on the video. If you don’t care about the display, turn it off and you will see a huge improvement. Third, dlib scales the frame by 2 and because this image is huge now, the processing speed decreases. Turn that off. Finally, I rewrote the display using OpenCV ( drawline ) and got further improvement. Hope this helps.

      • Nigel Lima

        hi, Satya. Thanks for your answer.

        I actually compiled it using cmake and ran this on the OSX terminal. By the way, I would like to compile it to Xcode project and use the landmark detection in another larger project, but I’m quite new to this image processing thing. Could you give me a hint of how compiling this to xcode?

        Best,

    • Also a lot of time is spent in face detection ( and not landmark detection ). You can cheat and detect faces ever 5th frame but run the landmark detector on every frame using the location of the last detected face.

  • DavidBlancarte

    Hi!! Thanks for the information !
    I am trying to create a lie detector with the movement of the iris so i hope this library let me detect and follow the iris

  • ricardo

    hi! can you use dlib on an existing project within XCode?

    • Rafael Vareto

      I used CMake in order to make it work with Xcode.

  • Phani Ivatury

    Has any one applied the techniques for other deformable objects other than faces?

    • I have applied a similar technique to other deformable objects. We were dealing with depth data. Unfortunately due to my confidentiality agreement I cannot talk about it in detail, but I am reasonably certain that this technique will work on other objects pretty well.

      • Phani Ivatury

        Thanks for your reply. I am trying extract keypoints to set of rigid objects (with lots of corners). I have to tune the parameters. Btw, how many labelled images does it generally require?

  • Phani Ivatury

    Thanks! Satya. I am trying extract keypoints to set of rigid objects (with lots of corners). I have to tune the parameters. Btw, how many labelled images does it generally require?

    • It depends a lot on the data. E.g. for facial landmark detection > 1000 images usually gives good results. But if you want a lot of non-frontal variations you can use about 2000 images. You can start with a few hundred images and have a large test set. If the algorithm fails on a test image, put that test image in the training set and train again after you have collected a hundred or so hard examples. This way you can systematically improve the quality of your results.

      • Phani Ivatury

        Thanks!

  • Rafael Vareto

    I’ve been trying to use train_shape_predictor_ex from Dlib, however it’s not working properly. In the xml file I have a list of images (each contain 68 fiducial points following the oder displayed in the attached picture). When I run face_landmark_detection_ex I get all points mixed up. Does anybody know the the fiducial points are arranged in wrong order?

    • Nigel Lima

      How did you get the result of the last image? The one including the landmarks and their respectively point numbers?
      Thanks in advance

  • Phani Ivatury

    You might want try the imglab tool in the dlib tools to check if the points are in the same sequence. https://github.com/davisking/dlib/tree/master/tools/imglab

    • Rafael Vareto

      The images provided in the Dlib examples/faces folder are too small to see all facial landmarks in the imglab tool, however, they seem to be in the exactly same order. I just have no clue why they still look strange when I run face_landmark_detection_ex. The picture and the a XML file sample I’ve generated can be downloaded, respectively, at https://dl.dropboxusercontent.com/u/12547094/001_01.jpg and https://dl.dropboxusercontent.com/u/12547094/testing_with_face_landmarks_small.xml

    • Rafael Vareto

      The images provided in the Dlib examples/faces folder are too small to see all facial landmarks in the imglab tool, however, they seem to be in the exactly same order. I just have no clue why they still look strange when I run face_landmark_detection_ex. The picture and the a XML file sample I’ve generated can be downloaded at https://dl.dropboxusercontent.com/u/12547094/Archive.zip

      • Phani Ivatury

        Is the mixing up consistent for all images? Are the points locations correct?

  • Anna Zhao

    How would you go about trying to find the top of the forehead? (Sorry I’m a beginner in CV)
    I’d like to see a post on face morphing and substitution!

  • Phani Ivatury

    How about trying to use annotations to do that as well. There might be other approaches to do that but I think this might be more relevant to the discussion on the thread,

  • baokun zhang

    the step “std::vector faces = detector(cimg);” is too
    cost time, need 25 seconds , but your demo is so fast, would you tell me
    the reason, thank you very much.

  • baokun zhang

    hello how can i put the “PTS” into the “train.xml” by the imglab

  • cefoot

    Hi, I’m loving to follow your writings and it would be great if you could write an article about aligning faces 🙂

    • Thanks. Check out my post on face morphing ( http://www.learnopencv.com/face-morph-using-opencv-cpp-python/ ). With small modification, it will work for face alignment and face averaging.

      • baokun zhang

        when i build the train.xml ,the imglab only read the “*.jpg” but not “*.pts”, could you tell me how to put the annotations of the “*.pts” in the “train.xml ” by the imglab, or I should be write a new program for that, thanks。

        • Baokun : I have trained my own model but I can’t remember the details. I will look it up over the weekend and let you know.

          • baokun zhang

            thank you, I want train the model on the helen subset .the sunset includes “*.jpg” and “*.pts”

  • Rafael Vareto

    Is it possible to use http://dlib.net/train_shape_predictor_ex.cpp.html and http://dlib.net/face_landmark_detection_ex.cpp.html to training with different landmark numbers? Both detect 68 landmarks in a frontal face by default. I would like to detect 39 landmarks only (I can train all right using the train_shape_predictor_ex however face_landmark_detection_ex outputs the attached message.

    • Sorry for this very late reply. You have to change face_landmark_detection_ex.cpp so that it renders only 39 points and not 68 points.

  • xiao9377

    I use qt on windows.I have included all header files and added source.cpp. What’s reason with the error?
    #error “This file has been replaced. Instead you should add dlib/all/source.cpp to your project”

  • Sara Sara

    Hi I am currently working on a project where I have to extract the facial expression form video and images.
    I tried Dlib but i didn’t know how to extract the coordinates of the corners of the eyes and mouth . Can you please help me.
    Thanks

    • Sara Sara

      and can you please explain to me this output

      • Sara Sara

        are those the positions of the point compared to all the image or just the face??

      • son nguyen minh

        when you detect facial landmark, you will receive 68 points. The points is fixed for every face. (i think so, because i check it with 3 faces). In your console, it only show 2 first points in received 68 points, and you can show more. To find the corners , you can choose left_most point and right_most point.

        • Sara Sara

          after calculating the distance between the points for the neutral face and for the face with an expression, how do we classifiy the results (how do i know witch emotion does the face express) ?

          • son nguyen minh

            Hi sara. What do you use for learning algorithms? I think you need learning a distance area for each emotion face. If not, why don’t you use classification problem with superviser.
            p/s: i am only a student.

          • Sara Sara

            i am just a newbie on this field so i don’t know exactly what to do next. should i use SVM for learning and what is the input for the learning ?

          • son nguyen minh

            input is vector of faces. you try to use sift descriptor for each points which you just used facial landmark. Then, you classify all them by SVM.

          • bob

            EmoNets: Multimodal deep learning approaches for emotion recognition in video

            http://arxiv.org/pdf/1503.01800v2

    • son nguyen minh

      Hi. I just check with 2 faces. Result is:
      sequence point: 17 oranges –>10 eyes brown –> 9 nose –>12 eyes–>20 mouth point == 68 point.

  • bob

    EmoNets: Multimodal deep learning approaches for emotion recognition in video

  • aakash nandi

    Thank you Sir. Your instructions and explanations are really lucid and easy to follow.

  • aakash nandi

    Sir,I am complete newbie to OpenCV. I have managed to configure code-blocks to compile and run OpenCV programs from

    http://jonniedub.blogspot.in/2013/01/setting-up-codeblocks-ide-for-use-with.html

    However i didn’t find any step by step instruction to configure it with the CLM Framework. Could you please help me.

  • Bigelow

    Hello.
    Are there analogs dlib on java?

  • Gopi Jayaraman

    Hi, using dlib, can we able to save the image with face landmark overlay? I see it only displays in screen.
    I am trying to get the face landmark from dlib then erase some of the facial features like eye brow, eye, mouth, … and fill that erased location with skin color.

  • 李凯

    Thanks for your great post, I have one question

    Facial landmarks can be used to align facial images to a mean face shape, so that after alignment the location of facial landmarks in all images is approximately the same.

    How to transform all faces to the mean face?
    How to define the mean face shape?
    Is there some code about this?
    Thanks!

  • hungbv

    Thank you Sir. I’m newbie of OpenCV. Could you please write a tutorial using dlib for iOS?

    • Thats a great idea. I have been thinking about it for a while. Let me see if I can do that in the next month or two.

      • hungbv

        Thank Sir. Hope you could write it as soon as possible

      • Virut Sontisuk

        I have been waiting

  • Ahlem AH

    thank you sir verry verrryy mutch your post was verry helpfull for me, and i will be glad if you can answer me:
    i need to get the x or y coordinate separately for exemple when i write cout << shape.part(38)<< endl; it gives (160.115) but i want to have the 160 .
    thank you sir verry much again and i wish good continuation for you.

    • Thanks.

      You can just round the number round(shape.part(38))

      http://www.cplusplus.com/reference/cmath/round/

      • Ahlem AH

        thank you again sir thank you verry much !!!!!

      • Ahlem AH

        it does not work for me :'( HELLLP PLEASE !!
        when i tried with cout <<" the coordinate :" << round(shape.part(38))<< endl
        i get x ans y coordinate together
        but need to have x and y separately to continue my work HELLLP PLEASE !!

  • Anand Marattukalam

    Hi Satya, thanks a lot for your wonderful post.
    I am a beginner in opencv, so what I may be asking could be stupid, but is there a way to find the type of nose(pointed, hooked), color of the eyes, hair etc?, if so could you kindly post a way of how to do it

    • Phani Ivatury

      I wonder how DL approaches would perform for a custom classification task as opposed to the traditional CV approaches. How different would the algorithm be if the classification task /labels change.

      • Anand Marattukalam

        Thanks for the reply sir, I didn’t quite understand what you meant, again I am a beginner in image recognization, I just wanted to know if it was feasible to categorize a nose(pointed, Roman), color of eyes, hair etc.

        • Phani Ivatury

          Sorry. Pls ignore my reply

        • Yes it is possible. To do that you have to get several images of pointed nose, Roman nose and other kinds of noses you want to classify. These days the state of the art algorithms for image classification are based on Deep Learning ( Google it ). There are some free frameworks to do Deep Learning. You can see my post on NVIDIA DIGITS here. The post is for beginners in mind.

          http://www.learnopencv.com/deep-learning-example-using-nvidia-digits-3-on-ec2/

          • Anand Marattukalam

            Thanks for the reply sir, I deeply appreciate it

  • Phani Ivatury

    Which of these approaches are the fastest on android if we could live with some loss in accuracy? Are there any open-source implementations for the same?

  • Arjun k m

    how can i use this landmark for facial emotion detection

    • Daniel Colceag

      You can use the library to get the facial landmarks, then match them to FACS (Facial Action Coding System). You can quantize the strength of the Action Unit between -1 and 1. Combinations of Action Units are marked as facial expressions. Use a database of emotion labeled portraits to extract the features of each emotion. Train a classifier to learn the emotion features and use it to classify on-line.

      • Arjun k m

        Thank

    • Phani Ivatury

      Here is another way to do it without using landmarks, https://github.com/mihaelacr/pydeeplearn

      • Arjun k m

        Thank you:)

  • Teresa Kozera

    Hello 🙂 I would like to print something on the screen using dlib functions (iOS project). Unfortunately, for some reason I can’t include in my project ‘dlib/gui_widgets.h’- I am getting the errors: ‘DLIB_NO_GUI_SUPPORT is defined so you can’t use the GUI code. Turn DLIB_NO_GUI_SUPPORT off if you want to use it.’ and ‘Also make sure you have libx11-dev installed on your system’. Do you have any other tip how I could print the coordinates of multiple points close to that points (drawn by ‘using draw_solid_circle’ function)? I would appreciate any help. 🙂

  • Sanjay Shreedharan

    I am successfully done with “cmake –build . –config Release”, but still am not able to find “./compile_dlib_python_module.bat” in the folder “dlib/python_examples “. Could you please help me with it ?

    • Phani Ivatury

      You could use pip too. I am not sure if you could specify the optimization flag through pip though!

    • Colin

      +1 I am having the same issue. There is no ‘compile_dlib_python_module.bat’ file in ‘dlib/python_examples’. What gives?

      • soyoung shin

        Thank you.!

    • soyoung shin

      did you solve this problem? I also have same issue..

  • Maurizio

    Hi Satya, is there a way to extract the head pose (yaw, pitch and roll) from dlib’s result?

  • Shubh Agrawal

    Please edit this line :
    ./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat faces/*.jpg

    this gives error of faces not being found. Since the executable is placed within build folder, the path to dataset becomes ../faces/*.jpg

    Hence,
    ./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat ../faces/*.jpg

    Please correct me if I am wrong. I experienced this issue.

  • jonas

    Hi Satya,

    thanks for providing code to all your projects, I appreciate that!
    Did you by any chance check out this project: https://github.com/ishay2b/VanillaCNN and know anything about performance and accuracy?

    Since detecting the head pose of a profile face is quite hard for this algorithm (it works only when both eyes are visible). I was thinking to expand this tracking algorithm using multiple cameras (I think my application differs a bit). What would be your approach when using two or more cameras. Calibrating and independently evaluating and then comparing? That seems CPU-heavy. Any ideas or things you can point me to?
    Many thanks!

    • We briefly looked at Deep Learning based approaches for Facial Landmark Detection but have not used dug deep into performance etc.

      Multiple cameras add a lot of complexity. Before I answer that question, what application are you thinking about ?

  • Perikles Gkiokas

    Hello,
    I am building an AAM (I noticed that AAMs are part of your research) , so I would like to ask you if I can use the Dlib library on Matlab in order to extract the shape parameters of a landmarked area from a picture.
    Thank you.

    • Dlib is pure c++ and you can use mex to integrate it with MATLAB.

  • Mo

    Hi Satya,

    Really awesome library and thank you for you contribution.I’m implementing Dlib and it looks promising!
    Is there any way to also detect hairline edge with Dlib? I tried to look into this but couldn’t find the right place. Appreciate if you can point me to the right direction.

    Thank you

    • Hairline is tough. But if the hair is always pulled back, you can train a new landmark detector with a few points on the hairline.

      • Caroline Wang

        Hi Satya, would love to learn how to train landmark detector for hairline. May you share some advice? Thank you!

  • thinh

    Can I get more than 68 points of facial landmark. I see that in Face++, they can get 83 points. Is there any library help me find 83 points or even more (for example: some points on forehead)? Thanks!

  • Michelle

    Hey Satya, thanks for the great post and the instructions! This is so interesting that I wanted to try it myself. So I followed your instructions, and tried to build examples on OSX el capitan but failed. At the step “cmake –build . –config Release”, I get an error:
    “[ 87%] Building CXX object CMakeFiles/dnn_imagenet_train_ex.dir/dnn_imagenet_train_ex.cpp.o
    In file included from /Users/HomeFolder/dlib/examples/dnn_imagenet_train_ex.cpp:12:
    In file included from /Users/HomeFolder/dlib/dlib/../dlib/dnn.h:14:
    In file included from /Users/HomeFolder/dlib/dlib/../dlib/dnn/layers.h:8:
    /Users/HomeFolder/dlib/dlib/../dlib/dnn/core.h:2444:45: fatal error: recursive template instantiation exceeded maximum depth of 256
    using next_type = typename std::remove_reference::type;”

    Do you happen to know why? I googled and looked at many posts from stackoverflow and none seems to be relevant to this issue. Thanks a lot!

  • tosarm

    How can I save eye area landmarks ?

  • Khương Thịnh Kiều

    Hello Satya Mallick,
    Thank for your sharing.

    I’ve built and run successfully some examples of Dlib such as face_detection_ex, face_landmark_detection_ex, webcam_face_pose_ex and your own example webcam_face_pose_fast.

    But, on my PC, the examples don’t run smoothly as your above video. When I try to track facial landmark using webcam, it’s really slow. Maybe, just 3 or 4 frames per second.

    What should I do to improve the performance? How can I allow the examples to use my GPU?
    My System Information:
    + Windows 8.1 Pro
    + CPU i5-2500 3.3 GHZ (4 CPUs)
    + RAM 8G
    + GPU NVIDIA GeForce GTX 670

    Thank you,

  • Minu

    Hello there! Nice work!
    The thing is i’m trying to get the landmarks from a video input (like a webcam), but i’m stucked on finding the code for Dlib-Python video landmark program (like the one you’re using in the video). All i found are designed for images. Can you please be kind and help me with some fast instructions or code advice?
    Thanks!

  • Kevin Wang

    Nice library! I finally managed to run “./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat ../faces/*.jpg”. I can see the number of parts (which is 68) on screen. But where did all these landmarks go? Do I can get to store these information? Thanks!

  • Infa tum

    Greetings. I’ve followed the instructions, but don’t know why there are no such ./webcam_face_pose_ex file. I run ls and get this list: 3d_point_cloud_ex matrix_expressions_ex
    assignment_learning_ex max_cost_assignment_ex
    bayes_net_ex member_function_pointer_ex
    bayes_net_from_disk_ex mlp_ex
    bayes_net_gui_ex model_selection_ex
    bridge_ex mpc_ex
    bsp_ex multiclass_classification_ex
    CMakeCache.txt multithreaded_object_ex
    CMakeFiles object_detector_advanced_ex
    cmake_install.cmake object_detector_ex
    compress_stream_ex one_class_classifiers_ex
    config_reader_ex optimization_ex
    custom_trainer_ex parallel_for_ex
    dir_nav_ex pipe_ex
    dlib_build pipe_ex_2
    dnn_imagenet_ex quantum_computing_ex
    dnn_imagenet_train_ex queue_ex
    dnn_inception_ex random_cropper_ex
    dnn_introduction2_ex rank_features_ex
    dnn_introduction_ex running_stats_ex
    dnn_mmod_dog_hipsterizer rvm_ex
    dnn_mmod_ex rvm_regression_ex
    dnn_mmod_face_detection_ex sequence_labeler_ex
    empirical_kernel_map_ex sequence_segmenter_ex
    face_detection_ex server_http_ex
    face_landmark_detection_ex server_iostream_ex
    fhog_ex shape_predictor_68_face_landmarks.dat
    fhog_object_detector_ex sockets_ex
    file_to_code_ex sockstreambuf_ex
    graph_labeling_ex std_allocator_ex
    gui_api_ex surf_ex
    hough_transform_ex svm_c_ex
    image_ex svm_ex
    integrate_function_adapt_simp_ex svm_pegasos_ex
    iosockstream_ex svm_rank_ex
    kcentroid_ex svm_sparse_ex
    kkmeans_ex svm_struct_ex
    krls_ex svr_ex
    krls_filter_ex threaded_object_ex
    krr_classification_ex thread_function_ex
    krr_regression_ex thread_pool_ex
    learning_to_track_ex threads_ex
    least_squares_ex timer_ex
    linear_manifold_regularizer_ex train_object_detector
    logger_custom_output_ex train_shape_predictor_ex
    logger_ex using_custom_kernels_ex
    logger_ex_2 video_tracking_ex
    Makefile xml_parser_ex
    matrix_ex

    I’ve run all the instructions, which you have gave bellow((( What am i doing wrong?

  • Marium Hashmi

    this might be a silly question but how to run commands like these on cmd?
    ./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat faces/*.jpg
    i m trying to run
    ./face_detection_ex faces/*.jpg
    but it says
    https://uploads.disquscdn.com/images/f08dde471997d751e8f59f934236d7abdedca58c82d5844e2d2d82bf4a369503.jpg

    • krazy

      Hello @Marium it seems you’re trying to run those examples on Windows however these instructions are all for linux or mac osx. If you have built the examples on windows however you may try emitting the ./ from the begining and add .exe to the end of the program name once you are in the same folder as the examples. Just check in the folder to make sure that the examples have .exe at the end of their names otherwise they will not run on windows. If you need anymore help please run the command dir in cmd and post a screenshot of the output as a reply.

  • Birendra Kumar Singh

    I am not able to successfully install dlib in my system.
    I was able to make and get the dlib.so file generated. However, when I try to import dlib in my python code – I get the below error
    ImportError: /usr/local/lib/python2.7/dist-packages/dlib-19.2.99-py2.7-linux-x86_64.egg/dlib/dlib.so: undefined symbol: _ZN4dlib12pixel_traitsINS_9rgb_pixelEE3numE

    • Phani Ivatury

      Thats a good one! Can you nm the dlib.so and see. You might also want to use C++filt to find the function prototype. It could be the order specified but thats strange.

  • Ekkasit Exit Pinyoanuntapong

    Hi SATYA MALLICK ,
    Thanks for your great tutorial. But one thing I want to know. There is any example for export landmark in text file (for using with your Faceswap tutorial)?

  • Hoang Nguyen

    Hi Satya,
    Thanks for your greate post! I have tested Dlib and found that it was really fast as I need. However, the face landmarks detected were jumping alot. Have you fix that problem? or could you recommend me how to fix it?

    • Thanks Hoang.

      Here are a few suggestions
      1. First check if the face detector bounding box is jumping around a lot. If so, you should not detect the face in each frame and use tracking instead ( Try MEDIANFLOW, http://www.learnopencv.com/object-tracking-using-opencv-cpp-python ). Alternatively you can use Kalman Filter to track the center of the box and that will smooth things out.

      2. If the face box is constant, but the points are still jiggling you can use feature trackers. E.g. you can use calcOpticalFlowPyrLK to track a feature point. If the point tracked by optical flow and the landmark detector are within some threshold, you can use choose the location predicted by optical flow, else use the landmark detector result ( because the point might have moved significantly ).

      Hope this helps.

      • Hoang Nguyen

        Thanks Satya.
        I think that using Optical Flow will be too slow with mobile device. However, the first suggesstion is a good idea. I will try this!

        • Keep in mind this is sparse optical flow. You are not calculating the flow on the entire image but only on 68 points using KLT .

          • Hoang Nguyen

            Yep, I will try both suggesttions and reply you about the results. Thank you very much!

  • Simon Hill

    Hi, great article ! I’m wondering what kind of computer power is needed to run 24/7 facial landmarks calculations. I’m planning on runnning Debian, do I need a lot of horsepower ? Thanks a lot !

    • You don’t need a lot of horsepower. I can run this on Intel Core i7 ( 2.5 GHz ) in real time and if I don’t bother with the display of the image it is much much faster. The landmark detection is very fast 1000 fps and bottleneck the face detector needed to initialize it.

  • Priyank Sharma

    I want to querry landmark points using f200 realsense camera…Please help me on how to do it

  • Hola no me funciona para python ayuda por favor

  • Joel Garcia

    hey guy, how can i make a lie detector with openCV? is there an algorithm? im using java. thanks

    • No, lie detection is not easily solvable using visual cues alone.

  • Grina Wiem

    Hi, after the
    facial landmark detection is there a way to extract the expression (happy , sad, anger … ) from dlib’s result?

  • soyoung shin

    Please help.. I have this problem..(ubuntu14.04)

    sophie@sophie-MacBookAir:~$ cd dlib/python_examples
    sophie@sophie-MacBookAir:~/dlib/python_examples$ ./compile_dlib_python_module.bat
    bash: ./compile_dlib_python_module.bat: No such file or directory

  • soyoung shin

    Hello. I stuck.. please help.!

    sophie@sophie-MacBookAir:~/dlib/examples/build$ tar xvjf shape_predictor_68_face_landmarks.dat.bz2
    tar: This does not look like a tar archive
    tar: Skipping to next header
    tar: Exiting with failure status due to previous errors
    —-> so I changed ‘tar xvjf’ to ‘bzip2 -d’

    sophie@sophie-MacBookAir:~/dlib/examples/build$ bzip2 -d shape_predictor_68_face_landmarks.dat.bz2
    —-> it looks okay.

    sophie@sophie-MacBookAir:~/dlib/examples/build$ ./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat faces/*.jpg
    bash: ./face_landmark_detection_ex: No such file or directory
    —-> I don’t know how to solve this… please help.

  • Yan U. S. Correa

    Hello, I’m using the dlib facial landmarks however the dots become unstable when I’m in a very clear environment or so if the background of the image is clear but works well where the background is dark like a black curtain for example. What can I do?

  • Tanmoy Sadhukhan

    In Facmorph example, I found points generated in a txt file for each image, how can I generate such text file for images?

    • 68 points can be automatically generated using dlib’s facial landmark detector. But for that example, I hand marked a few more points.

  • Lokesh Sinha

    I am trying to execute this(https://github.com/spmallick/dlib) on my server (Ubuntu) and MAC terminal. Both cases I get this error-

    (cv) MBP:build$ ./face_landmark_detection_to_file shape_predictor_68_face_landmarks.dat imgs/
    No protocol specified
    1463 FATAL [1] dlib.gui_core: Unable to connect to the X display.

    exception thrown!
    Failed to initialize X11 resources
    —–
    I have successfully done cmake and binaries have been generated. Please can you suggest on how to remove the dependencies of the X11? I have tried to do it, but, it throws up other exceptions(maybe I’m not doing it right)

  • mehmet ali atici

    Hi Satya,
    I want to create my own face landmark detector that will detect about 90 points. I will prepare the file training_with_face_landmarks.xml according to my images and their associated landmark coordinates.
    My question is: the code on page http://dlib.net/train_shape_predictor.py.html needs to be modified since I will detect more than 68 points?

  • andres

    hi SATYA MALLICK,
    i configure the visual studio with the dlib library and i tested ./webcam_face_pose_ex and this works well, when i open the other example with a image, this does not show anything, and does not throw any errors
    why?

    • What is the file name ?

      • andres

        the file is face_landmark_detection_ex, but when I run the program a black window appears and disappears without showing anything, also no mistakes