• Skip to primary navigation
  • Skip to content
  • Skip to primary sidebar

Learn OpenCV

OpenCV examples and tutorials ( C++ / Python )

  • Home
  • About
  • Resources
  • AI Consulting
  • Courses

Heatmap for Logo Detection using OpenCV (Python)

September 5, 2018 By Nadav Ben-Haim 5 Comments

In this post, we will learn how to create a heatmap to analyze annotations in a video sequence.

We first elaborate on why this would be useful, give a real world application, and follow up with a tutorial and implementation in Python.

This is a guest post by Nadav Ben-Haim. He is the co-founder of Orpix — a Computer Vision company.

Why use Heatmaps for Logo Detection Analysis?

In computer vision, we often need to annotate the location of objects in a video using bounding boxes, polygons, or masks. Sometimes these annotations are produced by human annotators for creating training data and sometimes the annotations are automatically generated as a result of object detection and tracking.

Analyzing annotations frame by frame is not always a good idea because sometimes a video sequence can be several minutes or hours long. At other times, the videos are short, but there are thousands of them ( e.g. in a training set ). This is where Heatmaps can be very useful.

Heatmaps provide for a quick visual way to analyze the screen location of object exposure across an entire video. The parts of a frame where an object appears more frequently (i.e. high exposure areas) are colored red (hot) and conversely, the parts of the video where an object appears infrequently are colored blue (cool).

Heatmaps for Exposure of Logos

At my company, Orpix, we use deep learning methods to detect brands and logos in digital media for sponsorship valuation.  A sample of our logo detection is shown below.

Sample of orpix logo detection for sponsorship valuation
Sample logo detection output automatically generated by Orpix Sponsorship Valuation

Brands that sponsor events place their logos on digital boards and on clothing. Market researchers assess the quality of exposure a brand gets based on a variety of factors —

  1. Exposure Time: The more time a logo is exposed on screen, the more successful an ad is deemed.
  2. Location: There are a few dimensions to location of a logo. First, if the logo is in the center of a frame, it is more likely to grab attention. Furthermore, if a logo is on the clothing of a prominent athlete, the exposure increases in value.
  3. Size & Clarity : When it comes to advertisement, size does matter! Needless to say an ad that covers a large screen space is much more valuable than the one that does not. Similarly, a blurry ad is less valuable than one that is clearly seen.
  4. Spotlight : Advertisers like to see their logo without other logos sharing the limelight. They often buy the entire visible ad space as shown in the sample output above.

Computing a heatmap for a brand’s exposure in a video can help in part to quickly value sponsorship across the entire video.

Logo Analysis for the 2018 Fifa World Cup Finals

The 2018 Fifa World Cup entertained millions of people around the world for weeks. In many countries, work stopped when an important match was on.

While most people were glued to the TV debating the odds of their favorite team, many teams of data scientists were working behind the scenes analyzing the effectiveness of sponsorship campaigns shown during the match.

At Orpix, we ran a recording of the live stream aired in the United States through our logo detection pipeline. The Fifa World Cup final lasted for 2 hours, 26 minutes and 13 seconds. We sampled the match time at 1 frame per second for analysis which produced a total of 8773 frames.

Nike Logo Detection Analysis

No points for guessing which company logo topped the list. It was obviously Nike — the biggest sporting goods company in the world. Their logo appeared on the players’ clothing on 1221 frames — an astounding 14% of the total broadcast time. The Nike exposure heatmap is shown below.

Nike Exposure Heatmap
Nike Exposure Heatmap

While Nike did receive the largest share of exposure throughout the game, the heatmap demonstrates that the distribution of exposure was spread out fairly evenly across the screen.  This can also be easily inferred from looking at the highest exposure of 36 seconds and comparing it to the total exposure time of 1221 seconds for the entire game.  The heatmap is interesting because you cannot help but be curious about the hot spot in middle-right of the screen. Turns out Nike got a lot of exposure when players were introduced as shown in the image below.

Nike Jersey Exposure
Nike Jersey Exposure

VISA Logo Detection Analysis

If you cannot get your logo on players’ clothing, your next best option is to advertise on digital boards. Credit card processing company VISA topped the list of logos detected on digital boards (769 seconds), closely followed by sporting giant Adidas.

In the example below, the Visa logo appears for a total of 769 seconds. The heatmap shows that in 80 (indicated by red) of those seconds, the Visa logo appeared in the upper left part of the screen. When contrasted with the Nike heatmap, you can see that the exposure Visa obtained was concentrated much more heavily in a specific location throughout the game (as denoted by the ratio of highest exposure on the heatmap of 80 to total exposure of 769).  This is because when a logo appears on a digital board, it is not moving nearly as much as a logo on a soccer player’s jersey.

Visa logo exposure Fifa World Cup 2018
Visa logo exposure Fifa World Cup 2018

For a full sample report of the World Cup Final that Orpix generates automatically, click here.

Logo Exposure Heatmap Code Tutorial

Download Code
To easily follow along this tutorial, please download code by clicking on the button below. It’s FREE!

Download Code

In this section, we will show a simple way to create a heatmap for Visa logos during the 2018 World Cup Final using annotations automatically generated with Orpix Sponsorship Valuation solution, as shown in the image above.

To get started, you will need to have the following python packages installed

  • opencv-python
  • matplotlib
  • numpy

Code Overview

We provide you with:

  • generate_heatmap.py – a script to generate the heatmap
  • Annotations for Visa during the 2018 World Cup Final.  You can download the data here.

Unzip the data into the same directory as the script location.

Below are details on the script’s input and output.

Input:

labels.txt – the provided file contains references to images and associated annotations.  The file is of the format of one image per line as follows:

image_path number_of_annotations x1 y1 x2 y2 x3 y3 x4 y4 …. (repeat the coordinates for number_of_annotations on this line)

In our case, the labels.txt references frames in the visa_frames directory along annotations of Visa for each frame.  These are all of the frames where Visa occurred during the 2018 World Cup Final, sampled at one frame per second.

Note: The reason we specify 4 (x,y) coordinates instead of x,y,width,height is for added flexibility, as we output quadrilaterals in our logo detection solution.

Output:

highlighted frames – For each frame specified in labels.txt, we highlight the associated annotations on the image and save the drawn image to “output” directory.  This is not required to compute the heatmap, but is done to aid the user in understanding, and also good for debugging.

heatmap – At the end of the script a heatmap is saved to the working directory as “heatmap.png”.  We use matplotlib since it’s quite easy to create a nice heatmap with good colors, and a legend as well.

Simply run the script as:

>> python generate_heatmap.py

Implementation Details

First we need to initialize the data we will use to create the heatmap.  We will use a single channel floating point numpy array the same size of the video frames we are processing to accumulate exposure time per pixel.

#keeps track of exposure time per pixel.  Accumulates for each image
#gets initialized when we process the first image
accumulated_exposures = None
    
#frames were sampled at one second per frame.  If you sampled frames from a 
#video at a different rate, change this value.
#
#if you sampled frames at 10 frames per second, this value would be 0.1
#
seconds_per_frame = 1.0 

Next, for each line in the labels.txt, we produce a mask where the annotation area has value of seconds_per_frame, and 0 everywhere else,
and we add that to accumulated_exposures array which we will use to create the heatmap.

#parse the line using helper function
frame_path, labels = parse_line(line)

print "processing %s" % frame_path

#load the image
frame = cv2.imread(frame_path)

#this is where the highlighted images will go
if not os.path.exists('output'):
    os.mkdir('output')


#if the heatmap is None we create it with same size as frame, single channel
if type(accumulated_exposures) == type(None):
    accumulated_exposures = np.zeros((frame.shape[0], frame.shape[1]), dtype=np.float)

#we create a mask where all pixels inside each label are set to number 
#of seconds per frame that the video was sampled at.
#So as we accumulate the exposure heatmap counts, each pixel contained 
#inside a label contributes the seconds_per_frame to the overall 
#accumulated exposure values
maskimg = np.zeros(accumulated_exposures.shape, dtype=np.float)
for label in labels:
    cv2.fillConvexPoly(maskimg, label, (seconds_per_frame))

#highlight the labels on the image and save.
#comment out the 2 lines below if you only want to compute the heatmap
highlighted_image = highlight_labels(frame, labels, maskimg)
cv2.imwrite('output/%s' % os.path.basename(frame_path), highlighted_image)

#accumulate the heatmap object exposure time
accumulated_exposures = accumulated_exposures + maskimg

Lastly, when we finished accumulating exposures for all frames, we can produce the heatmap using matplotlib.

#
#create final heatmap using matplotlib
#    
data = np.array(accumulated_exposures)
#create the figure
fig, axis = plt.subplots()
#set the colormap - there are many options for colormaps - see documentation
#we will use cm.jet
hm = axis.pcolor(data, cmap=plt.cm.jet)
#set axis ranges
axis.set(xlim=[0, data.shape[1]], ylim=[0, data.shape[0]], aspect=1)
#need to invert coordinate for images
axis.invert_yaxis()
#remove the ticks
axis.set_xticks([])
axis.set_yticks([])
    
#fit the colorbar to the height
shrink_scale = 1.0
aspect = data.shape[0]/float(data.shape[1])
if aspect < 1.0:
    shrink_scale = aspect
clb = plt.colorbar(hm, shrink=shrink_scale)
#set title
clb.ax.set_title('Exposure (seconds)', fontsize = 10)
#saves image to same directory that the script is located in (our working directory)
plt.savefig('heatmap.png', bbox_inches='tight')
#close objects
plt.close('all')   

And that’s it! You now have an exposure heatmap that you can use to analyze for object location in video!

Subscribe & Download Code

If you liked this article and would like to download code (C++ and Python) and example images used in this post, please subscribe to our newsletter. You will also receive a free Computer Vision Resource Guide. In our newsletter, we share OpenCV tutorials and examples written in C++/Python, and Computer Vision and Machine Learning algorithms and news.

Subscribe Now

Filed Under: Application, Deep Learning, Object Detection, OpenCV 3, Tutorial Tagged With: heatmap, logo detection, matplotlib

Comments

  1. Jae Duk Seo says

    September 5, 2018 at 10:33 am

    This is actually super cool!

    Reply
    • Satya Mallick says

      September 10, 2018 at 6:11 am

      Thanks!

      Reply
  2. Paul Oord says

    September 5, 2018 at 11:06 am

    From a Opencv perspective maybe very cool, but heatmaps don’t say anything about where and how people actually look on the screens. If a match is interesting, you don’t care about logo’s. Maybe you even (want to) avoid them… It’s a sort of visual pollution.

    Reply
    • Nadav Ben-Haim says

      September 5, 2018 at 11:38 am

      Heatmaps could be used to determine where a user looks but you would need eye tracking data for that. From the advertisers perspective though, they are interested to know where on screen their logos appeared and for how long, so for that, heatmaps can be a quick and visual way to analyze that information.

      Reply
      • Paul Oord says

        September 10, 2018 at 2:23 am

        Advertisers want a quick way to get ‘numbers’ but don’t know what these mean. So they continue putting their logos everywhere they ‘think’ it’s adequate. Even on the clothes we wear, I don’t like that.

        Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Join Course

Computer Vision for Faces

Resources

Download Code (C++ / Python)

Disclaimer

This site is not affiliated with OpenCV.org

I am an entrepreneur with a love for Computer Vision and Machine Learning with a dozen years of experience (and a Ph.D.) in the field.

In 2007, right after finishing my Ph.D., I co-founded TAAZ Inc. with my advisor Dr. David Kriegman and Kevin Barnes. The scalability, and robustness of our computer vision and machine learning algorithms have been put to rigorous test by more than 100M users who have tried our products. Read More…

Recent Posts

  • Gender & Age Classification using OpenCV Deep Learning ( C++/Python )
  • Invisibility Cloak using Color Detection and Segmentation with OpenCV
  • Fast Image Downloader for Open Images V4
  • Deep Learning based Text Detection Using OpenCV (C++/Python)
  • Video Stabilization Using Point Feature Matching in OpenCV

Copyright © 2019 · Big Vision LLC