Multi-object tracking with dlib

In this tutorial, you will learn how to use the dlib library to efficiently track multiple objects in real-time video.

So far in this series on object tracking we have learned how to:

1. Track single objects with OpenCV

2. Track multiple objects utilizing OpenCV

3. Perform single object tracking with dlib

4. Track and count people entering a business/store

We can of course track multiple objects with dlib; however, to obtain the best performance possible, we need to utilize multiprocessing and distribute the object trackers across multiple cores of our processor.

Correctly utilizing multiprocessing allows us to improve our dlib multi-object tracking frames per second (FPS) throughput rate by over 45%!

To learn how to track multiple objects using dlib, just keep reading!

Multi-object tracking with dlib

In the first part of this guide, I’ll demonstrate how to can implement a simple, naïve dlib multi-object tracking script. This program will track multiple objects in video; however, we’ll notice that the script runs a bit slow.

To increase our FPS throughput rate I’ll show you a faster, more efficient dlib multi-object tracker implementation.

Finally, I’ll discuss some improvements and suggestions you can make to enhance our multi-object tracking implementations as well.

Project structure

To get started, make sure you use the “Downloads”* *section of this tutorial to download the source code and example video.

From there, you can use the tree command to view our project structure:

$ tree.├── mobilenet_ssd│ ├── MobileNetSSD_deploy.caffemodel│ └── MobileNetSSD_deploy.prototxt├── multi_object_tracking_slow.py├── multi_object_tracking_fast.py├── race.mp4├── race_output_slow.avi└── race_output_fast.avi 1 directory, 7 files

.

│ ├── MobileNetSSD_deploy.caffemodel

├── multi_object_tracking_slow.py

├── race.mp4

└── race_output_fast.avi

1 directory, 7 files

The mobilenet_ssd/ directory contains our MobileNet + SSD Caffe model files which allow us to detect people (along with other objects).

We’ll review two Python scripts today:

multi_object_tracking_slow.py : The simple “naïve” method of dlib multiple object tracking.

multi_object_tracking_fast.py : The advanced, fast, method which takes advantage of multiprocessing.

The remaining three files are videos. We have the original race.mp4 video and two processed output videos.

The “naïve� dlib multiple object tracking implementation

The first dlib multi-object tracking implementation we are going to cover today is “naïve� in the sense that it will:

1. Utilize a simple list of tracker objects.

2. Update each of the trackers sequentially, using only a single core of our processor.

For some object tracking tasks this implementation will be more than sufficient; however, to optimize our FPS throughput rate, we should distribute the object trackers across multiple processes.

We’ll start with our simple implementation in this section and then move on to the faster method in the next section.

To get started, open up the multi_object_tracking_slow.py script and insert the following code:

# import the necessary packagesfrom imutils.video import FPSimport numpy as npimport argparseimport imutilsimport dlibimport cv2

from imutils.video import FPS

import argparse

import dlib

We begin by importing necessary packages and modules on Lines 2-7. Most importantly we’ll be using dlib and OpenCV. We’ll also use some features from my imutils package of convenience functions such as the frames per second counter.

To install dlib, follow this guide. I have a number of OpenCV installation tutorials available as well (even for the latest OpenCV 4!). You might even try the fastest way to install OpenCV on your system via pip.

To install imutils , simply use pip in your terminal:

$ pip install –upgrade imutils

Now that we (a) have the software installed, and (b) have placed the relevant import statements in our script, let’s parse our command line arguments:

9101112131415161718192021

# construct the argument parser and parse the argumentsap = argparse.ArgumentParser()ap.add_argument(“-p”, “–prototxt”, required=True, help=”path to Caffe ‘deploy’ prototxt file”)ap.add_argument(“-m”, “–model”, required=True, help=”path to Caffe pre-trained model”)ap.add_argument(“-v”, “–video”, required=True, help=”path to input video file”)ap.add_argument(“-o”, “–output”, type=str, help=”path to optional output video file”)ap.add_argument(“-c”, “–confidence”, type=float, default=0.2, help=”minimum probability to filter weak detections”)args = vars(ap.parse_args())

10

12

14

16

18

20

ap = argparse.ArgumentParser()

help=”path to Caffe ‘deploy’ prototxt file”)

help=”path to Caffe pre-trained model”)

help=”path to input video file”)

help=”path to optional output video file”)

help=”minimum probability to filter weak detections”)

If you aren’t familiar with the terminal and command line arguments, please give this post a read.

Our script processes the following command line arguments at runtime:

–prototxt : The path to the Caffe “deploy” prototxt file.

–model : The path to the model file which accompanies the prototxt.

–video : The path to the input video file. We’ll perform multi-object tracking with dlib on this video.

–output : An optional path to an output video file. If no path is specified then no video will be output to disk. I recommend outputting to an .avi or .mp4 file.

–confidence : An optional override for the object detection confidence threshold of 0.2 . This value represents the minimum probability to filter weak detections from the object detector.

Let’s define our list of CLASSES that this model supports as well as load our model from disk:

# initialize the list of class labels MobileNet SSD was trained to# detectCLASSES = [“background”, “aeroplane”, “bicycle”, “bird”, “boat”, “bottle”, “bus”, “car”, “cat”, “chair”, “cow”, “diningtable”, “dog”, “horse”, “motorbike”, “person”, “pottedplant”, “sheep”, “sofa”, “train”, “tvmonitor”] # load our serialized model from diskprint(“[INFO] loading model…“)net = cv2.dnn.readNetFromCaffe(args[“prototxt”], args[“model”])

detect

“bottle”, “bus”, “car”, “cat”, “chair”, “cow”, “diningtable”,

“sofa”, “train”, “tvmonitor”]

load our serialized model from disk

net = cv2.dnn.readNetFromCaffe(args[“prototxt”], args[“model”])

The MobileNet SSD pre-trained Caffe model supports 20 classes and 1 background class. The CLASSES are defined on Lines 25-28 in list form.

Note:Do not modify this list or the ordering of class objects if you’re using the Caffe model provided in the “Downloads”. Similarly, if you happen to be loading a different model, you’ll need to define the classes that the model supports here (order does matter). If you’re curious how our object detector works, be sure to refer to this post.

We’re only concerned about the “person” class for today’s foot race example, but you could easily modify Line 95(covered later in this post) below to track alternate class(es).

On Line 32, we load our pre-trained object detector model. We will use our pre-trained SSD to detect the presence of objects in a video. From there we will create a dlib object tracker to track each of the detected objects.

We have a few more initializations to perform:

# initialize the video stream and output video writerprint(“[INFO] starting video stream…“)vs = cv2.VideoCapture(args[“video”])writer = None # initialize the list of object trackers and corresponding class# labelstrackers = []labels = [] # start the frames per second throughput estimatorfps = FPS().start()

print(“[INFO] starting video stream…”)

writer = None

initialize the list of object trackers and corresponding class

trackers = []

 

fps = FPS().start()

On Line 36, we initialize our video stream — we’ll be reading frames from our input video one at a time.

Subsequently, on Line 37 our video writer is initialized to None . We’ll work more with the video writer in the upcoming while loop.

Now let’s initialize our trackers and labels lists on Lines 41 and 42.

And finally, we start our frames per second counter on Line 45.

We’re all set to begin processing our video:

4748495051525354555657585960616263646566

# loop over frames from the video file streamwhile True: # grab the next frame from the video file (grabbed, frame) = vs.read()  # check to see if we have reached the end of the video file if frame is None: break  # resize the frame for faster processing and then convert the # frame from BGR to RGB ordering (dlib needs RGB ordering) frame = imutils.resize(frame, width=600) rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)  # if we are supposed to be writing a video to disk, initialize # the writer if args[“output”] is not None and writer is None: fourcc = cv2.VideoWriter_fourcc(*“MJPG”) writer = cv2.VideoWriter(args[“output”], fourcc, 30, (frame.shape[1], frame.shape[0]), True)

48

50

52

54

56

58

60

62

64

66

while True:

(grabbed, frame) = vs.read()

# check to see if we have reached the end of the video file

break

# resize the frame for faster processing and then convert the

frame = imutils.resize(frame, width=600)

 

# the writer

fourcc = cv2.VideoWriter_fourcc(*“MJPG”)

(frame.shape[1], frame.shape[0]), True)

On Line 48 we begin looping over frames, where Line 50 actually grabs the frame itself.

A quick check to see if we’ve reached the end of the video file and need to stop looping is made on Lines 53 and 54.

Preprocessing takes place on Lines 58 and 59. First, the frame is resized to 600 pixels wide, maintaining aspect ratio. Then, the frame is converted to the rgb color channel ordering for dlib compatibility (OpenCV’s default is BGR and dlib’s default is RGB).

From there we instantiate the video writer (if necessary) on Lines 63-66. To learn more about writing video to disk with OpenCV, check out my previous blog post.

Let’s begin the object detection phase:

# if there are no object trackers we first need to detect objects # and then create a tracker for each object if len(trackers) == 0: # grab the frame dimensions and convert the frame to a blob (h, w) = frame.shape[:2] blob = cv2.dnn.blobFromImage(frame, 0.007843, (w, h), 127.5)  # pass the blob through the network and obtain the detections # and predictions net.setInput(blob) detections = net.forward()

# and then create a tracker for each object

# grab the frame dimensions and convert the frame to a blob

blob = cv2.dnn.blobFromImage(frame, 0.007843, (w, h), 127.5)

# pass the blob through the network and obtain the detections

net.setInput(blob)

In order to perform object tracking we must first perform object detection, either:

1. Manually, by stopping the video stream and hand-selecting the bounding box(es) of each object.

2. Programmatically, using an object detector trained to detect the presence of an object (which is what we are doing here).

If there are no object trackers (Line 70), then we know we have yet to perform object detection.

We create and pass a blob through the SSD network to detect objects on Lines 72-78. To learn about the cv2.blobFromImage function, be sure to refer to my writeup in this article.

Next, we proceed to loop over the detections to find objects belonging to the “person” class since our input video is a human foot race:

8081828384858687888990919293949596

# loop over the detections for i in np.arange(0, detections.shape[2]): # extract the confidence (i.e., probability) associated # with the prediction confidence = detections[0, 0, i, 2]  # filter out weak detections by requiring a minimum # confidence if confidence > args[“confidence”]: # extract the index of the class label from the # detections list idx = int(detections[0, 0, i, 1]) label = CLASSES[idx]  # if the class label is not a person, ignore it if CLASSES[idx] != “person”: continue

81

83

85

87

89

91

93

95

for i in np.arange(0, detections.shape[2]):

# with the prediction

 

# confidence

# extract the index of the class label from the

idx = int(detections[0, 0, i, 1])

 

if CLASSES[idx] != “person”:

We begin looping over detections on Line 81 where we:

1. Filter out weak detections (Line 88).

Ensure each detection is a “person” (Lines 91-96). You can, of course, remove this line of code or customize it to your own filtering needs.

Now that we’ve located each “person” in the frame, let’s instantiate our trackers and draw our initial bounding box(es) + class label(s):

9899100101102103104105106107108109110111112113114115116117118119

# compute the (x, y)-coordinates of the bounding box # for the object box = detections[0, 0, i, 3:7] * np.array([w, h, w, h]) (startX, startY, endX, endY) = box.astype(“int”)  # construct a dlib rectangle object from the bounding # box coordinates and start the correlation tracker t = dlib.correlation_tracker() rect = dlib.rectangle(startX, startY, endX, endY) t.start_track(rgb, rect)  # update our set of trackers and corresponding class # labels labels.append(label) trackers.append(t)  # grab the corresponding class label for the detection # and draw the bounding box cv2.rectangle(frame, (startX, startY), (endX, endY), (0, 255, 0), 2) cv2.putText(frame, label, (startX, startY - 15), cv2.FONT_HERSHEY_SIMPLEX, 0.45, (0, 255, 0), 2)

99

101

103

105

107

109

111

113

115

117

119

# for the object

(startX, startY, endX, endY) = box.astype(“int”)

# construct a dlib rectangle object from the bounding

t = dlib.correlation_tracker()

t.start_track(rgb, rect)

# update our set of trackers and corresponding class

labels.append(label)

 

# and draw the bounding box

(0, 255, 0), 2)

cv2.FONT_HERSHEY_SIMPLEX, 0.45, (0, 255, 0), 2)

To begin tracking objects we:

Compute the bounding box of each detected object (Lines 100 and 101). Instantiate and pass the bounding box coordinates to the tracker (Lines 105-107). The bounding box is especially important here. We need to create a dlib.rectangle for the bounding box and pass it to the start_track method. From there, dlib can start to track the object. Finally, we populate the trackers list with the individual tracker (Line 112).

As a result, in the next code block, we’ll handle the case where trackers have already been established and we just need to update positions.

There are two additional tasks we perform in the initial detection step:

Append the class label to the labels list (Line 111). In the event that you’re tracking multiple types of objects (such as “dog” + “person” ), you may wish to know what the type of each object is. Draw each bounding box rectangle around and class label above the object (Lines 116-119).

If the length of our detections list is greater than zero, we know we are in the object tracking phase:

121122123124125126127128129130131132133134135136137138139140141

# otherwise, we’ve already performed detection so let’s track # multiple objects else: # loop over each of the trackers for (t, l) in zip(trackers, labels): # update the tracker and grab the position of the tracked # object t.update(rgb) pos = t.get_position()  # unpack the position object startX = int(pos.left()) startY = int(pos.top()) endX = int(pos.right()) endY = int(pos.bottom())  # draw the bounding box from the correlation object tracker cv2.rectangle(frame, (startX, startY), (endX, endY), (0, 255, 0), 2) cv2.putText(frame, l, (startX, startY - 15), cv2.FONT_HERSHEY_SIMPLEX, 0.45, (0, 255, 0), 2)

122

124

126

128

130

132

134

136

138

140

# multiple objects

# loop over each of the trackers

# update the tracker and grab the position of the tracked

t.update(rgb)

 

startX = int(pos.left())

endX = int(pos.right())

 

cv2.rectangle(frame, (startX, startY), (endX, endY),

cv2.putText(frame, l, (startX, startY - 15),

In the object tracking phase, we loop over all trackers and corresponding labels on Line 125.

Then we proceed to update each object position (Lines 128-129). In order to update the position, we simply pass the rgb image.

After extracting bounding box coordinates, we can draw a bounding box rectangle and label for each tracked object (Lines 138-141).

The remaining steps in the frame processing loop involve writing to the output video (if necessary) and displaying the results:

143144145146147148149150151152153154155156

# check to see if we should write the frame to disk if writer is not None: writer.write(frame)  # show the output frame cv2.imshow(“Frame”, frame) key = cv2.waitKey(1) & 0xFF  # if the q key was pressed, break from the loop if key == ord(“q”): break  # update the FPS counter fps.update()

144

146

148

150

152

154

156

if writer is not None:

 

cv2.imshow(“Frame”, frame)

 

if key == ord(“q”):

 

fps.update()

Here we:

Write the frame to video if necessary (Lines 144 and 145). Show the output frame and capture keypresses (Lines 148 and 149). If the “q” key is pressed (“quit”), we break out of the loop.

• Finally, we update our frames per second information for benchmarking purposes (Line 156).

The remaining steps are to print FPS throughput information in the terminal and release pointers:

158159160161162163164165166167168169

# stop the timer and display FPS informationfps.stop()print(“[INFO] elapsed time: {:.2f}”.format(fps.elapsed()))print(“[INFO] approx. FPS: {:.2f}”.format(fps.fps())) # check to see if we need to release the video writer pointerif writer is not None: writer.release() # do a bit of cleanupcv2.destroyAllWindows()vs.release()

159

161

163

165

167

169

fps.stop()

print(“[INFO] approx. FPS: {:.2f}”.format(fps.fps()))

check to see if we need to release the video writer pointer

writer.release()

do a bit of cleanup

vs.release()

To close out, our fps stats are collected and printed (Lines 159-161), the video writer is released (Lines 164 and 165), and we close all windows + release the video stream.

Let’s assess accuracy and performance.

To follow along and run this script, make sure you use the “Downloads� section of this blog post to download the source code + example video.

From there, open up a terminal and execute the following command:

$ python multi_object_tracking_slow.py –prototxt mobilenet_ssd/MobileNetSSD_deploy.prototxt \ –model mobilenet_ssd/MobileNetSSD_deploy.caffemodel \ –video race.mp4 –output race_output_slow.avi[INFO] loading model…[INFO] starting video stream…[INFO] elapsed time: 24.51[INFO] approx. FPS: 13.87

–model mobilenet_ssd/MobileNetSSD_deploy.caffemodel \

[INFO] loading model…

[INFO] elapsed time: 24.51

It appears that our multi-object tracker is working!

But as you can see, we are only obtaining ~13 FPS.

For some applications, this FPS throughput rate may be sufficient — however, if you need faster FPS, I would suggest taking a look at our more efficient dlib multi-object tracker below.

Secondly, understand that tracking accuracy isn’t perfect. Refer to the third suggestion in the “Improvements and Suggestions” section below as well as read my first post on dlib object tracking for more information.

The fast, efficient dlib multi-object tracking implementation

If you run the dlib multi-object tracking script from the previous section and open up your system’s activity monitor at the same time, you’ll notice that only one core of your processor is being utilized.

In order to speed up our object tracking pipeline we can leverage Python’s multiprocessing module, similar to the threading module, but instead used to spawn processes rather than threads.

Utilizing processes enables our operating system to perform better process scheduling, mapping the process to a particular processor core on our machine (most modern operating systems are able to efficiently schedule processes that are using a lot of CPU in a parallel manner).

If you are new to Python’s multiprocessing module I would suggest you read this excellent introduction from Sebastian Raschka.

Otherwise, go ahead and open up mutli_object_tracking_fast.py and insert the following code:

# import the necessary packagesfrom imutils.video import FPSimport multiprocessingimport numpy as npimport argparseimport imutilsimport dlibimport cv2

from imutils.video import FPS

import numpy as np

import imutils

import cv2

Our packages are imported on Lines 2-8. We’re importing the multiprocessing library on Line 3.

We’ll be using the Python Process class to spawn a new process — each new process is independent from the original process.

To spawn this process we need to provide a function that Python can call, which Python will then take and create a brand new process + execute it:

def start_tracker(box, label, rgb, inputQueue, outputQueue): # construct a dlib rectangle object from the bounding box # coordinates and then start the correlation tracker t = dlib.correlation_tracker() rect = dlib.rectangle(box[0], box[1], box[2], box[3]) t.start_track(rgb, rect)

# construct a dlib rectangle object from the bounding box

t = dlib.correlation_tracker()

t.start_track(rgb, rect)

The first three parameters to start_tracker include:

box : Bounding box coordinates of the object we are going to track, presumably returned by some sort of object detector, whether manual or programmatic.

label : Human-readable label of the object.

rgb : An RGB-ordered image that we’ll be using to start the initial dlib object tracker.

Keep in mind how Python multiprocessing works — Python will call this function and then create a brand new interpreter to execute the code within. Therefore, each start_tracker spawned process will be independent from its parent. To communicate with the Python driver script we need to leverage either Pipes or Queues. Both types of objects are thread/process safe, accomplished using locks and semaphores.

In essence, we are creating a simple producer/consumer relationship:

1. Our parent process will produce new frames and add them to the queue of a particular object tracker.

2. The child process will then consume the frame, apply object tracking, and then return the updated bounding box coordinates.

I decided to use Queue objects for this post; however, keep in mind that you could use a Pipe if you wish — be sure to refer to the Python multiprocessing documentation for more details on these objects.

Now let’s begin an infinite loop which will run in the process:

17181920212223242526272829303132333435363738

# loop indefinitely – this function will be called as a daemon # process so we don’t need to worry about joining it while True: # attempt to grab the next frame from the input queue rgb = inputQueue.get()  # if there was an entry in our queue, process it if rgb is not None: # update the tracker and grab the position of the tracked # object t.update(rgb) pos = t.get_position()  # unpack the position object startX = int(pos.left()) startY = int(pos.top()) endX = int(pos.right()) endY = int(pos.bottom())  # add the label + bounding box coordinates to the output # queue outputQueue.put((label, (startX, startY, endX, endY)))

18

20

22

24

26

28

30

32

34

36

38

# process so we don’t need to worry about joining it

# attempt to grab the next frame from the input queue

 

if rgb is not None:

# object

pos = t.get_position()

# unpack the position object

startY = int(pos.top())

endY = int(pos.bottom())

# add the label + bounding box coordinates to the output

outputQueue.put((label, (startX, startY, endX, endY)))

We loop indefinitely here — this function will be called as a daemon process, so we don’t need to worry about joining it.

First, we’ll attempt to grab a new frame from the inputQueue on Line 21.

If the frame is not empty, we’ll grab the frame and then update the object tracker, allowing us to obtain the updated bounding box coordinates (Lines 24-34).

Finally, we write the label and bounding box to the outputQueue so the parent process can utilize them in the main loop of our script (Line 38).

Back to the parent process, we’ll parse our command line arguments:

40414243444546474849505152

# construct the argument parser and parse the argumentsap = argparse.ArgumentParser()ap.add_argument(“-p”, “–prototxt”, required=True, help=”path to Caffe ‘deploy’ prototxt file”)ap.add_argument(“-m”, “–model”, required=True, help=”path to Caffe pre-trained model”)ap.add_argument(“-v”, “–video”, required=True, help=”path to input video file”)ap.add_argument(“-o”, “–output”, type=str, help=”path to optional output video file”)ap.add_argument(“-c”, “–confidence”, type=float, default=0.2, help=”minimum probability to filter weak detections”)args = vars(ap.parse_args())

41

43

45

47

49

51

ap = argparse.ArgumentParser()

help=”path to Caffe ‘deploy’ prototxt file”)

help=”path to Caffe pre-trained model”)

help=”path to input video file”)

help=”path to optional output video file”)

help=”minimum probability to filter weak detections”)

The command line arguments for this script are exactly the same as our slower, non-multiprocessing script. If you need a refresher on the arguments, just click here. And furthermore, read my post about argparse and command line arguments if you aren’t familiar with them.

Let’s initialize our input and output queues:

# initialize our lists of queues – both input queue and output queue# for every object that we will be trackinginputQueues = []outputQueues = []

for every object that we will be tracking

outputQueues = []

These queues will hold the objects we are tracking. Each process spawned will need two Queue objects:

1. One to read input frames from

2. And a second to write results to

This next block is identical to our previous script:

596061626364656667686970717273747576

# initialize the list of class labels MobileNet SSD was trained to# detectCLASSES = [“background”, “aeroplane”, “bicycle”, “bird”, “boat”, “bottle”, “bus”, “car”, “cat”, “chair”, “cow”, “diningtable”, “dog”, “horse”, “motorbike”, “person”, “pottedplant”, “sheep”, “sofa”, “train”, “tvmonitor”] # load our serialized model from diskprint(“[INFO] loading model…“)net = cv2.dnn.readNetFromCaffe(args[“prototxt”], args[“model”]) # initialize the video stream and output video writerprint(“[INFO] starting video stream…“)vs = cv2.VideoCapture(args[“video”])writer = None # start the frames per second throughput estimatorfps = FPS().start()

60

62

64

66

68

70

72

74

76

detect

“bottle”, “bus”, “car”, “cat”, “chair”, “cow”, “diningtable”,

“sofa”, “train”, “tvmonitor”]

load our serialized model from disk

net = cv2.dnn.readNetFromCaffe(args[“prototxt”], args[“model”])

initialize the video stream and output video writer

vs = cv2.VideoCapture(args[“video”])

 

fps = FPS().start()

We define our model’s CLASSES and load the model itself (Lines 61-68). Remember, these CLASSES are static — our MobileNet SSD supports these classes and only these classes. If you want to detect + track other objects you’ll need to find another pretrained model or train one. Furthermore, the order of this list matters! Do not change the ordering of the list unless you enjoy being confused! I would also recommend reading this tutorial if you want to further understand how object detectors work.

We initialize our video stream object and set our video writer object to None (Lines 72 and 73).

Our frames per second calculator is instantiated and started on Line 76.

Now let’s begin looping over frames from the video stream:

7879808182838485868788899091929394959697

# loop over frames from the video file streamwhile True: # grab the next frame from the video file (grabbed, frame) = vs.read()  # check to see if we have reached the end of the video file if frame is None: break  # resize the frame for faster processing and then convert the # frame from BGR to RGB ordering (dlib needs RGB ordering) frame = imutils.resize(frame, width=600) rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)  # if we are supposed to be writing a video to disk, initialize # the writer if args[“output”] is not None and writer is None: fourcc = cv2.VideoWriter_fourcc(*“MJPG”) writer = cv2.VideoWriter(args[“output”], fourcc, 30, (frame.shape[1], frame.shape[0]), True)

79

81

83

85

87

89

91

93

95

97

while True:

(grabbed, frame) = vs.read()

# check to see if we have reached the end of the video file

break

# resize the frame for faster processing and then convert the

frame = imutils.resize(frame, width=600)

 

# the writer

fourcc = cv2.VideoWriter_fourcc(*“MJPG”)

(frame.shape[1], frame.shape[0]), True)

The above code block is, yet again, identical to the one in the previous script. Be sure to refer above as needed.

Now let’s handle the case where we have no inputQueues :

99100101102103104105106107108109110111112113114115116117118119120121122123124125126127

# if our list of queues is empty then we know we have yet to # create our first object tracker if len(inputQueues) == 0: # grab the frame dimensions and convert the frame to a blob (h, w) = frame.shape[:2] blob = cv2.dnn.blobFromImage(frame, 0.007843, (w, h), 127.5)  # pass the blob through the network and obtain the detections # and predictions net.setInput(blob) detections = net.forward()  # loop over the detections for i in np.arange(0, detections.shape[2]): # extract the confidence (i.e., probability) associated # with the prediction confidence = detections[0, 0, i, 2]  # filter out weak detections by requiring a minimum # confidence if confidence > args[“confidence”]: # extract the index of the class label from the # detections list idx = int(detections[0, 0, i, 1]) label = CLASSES[idx]  # if the class label is not a person, ignore it if CLASSES[idx] != “person”: continue

100

102

104

106

108

110

112

114

116

118

120

122

124

126

# create our first object tracker

# grab the frame dimensions and convert the frame to a blob

blob = cv2.dnn.blobFromImage(frame, 0.007843, (w, h), 127.5)

# pass the blob through the network and obtain the detections

net.setInput(blob)

 

for i in np.arange(0, detections.shape[2]):

# with the prediction

 

# confidence

# extract the index of the class label from the

idx = int(detections[0, 0, i, 1])

 

if CLASSES[idx] != “person”:

If there are no inputQueues (Line 101) then we know we need to apply object detection prior to object tracking.

We apply object detection on Lines 103-109 and then proceed to loop over the results on Line 112. We grab our confidence values and filter out weak detections on Lines 115-119.

If our confidence meets the threshold established by our command line arguments, we consider the detection, but we further filter it out by class label . In this case, we’re only looking for “person” objects (Lines 122-127).

Assuming we have found a “person” , we’ll create queues and spawn tracking processes:

129130131132133134135136137138139140141142143144145146147148149150151152153154

# compute the (x, y)-coordinates of the bounding box # for the object box = detections[0, 0, i, 3:7] * np.array([w, h, w, h]) (startX, startY, endX, endY) = box.astype(“int”) bb = (startX, startY, endX, endY)  # create two brand new input and output queues, # respectively iq = multiprocessing.Queue() oq = multiprocessing.Queue() inputQueues.append(iq) outputQueues.append(oq)  # spawn a daemon process for a new object tracker p = multiprocessing.Process( target=start_tracker, args=(bb, label, rgb, iq, oq)) p.daemon = True p.start()  # grab the corresponding class label for the detection # and draw the bounding box cv2.rectangle(frame, (startX, startY), (endX, endY), (0, 255, 0), 2) cv2.putText(frame, label, (startX, startY - 15), cv2.FONT_HERSHEY_SIMPLEX, 0.45, (0, 255, 0), 2)

130

132

134

136

138

140

142

144

146

148

150

152

154

# for the object

(startX, startY, endX, endY) = box.astype(“int”)

 

# respectively

oq = multiprocessing.Queue()

outputQueues.append(oq)

# spawn a daemon process for a new object tracker

target=start_tracker,

p.daemon = True

 

# and draw the bounding box

(0, 255, 0), 2)

cv2.FONT_HERSHEY_SIMPLEX, 0.45, (0, 255, 0), 2)

We first compute the bounding box coordinates on Lines 131-133.

From there we create two new queues, iq and oq (Lines 137 and 138), appending them to inputQueues and outputQueues respectively (Lines 139 and 140).

From there we spawn a new start_tracker process, passing the bounding box, label , rgb image, and the iq + oq (Lines 143-147). Don’t forget to read more about multiprocessing here.

We also draw the detected object’s bounding box rectangle and class label (Lines 151-154).

Otherwise, we’ve already performed object detection so we need to apply each of the dlib object trackers to the frame:

156157158159160161162163164165166167168169170171172173174175176177178

# otherwise, we’ve already performed detection so let’s track # multiple objects else: # loop over each of our input ques and add the input RGB # frame to it, enabling us to update each of the respective # object trackers running in separate processes for iq in inputQueues: iq.put(rgb)  # loop over each of the output queues for oq in outputQueues: # grab the updated bounding box coordinates for the # object – the .get method is a blocking operation so # this will pause our execution until the respective # process finishes the tracking update (label, (startX, startY, endX, endY)) = oq.get()  # draw the bounding box from the correlation object # tracker cv2.rectangle(frame, (startX, startY), (endX, endY), (0, 255, 0), 2) cv2.putText(frame, label, (startX, startY - 15), cv2.FONT_HERSHEY_SIMPLEX, 0.45, (0, 255, 0), 2)

157

159

161

163

165

167

169

171

173

175

177

# multiple objects

# loop over each of our input ques and add the input RGB

# object trackers running in separate processes

iq.put(rgb)

# loop over each of the output queues

# grab the updated bounding box coordinates for the

# this will pause our execution until the respective

(label, (startX, startY, endX, endY)) = oq.get()

# draw the bounding box from the correlation object

cv2.rectangle(frame, (startX, startY), (endX, endY),

cv2.putText(frame, label, (startX, startY - 15),

Looping over each of the inputQueues , we add the rgb image to them (Lines 162 and 163).

Then we loop over each of the outputQueues (Line 166), obtaining the bounding box coordinates from each independent object tracker (Line 171). Finally, we draw the bounding box + associated class label on Lines 175-178.

Let’s finish out the loop and script:

180181182183184185186187188189190191192193194195196197198199200201202203204205206

# check to see if we should write the frame to disk if writer is not None: writer.write(frame)  # show the output frame cv2.imshow(“Frame”, frame) key = cv2.waitKey(1) & 0xFF  # if the q key was pressed, break from the loop if key == ord(“q”): break  # update the FPS counter fps.update() # stop the timer and display FPS informationfps.stop()print(“[INFO] elapsed time: {:.2f}”.format(fps.elapsed()))print(“[INFO] approx. FPS: {:.2f}”.format(fps.fps())) # check to see if we need to release the video writer pointerif writer is not None: writer.release() # do a bit of cleanupcv2.destroyAllWindows()vs.release()

181

183

185

187

189

191

193

195

197

199

201

203

205

if writer is not None:

 

cv2.imshow(“Frame”, frame)

 

if key == ord(“q”):

 

fps.update()

stop the timer and display FPS information

print(“[INFO] elapsed time: {:.2f}”.format(fps.elapsed()))

 

if writer is not None:

 

cv2.destroyAllWindows()

We write the frame to the output video if necessary as well as show the frame to the screen (Lines 181-185).

If the “q” key is pressed, we “quit”, breaking out of the loop (Lines 186-190).

If we do continue processing frames, our fps calculator is updated on Line 193, and then we start the process at the beginning of the while loop again.

Otherwise, we’re done processing frames, and we display the FPS throughput info + release pointers and close windows.

To execute this script, make sure you use the “Downloads� section of the post to download the source code + example video.

From there, open up a terminal and execute the following command:

$ python multi_object_tracking_fast.py –prototxt mobilenet_ssd/MobileNetSSD_deploy.prototxt \ –model mobilenet_ssd/MobileNetSSD_deploy.caffemodel \ –video race.mp4 –output race_output_fast.avi[INFO] loading model…[INFO] starting video stream…[INFO] elapsed time: 14.01[INFO] approx. FPS: 24.26

–model mobilenet_ssd/MobileNetSSD_deploy.caffemodel \

[INFO] loading model…

[INFO] elapsed time: 14.01

As you can see, our faster, more efficient multi-object tracker is running at 24 FPS, an improvement by over 45% from our previous implementation 🚀!

Furthermore, if you open up your activity monitor while this script is running you will see that more of your system’s overall CPU Is being utilized.

This speedup is obtained by allowing each of the dlib object trackers to run in a separate process which in turn enables your operating system to perform more efficient scheduling of the CPU resources.

Improvements and suggestions

The dlib multi-object tracking Python scripts I’ve shared with you today will work just fine for processing shorter video streams; however, if you intend on utilizing this implementation for long-running production environments (in the order of many hours to days of video) there are two primary improvements I would suggest you make:

The first improvement would be to utilize processing pools rather than spawning a brand new process for each object to be tracked.

The implementation covered here today constructs a brand new Queue and Process for each object that we need to track.

For today’s purposes that’s fine, but consider if you wanted to track 50 objects in a video — this implies that you would spawn 50 processes, one for each object. At that point, the overhead of your system managing all those processes will destroy any increase in FPS throughput. Instead, you would want to utilize processing pools.

If your system has N processor cores, then you would want to create a pool with N – 1 processes, leaving one core to your operating system to perform system operations. Each of these processes should perform multiple object tracking, maintaining a list of object trackers, similar to the first multi-object tracking we covered today.

This improvement will allow you to utilize all cores of your processor without the overhead of having to spawn many independent processes.

The second improvement I would make is to clean up the processes and queues.

In the event that dlib reports an object as “lost� or “disappeared� we are not returning from the start_tracker function, implying that that process will live for the life of the parent script and only be killed when the parent exits.

Again, that’s fine for our purposes here today, but if you intend on utilizing this code in production environments, you should:

Update the start_tracker function to return once dlib reports the object as lost. Delete the inputQueue and outputQueue for the corresponding process as well.

Failing to perform this cleanup will lead to needless computational consumption and memory overhead for long-running jobs.

The third improvement is to improve tracking accuracy by running the object detector every Nframes (rather than just once at the start).

I actually demonstrated this in my previous post on people counting with OpenCV. It requires more logic and thought, but yields a much more accurate tracker.

I elected to forego the implementation for this script so that I could teach you the multiprocessing method concisely.

Ideally, you would use this third improvement in addition to multiprocessing.

Summary

In this tutorial, we learned how to utilize the dlib library to perform multi-object tracking.

We also learned how to leverage multiprocessing to:

1. Distribute the actual object tracker instantiations to multiple cores of our processor,

2. Thereby leading to an increase in FPS throughput rate by over 45%.

I would encourage you to utilize the multiprocessing implementation of our dlib multi-object tracker for your own applications as it’s faster and more efficient; however, you should refer to the “Improvements and suggestionsâ€� section of this tutorial where I discuss how you can further enhance the multi-object tracking implementation.

If you enjoyed this series on object tracking, be sure to enter your email in the form below to download today’s source code + videos as well as to be notified of future tutorials here on PyImageSearch.

Downloads: