Building an object recognition mobile app

Monday. January 29, 2018 - 2 mins

Introduction

This past weekend, I participated in DeltaHacks IV at McMaster, and got a chance to meet, learn and collaborate with a lot of awesome people. Our team decided to tackle one of the challenges posted, to improve workplace safety. Our idea was to create an object recognition mobile application that employees could use to easily identify the dangers and safety protocols required to operate the equipment around them.

Together with Ryan and Sujan, we built SafetyFirst.

The technology

A lot of this app was made possible through the work done by Google’s research team. In 2017, their team released several object recognition models trained on the COCO dataset. While they all work well, we chose to use the “Single Shot Multibox Detector (SSD) with MobileNet” due to its rapid prediction time. After trying a few others, we found that this model could provide reliable results in seconds running on just a CPU.

To use this model, we built a simple webserver using Flask to expose a single endpoint. This endpoint, when deployed receives images base64 encoded and embedded within a JSON object. At this point, the image is decoded in the backend and sent through the network for classification, and bounding box prediction. Using PIL, the bounding boxes of each prediction are drawn on the image before sending it back to the user, along with prediction data. To deploy this application, we used Heroku. Although I had never used it before, it’s surprisingly simple to expose a Flask application through there.

The front end of this application was built using Android Studio. This was entirely built by Sujan, and he did an amazing job. This app currently allows users to take pictures, send it to the webserver for prediction, and then display the results. For each of our predictions, Ryan setup a MongoDB document collection that was exposed through a RESTful API. MongoDB was another tool that I had just learned about, and was impressed with how easy and quick it was to setup.

Feel free to poke around or fork our git repo. Happy hacking!

Acknowledgements

A big thanks to DeltaHacks IV organizing committee, your hard work made all of this possible. Another thank you to AccelorMital Dofasco for awarding us with the Workplace Safety prize. And finally a thanks to all the judges that deemed us worthy as one of the top six finalists.

Nishanth Merwin

A researcher tackling biology one keystroke at a time