A Novel Application of Deep Learning in a Robotic Arm for the Classification and Sorting of Recyclables

Abstract

Currently, the world disposes 1.3 billion tonnes of waste per year, with a projected increase to 2.2 billion tonnes by 2025 (World Bank, 2018). Out of the total waste produced in Canada, only around 27% is diverted. The other 73% end up in either landfills or incinerators, both of which increase pollution, while damaging public health. However, statistics show that around 75% of waste can be diverted in one way or another (United States Environmental Protection Agency, 2018), with recycling accounting for a huge chunk of said diversion. It is quite apparent that the solution to our waste disposal problem is to have better recycling and waste-sorting facilities.

In order to improve the sorting of waste, a robotic arm was made. Transfer learning with AlexNet in Matlab was used on 1033 images to train the arm to recognize four different object classes. Bluetooth modules conveyed commands written in Keil5 and Microsoft Visual studio to the arm, enabling it to grab the objects and sort them into separate bins, based on the type of recycling treatment they need. Consultations with industry experts from major recyclable treatment companies such as GEEP Canada and Emterra Group, leading recycling technology companies such as MSS Optical Sorters (U.S.), as well as waste-management scientists, were undertaken to discuss the viability of incorporating this invention into existing recycling processes. Their feedback included many potential uses of this arm, such as reducing the number of workers exposed to health hazards, increasing the efficiency of pre-sorting, reducing contamination after sorting, and classifying e-waste. In general, responses were positive and encouraging.

Introduction

Current Consumer Problems

Although people generally are supportive of the concept of recycling, the current process prevents many from following through—people are expected to place only designated items into curbside recycling bins and transport all other recyclables to separate specialized treatment facilities. At the moment, these specialized treatment facilities are sparsely used, contributing to the 50% gap between recyclable and recycled waste. If more recyclables could be processed single-stream, recycling would be more convenient for the consumer and a larger percentage of waste could be diverted. This, of course, would necessitate more advanced sorting technologies in recycling plants.

Current Facility Challenges

Currently, mixed recyclables (paper, plastics, cardboard, glass, cans) are sorted using a combination of manual sorters and automatized conveyor belts utilizing magnets, eddy currents, tumblers, air-blowers, screens, and optical sorters. Workers are employed to remove items unsortable by the automatized process (e.g. soft plastics, rubber, foam, and heavy metallic objects) and indiscriminately send these materials to a landfill. Furthermore, workers manually remove wrongly-sorted items after they pass through various machines, ensuring quality control. While involved in this process, workers are exposed to toxic fumes, chemicals, and other hazards. The purpose of this project is to increase the amount of diverted waste by enabling the sorting of more types of objects.

Hypothesis

It was hypothesized that by employing the use of robotics and transfer learning, the amount of diverted waste could be increased, the efficiency of mixed recycling facilities could be improved, and the negative environmental effects on workers’ health could be reduced.

Background Research

After visiting several recycling facilities and reading reports from sources such as Statistics Canada, World Bank, and United States Environmental Protection Agency, it became clear that the processes currently in use rely on mechanical innovations (e.g. fibre and glass breaker screens and magnetic metal sorters), supported by optical sorters using near-infrared rays and spectrometers. Visiting the Emterra recycling plant in North Vancouver, BC unveiled a clear view of the general recycling process.

1.     Various cities unload curbside recyclables into a large pile—around 250 tonnes per day.

2.     Emterra trucks scoop up recyclables and transfer them to a conveyor belt.

3.     Waste goes through a presorting area, where workers remove obvious contamination.

4.     Waste passes through fibre screens, which separate paper and cardboard from other recyclables. Soft plastics get stuck in this screen and have to be manually removed.

5.     Objects pass over a glass-breaker screen, where glass is broken and falls between the cracks, while other recyclables are unharmed. As with before, any remaining soft plastics get stuck in this screen.

6.     Remaining objects (various containers) are separated by magnets, which target ferromagnetic cans, and eddy currents, which aim to sort nonferrous metallic objects.

7.     Optical Sorters with Near Infrared (NIR) shine light at the objects, distinguishing the identity of objects based on the reflected light. Once this is done, an air gun is used to separate different types of objects. During this process, tetra paks and hard plastic containers are separated.

8.     Following this sorting, some materials are compressed with a baler, while others are put into boxes. These bales and boxes will be sold to appropriate recycling markets. As of now, these sold materials have around 70% purity—if the buyer raises their purity standards, market share would be lost.

9.     Soft plastics persist in every part of the recycling process—in multiple areas of the conveyor belt, on various screens and bales, workers remove this material.

Based on multiple World Bank reports, sorting is a worldwide problem in the area of waste-management. Globally, open-dump, landfill, and incinerator use account for 81% of all waste treatment (below).

Right now, landfills and incinerators account for 3% of total carbon dioxide emissions and 11% of methane gas emissions—alarmingly, methane gas is over 30 times more potent than carbon dioxide in causing global warming (EPA). It is explicitly stated that by 2050, our total waste production would increase by 70% (World Bank)—if this trend continues, methane and carbon dioxide emissions will increase exponentially.

The imminence of the world’s waste problem is quite apparent. On the environmental level, greenhouse gas emissions lead to global warming, extreme weather and frequent natural disasters, such as hurricanes and forest fires. After examining the current recycling processes, sorting was identified to be a major problem and a solution was sought. Nowhere in the world has deep learning been implemented to better the process of sorting. With the assistance of deep learning, this robotic arm is able to distinguish accurately between different classes of objects, and sort these objects appropriately.

Multiple attempts at the classification of waste using deep learning have been made. Waste classification of general (curbside) recyclables based on images has been attempted using two pre-trained convolutional neural networks – Resnet and ImageNet. Research has also been conducted on the classification of biodegradable vs. non-biodegradable waste, as well as the recognition of different classes of papers. Nonetheless, no attempts at the sorting component have been made.

Method / Testing and Redesign

（1）General Design

This robotic arm has to be able to distinguish between objects and execute different commands based on the result from distinguishing.

An upper computer is attached to a camera, which in turn is attached to the top of the robotic arm. This upper computer is connected via Bluetooth (this method, as opposed to wired or wifi, was chosen due to its relatively low cost and ease of setup, as well as the short range of the communication) to the motion-controlling microcontroller at the base of the arm. The upper computer then cuts and processes these images, identifying the objects and signaling the arm (by signalling the microcontroller) to grab a given object, drop it into its corresponding box, and return to its starting position.

（2）Structural design

A six-axis robotic arm was used. Of the six (analog) servos, one controls the rotation of the base; three others control angles of joints within the arm, and two servos control the angle and openness of the grabber claw. They are enclosed by aluminum anglers, connectors, and tie brackets, which are connected by screws. A camera was attached using hot glue to next to the claw of the arm. Furthermore, three wooden boxes were made through AutoCAD, each for the deployment of specific items.

（3）Hardware Design

The hardware includes a microcontroller (STM32F103c8t6), Bluetooth module, six SG90 micro (analog) servos, camera, and USB-TTL convertor. A STM32 microcontroller was chosen due to the relatively low cost and the large amount of open libraries and resources. The upper computer analyses data collected by the camera (below) and sends six PWM commands to the microcontroller, thereby controlling the position of each servo and facilitating the process of grabbing given objects, rotating, and dropping objects in their corresponding boxes. The functions of the six servos are outlined below.

（4）Software Design

4.1 Servo Movement

The pre-supplied PWM library of Keil5 was used to control the servos, which have a speed of 300 deg/s and an accuracy of around ± 5%. Through Bluetooth, the microcontroller receives six PWM signals from the upper computer and controls servo rotation and claw position. This allows the claw to pick up given objects and transport them to their corresponding designation.

4.2 Image Recognition

4.2.1 Transfer Learning

Transfer learning (Fig.2) with a pretrained AlexNet convolutional neural network (Fig.1) fine-tunes the net’s capabilities in recognizing some classes of objects. To train the neural network, images of various objects such as pens, plastic bags, and batteries were uploaded. In the code (Fig.3), images were resized to their required dimension (227-227-3) and put through a series of filters, where key bits of information about the images are extracted. Subsequent to training, an accuracy number, ideally 1, is outputted.

4.2.2 Image Processing

Pictures of the background and objects are taken and transferred to the computer. Differences between the objects and the background are detected and black-and-white images of object outlines are outputted. These images are segmented using a Gaussian Mixture Model (below) and recognized through transfer learning with AlexNet.

4.3 Arm Movement

The arm is controlled by six servos, each controlled by PWM commands. Keil5 (the standard compiler for STM32-type microcontrollers) is used to control the microcontroller which controls the servos. The Upper Computer sends a PWM ratio to the chip via Bluetooth. Code in the pre-existing Keil5 Library converts the ratios into servo movement.

4.4 Upper Computer

Microsoft Visual Studio is used to control the Upper Computer (below).

The main interface to adjust PWM values of the servos is shown below.

After determining the PWM values of required positions, the values were inputted into the code, forming commands.

The upper then computer transfers image-recognition data and determines the command to send to the microcontroller, based on the object’s identity and location (below).

Results

The results shall be presented in three main sections.

I.       Image recognition rate and effects of external disruptions.

II.     The effectiveness of segmentation.

III.    The precision of grabbing and transporting objects based on their location and identity.

Image Recognition Rate and Effects of External Disruptions

Transfer learning with AlexNet was performed on a CUDA® enabled GPU. During training, images were put through multiple training cycles, called Epochs. Accuracy varied nonlinearly with the number of Epochs. An example relationship between these two variables is shown below.

One can see that the accuracy is highest at Epoch 3 and after Epoch 6. The direct training data was outputted from my Matlab code (left, below). Ideally, the accuracy is one—this arm got very close to this goal (0.97143). Once objects were recognized, their identities were outputted (right, below).

Accuracy also varied with light levels, with both extremely low and high light intensities being unideal. The ideal light intensity is between 500-750 Lux—should this arm be applied industrially, light conditions would have to be maintained within these bounds.

The Effectiveness of Segmentation

To process images, the background (Fig.1) and foreground (Fig.2) were both recorded. Then, a black-and-white “clean foreground” image was outputted (Fig.3). Segmentation was then done by a Gaussian Mixture Model, after which a segmented image was outputted (Fig.4).

The Precision of Grabbing and Transporting Objects Based on their Location and Identity

The robotic arm is able to recognize the identity and location of objects, grab them, and transport them to their allotted boxes.

Conclusion, Discussion & Analysis

Summary

Through the use of deep learning, a robotic arm was constructed that can increase the efficiency of mixed recyclable sorting facilities and reduce negative environmental effects on workers’ health.  In order to further improve this arm, consultation with experts in the field of waste sorting was performed and their opinions on the usage of this robotic arm were sought.

The viability of such an innovation being utilized by waste-management facilities was confirmed by experts that were communicated with.

Utilisation

The first expert that was consulted was Mr. Edwin Del Carmen, Assistant Manager of the Emterra plant in Surrey. Mr. Del Carmen mentioned that soft plastics, apart from becoming a nuisance by getting stuck in fibre screens, also contribute to the sheer amount of physical labour required. It is clear that reducing the contamination by soft plastics throughout the recycling process is key to plant efficiency—a definite application for my robotic arm. In general, the air quality in waste-management facilities is not ideal—some workers do wear protective masks while on their jobs. Utilisation of a robotic arm would reduce the number of workers being exposed to this health hazard.

Many other industry experts were contacted, all of whom expressed approval of my project and gave useful suggestions. Ms. Shuyue Liu, a scientist in the Shanghai Environmental Group, suggested that a row of multiple robotic arms would be able to increase the efficiency of pre-sorting. Mr. Felix Hottenstein of MSS Optical Sorters (a cutting-edge recyclable-sorting technology company) pointed out that this arm could also be used in quality control of sorted waste, presenting a cost-effective solution of removing contamination in the stream of recyclables after sorting, thereby greatly improving the value of waste. Mr. Jeff Stasiuk of GEEP Canada (an industry leader in e-waste recycling in Canada) suggested that this robotic arm would be a benefit to his e-waste facility. E-waste is mechanically crushed, but components are sorted manually. Separating plastic from other components is tedious work and could just as easily be done by robotic arms.

Improvements

While my robotic arm already has reasonable success in handling recyclables, this action needs to be refined should this arm be used in the industry. Refinement definitely is possible, and it could be done in the following ways.

1.     The speed of image recognition could be increased with improvements in computing power.

2.     To accommodate various positions and orientations of objects, reverse kinematics (already existing) could be used instead of simple division into grids.

3.     Although my arm is not fitted with suction, this technology could be employed with a minor adjustment. Suction would allow for better handling of paper and soft plastics.

4.     The ability to recognize objects despite complex layers and/or overlaps would be beneficial. However, more research needs to be done.

Bibliography & References

Bibliography

\”Alexnet.\” Pretrained AlexNet Convolutional Neural Network – MATLAB. Accessed August 21, 2018. https://www.mathworks.com/help/deeplearning/ref/alexnet.html?searchHighlight=alexnet&s_tid=doc_srchtitle.

\”Cdf.\” Cumulative Distribution Function for Gaussian Mixture Distribution – MATLAB. Accessed August 25, 2018. https://www.mathworks.com/help/stats/gmdistribution.cdf.html?s_tid=doc_ta/.

\”Global Mitigation of Non-CO2 Greenhouse Gases: Landfills.\” EPA, November 09, 2016. https://www.epa.gov/global-mitigation-non-co2-greenhouse-gases/global-mitigation-non-co2-greenhouse-gases-landfills.

Kernighan, Brian W., and Dennis M. Ritchie. The C Programming Language. Englewood Cliffs, NJ: Prentice Hall, 2009.

\”Microcontrollers & Microprocessors.\” Mainstream Performance Line, ARM Cortex-M3 MCU with 64 Kbytes Flash, 72 MHz CPU, Motor Control, USB and CAN – STMicroelectronics. Accessed August 19, 2018. https://www.st.com/en/microcontrollers/stm32f103c8.html.

\”Overview of Greenhouse Gases.\” EPA. Accessed November 8, 2018. https://www.epa.gov/ghgemissions/overview-greenhouse-gases.

Siciliano, Antonio. MATLAB: Data Analysis and Visualization. Singapore: World Scientific, 2009.

\”Transfer Learning.\” Transfer Learning Using AlexNet – MATLAB & Simulink. Accessed August 23, 2018. https://www.mathworks.com/help/deeplearning/examples/transfer-learning-using-alexnet.html?s_tid=srchtitle.

\”Understanding Global Warming Potentials.\” EPA, February 14, 2017. https://www.epa.gov/ghgemissions/understanding-global-warming-potentials.

Zhu, Y. \”Tutorial and Illustration: Gaussian Mixture Model.\” Tutorial and Illustration: Gaussian Mixture Model, November 13, 2014. http://yulearning.blogspot.com/2014/11/einsteins-most-famous-equation-is-emc2.html.

References    

Bircanoglu, Cenk, Meltem Atay, Fuat Beser, Ozgun Genc, and Merve Ayyuce Kizrak. \”RecycleNet: Intelligent Waste Sorting Using Deep Neural Networks.\” 2018 Innovations in Intelligent Systems and Applications (INISTA), July 2018. doi:10.1109/inista.2018.8466276.

Ching, Collin. \”How to Build an Image Classifier for Waste Sorting.\” Medium, March 29, 2019. https://towardsdatascience.com/how-to-build-an-image-classifier-for-waste-sorting-6d11d3c9c478.

Emterra. “Useful Terminology at Emterra.” Emterra Group. Accessed October 13, 2018.

Sakr, George E., Maria Mokbel, Ahmad Darwich, Mia Nasr Khneisser, and Ali Hadi. \”Comparing Deep Learning and Support Vector Machines for Autonomous Waste Sorting.\” IEEE, December 08, 2016. doi:10.1109/IMCET.2016.7777453.

Song, J., Ph.D. “Smartening Solid Waste Management: Lessons from Korea\’s Experience.” The World Bank Group. Accessed October 15, 2018.

Statistics Canada: Canada\’s National Statistical Agency / Statistique Canada : Organisme Statistique National Du Canada. October 5, 2018. https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=3810003201.

Statistics Canada: Canada\’s National Statistical Agency / Statistique Canada : Organisme Statistique National Du Canada. October 5, 2018. https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=3810003301.

Statistics Canada: Canada\’s National Statistical Agency / Statistique Canada : Organisme Statistique National Du Canada. October 5, 2018. https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=3810003401.

World Bank. “Islamic Republic of Afghanistan Rapid Assessment of Kabul Municipality\’s Solid Waste Management System.” The World Bank Group. Accessed November 21, 2018.

World Bank. “Municipal Solid Waste Management: A Roadmap for Reform for Policy Makers.” The World Bank Group. Accessed September 30, 2018.

World Bank. “Solid and Industrial Hazardous Waste Management Assessment: Options and Action Areas to Implement the National Strategy.” The World Bank Group. Accessed October 3, 2018.

World Bank. “What a Waste: A Global Snapshot to 2050.” The World Bank Group. Accessed September 29, 2018.

About the Author

Angela is a 14 year old student from Magee Secondary School in Vancouver, BC, Canada. She is fascinated by the subjects of Physics and Computer Science. When she has free time, she enjoys tinkering and fencing.