How we Used Gesture Recognition to Save Lives (And Tons of Gold and Diamonds!)

Hello, world! We are Warden AI Lab, a Latvia-based start-up studio. We would like to share with you how we use available democratized technology to help large and medium-sized enterprises save thousands of lives, millions of life-years, tons of gold and diamonds, and billions of dollars. We create thousands of new jobs, too, in AI/ML/DL/CV/MLOps.

We focus on gesture recognition and behavioral video analytics as well as create start-ups based on a turnkey solution (3–6 months) tailored to customers/industries.

Here are our ten cases, sad or funny, with lessons learned.

Case No.1: How Gesture Recognition prevents stealing

• Company: Gold processing plant
• Size: 8,000 employees
• Location: Eurasia
• Capacity: 16 tons of gold per year

A mine or quarry supplies gold-bearing ore to the gold mill where the ore is crushed, graded, sorted, and smelted into ingots. Crushed into sand, the ore goes to the gravity table where one of the sorting processes takes place. At this stage, the control over the employees was not very strict and as a result... the mill experienced a gold shortage of 7%. In other words, out of a kilogram of gold from the conveyor, only 930 grams made it to the final stage. More than a ton of gold "vanished" annually. A ton of gold costs $57 million. With the global gold production amounting to 3,000 tons per year, 20 tons of gold get siphoned off ($1 billion) – and this is at the plant, let alone the mine and transportation.

Attack Scenario 1:The malcontents would sometimes put a stick on the gravity table to build up a pile of gold sand and then scoop the sand with their hands straight into a pocket. Then they would put it in a handkerchief and make a tricky gear of yarn and nylon stockings to transport the sand through the ventilation system to the outside. In slang, this is called "letting the horse through the vent.”

Attack Scenario 2:The perpetrator would hide a bucket with gold sand in a tank with chemicals to be taken out of the security perimeter.

Attack Scenario 3: In the smelting shop, doré alloy would inadvertently be spilled on some item that has been deliberately put there by a steelworker, and the perpetrator would go home where a "golden shoe" that left the factory via ventilation would be waiting for him.

Solution:We installed 17 IP cameras and 3 servers with specialized software. The solution cost $800,000. The project roll-out took 7 months. It involved six specialists.

The video analytics software could recognize 20 target scenarios of “non-standard behavior”, such as when the operator’s hand is in the wrong place at the gravitation screener; the operator is not holding a special brush; the operator is touching the gold-accumulating box; the operator is sealing the gold concentrate carrier in a wrong way; etc.

Result: The installation of the video analytics system resulted in a loss reduction from 7% to 2%. It is still a mystery though where the remaining 300 kg of gold vanishes yearly. Extrapolating these results from one enterprise to global gold mining, the implementation of video analytics could reduce the loss of gold 3–7 times, from 20 tons per year to 3–6 tons.

Case No.2: How Gesture Recognition prevents armed uprising

• Company: Diamond mines
• Size: 1,500 employees
• Location: Africa

Issue:Mining truck drivers do not comply with work safety regulations and get behind the wheel tired and sleepy, which leads to the overturning of trucks or fatal accidents. This is especially the case during Ramadan when you are not allowed to eat during the daytime for religious reasons in sub-Saharan Africa. Statistically, this leads to two major accidents during Ramadan each year.

When on-site, we witnessed a driver falling asleep behind the wheel of a 100-ton dump truck and crashing into a repair and warehouse complex, killing two people in front of dozens of workers. The workers got up in arms, seized the mine, and blamed the accident on the management of the European firm. The British management team and their families were urgently evacuated by plane to a neighboring country.

Solution:Install 2 cameras on each dump truck to capture behavioral signs of fatigue. The cost of equipping one truck would amount to $80,000 on a turnkey basis, including communications equipment and climate hardware edition. Deploy an operations control center.

Result: The solution was not implemented for religious reasons – video cameras, presumably, steal souls. It’s like GDPR restrictions, only you can't comply with them.

Case No.3: How Gesture Recognition prevents in-transit theft

• Company: Diamond mines
• Size: 1,300 employees
• Location: Africa

Issue:A road train stops halfway between the mine and the ore processing and diamond extraction plant and reports a breakdown over the radio. Meanwhile, 30 to 40 specially trained people from a neighboring village unload what they consider the most promising pieces of ore, up to 2 tons per hour, if we rely on the truck scales. This would happen virtually every other night, in different spots along the 40-kilometer route, on random targets, contributing to the development of small jewelry businesses in the region more than any state-run development program.

Solution: Video analytics on board the dump trucks enabled monitoring of the vehicle sides and the driver's behavior to distinguish real breakdowns from spontaneous attacks and malicious intent. The system transmitted the data to the security task force. After 3 months and a series of high-profile incidents, in-transit ore losses were reduced by 72%. Where the remaining 28% go, we do not know, we would only advise having the truck scales checked more frequently.

Case No.4: How Gesture Recognition prevents misuse of hazardous equipment

• Company: Diamond mines
• Size: 1,300 employees
• Location: Africa

Issue:Imagine a work settlement with almost 10,000 people. Out of them, one person officially undergoes full training as a dump truck driver and receives all the certificates and papers. At night, the driver brings in an outsider to the premises who looks pretty much like the driver, puts him behind the wheel of the dump truck, and trains him. As a result, one driver with all the legal documents works on three sites almost simultaneously, when in fact the trucks are driven by the illegals who were trained in this improvised underground training center at night, with all the ensuing incidents, risks, and responsibilities pertinent to no one.

Solution: We installed cameras in mining trucks to monitor driver fatigue. The cameras were augmented with face recognition functionality. The collected proprietary library made it possible to recognize 319 incidents in 1 month. As a result, the number of accidents and breakdowns of dump trucks, average fuel consumption, and loss of tools decreased. The number of drivers was also cut down, by almost a dozen. As for the economics of this security and safety use case, attracting and training new people turned out to be incommensurably cheaper than it seemed to the HR department at first.

Case No.5: How Gesture Recognition prevents non-compliance with safety regulations at a power plant

• Company: Thermal power plant
• Size: 914 employees
• Location: Eurasia

Issue:The customer wanted to install a video analytics system to check the completeness of personal protective equipment used by employees when leaving the administration complex and going into the production area (7 checkpoints in total).

Solution:16 IP cameras and 1 server. The project costs totaled $200,000. The software automatically recognized the presence of a helmet on the head, gloves on the hands, boots on the feet, and distinguished jeans from protective pants.

Result:The solution detected gross violations, such as wearing hockey helmets with a color design similar to protective helmets; wearing safari helmets; or wearing shorts in a workshop.
We also identified a mini-pride of seven cats living in the shop.

To the amusement of some, we also discovered that every Friday the employees would bring a sauna broom to enjoy a free improvised sauna resulting from a pipe crack in the steam generation shop. Amazing!

After 14 months of operation, the number of incidents was down 83% in comparison to the previous year. The project paid off more than tenfold because persistent offenders were de-bonused and incidents avoided.

Case No.6: How Gesture Recognition prevents major breakdowns

• Company: Coal enrichment facility
• Size: 510 employees
• Location: Eurasia

Issue:Frozen oversize lumps of coal were supplied from the storage area to the conveyor belt and then to the crushing equipment, clogging its overlaps and causing shutdowns.

Solution:The solution would be an analytics system comprising 6 IP cameras and 1 server at a cost of $200,000 to monitor the conveyor belt and notify the operator about detected oversized lumps of coal (nicknamed "snowmen").

Result: To the customer, these costs seemed unreasonable. Sometime later, however, an especially large lump of frozen coal caused a most unwanted 10-ton cylinder breakdown. Production stopped for four months – the spilled coal had to be unloaded manually, using portable conveyors, since the machinery could not access the venue. Such incidents required more resources from the already understaffed factory. According to various estimates, the losses amounted to about $2 million.

Case No.7: How Gesture Recognition prevents fatal accidents in heavy industries

• Company: Steel mill
• Size: 21,848 employees
• Location: Eurasia

Issue:The company used equipment made in the 1960s. As a result, to meet the production targets some processes had to be completed "manually", i.e. people would go into the hazardous working area of the mill where objects of red-hot metal periodically flew out at a high speed, such as beams, rebars or other hot-rolled products. Every year, 12 to 16 people were brutally killed by what they called the "kebab" method.

Solution: Installation of a video analytics system comprising 5–10 cameras per shop. The system would detect a person accessing the hazardous zone or predict their trajectory – and automatically slow down or even safely shut down the production process. The price is $100,000 per shop. Taking into account social payments and other previous losses from incidents, the economics of this use case brings up to $500,000 a year in savings.

Case No.8: How Gesture Recognition reduces downtime on the railroad

• Company: Railway
• Size: 696,000 employees
• Location: Europe

Issue:As Frederick Taylor stated, there are more than 40 ways to shovel coal and only one right way to do it. The client wanted to check the manual actions of railroad employees against standard operating procedures. Which actions did employees perform? What practices did they follow? Did they use the proper tools? How long did it take them to perform an action? A documented benchmark is used to this end – a flow chart, or as the Germans fascinatingly simply put it, Arbeitsplanstammkarte.

Solution:Four huge servers receive video fragments collected at the locations where service procedures are performed on the road. Smart cameras ensure compliance with GDPR as they only send anonymized videos with blurred faces for processing. Next, the software recognizes typical gestures, tools, and the sequence of their application. Based on special markers, the software understands where one step of the procedure has ended and another one has started.

Result: We detected hundreds of gross violations, such as high-speed trains running on rails disconnected from sleepers; tools placed dangerously close to moving coaches; actions performed in a random sequence; or lifting heavy weights with your back, not with your legs, above all. In the same way, we learned that a lack of standardized shovels is a global problem, while standardized practice makes perfect and leads to a 2–4% increase in manual labor productivity annually. Also, when people are aware of being monitored, downtime reduces by more than 50%.

Case No.9: How Gesture Recognition helps with post-stroke rehabilitation

• Company: Post-stroke rehabilitation
• Location: Europe

Issue: In the European Union, stroke is the second leading cause of deathand a common cause of disability among adults. Every year, around 1.1 million people have a stroke, resulting in 440,000 deaths. Estimated stroke-related costs come to EUR 45 billion, including direct and indirect expenses on healthcare and performance losses. 5–7 million disability-adjusted life years are lost because of stroke per year. Latvia suffers most from stroke and its implications.

Solution:We developed a product to curb expenses on post-stroke rehabilitation. Trials show that neurologists can see 2–2.5 times more patients by using our product. With basic equipment and advanced gesture recognition technology, patients can get effective rehabilitation support.

A screen displays gamified tasks and gestures to be reproduced by a patient, while the software assesses the accuracy of the reproduced gestures, keeps a progress log, and provides visual or spoken prompts.

Unlike other costly solutions that employ 3D cameras and Kinect, our product uses affordable equipment (a regular computer + a web camera) and helps healthcare staff double their performance.

Case N.10: How Gesture Recognition facilitates the education process

• Company: Low-voltage household and consumer electronics manufacturer
• Size: 10,000 employees
• Location: Europe

Issue: Manual assembly of circuit breakers.

The company produces 150,000 units per year. Forty people assemble the devices. An operator at the assembling station has 50+ boxes with components. The boxes may look very similar, i.e. 10-ohm resistors look pretty much the same as 20-ohm resistors. A tired person can easily get confused and take the wrong component with different parameters causing accidents and fires.

Solution:We used behavioral analytics to implement a system designed to facilitate decision-making during assembling operations, i.e. which component to take next and from which box.

Equipment:2 cameras per desk, up to 8 cameras for server. The pilot cost EUR 120k.

Result: The error rates dropped by 61%, and training time was reduced from 1 week to 1 day.

Conclusion

On the one hand, we are happy to acknowledge that our efforts save lives and strengthen health. Every day, 600 workers are killed in work-related accidents globally, resulting in 200,000 deaths per year. Australia alone registers 120,000 injuries at work, with one injury costing on average $13,000 and 6 man-weeks of downtime. Available democratized solutions help businesses all over the world improve safety and take care of their people.

On the other hand, we embrace fast-growing markets, such as

• Workplace safety (the global market is projected to grow from an estimated $14 billion in 2022 to $26 billion by 2027)
• Video analytics (the segment is expected to grow from $7 billion in 2022 to $20 billion in 2027)
• Behavioral analytics (the market is to grow from $200 million in 2016 to $3.5 billion in 2024
• Gesture recognition (market size was valued at $14.08 billion in 2021and is anticipated to reach $70.18 billion by 2030

Which is very appealing to venture investors.

We strongly believe that the start-up mindset will help tackle humanity's major challenges and our mission is to contribute to the process and make a difference.