When Algorithms Assume
Artificial Intelligence is a major topic today, not just in communications and Contact Centers, but across business. Most of the current crop of business-focused “AI” is in fact Machine Learning (ML). In ML, a large data set is used to “train” the algorithm to predict an outcome based on a range of interacting variables. Using modern data sets that have millions and even billions of test sets and thousands of variables, the ML program is able to train a neural network to predict an outcome. Once the parameters of the neural network are set, the ML algorithm can then be used to predict the probability of an outcome in future data sets.
The capability of ML is incredible, given a large and valid database. ML for speech recognition has improved dramatically. But AI/ML is in much of our everyday lives, too. The suggestions of movies to watch or products to buy based on your previous watching/buying patterns are used to predict other things you may like. The pop-ups on Amazon or the “for you” from your video provider are examples. Beyond shopping, AI/ML is at the core of services like Waze that use a combination of real-time data and historical data sets to make an amazingly accurate prediction of when you will arrive. This technology is being implemented in an array of applications, from predicting failures in parts to monitoring complex eco-systems. While there are many discussions about AI becoming sentient and threatening, a prediction made by an ML solution is only as good as the underlying data set. ML doesn’t create new data; it analyzes existing data to predict an outcome with a new data point.
When implementing your first AI/ML project, before beginning to “train” the algorithm, experts emphasize that the most important step is to “clean” the data. Any erroneous or mis-entered data needs to be cleaned/corrected. In fact, in most AI/ML projects, getting a clean valid data set is one of the most critical, and time consuming, steps. As they say, “garbage in, garbage out.” The quality of the ML prediction is only as good as the underlying data set and the quality of the data sets for which prediction is to be made.
But what happens if the well cleaned and valid data set that is being used is invalidated by an event that changes the actual data relationships so significantly that the huge historical data set is no longer valid? Recently I experienced two examples of this, and I believe it is an important part of assuring the validity of an AI/ML implementation.
The first was in late March, in the beginning of the shelter-in-place in California. I needed to make a trip to Moss Landing, south of San Francisco, about 80 miles southeast of where I live. As I was going to get some photographs, I felt I could stay socially distant. While there, I stopped at a local fish monger (selling from a boat in the harbor with great social distancing) and bought a couple of Dungeness crabs, with the warning that they should be cooked in no more than 60-90 minutes as I set off home. As it was a Friday, normal commute traffic would be abysmal, resulting in a 2-hour plus journey. I was not really concerned as the early shelter traffic was minimal. When I put my home address into Waze, I was surprised by the projected trip time, Waze said it would take 2 hours and 5 minutes to get home. But, as I drove, the arrival time continued to get earlier and earlier. In the end, it was just over an hour drive with almost no traffic on the road. This was what I expected, but not what Waze predicted.
Clearly, Waze must use the historical arrival rate of thousands of commuters to predict future daily traffic levels based on historical data. And the data set clearly showed that, with typical commute levels, it would be a terrible drive. But, with 95% of commuters staying home, the algorithm was fooled. Similarly, as cars returned to the roads, the most recent data would be less useful, though the rate of change would be more gradual.
The second change was in the weather forecast. During the August fires in the Bay Area we also had a predicted increase in temperatures. On one specific day, weather.com predicted over 90 degrees for the next day. We woke up to the red sky day of smoke from a plethora of local fires. In fact, the sun never broke through the smoke that day and the temperature never went above about 70 degrees. Again, the smoke was such a significant change that the weather data set predictions were no longer valid. Smokey skies were not included in the weather ML algorithm’s test sets as part of the neural networks’ outcome predictors.
Both were the result of so called “Black Swan” events: a change so dramatic it was beyond the prediction range of common knowledge. A Black Swan event is something that common knowledge/belief says cannot exist or will never happen. In the 1600s, Europeans could not imagine black swans as all European swans are white. When Europeans arrived in Australia and saw black swans for the first time, the “all swans are white algorithm” was upended.
Similarly, the concept of a large-scale office shutdown due to a pandemic or fires that blackened the sky were both unimaginable before they happened. In both cases, the algorithms that were incredibly accurate using historical test sets became virtually useless as the data set was no longer valid for current conditions. One clear lesson from this is that a major event that impacts sunlight (fires, volcanoes, asteroids) can really have a major impact on temperature and the global eco-system. It was easy to imagine what life would be like if that red sky day were to become normal or worse for a year or more as a major volcano eruption might produce.
Beyond the life lesson that Black Swan events can really happen, for enterprises implementing AI/ML, this brings up an important point to consider. When you define the AI/ML data set, have you considered the business circumstances where that data set may be invalid, either due to an external or internal Black Swan type event? Understanding and defining potential Black Swan events will enable creating guard rails that will signal if the data set is no longer valid due to factors that were not in the data but which have arisen since the ML algorithm was trained.
One way may be to install instant checks of verifiable predicted variables. For example, if Waze had generated estimated car counts or speeds for specific freeway areas during the commute, those could be compared to the actuals. If the real-time variance goes over a specified percentage error rate an alarm could be triggered that could introduce either automated or human intervention. For example, if Waze saw a large number of bad estimates on these test cases in the first day, immediately a message could pop up indicating that “estimates may be long due to unpredictable traffic decreases.” While the accuracy might not have immediately improved, it would warn the customer that normal high levels of accuracy were currently not being delivered.
Another analysis factor is to closely monitor the trending in outcomes. By establishing in-bounds ranges for outcome percentages, major shifts in outcome predictions can be a warning sign also. AI/ML can also be used for the analysis of the aberrations and whether they are a valid change or a change that is outside of the data set model. As the most critical time for algorithms and their predictive value may be in times of stress (like the time on Waze being used by a first responder), understanding the factors in your business and business process that might invalidate the AI/ML models is critical.
Tags
Start YourCustomized Search
SOLUTION AREA
SOLUTION PROVIDERS
- 8x8 (40) Apply 8x8 filter
- Alcatel-Lucent Enterprise (50) Apply Alcatel-Lucent Enterprise filter
- AT&T (44) Apply AT&T filter
- AudioCodes (49) Apply AudioCodes filter
- Avaya (397) Apply Avaya filter
- Cisco (574) Apply Cisco filter
- Dell (11) Apply Dell filter
- Five9 (55) Apply Five9 filter
- Fuze (39) Apply Fuze filter
- Genesys (100) Apply Genesys filter
- HP (98) Apply HP filter
- IBM (171) Apply IBM filter
- Jabra (9) Apply Jabra filter
- Logitech (56) Apply Logitech filter
- Lumen (4) Apply Lumen filter
- Masergy (50) Apply Masergy filter
- Microsoft (766) Apply Microsoft filter
- Mitel (233) Apply Mitel filter
- NEC (128) Apply NEC filter
- Nectar (58) Apply Nectar filter
- Polycom (95) Apply Polycom filter
- Ramp (37) Apply Ramp filter
- RingCentral (127) Apply RingCentral filter
- Sennheiser (18) Apply Sennheiser filter
- Slack (13) Apply Slack filter
- Tata Communications (59) Apply Tata Communications filter
- Unify (186) Apply Unify filter
- Vonage Business (80) Apply Vonage Business filter
- Yealink (8) Apply Yealink filter
- Zoho (6) Apply Zoho filter
- Zoom (22) Apply Zoom filter
- Acme Packet (24) Apply Acme Packet filter
- Allworx (2) Apply Allworx filter
- Arkadin (22) Apply Arkadin filter
- Aspect (34) Apply Aspect filter
- BT (25) Apply BT filter
- CaféX (8) Apply CaféX filter
- CallTower (14) Apply CallTower filter
- Clarity Connect (10) Apply Clarity Connect filter
- Continuant (1) Apply Continuant filter
- Damaka (4) Apply Damaka filter
- Dialogic (5) Apply Dialogic filter
- Dimension Data (44) Apply Dimension Data filter
- Empirix (11) Apply Empirix filter
- Enghouse Interactive (17) Apply Enghouse Interactive filter
- Inference Solutions (9) Apply Inference Solutions filter
- IntelePeer (27) Apply IntelePeer filter
- IR (11) Apply IR filter
- Jive (21) Apply Jive filter
- Kurmi Software (21) Apply Kurmi Software filter
- Lifesize (33) Apply Lifesize filter
- Lightware (3) Apply Lightware filter
- Mavenir (6) Apply Mavenir filter
- Modality Systems (8) Apply Modality Systems filter
- Momentum (36) Apply Momentum filter
- Netfortris (5) Apply Netfortris filter
- NetSapiens (6) Apply NetSapiens filter
- NewVoiceMedia (31) Apply NewVoiceMedia filter
- Nureva (26) Apply Nureva filter
- NUWAVE (5) Apply NUWAVE filter
- Orange (32) Apply Orange filter
- OVCC (8) Apply OVCC filter
- Panasonic (18) Apply Panasonic filter
- PanTerra Networks (9) Apply PanTerra Networks filter
- ScanSource (21) Apply ScanSource filter
- SIPPIO (3) Apply SIPPIO filter
- Snom (20) Apply Snom filter
- Star2Star (8) Apply Star2Star filter
- StarLeaf (12) Apply StarLeaf filter
- Tadiran Telecom (2) Apply Tadiran Telecom filter
- TekVizion (9) Apply TekVizion filter
- Unimax (7) Apply Unimax filter
- Verint (44) Apply Verint filter
- Voice4Net (2) Apply Voice4Net filter
- VOSS (85) Apply VOSS filter
- Voxbone (14) Apply Voxbone filter
- West (28) Apply West filter
- XO Communications (3) Apply XO Communications filter
- Yorktel (17) Apply Yorktel filter
- Zultys (2) Apply Zultys filter
- 3CX (8) Apply 3CX filter
- ADDASOUND (1) Apply ADDASOUND filter
- Aerohive (1) Apply Aerohive filter
- Aryaka (1) Apply Aryaka filter
- Asurion (22) Apply Asurion filter
- Avnet (7) Apply Avnet filter
- Bandwidth (5) Apply Bandwidth filter
- Calabrio (5) Apply Calabrio filter
- Consilium Software (13) Apply Consilium Software filter
- Drum (5) Apply Drum filter
- ESI (6) Apply ESI filter
- Esna (16) Apply Esna filter
- Exinda (2) Apply Exinda filter
- EZuce (3) Apply EZuce filter
- GUnify (6) Apply GUnify filter
- Highfive (4) Apply Highfive filter
- Huawei (47) Apply Huawei filter
- Imagicle (3) Apply Imagicle filter
- IPCortex (1) Apply IPCortex filter
- KnoahSoft (1) Apply KnoahSoft filter
- KOVA (1) Apply KOVA filter
- Logmein (9) Apply Logmein filter
- Metropolis Technologies (4) Apply Metropolis Technologies filter
- Mutare (2) Apply Mutare filter
- NextPlane (27) Apply NextPlane filter
- Ooma (16) Apply Ooma filter
- Patton (11) Apply Patton filter
- Radish Systems (1) Apply Radish Systems filter
- Radisys (3) Apply Radisys filter
- Shango (1) Apply Shango filter
- SMART (163) Apply SMART filter
- Stack8 (1) Apply Stack8 filter
- Swyx (1) Apply Swyx filter
- TrueConf (4) Apply TrueConf filter
- UJET (13) Apply UJET filter
- Voximplant (3) Apply Voximplant filter
CONTENT TYPE
- BC Expert Insights Market (44) Apply BC Expert Insights Market filter
- BC Expert Insights Objective - Vendor Neutral (42) Apply BC Expert Insights Objective - Vendor Neutral filter
- BC Expert Insights Planning (15) Apply BC Expert Insights Planning filter
- BC Expert Insights Solution (12) Apply BC Expert Insights Solution filter
- BC Expert Insights Vendor (82) Apply BC Expert Insights Vendor filter
- BC Expert Insights Vendor Solution (150) Apply BC Expert Insights Vendor Solution filter
- BC Expert Roundtable (136) Apply BC Expert Roundtable filter
- Bcs Webinar (0)
- Bcs Webinar Registration (0)
- Best Practice (38) Apply Best Practice filter
- Buyer Guide (14) Apply Buyer Guide filter
- Case Study (29) Apply Case Study filter
- Executive Interview (145) Apply Executive Interview filter
- Expert Roundtable (453) Apply Expert Roundtable filter
- Guest Contributions (35) Apply Guest Contributions filter
- Multimedia (38) Apply Multimedia filter
- News Analysis (2081) Apply News Analysis filter
- Newsfeed Article (1303) Apply Newsfeed Article filter
- Newsfeed Article (1) Apply Newsfeed Article filter
- Thought Leadership (21) Apply Thought Leadership filter
- Vendor Collateral (211) Apply Vendor Collateral filter
- Vendor Resource Best Practices (24) Apply Vendor Resource Best Practices filter
- Vendor Resource Buyers Guides (2) Apply Vendor Resource Buyers Guides filter
- Vendor Resource Multimedia Content (4) Apply Vendor Resource Multimedia Content filter
- Vendor Resource White Paper (4) Apply Vendor Resource White Paper filter
- Webinar (13) Apply Webinar filter
- Webinars (7) Apply Webinars filter
- White Paper (64) Apply White Paper filter
MORE FILTERS
INDUSTRY
- Banking And Investment (800) Apply Banking And Investment filter
- Education (446) Apply Education filter
- Energy And Utilities (487) Apply Energy And Utilities filter
- Finance (12) Apply Finance filter
- Government (675) Apply Government filter
- Healthcare (482) Apply Healthcare filter
- Hospitality (180) Apply Hospitality filter
- Insurance (100) Apply Insurance filter
- Manufacturing (704) Apply Manufacturing filter
- Media/Publishing (422) Apply Media/Publishing filter
- None (43) Apply None filter
- Professional Services (745) Apply Professional Services filter
- Retail & Distribution (798) Apply Retail & Distribution filter
- Technology (1611) Apply Technology filter
- Transportation (110) Apply Transportation filter
Comments
There are currently no comments on this article.
You must be a registered user to make comments