Why is it so hard to adopt Machine Learning?

There are plenty of advocates of Machine Learning (ML) at the moment. Many of them have something to sell, which of course uses ML (along with Artificial Intelligence, AI).

ML should be able to help businesses though, particularly if we turn down the hype and see it for its strengths; it’s excellent as an enabler of automation, and business operations are full of repetitive tasks.

While larger organisations often have enterprise software systems to administer operations, most of the other businesses have systems that are assembled from collections of software products that may be off-the-shelf or bespoke. Again, this is fertile ground for enhancing integration between systems through automation.

But why do companies find it difficult to achieve automation with ML?

I think that we forget how capable humans are when compared to computers.

IT systems definitely have better memories for transactional data, and the recall of data is far superior to what we can achieve. However, when we see a process and decide that it is automate-able, the challenge suddenly becomes greater.

For instance, if we think about daily operations such as processing employee timesheets, or absence notes, or purchase requisitions, etc. these all seem straightforward when carried out by staff. The actual complexities of these processes are hidden by the capabilities of the person who is trained to process them.

How many actual variations in these process steps can occur?

What tiny decisions are reasoned about, to enable the process to continue, that would flummox a rule-based, automated system?

It’s only when we look at the detail that we see the value of staff and the extent of their contribution to our daily operations. This can be difficult to automate without considering the wider case for how we must fuse data from different sources so that the process step can be concluded.

Too often we approach a process with a view to automating it, and we unwittingly restrict ourself to the data that is required to complete the transaction. The wider set of contextual data – outside, but surrounding the process data – is not captured, and as such the ability to automate is also restricted.

Looking at a broader picture also give ML a better chance of working. Many applications in themselves produce insufficient data upon which a useful ML model can be trained, and this becomes yet another reason for businesses to distrust the apparent alchemy of ML.

Essentially, ML adoption is hard because it involves change. We need to change our outlook when it comes to data; we should embrace the complexity and start thinking about how we can usefully fuse data sources together, after decades of creating silos to manage it.