Data, analysis, computing and creativity

Asking questions of data can be hard work. We can often struggle to frame the ‘correct’ question, that will give us the answer we need. And while we have tools that can ‘drill-down’ and `slice-and-dice’, when it comes to the crunch it can seem impossible to make any headway with a dataset and discover anything new.

Of course it might be that the insight doesn’t lie within the data that has been collected. But it is useful to have a strategy that enables us to know when to stop, re-frame the question, and start analysing again.

This is a problem that people in the creative industries face. It’s perhaps not described as a problem as such; it’s part of the creative process that people find frustrating, invigorating and ultimately, rewarding.

It is interesting to observe how people approach problems. Some domains have a rigid view that to solve a problem, you must have some degree of expertise in the field of investigation.

I’m sure that many people in the creative industries enjoy the fact that they may face challenges in a variety of domains. That is part of the whole experience for them.

I have witnessed people with very strong data analysis skills, who can provide insight into a domain specific problem that they are not familiar with. They use their limitation as a means to be creative; they can’t provide a response based on their domain knowledge, so they work with what they do have.

They channel the limitation, focus on the data, look for patterns and outliers. They select and apply a number of methods that display the data in different forms. They draw conclusions based in what they see in front of them, so that they can ask pertinent questions of the domain experts. Those questions can yield different answers from the experts, refining the investigation and opening up new lines of enquiry.

Musicians often talk of embracing limitations to foster creativity, especially those who use technology to create music. As music technology equipment becomes less expensive it can be tempting to acquire more equipment. And how much of this applies to us as we look for the next bit of insight from our data?

IIoT devices are becoming cheaper, so we can use those in an almost disposable way now. We are awash with data analysis tools that are free – no expense required to acquire them, and plenty of free online resources that show us how to get started with them.

Quickly we can get into ‘seeking solutions’ mode, rather than actually concentrating on the processes of enquiry. Creativity is the lubricant that we need for data science, and it’s worthwhile arming ourselves accordingly.

We can achieve a lot more with our data if we stop acquiring tools and focus on acquiring:

• basic data analysis skills;
• an understanding of creative problem-solving processes;
• open questioning techniques that can help define the questions to be asked;
• experience of the above with small-scale pilot investigations.

One way forward might be to consciously approach some data analysis using the lowest-cost approach possible. Cheap sensors, small embedded devices like Arduino or Raspberry Pi, and your favourite spreadsheet, together with some basic stats, provide sufficient capability and focus to prevent unnecessary distractions.

The last thing we need when looking for a needle in a haystack is distraction. Let’s not be distracted from the outset and let creativity guide us.