In many fields (not just Data Science) people talk about the need to have problem-solving skills as a key requirement for all roles. I can’t recall coming across any explanation of what these skills are, which makes it hard to help people develop them.
Off the top of my head, I would say that the following non-exhaustive list constitute problem-solving skills:
Make sure you understand the end goal.
Sometimes you can identify how to bypass the problem entirely by realising the goal has been misunderstood. There could be misalignment, wrong choice of metrics or tools, or a misunderstanding of what a good outcome looks like. Miscommunication happens a lot.
Tracing the root cause.
Some professions call this root-cause analysis or “the 5 whys”. Each question takes you deeper to the root cause of the issue.
Problem: We spend a lot of time cleaning our text data.
Why #1: Our automated cleaning can’t catch all the exceptions.
Why #2: We frequently have new variations in our input to account for.
Why #3: We don’t have data validation on the form input.
Why #4: We can’t agree with the Marketing team on what the selections should be.
Why #5: Marketing team doesn’t have shared metrics they’re working towards.
So instead of trying to build more robust and advanced data cleaning systems, it seems like it’s better to work with the Marketing team to translate their objectives into shared metrics that you can build towards.
Being able to break a large problem down into smaller components.
This makes each one easier to address.
If someone says a report they receive isn’t useful, it could be for a variety (or combination) of reasons: irrelevant content, they don’t get the info they need or it’s presented in the wrong way; wrong frequency, they need it more or less often; wrong recipient, someone else actually needs the information; wrong format, report should actually be an alert etc. Now you can hone in on fixing the right problem.
Figuring out how you will test if the problem is solved.
This is partly addressed by first breaking the problem down into its components, and then by choosing a fix that helps you identify if the problem is solved. One common anecdote you might hear is the problem of a data team not being able to maintain all the dashboards in production and that they don’t know if anyone uses them. The test often flouted is to take a dashboard offline and see if anyone complains.
Recognizing when you haven’t broken down the problem enough.
If you’re struggling to pin down what a good test of a fix is, or what a successful result looks like, then you may need to break it down further.
Using the process of elimination to understand which components you’ve solved.
When testing a fix, you’ll sleep much better knowing which problem you fixed if you can eliminate the causes individually. This isn’t always possible but e.g. if you changed a report format, and the recipient, and the content, and the frequency all at once, and it works, then you won’t know which element/s were the problem. This makes it hard to prevent the same issue happening again.
Developing thought experiments to explore “what if?”.
I find these handy by testing extremes to figure out the boundary of the issue e.g. what if X always happens? What if it never happens? What if the value was infinite, what if it was zero? What if it didn’t exist at all?
Recognising all the resources at your disposal to solve the issue, and searching for new ones if needed.
This could be time, money, or more often than not, expertise of others.
These skills have come in handy for me across my professional and personal life, and will continue to be useful in future.