Some simple AI's have "explain" s/w that allows them to expand on how they arrived at a particular answer. But, usually, the explanation generated in this way is just as obscure as the original answer that you're asking the AI to explain.

Here's a non-linear planner solving the simple "number puzzle".

The object is to sort the numbers from 1-15 (plus "empty") into order.

A human usually does it by concentrating on simple goals like "get number 1 into the first position, then get 2 into its position, then ....". But this particular software (s/w) tries to solve all goals in parallel -- the efficient way.

Dim lights Embed Embed this video on your site

Looking at the way it's solving the puzzle is surprising. It's hard to see why the s/w is making SOME moves while other moves seem quite obvious. But in the end, you can see it has been making moves to get all the numbers into their right positions as efficiently as it can (the solution it finds is still no guaranteed to be optimal).

Yes, there's quite a communication gap between Artificial Intelligence and Human Intelligence...