Goalposts

Writing the code is not the real problem, understanding what code should be written is the real challenge.

So when AI can tease out my clients requirements when they have no clue, or even worse are adamant they know what they need, but are clueless, then I'll be concerned.

And furthermore, try getting regulatory approval in say, the banking or machinery safety sector for a system that you can't explain in detail what it will do for any given circumstance.

This problem of explainability already exists, one of our local banks gave a talk at AI meetup and said they were able to get results with AI/ML that they couldn't roll out because they could not explain adequately to the regulator the exact results it would give in certain circumstances.