Human psychology, shaped by evolution, tends to treat the unknown as potentially dangerous. This partly explains public caution toward artificial intelligence (AI) and the common perception that AI systems are impenetrable "black boxes." In an edited excerpt from his book Robot, nem pilóta, AI researcher Luc Julia argues that the black-box concept is often mistaken and not unique to AI.
Gaston Julia, Benoît Mandelbrot and the fractal lesson
The history of fractals illustrates how the feeling of a "black box" can vanish once something is presented understandably. Gaston Julia published a mathematical formula in 1914 whose meaning was hard to convey to non-specialists. In 1955, Benoît Mandelbrot, a former student of Julia, visualized the same formula using a computer, producing a fern-like image that instantly made the mathematics comprehensible. The example shows that with time and clear presentation, the illusion of a black box can disappear.
What fuels the AI black-box myth?
The author names two related phenomena behind the myth: perceived unpredictability and perceived mystery. Modern AI systems operate with "billions of data points and parameters," and designers build algorithms to shape their behaviour. Nevertheless, unexpected situations can arise—bugs, deviations from planned operation, or simply randomness—where the system behaves differently from what was anticipated. These occurrences, however, do not necessarily mean the system is fundamentally unexplainable.
The text notes that the black-box label would only apply if no one could interpret or explain a system’s operation. Since AI creators can usually account for the data and algorithms they used, these systems are not absolute enigmas.
Sources of unpredictability
The article distinguishes three main sources of unexpected AI behaviour:
- Intentional randomness: Generative AI often includes randomness, which is necessary to produce varied outputs from the same prompt. In such cases, unpredictability is a designed feature.
- Design and programming errors: If operational constraints are not specified precisely, a system can become "uncontrollable," or it may suffer from logical errors or corrupted data.
- Security flaws and failures: Poorly protected systems or hardware/software failures can cause behaviour to deviate from intention.
The author emphasizes that these deviations ultimately trace back to human decisions or mistakes. Even seemingly inexplicable behaviour can be explained retrospectively by uncovering the design choices or the presence of randomness.
A concrete example: Tesla's screwing robots
As an example, the article discusses Tesla's screwing robots. These robots are effective at locating a screw, identifying the car component in front of them, and screwing the fastener into the correct place. The robot does not "think"—it executes programmed instructions such as "move your arm right" or "tighten."
The author points out that accidents remain possible if randomness is programmed in or errors occur. But such incidents do not imply the robot chooses to commit harmful acts; any unexpected behaviour would stem from designer decisions or malfunctions, not an autonomous intent. From this example the article draws three conclusions:
- there is no inherently unexplainable action;
- an unexpected event usually results from a designer’s decision or fault;
- humans determine what is permitted and what is not.
Is explainability an achievable goal?
Explaining AI behaviour in detail can be complex and time-consuming due to vast datasets and computational complexity, but it is not impossible. The article highlights two important points:
- AI designers select the data and algorithms, so the operation of systems is a question of design rather than mysticism.
- It is possible to build an AI system capable of tracking and explaining another AI’s operations—AI can be used to interpret AI.
The historical analogy is that humans have adapted to technologies that once seemed to put us at a disadvantage: instead of abandoning control, we develop tools to regain oversight, just as law enforcement adopted cars to pursue criminals who used cars.
Conclusions
In Luc Julia’s view, the characterization of AI as a "black box" is often more an issue of perception and communication than an absolute technical reality. Most unexpected AI behaviour can be traced back to human design choices, randomness intentionally introduced by designers, or failures—and there are methods to monitor and explain systems, including the use of other AI tools. "Unexplainability" is therefore not an intrinsic property of AI but often the result of insufficient effort devoted to transparent explanation.
The above text is an edited excerpt from Robot, nem pilóta by AI researcher Luc Julia.
