elektropas.com

The most common mistake in your conversation with your software vendor about AI

Many companies only ask their software vendor whether a system 'uses AI' and accept a brief 'no' or 'yes, but harmless' at face value. This mistake can be prevented with the right questions and documentation.

The most common mistake in the conversation with your software vendor about AI is that companies settle for a short answer to the wrong question. The question asked is 'does your system use AI?', the vendor replies 'no' or 'a bit, but that's harmless', and that's the end of it. No one documents what exactly was asked, what the answer entailed, or whether that answer is based on anything.

You can prevent this mistake by structuring the conversation differently: not as a yes-or-no question, but as a question seeking substantiation. What exactly does the system do, what decisions does it make or support, and on what basis does the vendor claim that this does or does not fall under the AI Regulation. You document that answer in writing, not for the sake of it, but because you must later be able to show what question you asked and what answer you received.

Why this conversation often goes wrong

The conversation with a software vendor usually takes place via an account manager or helpdesk, not via someone familiar with the AI Regulation. That person wants to reassure you, and a brief reassurance is given more quickly than a precise answer. Moreover, much software contains AI features that have been built in by an underlying vendor, for example a language model that has been purchased as a separate component. The party you are speaking with may not always know exactly what is happening under the hood.

The result is that companies receive an answer that is factually correct for the person giving it, but not complete for the situation in which the product is used. search the knowledge base for your type of software shows that the Regulation distinguishes between different roles and risk categories, and that the answer to the question 'is this AI' can rarely be given in one word.

What the mistake looks like in practice

An installation company purchases a scheduling package with a function that automatically allocates maintenance appointments. The question to the vendor is 'does this contain AI', the answer is 'no, that's just smart scheduling'. Six months later, the scheduling function turns out to have been expanded with a predictive model, without a separate discussion about it. No one at the installation company noticed this, because the original reassurance was treated as final.

This situation is illustrative, not exceptional: software changes, vendors add features through updates, and an answer that was correct at time one may not be correct at time two. Anyone who conducts the conversation as a one-time question rather than a recurring topic risks continuing to use an outdated answer.

What supervisory authorities consider important in such conversations

When assessing such questions, supervisory authorities typically look at traceability: can a company show what question it asked, to whom, when, and on what information the answer was based. An oral assurance without documentation is difficult for a supervisory authority to assess, simply because there is nothing to assess. This is not a judgment on the content of the answer, but on the absence of a trail.

So the mistake is not necessarily that a company receives a wrong answer, but that the company cannot demonstrate that it asked the question and what was answered. For your own situation and the question of what documentation is precisely needed, we refer you to a lawyer or the competent supervisory authority.

How to prevent it

The first step is to make the question more concrete: not 'is this AI', but 'which parts of this system make or influence a decision, and what technique is it based on'. The second step is to have the answer confirmed in writing, for example by email or in the agreement itself. The third step is to repeat the conversation with every major update, not just at the first purchase.

Many companies underestimate how much time this takes once it is properly set up: a few fixed questions, a fixed moment in the update process, and a place to keep the answers. Those who do not have this can use get an initial indication with a few questions where their own software roughly stands, and with the most frequently asked questions on this subject get an idea of what others in the same position encounter.

Finally, it helps to determine in advance who within your company will conduct and keep track of this conversation. At an installation company, this is often the director himself, at a larger office the IT or procurement manager. As long as that role has not been assigned, the conversation with the software supplier remains dependent on chance, and that is precisely where the most common mistake comes from.

Do you want to know where your own software roughly stands and which questions are relevant for your situation? Take the free quick-scan or read the topics that are closest to your situation via an explanation of how we compile this information.

This article is general information and not legal advice.