How do I assess an AI vendor before signing?

Ask what the system does when it is uncertain, and ask to see it fail. Vendors with a working product answer both immediately.

A capable vendor can tell you the confidence threshold at which their system escalates to a person, roughly what proportion of cases that is in production for a customer like you, and what happens to the ones it gets wrong. A vendor selling a thin wrapper will redirect to accuracy percentages measured on their own benchmark.

Three more separate quickly. Ask which model or models sit underneath and what happens when the provider deprecates one. Ask for a reference customer of similar size in a similar industry, and actually call them. Ask what the system cannot do, and treat a vendor who claims no limitations as having told you they do not know.

The point is not to catch anyone out. It is that these questions are easy to answer for a vendor with an operating product and hard for one with a demo.

Four: what data of yours the system ingests, where it is stored and processed, whether it is used to train models, and what happens to it when you leave.

Get the training answer in writing and get it specific. A statement that the vendor does not train on customer data can be entirely true of the vendor while untrue of the model provider sitting behind them, so ask about the whole chain, not only the party you are contracting with.

Storage and processing location matters for Australian businesses with privacy obligations or government-adjacent clients. Ask which jurisdiction, not which cloud provider.

Retention and deletion is the question most often left vague. Ask how long data persists after cancellation, what format you get it back in, and whether deletion covers derived data such as embeddings, indexes and logs or only the original records. A vendor who has answered this before will have a document. A vendor who has not will offer to find out, and that is also useful information.

Compare the total cost over a realistic term rather than the pilot price, and make every vendor quote against the same volume assumptions.

AI pricing hides in three places. Usage that scales with something you cannot predict at signing, such as tokens, documents or seats that grow with headcount. Implementation and integration work quoted separately, or not quoted at all. And the internal cost of your own people supporting the thing, which no vendor quotes and which is frequently the largest line of the three.

Ask each vendor to price the same scenario: your expected volume in the second year, not the first. Ask what happens to unit pricing if volume doubles, and what happens if it halves. Ask which numbers are contractual and which are estimates.

Then compare like for like. A cheaper platform that needs a person to supervise it is not cheaper. This is the point where an independent view earns its keep, because whoever runs the comparison should have no interest in which vendor wins.

The traps are data you cannot extract in a usable form, workflows rebuilt inside the vendor's platform, and contract terms that make leaving expensive rather than merely inconvenient.

Data extraction is the obvious one and the easiest to test. Ask for a sample export before you sign, not a description of one. Check whether it includes the things that took effort to create, such as configurations, prompt libraries, labels and human corrections, or only the raw records you put in.

The subtler trap is process. Once your team's daily work lives inside a vendor's interface, the switching cost is retraining and rebuilt process, which usually dwarfs the data migration. That is not a reason to avoid the vendor. It is a reason to understand the size of it before you sign rather than two years later.

Then read the term structure: auto-renewal windows, notice periods, uplift clauses, and any charge for migration assistance on the way out. Ask for a shorter initial term. A confident vendor will usually agree.

When the decision is hard to reverse, when the vendors are difficult to compare on their own materials, or when the people advising you also stand to benefit from the outcome.

That last case is the common one. Your MSP may resell one of the products under consideration, or may be the party who would implement it. Their recommendation can still be the right one, and this is not a reason to distrust them, but it is a reason to have the decision reviewed by someone with nothing riding on it.

An independent assessment from Benchmark Technology is fixed price, quoted up front after a scoping conversation, with no vendor commissions of any kind. The output is a report: your shortlist assessed against your actual requirements, the risks in each option stated plainly, and a ranked recommendation your internal tech function or MSP can act on. Where a capability gap needs filling, we can introduce providers from our network, and those introductions carry no commission either.