Expertise location systems: finding the expert, not the document

Expertise location is one of the oldest categories in enterprise software and one of the least successfully executed. AT&T built a referral network from bibliographic data in the nineties. MITRE published a technical report on expert finding systems in 2006. Organisations have been buying expertise locators, and quietly retiring them, for thirty years.

The category deserves another look, not because the idea was wrong but because the input it always needed has only recently existed.

What it is meant to do

The job is narrow: given a subject, return the people in this organisation who know about it, ranked usefully.

That is different from search, and the difference is the point. Document search returns eleven files that mention the subject and leaves the searcher with reading to do and a judgement to make about which of them represents real depth and who was actually responsible. An expertise locator returns a person, which is the end of the search rather than the middle of it.

In a professional services firm the case is unusually strong, because expertise is the product. The alternative currently in use is a broadcast email that interrupts a hundred people, returns answers from the available rather than the expert, and fails silently when the right person does not see it.

Search finds the document. The searcher still has to work out who to ring.

Why it failed the first two times

Generation one: self-reported profiles. Everyone completes a skills profile. This fails immediately and for a structural reason: in an organisation where staffing follows stated capability, describing yourself narrowly is career limiting. Everyone claims the widest defensible range, all of them honestly, and the directory loses the ability to discriminate. A search returns forty names, which is as useless as returning none. Then it decays, because nobody returns to their profile after an engagement to add what they just learned.

Generation two: document proximity. Index the document store, treat authorship and keyword co-occurrence as an expertise signal. Better, because it requires nothing of anyone, and still wrong in a specific way: the author field usually names whoever last saved the file, which is frequently an analyst doing the formatting. Mention frequency correlates with expertise about as well as email volume correlates with contribution.

Both generations failed the same test. They produced a name without producing a reason to believe it, so the person who received the recommendation checked it by emailing the recommended expert, which is a shorter broadcast but a broadcast nonetheless.

The five properties that make one work

Working forwards from those failures, an expertise locator in a firm has to have all five of these. Four out of five is not a partial result, it is a system nobody uses.

1. Nobody maintains it. Any profile a person has to update will be accurate on the day it is created and wrong within a year, and wrong in a biased direction, since the busiest people update least.

2. Expertise is inferred from delivery. Not from claims, not from proximity to keywords. From engagements completed, with the person's role in each one, because leading an engagement is different evidence from being copied on it.

3. Every recommendation shows its evidence. The name arrives with the engagements behind it: client, date, role, document. This is what removes the confirmation email, and it is the difference between a suggestion and an answer.

4. Ranking reflects weight. Several engagements across several clients outrank one engagement, which outranks a mention. Without ordering, a system that returns twelve names has reproduced the self-reported directory problem in a new interface.

5. It can return nothing. If the firm has no proven experience, the honest answer is the valuable one. A system that always produces a name cannot be relied on for the negative, and the negative, "we have not done this, price it accordingly", is worth as much commercially as the positive.

The fifth is the one most products get wrong, and it is not an oversight. A system that returns nothing looks broken in a demo. It is, however, the property that determines whether the tool can be used for decisions that matter.

An expertise system that cannot say "we have not done this" cannot be trusted when it says we have.

Why now, and not in 2006

The input is what changed.

Early systems had HR records, publication lists and a document store. None of those describe delivery. What they lacked was any way to read ordinary working material, the proposal and the report and the thread where the scope changed, and extract from it the fact that a named colleague led a named piece of work for a named client, evidencing a general capability.

That extraction is now reliable enough to build on, with one condition attached: the extracted facts must remain tied to their sources. Extraction without provenance is worse than the self-reported directory, not better, because at least a self-reported directory has a human being who can be asked. An inferred claim with nothing behind it cannot be checked, and the first one that turns out to be wrong ends the deployment.

What this looks like in OrgAtlas

Expertise is not a feature here, it is a consequence of the structure. Colleagues, engagements and capabilities are linked, each link names the relationship and carries the document behind it, and each item carries a gauge of how much proven evidence stands behind it.

So the answer to "who has done this" is a list of colleagues in evidence order, each openable to the engagements that put them there. Nobody wrote a profile. Nobody ranked anyone. And a capability with nothing behind it is drawn hollow rather than omitted, so the answer "the firm has no evidence of this" is available and visible rather than being indistinguishable from a system with no data.

How to evaluate one

Four questions, in order of how much they tell you.

1. What does a person have to do for their record to be current? Any answer involving them is the answer to how this will fail.
2. Where does the expertise signal come from? Self report, keyword proximity, or delivered work. These are three different products with the same category name.
3. Ask it something the firm has never done. Watch whether the interface can express absence, or whether it returns the nearest plausible names.
4. For a name it does return, how many clicks to the proof? If the answer is more than one, the recommendation will be verified by email and you have not replaced the broadcast.

Related: what the broadcast email actually costs, and why it persists