March 2, 20265 min

Retrieval is an editing problem

Ranking, chunk boundaries, and knowing when to say the data does not cover this.

Building a question answering layer over the city's open data taught me that the model was never the hard part. The hard part was deciding what a chunk is, and what to do when nothing good comes back.

A chunk that crosses a table boundary poisons the answer. A chunk that stops one row short of the total makes the system confidently wrong about a sum. Both look identical in a similarity score.

The most valuable behaviour we shipped was refusal. When ranking confidence sits under the floor, the answer is that the data does not cover this, with a link to what does exist. People trust that more than a paragraph of hedging.

Same discipline as photography and as music. Decide what stays, then take one more thing out.

retrieval, public sector

Leave a note