Building a question answering layer over the city's open data taught me that the model was never the hard part. The hard part was deciding what a chunk is, and what to do when nothing good comes back.
A chunk that crosses a table boundary poisons the answer. A chunk that stops one row short of the total makes the system confidently wrong about a sum. Both look identical in a similarity score.
The most valuable behaviour we shipped was refusal. When ranking confidence sits under the floor, the answer is that the data does not cover this, with a link to what does exist. People trust that more than a paragraph of hedging.
Same discipline as photography and as music. Decide what stays, then take one more thing out.