Tag

#evaluation

June 2, 20265 min read

An adversarial persona suite: how I attack my own chatbot

In short: I keep a file of attacks on my own chatbot, each one tied to a written rule, and run them through the real conversation graph against a live model. A broken hard rule fails the run with its…

May 5, 20264 min read

Citations that point at the wrong paragraph

In short: a citation can be wrong in at least seven different ways, and only two of them involve the model. Most come from how long documents were cut into chunks. Name each failure, and give each one…

April 7, 20265 min read

How I evaluate retrieval before anyone asks a question

In short: before a single user types a question, I write the questions myself, mark which passage answers each one, and measure how often that passage comes back in the top k. That one number, recall…

February 10, 20264 min read

The AI feature that cut churn, and the one that didn't

In short: on a platform product I led AI on, a round of AI features came with lower churn and higher feature adoption. Those are two different measures, and a feature can win one without touching the…

January 27, 20264 min read

What the buyer actually bought

In short: I sold my startup and its technology, and I do not think the buyer was paying for clever code. They were paying for the parts that keep the product working without me: the pipeline that…

December 16, 20255 min read

Rules vs transformers for extracting metadata from legal documents

In short: on the same Spanish-language legal documents, the transformer path was more accurate and slower. The rules were fast, cheap and easy to explain. I kept both: rules by default, the…

August 26, 20255 min read

Selecting papers for a review without reading all of them

In short: export every search hit as a .bib file, let a language model screen titles and abstracts against written criteria, then hand-check a random sample to see how often it's right. The model…

August 12, 20255 min read

Leakage: the bug that makes every metric lie

In short: leakage is when information from the test set, or from the answer itself, sneaks into training. The three kinds I check for every time are the same person on both sides of the split,…

June 3, 20255 min read

Class imbalance in clinical data: what actually worked

In short: oversampling and class weights mostly move you along the same precision-recall curve. What actually worked was picking the decision threshold on purpose, on a validation set, against a…

April 22, 20254 min read

Explaining a model to a doctor

In short: doctors don't want to see how the model works. They want to know why it said this about this patient, in units they already use, and whether they should trust it. A simple bar chart with…

April 8, 20254 min read

Monitoring a model that listens to patients

In short: for a voice model in healthcare, the true labels arrive late or never, so you can't watch accuracy. You watch the recordings coming in, the scores going out, and the plumbing in between, and…

February 25, 20255 min read

The clinician said no, and was right

In short: a clinical model can score beautifully because it has learned something the clinic already decided. The fastest way to catch that is not a better metric. It is ten minutes with the person…

December 31, 20244 min read

The boring checklist that saved every project

In short: before I train anything I want three things: a record of where every piece of data came from, a dumb baseline to beat, and one number the people paying for the work actually care about. None…

September 24, 20244 min read

The baseline before the fancy model: medical image segmentation

In short: before training a U-Net, write a threshold with a little cleanup and score it with the same metric. It's quick to write, it tells you how hard the problem really is, and on high-contrast…

August 27, 20244 min read

Reading papers with an LLM without getting lied to

In short: give the model only the part of the paper that holds the study, ask narrow questions that demand quotes and an explicit NA, and check every quote against the text. It still breaks on numbers…