Laya test: synthetic data lifts 56%, hurts 30%

The experiment started from a practical constraint: routing free-form questions to the right table among roughly 1,000, fast, on local hardware, without shipping schema metadata to an external API. Laya looked like a clean fit on paper, since it is a non-autoregressive model that takes a state and a typed question with a defined set of options and returns a calibrated choice in one forward pass. The 512-token input limit, however, pushed the setup into a two-stage design: retrieve about 15 candidates, then let Laya decide among them.

Retrieval turned out to set the ceiling, and lexical search struggled with schema vocabulary, where users say "daily sales by store" but the table is named XQR7_DTV_AGG. A small embedding model closed much of that gap cheaply, while roughly 30 hand-written abbreviation expansions added only one point. The author also found that concatenating the last two user utterances roughly doubled every metric, because follow-up questions cannot be resolved without history.

Fine-tuning was meant to be the payoff, and the documentation was free: splitting "typical phrasing" strings from each table's docs yielded about 25,000 query-table pairs. Training on a 56-thread Xeon took about 17 hours per epoch, close to three days for four epochs. Validation accuracy rose 56%, but accuracy on the 115 real conversation turns fell 30%, because the model learned the terse keyword style of the synthetic data rather than conversational traffic. The table-held-out validation set never flagged this, since its queries came from the same synthetic distribution. The author's recommended path is to fix the training data's style rather than the model, to invest in retrieval before the decision stage, and to use Laya mainly as a confidence gate that answers locally when sure and escalates to a larger LLM when not.