New MedMT-Bench benchmark tests whether LLMs can handle extended medical conversations with 22+ dialogue turns while maintaining safety and accuracy.

MedMT-Bench introduces 400 test cases simulating real medical diagnosis and treatment processes, averaging 22 dialogue rounds with up to 52 rounds maximum

Benchmark addresses critical gaps in existing medical benchmarks by stress-testing long-context memory, interference robustness, and safety defenses required in high-stakes medical applications

Test cases cover 5 types of difficult instruction-following challenges and were refined through manual expert editing for real-world consistency

Evaluation uses an LLM-as-judge protocol with instance-level rubrics and atomic test points, validated against expert annotations for reliable assessment