lru_cache breaks async calls on second await, dev finds

The article walks through a debugging session whose starting point is simple: the author added a cache and only the second call broke. The reason sits in how Python's functools.lru_cache works. Calling an async def does not run the function; it returns a coroutine, and the awaited result only appears after await. lru_cache does not await anything, so it saves that coroutine object as its return value. Reusing it means awaiting a coroutine that has already been consumed. The author also checked the copy of Lib/functools.py shipped with Python and confirmed the pure-Python implementation likewise stores the function's return value.

The suggested answer is to cache an asyncio Task instead of a value. A Task, unlike a coroutine, can be awaited many times, which is what makes sharing it work for parallel callers. Two details in the test carry weight: the sleep(0) deliberate yield lets a second call arrive before the first finishes, and there is no await between checking the dictionary and registering the Task, so within one event loop a second once cannot slip in and create a duplicate. Failed Tasks are removed, but only after confirming identity, so the next call retries rather than replaying the same exception.

The author is explicit about the boundary. Storing only the result is enough for sequential callers, but in parallel both callers see an empty cache and start work anyway. Keys should include the model name, generation settings, and input, and this sharing should be avoided where a fresh answer is wanted each time. Whether this pattern holds up in real LLM batch processing, where error and cancellation behavior is messier than the test script, appears to be the open question.