AIToday
Large Language ModelsOpen-Source AIQiita 機械学習Published: Nov 9, 2026, 10:01 JST

Rockchip rknn-llm demo for Qwen2-VL-2B-Instruct runs only on transformers 4.51.3

Send on LINE
Rockchip rknn-llm demo for Qwen2-VL-2B-Instruct runs only on transformers 4.51.3

3 Key Points

  1. What happened

    Testing Rockchip's rknn-llm multimodal demo for Qwen2-VL-2B-Instruct, the author found transformers 5.8.0 throws an AttributeError, 4.57.0 and 4.54.x throw a TypeError, and only 4.51.3 works.

  2. Why it matters

    The same repository mixes scripts written for two different library generations, so users must pin separate environments per step.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

The root of the trouble is that rknn-llm is not internally consistent. The export script follows an older contract, reading model.visual directly, while the quantization script is already written for the 5.x module layout (model.model.language_model). Each generation shows why: 5.8.0 moved visual to model.model.visual, 4.57.0 made VisionAttention unpack position_embeddings unconditionally, and 4.54.x had already removed the fallback that derived it from rotary_pos_emb. Only 4.51.3 satisfies both conditions, confirmed by reading the source and by an actual export.

Rather than patch around it, the author kept two rules: no patches to upstream scripts or the model, and verification from source rather than guesswork. An alias for .visual was tried first, but even when the attribute error was bypassed, the attention contract still failed, and altering the embedding flow risked silently wrong embeddings that would contaminate later benchmarks. Pinning versions solved it, at the cost of running 4.51.3 for the Vision export and 5.8.0 for quantization embeddings in parallel.

With the Vision tower exported (a 53MB ONNX graph plus 2,661,085,184 bytes of weights, passing onnx.checker), the remaining work is quantization and RKLLM export, then board measurements of per-call Vision encoding time and prefill/decode tokens/s, and a 24-question quality check that includes Chinese signage and OCR cases.

FAQ
Which transformers version actually works for the Vision export?
Version 4.51.3 works, together with torch 2.6.0, because its source has both self.visual and the compatibility handling for position_embeddings.
Why not just patch the upstream script or monkey-patch the model?
The author tried an alias for .visual, but the attention contract still failed, and silent wrong embeddings could contaminate later benchmarks, so version pinning was used instead.
Did the export succeed in the end?
Yes: the Vision tower exported cleanly as a 53MB ONNX graph plus 2,661,085,184 bytes of weights, and onnx.checker passed.
Qiita 機械学習Read Original Article
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleLinum's P-JiT image model trained for $21K, open weights