Researchers develop framework to understand and improve semantic organization in CLIP and other vision-language model embeddings

New post-hoc framework extracts and explains semantic hierarchies from vision-language model (VLM) encoders like CLIP by clustering class centroids and naming internal nodes

Proposes evaluation methods comparing extracted hierarchies against human ontologies using tree-level and edge-level consistency measures to assess plausibility

Introduces ontology-guided alignment technique that applies lightweight embedding-space transformations to align VLM hierarchies with desired semantic structures

Demonstrates utility through explainable hierarchical tree-traversal inference with uncertainty-aware early stopping (UAES) for improved classification

Addresses the gap in understanding semantic organization of shared image-text embedding spaces that enable strong zero-shot classification and retrieval