Claude extracts traits from 145 data science job posts

The project started from a practical frustration: the author could not tell what traits data scientist roles actually demand, and found existing online write-ups hard to trust because their evidence was unclear. Reading every posting by hand was infeasible at scale, so the author built a pipeline that uses a large language model to pull structured elements — label, type (skill, experience, attitude), and strength (required, preferred, unknown) — from the free-text 'desired profile' column.

A recurring problem with such text is wording variation: 'can drive oneself' and 'can act proactively' mean the same thing but are different strings. Converting labels to embeddings and clustering them by cosine distance was intended to merge these variants, but the results show it only partly worked. The largest cluster alone held 196 labels spanning data analysis expertise, document-creation skills, and even interest in becoming a data scientist — a sign the threshold of 0.35 with average linkage was too coarse. Conversely, 136 of 357 clusters held a single posting, a sign that some variants stayed apart.

The author is careful not to overclaim on the must-have gap between skill and attitude groups, stating that the cause of the difference has not yet been isolated. The next steps are a threshold comparison and an HDBSCAN run, with the caveat that HDBSCAN would exclude one-off labels from the counts — a trade-off the author flags up front.