Evaluating Data Suitability for Rag Systems in Manufacturing: A Comparision between Human and Llm Judgments

Retrieval-Augmented Generation (RAG) is widely used for manufacturing assistance, but its effectiveness depends on selecting retrievable text units. We test whether humans or Large Language Models (LLMs) can judge which case descriptions are better suited as RAG inputs. We constructed 100 synthetic manufacturing service cases, each paired with a realistic query and two comparable problem–solution variants differing in contextual completeness, granularity, and quality. Five engineers and five LLMs chose the variant expected to be more retrievable and useful. As a reference, both variants were indexed in a minimal retrieval setup with one chunk per case and evaluated with MRR@3, treating the case-matching chunk as the only relevant item among distractors. LLMs showed much higher within-group agreement than humans, yet neither cohort consistently matched retrievalderived winners. Ties were frequent; on non-tied cases, majority decisions fell below chance and were significantly worse than random guessing in one embedding setting, while no individual rater achieved above-chance performance. Overall, the findings indicate that perceived RAG-fitness is not a reliable proxy for retrieval performance and should be grounded in retrieval-based evaluation under the target deployment setup.