Abstract: In this paper, we report an experiment to investigate the performance of ChatGPT in the task of classifying domain terms according to the categories of upper-level ontologies. The experiment consisted of (1) starting a conversation in ChatGPT with a contextual prompt listing the categories of an upper-level ontology along with their definitions, (2) submitting a follow-up prompt with a list of terms from a domain along with informal definitions, (3) asking ChatGPT to classify the terms according to the categories of the chosen upper-level ontology and explain its decision, and (4) comparing the answers of ChatGPT with the classification proposed by experts in the chosen ontology. Given the results, we evaluated the success rate of ChatGPT in performing the task and analyzed the cases of misclassification to understand the possible reasons underlying them. Based on that, we made some considerations about the extent to which we can employ ChatGPT as an assistant tool for the task of classifying domain terms into upper-level ontologies. For our experiment, we selected a set of 19 terms from the manufacturing domain that were gathered by the Industrial Ontologies Foundry (IOF) and for which there are informal textual definitions reflecting a community view of them. Also, as a baseline for comparison, we resorted to publicly available classifications of such terms according to DOLCE and BFO upper-level ontologies, which resulted from a thorough ontological analysis of those terms and informal definitions by experts in each of the ontologies.

Keywords: Ontologies, ChatGPT, DOLCE, BFO, Industrial Ontologies Foundry, IOF, Large Language Models, LLMs, Term Classification and Manufacturing

Authors: Fabrício Henrique Rodrigues, Alcides Gonçalves Lopes Junior, Nicolau Oyhenard dos Santos, Luan Fonseca Garcia, Joel Luís Carbonera, Mara Abel

Journal/Conference: ER 2023 – Advances in Conceptual Modeling

Publication: https://doi.org/10.1007/978-3-031-47112-4_24