Table 2 Computer-based calculation of the scores in the six categories

Use of semantic qualifiers (SQ)

Identification of semantic qualifiers in the statements based on the list provided by Connell et al. [11] and application of rules to compare results, occurrences and the semantic context with the NLP tree.

<  2 SQ: Score = 0

> = 2 and < =4 SQ: Score = 1

>  4 SQ: Score = 2

Appropriate narrowing of differential diagnosis

Identification of findings, differential diagnoses, and anatomical terms based on an adapted MeSH thesaurus and comparison of the result with analysis of the expert statement and VP metadata.

(found terms of expert - terms of learner matching with expert -) / found terms of expert:

>  0.75: Score = 0

<= 0.75 and > = 0.25: Score = 1

<  0.25: Score = 2

Transformation of information

Identification of transformed terms and non-transformed terms based on a list of SI units and the MeSH thesaurus and comparison with transformed terms by expert and overall length of the statement.

(transformed terms - non-transformed terms /2)/ (transformed terms of expert + text length factor)

<  0.16: Score = 0

> = 0.16 and < = 0.7: Score = 1

>  0.7: Score = 2

Factual accuracy

Identification of contradicting use of SQ in the learner and expert statement

contradicting information found: score = 0, else score = 1.

Patient name used

Identification of a person token in the NLP tree

person identified: score = 1, else score = 0.

Global rating

Sum of the five categories

Sum <=2: Score = 0

Sum > 2 and < =5: Score = 1

Sum > 5: Score = 2