Goals: The Joanna Briggs Institute has implemented training programs on systematic control methods, including verification of qualitative studies with software from the Joanna Briggs Institute for Qualitative Assessment and Evaluation in the United Kingdom, Spain, the United States, Canada, Thailand, Hong Kong, China and Australia. As part of the training, participants worked as a couple to carry out a blind critical assessment, followed by a process of transmission agreement; Obtaining qualitative knowledge and the completion of a metasynthesis process in two qualitative studies. These studies were verified by 18 pairs of experts from different cultures and contexts. The results of metasynthesis were analyzed to determine the extent to which an interreviewer was achieved between these 18 pairs. There are several operational definitions of “inter-rated reliability” that reflect different views on what a reliable agreement between advisors is.  There are three operational definitions of agreements: therefore, the common probability of an agreement will remain high, even in the absence of an “intrinsic” agreement between the advisers. A useful interrater reliability coefficient (a) is expected to be close to 0 if there is no “intrinsic” agreement and (b) increased if the “intrinsic” agreement rate improves. Most probability-adjusted match coefficients achieve the first objective. However, the second objective is not achieved by many well-known measures that correct the odds.  As you can see, we have an agreement between the two methods, 17 times out of 26, which is an agreement of 65.4%. I suppose the higher the agreement here, the better, but we can discuss the objectives of this agreement if you have any other questions.
Another approach to concordance (useful when there are only two advisors and the scale is continuous) is to calculate the differences between the observations of the two advisors. The average of these differences is called Bias and the reference interval (average ± 1.96 × standard deviation) is called the compliance limit. The limitations of the agreement provide an overview of how random variations can influence evaluations. There are several formulas that can be used to calculate compliance limits. The simple formula given in the previous paragraph that works well for sample sizes over 60, is subsequent extensions of the approach containing versions that could deal with “partial credit” and ordination scales.  These extensions converge with the intra-class correlation family (ICC), which allows us to estimate reliability for each level of measurement, from the notion (kappa) to the ordinal (or ICC) at the interval (ICC or ordinal kappa) and the ratio (ICC).