No. Most SR authors don't include Cohen's Kappa score for inter-rater reliability. Even the Cochrane Handbook doesn't require it.
Cochrane 8.2.3:
Although not required, if review authors wish to calculate measures of agreement (e.g. kappa statistics) for the answers to the signaling questions, we recommend treating ‘Yes’ and ‘Probably yes’ as the same response, and ‘No’ and ‘Probably no’ as the same response.
