You are right. Please consider the following two papers. The first one discusses the measure of consistency. The second one provides some empirical guides.
1) http://lpl.psy.ohio-state.edu/documents/Buckeye.Corpus-5.pdf
2) http://cslu.cse.ogi.edu/corpora/docs/labeling.pdf
1) http://lpl.psy.ohio-state.edu/documents/Buckeye.Corpus-5.pdf
2) http://cslu.cse.ogi.edu/corpora/docs/labeling.pdf
