Compensation benchmarks are invaluable tools to help companies set fair and competitive compensation that’s in line with labor markets. Traditionally, compensation benchmarking providers have emphasized sample size as a primary indicator of a benchmark’s reliability: the larger the sample size, the more reliable the compensation benchmark. However, that’s only part of the picture. A comprehensive understanding of sample size, data distribution, and the impact of outliers is crucial for understanding the validity of compensation benchmarks and making well-informed compensation decisions based on them.
The Significance of Sample Size
Sample size – often referred to as the “n” – has long been the go-to metric for assessing data reliability. The historical approach has been that increasing sample size is the only way to increase a data set’s statistical confidence, improve representativeness, and support robust decision-making. Sample size does matter, but by only considering the sample size, users of compensation benchmarking can be tricked into a false sense of reliability of a compensation benchmark.
The Role of Data Distribution
Beyond sample size, how data is distributed within a compensation benchmark is important. Different distribution patterns have profound impacts on the accuracy of compensation benchmarks:
Let’s look at an example. In the image below, Benchmark A and Benchmark B have the same benchmark value for the 50th percentile. They also have the same sample sizes. However, you can see that the data points in Benchmark A actually cluster below the 50th percentile and right above the 50th percentile; if you were to pay at the 50th percentile based on the data for Benchmark A, you would find that a lot of companies are paying way below or above that benchmark for very similar roles in the industry. Pave’s benchmarking confidence labels will tell you that Benchmark A should be used with more caution than Benchmark B.
To help companies make well-informed compensation decisions, a benchmarking data set must paint a complete picture of both sample size and distribution patterns. This gives users an indication of the overall confidence of a benchmark.
Modern benchmarking providers understand the interplay between sample size and distribution. They indicate a benchmark as reliable (or not) after considering both the number of data points for the benchmark and the distribution pattern of those data points.
At Pave, we've always used these confidence scales internally to determine how reliable our data is, and now giving users insight into how we measure data confidence. Within the Pave app, you’ll see a confidence scale – labeling each compensation benchmark from “Very High Confidence” to “Low Confidence” – as well as a sample size for the benchmark. This is to guide users on the holistic confidence in the benchmark.
The most accurate compensation benchmarks must encompass both sample size and data distribution. Ignoring either of these factors can lead to flawed decisions. By balancing these elements, compensation leaders can confidently establish fair and precise compensation benchmarks tailored to their unique organizational needs.