Adverse Impact Analysis

I found this adverse impact analysis report and found it to be a good summary of how an adverse impact study is done. The report cites two criteria used when conducting an adverse impact study. The first is the 4/5ths rule, which states that protected class must be at least 80% as successful as other groups (the article uses the example that women must be at least 80% as successful in men in determining growth potential). The second criteria is to see if the differences in results is statistically significant. I think this second criteria has the tendency to place more responsibility on bigger companies. It's been a while since I took statistics, but I'm pretty sure that smaller the sample size, the harder it is to prove statistical significance within a 95% confidence level. For instance, in the case of hiring procudures, smaller companies will most likely not have as many applicants and may not have enough data points to prove statistical significance. What do you all think?