Surviving the Big Data HR Test

I've known a bunch of different human resources professionals throughout my career and have tended to have good working relationships with them. In fact, I've found that I could be honest with them when suggesting ideas or airing complaints, maybe because I wouldn't tell them something that I wouldn't tell the big boss to their face.

One thing I have learned about those HR people is that they aren't gamblers. Their idea of risk is limited to decisions such as whether to move to a new forms processing system or to do the "trust your coworker when you fall backward" exercise at the company meeting.

So I was shocked to read about the position HR leaders are taking when it comes to big data and personnel decisions. Let's face it, in using big data to hire people or even support decisions on promotions, there are privacy considerations and even legal worries that can come up.

But by golly, those HR pros are for it, warts and all.

The website Human Resource Executive Online reported on an Equal Employment Opportunity Commission hearing on big data. Reading some of the comments from government officials, HR leaders, and attorneys who testified, the bottom line seems to be that big data-driven HR systems can't be any worse than the already flawed systems employers have been using for decades.

Plus, they said a technology and big data-based system makes it easier to defend against complaints about fairness in personnel decisions because the data is more tangible than human opinions.

"Big data can eliminate existing major problems, and while it may create some new unexpected issues, we're making progress," said Marko Mrkonich, a shareholder in the Minneapolis office of law firm Littler. "I think that's the right way to look at this."

Other speakers advised that HR teams start by using big data in a test-bed environment so they can look for holes such as subtle bias, in their models, and that independent experts should be called in to provide an additional sanity check for fairness.

The EEOC hearing was designed to explore the fairness of big data HR systems and how to avoid designs that would have a disproportionate impact on a protected group -- the so-called "disparate impact" theory. It's a concern that has been voiced about some other big data applications, including those that retailers use to do promotional pricing or lenders use to set interest rates. Do those systems represent even unintentional bias?

Big data has been edging its way into HR for a while now, approaching in stealth mode. Criminal record checks and credit reports once were used only for filling a handful of sensitive jobs. Not any more; those have become standard for many hires today. That's big data.

Closer to the big data model is the background check that delves into a candidate's social media activity, whether it is done by HR or by a hiring manager. That's where the candidate learns that the party photo with said candidate holding a bong or wearing a two-can beer hat might not have been a good idea.

As that type of review becomes more common, it also will have to become more structured, which raises the old issue of how to give structure to unstructured data. Does the company do social media analyses for every candidate? Just for young people? Just for certain jobs? Just for people who raise red flags about politics in the interview process?

An even bigger issue is how managers quantify what they find. Does bad credit count for a certain number of negative points in the overall evaluation? Does that recent drinking party pic carry less weight for a new college grad than it does for a 40-something middle management candidate? Does a candidate get positive points if their social media activity highlights charitable work or community leadership? If you assign point values to what you find on social media, do you do the same for professional experience and industry awards?

Big data can serve a valuable role in hiring and other personnel decisions. Yet we are still applying metrics to human beings. These models will have to be tested, refined, tested again, and modified over time. Despite all that data, in the end someone will have to decide whether that person over there is right for the job. What we have to see is just how deep the data goes in our decision processes.