Data scientists in big demand as numbers pile up

Bloomberg NewsJune 11, 2013 

— Jobs centered on data have been falling into Ana Bertran Ortiz’s lap since she finished her electrical engineering Ph.D. in 2007.

The jobs all come from her command of statistics, making her a beneficiary of the growing demand by U.S. employers for so-called data scientists who can analyze and manipulate the mountains of information generated and stored in the Internet age. Harvard Business Review last year called this profession “the sexiest job of the 21st century.”

One measure of demand: Hours billed for work in statistical analysis grew by 522 percent in the first quarter compared with the same period in 2011, according to data compiled for Bloomberg by oDesk Corp., which runs an online service connecting employers with remote freelancers. Time billed on oDesk for all categories of work in the same time span grew by 135 percent.

“In most areas of the modern economy, math and statistics have never been more important,” said Susan Athey, an economics professor at the Stanford Graduate School of Business near Palo Alto, Calif. “As firms get more and more data-driven, there become fewer and fewer careers that don’t require those skills.”

Bertran Ortiz got her start designing algorithms for a NASA lab to measure sea levels from outer space and orchestrate the landing of the Mars rover. She’s researched how to design flight paths to get more information from radar signals, and helped hone a mobile application that forecasts weather in 10-minute increments. She’s now working on software that automatically diagnoses glitches in the networks that house the world’s ever-expanding trove of information.

“I knew that in electrical engineering, it was very important to understand the randomness of data,” Bertran Ortiz said. “But I didn’t think it would become so important outside of my field.”

Unlike statisticians of a previous generation, data scientists work with information sets so big – far too large and unwieldy to fit into an Excel spreadsheet – that they need to write extensive computer code to extract the right segments.

Often, the data are on a scale that requires multiple servers to even access the numbers. After that, the analysts run calculations – correlations, regressions, t-tests, machine learning algorithms – to discover the patterns they’re looking for.

National security

The scope of data collection is widening in the private and public sectors, a shift that was highlighted recently when the Guardian and Washington Post disclosed the existence of secret U.S. government programs that collect data on U.S. residents’ telephone calls and foreign nationals’ Internet activity. James R. Clapper, the director of national intelligence, subsequently confirmed the existence of the programs.

The national security industry is among the biggest employers of big-data professionals, according to an analysis from Burning Glass, a Boston-based job-matching company. One of the best-known companies specializing in big-data analysis is Palantir Technologies, which made its name offering terrorism analysis software to the Pentagon and the Central Intelligence Agency.

Douglas Puett runs pattern analyses for Pulse, a news aggregator that was acquired by Mountain View, Calif.-based LinkedIn in April. Every day, the 25-year-old peers into the mobile application’s logs to track figures, including which news outlets are keeping readers most engaged.

Armed with those numbers, Puett helps Pulse’s product team make decisions such as how to tweak the algorithms that suggest news stories. He also helps design, run and analyze experiments to make sure product changes lead to positive results.

Shortage by 2018

The challenge for employers is that there aren’t enough Douglas Puetts out there, with the multiplying trove of information likely to further exacerbate the shortage of these analysts. By 2020, all the digital data created, replicated and consumed in a single year will grow to 40,000 exabytes, or 40 trillion gigabytes, according to a December study by technology research firm IDC. That’s a 300-fold increase from the 130 exabytes in 2005.

By 2018, the U.S. may face a shortage of as many as 190,000 people with deep analytical skills and 1.5 million managers and analysts who know how to use big data to make decisions, McKinsey Global Institute said in a report in 2011.

“It’s so cross-functional and you need multiple skills – you need programming, you need statistics, you need visualization, you need database skills,” said Harpinder Singh Madan, co-founder and head of product and marketing at Slice, a Palo Alto-based startup that helps consumers track and analyze their emailed receipts. “The bottom line is that there’s no institution that trains for this.”

News & Observer is pleased to provide this opportunity to share information, experiences and observations about what's in the news. Some of the comments may be reprinted elsewhere in the site or in the newspaper. We encourage lively, open debate on the issues of the day, and ask that you refrain from profanity, hate speech, personal comments and remarks that are off point. Thank you for taking the time to offer your thoughts.

Commenting FAQs | Terms of Service