Big data is everywhere and it's getting more complex. Mathematicians seem to think we have effective ways to analyze it, but are still in need of developing the tools to reach conclusions. Just look at what happened as people tried to sign up for the online insurance marketplace. Computer code glitches shut it down and President Obama promised a tech surge for the problem.
The government has hired experts from the private sector and other federal agencies to rewrite the flawed code. Health and Human Services Secretary Kathleen Sebelius was in Cincinnatilast week apologizing and reporting that the product was getting better. Next week she will testify in front of a congressional committee.
Yale math professor Ronald Coifmanpointed to the infrastructure issues with the federal websites and said, it's a complex issue and people were too ambitious. He told Quanta Magazinewhat is really needed is "the big data equivalent of a Newtonian revolution, on par with the 17th century invention of calculus."
Coifman uses shapes to quantify big data and so does Stanford mathematics professor Gunnar Carlsson. He says big data is more complicated than the algebraic and mathematical models people are used to. He says it requires new math and engineering. "I think there is room for creativity from a lot of sources you know, within, the academic community and other communities in being creative about how we understand and represent this data in a useful way."
One of those ways is Topological Data Analysis (TDA)
- uses shapes to represent data
- gives an overview of a massive amount of data
- big light vs flashlight to see big picture
- allows average person to understand complicated data.
Carlsson has a company that is developing software so the average person can interpret data. It's called Ayasdi.
Not everybody favors TDA. Others prefer network analysis, or creating networks of relations between people and things to look for hidden data. There may be more than one way to effectively analyze big data. After all, Coifman says living creatures deal with big data all the time.
