Catch up on stories from the past week (and beyond) at the Slashdot story archive

 


Forgot your password?
Close
typodupeerror
×
Microsoft Software Science News

Researchers Mine Old News To Predict Future Events 99

Posted by samzenpus from the what's-old-is-new dept.
hypnosec writes "Microsoft Research has teamed up with the Technion-Israel Institute of Technology to develop software that can predict events like outbreaks of disease or violence by mining data from old news and the web. The project, if successful, will result into a tool that would provide information that is more than just educated guesses or intuition. The team consisting of Eric Horvitz from Microsoft Research and Kira Radinsky from Technion-Israel Institute tested the program with articles from New York Times spanning over 20 years from 1986-2007."
This discussion has been archived. No new comments can be posted.

Researchers Mine Old News To Predict Future Events More

Researchers Mine Old News To Predict Future Events

Comments Filter:

  • Past performance... (Score:5, Funny)

    by Overzeetop ( 214511 ) on Sunday February 03, 2013 @07:38PM (#42781169) Journal

    ... is Not Necessarily Indicative of Future Results.

    Do these guys not even read their own prospectus?

    • Re:Past performance... (Score:4, Interesting)

      by Trepidity ( 597 ) <delirium-slashdot@@@hackish...org> on Sunday February 03, 2013 @08:00PM (#42781355)

      True, though it can to the extent that there are recurring patterns and you find the right ones.

      On the other hand, there's also a circularity problem. Say you find, from analyzing 20 years of correlations, that certain events tend to happen some period after certain news reports. This might impact whether that relationship continues to hold in the future. That's already quite common for financial events: if you can reliably predict that when News Report Type X happens (for a possibly complex "X"), then Stock Move Y will happen, you can profit from it, but only until it becomes known by enough people, after which the arbitrage opportunity will close.

      • On a microscopic level (one person, one company, one country's economy...) the SNR is pretty bad. But it would be interesting to see whether the data fits patterns long noticed by civilizations. The Chinese "Wealth never survives three generations" and similar have a grain of truth. The causes and effects of prosperity are predictable:
        • Hard work and careful use of resources leads to wealth in the grandfather's generation
        • This wealth is handed down to the next generation who enjoy comfort without the hard wo

    • maybe they should read more Asimov as well.
      there was a second Foundation who's job it was to make sure the predictions came true.

    • Re: (Score:2)

      by flyneye ( 84093 )

      Well , as long as we're not predicting global warming, the superbowl or any of the other myriad of things out there that require far more criteria than is (mis)represented by news media. My main concern is the level of propaganda printed over the timeframe covering everything from politics to commerce to science and even religion that
      get the "ol' quicky newsclown one sided, one eyed coverage" by personnel biased and unfit to cover grandmothers tea party, let alone add criteria to digital clarvoyance. Their

      • Re: (Score:2)

        by foobsr ( 693224 )
        unbiased scientific terms

        Dark matter theory vs. MOND - who is owned by the bias?

        CC.

    • I just want to know when Hari Seldon [wikipedia.org] joined Microsoft?

    • Re: (Score:1)

      by Dabido ( 802599 )
      Read Didier Sornette's http://www.google.com.au/url?sa=t&rct=j&q=sornette&source=web&cd=1&cad=rja&ved=0CDIQFjAA&url=http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FDidier_Sornette&ei=W-0RUfisEMmUiAesz4DgCQ&usg=AFQjCNFfRpV9pxxFO-xtIOYzfYFJ4k7PNw&sig2=1uKuO63XsPwMBFg9D_kl9Q&bvm=bv.41934586,d.aGc [google.com.au] work into predicting the stock exchange (he started out by predicting earthquakes and volcanic eruptions). I'm sure his work would be excellent in this field too. Sure, past event

  • For example, did you know... (Score:5, Insightful)

    by Anonymous Coward on Sunday February 03, 2013 @07:38PM (#42781171)

    There will be a Presidential election in 2016 and 2020

    An NFL team will win the Super Bowl.

    Many new TV shows will be cancelled each fall.

    Old people will die.

    • More nightly reports of murders and burglaries may be increasing your city's crime rates.

      Viewers who are asked teasing questions about the weather makes them more likely to keep watching until the end of the news.

      You can predict a candidate's election success by how much air time is spent on them during the evening news.

      What could be lurking under you kitchen sink that might kill you, won't do so until after a word from our sponcers.

  • The Observer from NZ (Score:1)

    by Anonymous Coward

    Cliff has been doing this for years. The most interesting aspect is that there is weighting given to emotional "charge".
    http://www.halfpasthuman.com/

  • Asimov was here (Score:5, Insightful)

    by Minion of Eris ( 1574569 ) on Sunday February 03, 2013 @07:49PM (#42781275)
    The shade of Asimov raises his head..... Does this seem a little like Psychohistory to anyone else? Where's the Mule?

    • The shade of Asimov raises his head..... Does this seem a little like Psychohistory to anyone else? Where's the Mule?

      More importantly, which is the empire that will break down?

      • Re: (Score:2)

        by foobsr ( 693224 )
        which is the empire that will break down

        Easy, the US - mind that it probably takes a century.

        CC.

    • Re: (Score:2)

      by MrLizard ( 95131 )

      OK, good, someone beat me to it. I'd be disappointed otherwise.

    • Re:Asimov was here (Score:4, Interesting)

      by gmuslera ( 3436 ) on Sunday February 03, 2013 @08:41PM (#42781585) Homepage Journal
      More important than the mythical Mule (unless telepathy really exists and to that extent), the important message in Asimov's Psychohistory is that predicted people shouldnt be aware of the predictions on them, and that includes the government. Is a good way to invalidate predictions, acting with the knowledge of the prediction instead of acting "naturally", whatever be it.

      • well known by stock market types. which makes me wonder what the purpose of financial reporting is.

  • Pythia was here (Score:1)

    by Anonymous Coward

    "All this has happened before. All this will happen again."

  • Minority Report references in 3..2..1

    • would you consider a Douglas Adams?

      “Anything that happens, happens.
      Anything that, in happening, causes something else to happen, causes something else to happen.
      Anything that, in happening, causes itself to happen again, happens again.
      It doesn’t necessarily do it in chronological order, though.”

      • See Sig

        Memory is deceptive because it is colored by today's events. - Albert Einstein (For AC's)

        • (For AC's)

          And for future readers finding your post long after you've changed your signature to something completely different.

          Nobody should ever refer to sigs (not their own, and certainly not other people's) without quoting them.

  • ...and will happen again.

    Hey that makes a pretty good tag line...someone should make a scifi tv series based on that concept. Maybe two even!

  • Science! (Score:3)

    by Intrepid imaginaut ( 1970940 ) on Sunday February 03, 2013 @08:06PM (#42781397)

    Discovers experience.

  • Is this the softwar that comes up with republicrat propaganda?

  • Here, Let Me Save Them Some Time (Score:5, Informative)

    by Nova Express ( 100383 ) <lawrenceperson.gmail@com> on Sunday February 03, 2013 @08:27PM (#42781495) Homepage Journal

    Unrest in Middle East
    African Regime Unstable, May Collapse
    Congressman Indicted
    Experts Say Hurricane "Extremely Dangerous"
    Audit Finds Serious Misuse of Funds
    Green Energy Firm Declares Bankruptcy
    Apple's Latest Product Selling Like Hotcakes
    Patch Released for Serious Windows Vulnerability
    Unemployment Rises, Unexpectedly

    There's your headlines for the year.

    • You forgot: "It will snow in winter." At least here in Europe it is a serious, unpredictable problem which surprises the people every year.
    • There seems to be some kind of collective ability to predict the future based on linguistics. This guy has been doing it for more than a decade. http://halfpasthuman.com/altaprocess.html [halfpasthuman.com] He has some major hits, and I enjoy following him as a hobby, though you could say I'm agnostic on the matter.
      • I am super-serious about evidence-based reasoning, atheism, and accepting painful truths, but sometimes I get weird feelings about time and consciousness.
        A week ago I put on Buffalo 66 for someone, and this person said, "Blue Bird," out loud when a frame of the movie showed a Blue Bird brand bus. Then he asked me about this brand, and I honestly said I had no idea, never noticed the name before. The following night, very late, I walked by a normally vacant street, and there was another BB, idling, full of o

    • You forgot "Lance Armstrong movie set to break previous sales record", and "Windows 9 releases classical desktop edition earlier than expected".

  • ...to the land of tomorrow!
  • Its taken them this long to realize that the past and the future are very strongly correlated and that those in power haven't learned from their own mistakes? I mean, every war that the US has gone to since WWII has been played out multiple times in history, economies run on the same basic patterns, etc.

  • What better place for mining old news?

    Eureka! We've struck gold!

  • Disease outbreaks? (Score:3)

    by Hartree ( 191324 ) on Sunday February 03, 2013 @10:06PM (#42782009)

    You mean by looking at twenty years of newspaper articles they were able to predict that there will be a large increase in the number of cases of influenza during November 2013 to March 2014 compared to the preceding 5 months?

    Or, when disaster causes infrastructure to break down and crowds refugees into unsanitary temporary housing there's a high likelyhood of more cholera breaking out than at other times?

    Gee. Color me impressed.

    How is this greatly different than many different types of analysts have been doing for decades via headline counts in world newspapers and the like? (See John Naisbitt of Megatrends fame, for example. And he certainly wasn't the first.)

    The math used to find the probabilities may be a bit better, and it may be more automated, but it's not particularly new.

  • Yes, i'm sure this audience will always quote that old hack Asimov but perhaps Mark Twain is better - "history does not repeat itself, but it does rhyme". Humans respond to much the same situations in much the same way - emphasis on 'much', because there are always differences of culture, place and circumstance.

    That is why we study history - to guide us in our own decisions. Do some research on Heine and Neitzche and the Doctrine of Eternal Recurrance which ponts to recurring patterns of human behaviour,

    • Re: (Score:2)

      by Anonymous Coward

      But tell me please - this baby boomer asks why do so few gen X and Ys show any interest in history?

      Because the Founding Fathers were Christian through and through; the Civil War was about slavery and nothing else; and we're the chosen people, spreaders of truth and justice and democracy, having never committed wholesale genocide or locked up our own citizens in concentration camps.

      Also, they hate us for our freedumbs.

      Seriously, though, most history teachers bore kids to death by doing little more than demanding they memorize places, names and dates. While some of them are shitheads who get off on this s

    • this baby boomer asks why do so few gen X and Ys show any interest in history?

      ...probably because there are few jobs where history is of direct relevance and practical usage.

      If you're investing a considerable sum in money and precious years of your youth then getting a real payback is a valid target; not everyone is rich enough to be able to dedicate resources to indulge in "intellectual masturbation".

      This isn't to decry history, or to deny that many find it interesting - just one potential explanation on

  • Will it predict the future, or media cycles? (Score:3)

    by dorpus ( 636554 ) on Sunday February 03, 2013 @10:21PM (#42782091)

    I have two family members in the journalism business. The media business has a cycle of covering particular topics and moving on when the public gets bored of it. Plenty of news does not get coverage at all, at least by English language media, if the country is too remote or the topic is too cliche.

  • Only as good as the source material (Score:3)

    by RogueWarrior65 ( 678876 ) on Sunday February 03, 2013 @10:22PM (#42782093)

    The NYT? For only the last 20 years? That's worse that basing global warming predictions on just the last 20 years.

    • Newspapers focus on big events not gradual changes. So a study like this is going to miss non-events until there are enough natural disasters to report a definite trend. But it will always be easy to be wise after the fact by selecting keywords related to the event you want to trace historically and thus prove your method could have predicted the present.

      • That assumes that the paper is reporting everything. Not only is it physically impossible for papers to do so but editorial policy will emphasize one type of story and demphasize or completely ignore another. In addition, does this prediction system take into account the location and size of story? What shows up on page one above the fold in one paper (with one editorial viewpoint) may only appear several pages down in another and the number of words can also be vastly different. A paper could easily ch

    • I think we're a little more short-term than climate change.

      • Hardly. Back in the 80s and early 90s, nobody ever heard of global warming. In fact, during the 70s, there was a lot of ginned up hand-wringing about another ice age.

        • Hardly. Back in the 80s and early 90s, nobody ever heard of global warming.

          Public recognition, regardless of what any ad agency may tell you to the contrary, does not regulate reality.

          In fact, during the 70s, there was a lot of ginned up hand-wringing about another ice age.

          No. There was ONE article about it in (I believe) Time Magazine.

  • Reminds me of the movie "Paycheck" but other than that it has been pointed out to me by a financial adviser that FDR converted this country from a republic to a democracy as the people gave him credit for bringing the US out of the great depression. Only it wasn't the government but the oil industry. This is the history part. The future part is strongly suggesting Obama is going to convert this country from a democracy to Socialism and he too will take credit for turning the economy around, and again it won

  • There is nothing Microsoft Research ever will discover that will ever be of use to the rest of us in general. They research for Microsoft, not for us, and almost all of their output is ignored even by Microsoft. When it's a product then look at it in terms of whether it's useful to you despite the completely unnecessary but mandatory IE/DirectX interface. It probably won't even be worth loading up Windows in a VM to have IE to try it out with.
  • But, isn't Zapaday.com already doing that?

  • Made mostly of cellulose, and some of the most expensive fluids known to mankind, this newly discovered vessel is capable of transmitting information from past to present and can acurately predict many future events.

    Scientists dub "Calendar" as discovery the year.

  • Isn't this a job for the Time Bandets?

  • Was that nothing had essentially changed from the beginning of man, so you could scientifically use Greek or Roman battles to navigate the contemporary wars with Pisa and Cesare Borgia.

    Usually, though, such an enterprise would only be fruitful after the event (e.g. no scientific predictions) on account of Fortuna. Notably, Machiavelli died bitter and beaten.

    Point of this is that you cannot predict the future; perhaps at best a possible outcome with some better chance than other counterfactual scenarios, whi

  • "Garbage in, Garbage out".

    But, it's probably worth a try...

Slashdot Top Deals

What is research but a blind date with knowledge? -- Will Harvey

Close