×

Announcing: Slashdot Deals - Explore geek apps, games, gadgets and more. (what is this?)

Thank you!

We are sorry to see you leave - Beta is different and we value the time you took to try it out. Before you decide to go, please take a look at some value-adds for Beta and learn more about it. Thank you for reading Slashdot, and for making the site better!

Statistics Losing Ground To CS, Losing Image Among Students

Unknown Lamer posted about 4 months ago | from the big-bad-data dept.

Stats 115

theodp (442580) writes Unless some things change, UC Davis Prof. Norman Matloff worries that the Statistician could be added to the endangered species list. "The American Statistical Association (ASA) leadership, and many in Statistics academia," writes Matloff, "have been undergoing a period of angst the last few years, They worry that the field of Statistics is headed for a future of reduced national influence and importance, with the feeling that: [1] The field is to a large extent being usurped by other disciplines, notably Computer Science (CS). [2] Efforts to make the field attractive to students have largely been unsuccessful."

Matloff, who has a foot in both the Statistics and CS camps, but says, "The problem is not that CS people are doing Statistics, but rather that they are doing it poorly. Generally the quality of CS work in Stat is weak. It is not a problem of quality of the researchers themselves; indeed, many of them are very highly talented. Instead, there are a number of systemic reasons for this, structural problems with the CS research 'business model'." So, can Statistics be made more attractive to students? "Here is something that actually can be fixed reasonably simply," suggests no-fan-of-TI-83-pocket-calculators-as-a-computational-vehicle Matloff. "If I had my druthers, I would simply ban AP Stat, and actually, I am one of those people who would do away with the entire AP program. Obviously, there are too many deeply entrenched interests for this to happen, but one thing that can be done for AP Stat is to switch its computational vehicle to R."

Sorry! There are no comments related to the filter you selected.

Well (1)

Anonymous Coward | about 4 months ago | (#47764169)

My margin of error is pretty high so things never really seem to turn out how I expect them to turn out.

As a statisticians (3, Interesting)

Anonymous Coward | about 4 months ago | (#47764197)

As a statisticians, you should know better that you don't make your point with a succession of anecdotes as

- A few years ago, for instance, I attended a talk by a machine learning specialist who had just earned her PhD at one of the very top CS Departments. in the world. She had taken a Bayesian approach to the problem she worked on, and I asked her why she had chosen that specific prior distribution. She couldn’t answer – she had just blindly used what her thesis adviser had given her–and moreover, she was baffled as to why anyone would want to know why that prior was chosen.
- But there is no substitute for precise thinking, and in my experience, many (nominally) successful CS researchers in Stat do not have a solid understanding of the
fundamentals underlying the problems they work on. For example, a recent paper in a top CS conference incorrectly stated that the logistic classification model cannot handle non-monotonic relations

Re:As a statisticians (3, Funny)

BorisSkratchunkov (642046) | about 4 months ago | (#47764215)

Considering how small the population size for machine learning researchers in academia can be, it is very likely that anecdotes can constitute a satisfactory sample.

Re:As a statisticians (4, Insightful)

Anonymous Coward | about 4 months ago | (#47764295)

Machine learning is an example in the article. This is a blatant attack on all CS students, researchers and professors.

Let’s consider the CS issue first. Recently a number of new terms have arisen, such as data science, Big Data, and analytics, and the popularity of the term machine learning has grown rapidly.

He seems to not really know CS. Statistics and probability are a tool to CS since the very inception. This is no news.

Re:As a statisticians (0)

Anonymous Coward | about 4 months ago | (#47764335)

In just one journal (Journal of Machine Learning Research [jmlr.org] ), there is hundreds of papers [jmlr.org] . 2 anecdotes is insignificant. This guy have no shame.

Re:As a statisticians (1)

Anonymous Coward | about 4 months ago | (#47764587)

According to you, he's attacking himself: He is a CS professor (with a PhD in statistics) in a CS department, who thinks that computer science needs to make better use of the tools of statistics and probability.

Re:As a statisticians (0)

Anonymous Coward | about 4 months ago | (#47764675)

Let’s consider the CS issue first. Recently a number of new terms have arisen, such as data science, Big Data, and analytics, and the popularity of the term machine learning has grown rapidly.

He seems to not really know CS. Statistics and probability are a tool to CS since the very inception. This is no news.

You're responding to a point he never made. 'Data science', 'Big Data', and 'analytics' *are* terms which were recently added to the IT trade lexicon, and machine learning *is* growing rapidly in popularity. So he's correct on the points that you quoted.

Re:As a statisticians (0)

Anonymous Coward | about 4 months ago | (#47764933)

Re:As a statisticians (2, Informative)

93 Escort Wagon (326346) | about 4 months ago | (#47765041)

You're arguing against a point he didn't make. He didn't say those terms were recently created - he stated they recently were added to the "IT trade lexicon", which is true.

10-15 years ago you didn't hear people bandying about the terms "big data" or "data science".

Re: As a statisticians (0)

Anonymous Coward | about 4 months ago | (#47765133)

As a scientists forced to become a SAN and cluster admin, it's because how data is generated changed.
Film was replaced by digital phosphor and fluoro digital imaging systems. Sequencing progressed from tens of bases to a hundred Gbases per experiment. Cameras migrated from Kodak slides to high bit depth digital movies. I'm a small lab admin. We generate about 2TB of new data to process every day. Scientist can now generate data in volumes they aren't equipped to handle. It was a good time to be in science.

Re: As a statisticians (1)

bill_mcgonigle (4333) | about 4 months ago | (#47766781)

what's funny is that CS nerds and stats nerds work very hard together to enable hard drive firmwares that permit the very dense and cheap storage that scientists and statisticians need. Not to mention the broad applicability of coding theory to every other discipline. TFA might have a point on the margins but by and large he's trolling academia (which is working to bring attention to his issue).

Re:As a statisticians (0)

Anonymous Coward | about 4 months ago | (#47765223)

All my apologies, I did not noticed CS was now IT trade.

Let’s consider the CS issue first. Recently a number of new terms have arisen, such as data science, Big Data, and analytics, and the popularity of the term machine learning has grown rapidly.

For sure, my argument was cleary a straw man by misrepresenting CS as not IT Trade and confusing buzz words with science.

Re:As a statisticians (3, Insightful)

u38cg (607297) | about 4 months ago | (#47765031)

Get real. Anyone doing "statistics" who doesn't understand the concept of a prior is just pretending to do statistics. That is a problem.

Re:As a statisticians (0)

Anonymous Coward | about 4 months ago | (#47766237)

>Anyone doing "statistics" who doesn't understand the concept of a prior is just pretending to do statistics.

Please explain.

Sure, we could lose 50% of our statisticians (5, Funny)

NotDrWho (3543773) | about 4 months ago | (#47764199)

But there's only a 25% chance of that.

Re:Sure, we could lose 50% of our statisticians (0)

Anonymous Coward | about 4 months ago | (#47764359)

So it's either a coin toss or a crap shoot. The CS students should let /dev/urandom decide.

Re:Sure, we could lose 50% of our statisticians (1)

Austerity Empowers (669817) | about 4 months ago | (#47765951)

I question whether the distribution of /dev/urandom provides the distribution necessary to ensure the desired outcome.

Re:Sure, we could lose 50% of our statisticians (1)

Anonymous Coward | about 4 months ago | (#47764409)

but that applies 100% of the time.

Re:Sure, we could lose 50% of our statisticians (0)

Anonymous Coward | about 4 months ago | (#47765415)

110%, because Coach Smith always told us that's what we needed to give to win.

Re:Sure, we could lose 50% of our statisticians (1)

HornWumpus (783565) | about 4 months ago | (#47768911)

Whenever someone asks for 110% I give it to them. 25% mon-thurs and 10% on friday.

Re:Sure, we could lose 50% of our statisticians (1)

mysidia (191772) | about 4 months ago | (#47767043)

But there's only a 25% chance of that.

And that's just the average probability, with the actual probability for a given sample in terms of percentage points having a standard deviation of + / - 25 percentage points.

Agreed (0)

Anonymous Coward | about 4 months ago | (#47764213)

Yeah, I'd definitely switch AP Stat's computational vehicle to R. I might even take it straight to S or T if it could be done at reasonable cost.

Re:Agreed (2)

Bigbutt (65939) | about 4 months ago | (#47764369)

But what about Q?

[John]

Re:Agreed (4, Funny)

Chris Mattern (191822) | about 4 months ago | (#47764777)

But what about Q?

We'll have to ask M.

Re:Agreed (1)

Hotawa Hawk-eye (976755) | about 4 months ago | (#47764977)

Hmm ... John de Lancie in the next Bond film as the gadgeteer for the CIA, with whom MI6 partners on a mission of importance to both agencies? I'd see that.

Statistics as standalone field (4, Insightful)

sinij (911942) | about 4 months ago | (#47764227)

Statistical analysis is now more complex, and statistics are better understood in science than a decade ago. There are number of software packages and libraries that simplifies and standardizes techniques.

Correctly applying all of these require subject matter expertise. You need to understand what you analyzing. As a result pure statistician is not very useful - generic analysis can be performed by software, in-depth analysis requires specific knowledge.

This is not unlike complaining that assembly coding is dying. Well, yes, we now have less need to code everything that way because we have better tools.

Re:Statistics as standalone field (4, Insightful)

Anonymous Coward | about 4 months ago | (#47764405)

Correctly applying all of these require subject matter expertise. You need to understand what you analyzing. As a result pure statistician is not very useful - generic analysis can be performed by software, in-depth analysis requires specific knowledge.

From my experience, statisticians tend to be far more successful acquiring subject matter expertise than people in other fields have in using proper statistical procedures for their problems.

It's like saying mathematicians are not useful because calculators. It's simply not true, and while software can perform generic analysis, it is only quite a tiny part of doing a statistical problem correctly. What we have now are coders who think that computers can set up and interpret their problems correctly, and thus we have an increase in bad results.

Re:Statistics as standalone field (1)

sinij (911942) | about 4 months ago | (#47764683)

Yes, there are statisticians that end up working in other fields and they tend to be better at statistics than a typical practitioner in their adapted area of expertise. Thing is, these statisticians are no longer in the field of Statistics, they are researchers in these other fields.

Re:Statistics as standalone field (0)

Anonymous Coward | about 4 months ago | (#47764435)

Who will complain against some old people dying [wikipedia.org] furthermore french ? Anyway I never noticed they were coding.

Re:Statistics as standalone field (4, Insightful)

wisnoskij (1206448) | about 4 months ago | (#47764569)

I completely disagree. Pretty much everyone is complete shit at statistics. It is a very very advanced and unique field that is continually and horribly bungled by scientists and everyone else. We need statisticians, that said I cannot imagine anyone wanting to go into stats.

Re:Statistics as standalone field (3, Insightful)

sinij (911942) | about 4 months ago | (#47764613)

Following is anecdote, but when someone I knew approached multiple statisticians with a model question (related repeated measures), the understanding of concept was not there. If your view that "everyone is complete shit at statistics", that should include statisticians.

Re:Statistics as standalone field (1)

Anonymous Coward | about 4 months ago | (#47764767)

Amen! Again anecdotal, but my experience is that junior statisticians, i.e. recent graduates, are for more likely to be able to perform the statistics you need in business, health research, etc than a statistician with postgraduate qualifications. I suppose it's a case of a little knowledge going a long way. The junior guy has more than enough knowledge to deal with 80% of what most people might need to do. It's rare that business or the health sciences need anything more complex than an ANOVA to make their decisions, assuming they've captured their data correctly. Ask your local stats professor a basic question: how do I tell if my new business process is effective? or How do I tell if my experimental treatment affects fur growth on rats? are rarely answered simply and in context. A t-test is probably sufficient for both cases, and a junior statistician will blurt that out first. The professor will pontificate and not give you a straight answer.

I had a simple case where I had two datasets produced by different software algorithms running on the same raw data. No algorithm was particularly reliable, and I wanted to combine their reliabilities, i.e. combine the p-values. Fischer's method does this just fine, and yes my data matched the assumptions. Could one qualified statistician tell me how to do this? NO! It was a an undergraduate who had just learnt about it who told me about it, and showed me a textbook to look it up in.

Re:Statistics as standalone field (1)

HornWumpus (783565) | about 4 months ago | (#47768957)

What you say is generally true of professors. Not just statisticians.

But it's reasonable. The professors assume you already know the simple answer and give the somewhat complete one.

Re:Statistics as standalone field (1)

radtea (464814) | about 4 months ago | (#47769923)

If your view that "everyone is complete shit at statistics", that should include statisticians.

This has been my experience as well. I would go so far as to say that statisticians understand probability less well than most working experimental scientists. They are overly-enamoured of abstract models and rarely dig down to the raw probability distributions underneath, which is what working scientists actually care about.

Re:Statistics as standalone field (1)

JeffSh (71237) | about 4 months ago | (#47765205)

it seems to me that statisticians are CS overlapping with practical insight.

When a report is prepared, it takes consideration to define all of the inputs and modifiers that lead to a successful statistical analysis. Without this hard-to-define inputs, i can see how and why a CS-only based approach to stats fails.

Re:Statistics as standalone field (1)

disposable60 (735022) | about 4 months ago | (#47765537)

But I thought Econ was the Dismal Science.

Re:Statistics as standalone field (1)

Anonymous Coward | about 4 months ago | (#47766033)

I want to learn statistics, but it's so ... big. There just aren't enough hours in the day as a working software-industry professional to do something as big and complex as learning statistics. After a day of coding and debugging, my brain is fried. Opening a probability book just doesn't work. I guess you need to learn it while you're still in school, and your job is studying. I took a statistics class in college, but can't remember anything about it.

Besides, "statistics" is an umbrella term for probability, combinatorics, regression, and so on. Each field could take years to master.

Statistics as standalone field (3, Informative)

aaaaaaargh! (1150173) | about 4 months ago | (#47765013)

Quite the opposite is the case. Unless we are talking about experiments with terrabytes of data most software packages are complete overkill anyway, you could make your statistics with a pocket calculator instead. The problem is the conceptual work. Most institutes and individual scientists would be much better off if they employed a well-trained full-time statistician. Provided they were interested in correct and robust results rather than getting one more pilot study published as soon as possible (which will in turn be based on an insignificantly small non-random sample using an inadequate model).

Re:Statistics as standalone field (1)

Rich0 (548339) | about 4 months ago | (#47766783)

I think you hit the nail on the head. Nobody cares about getting it right, they just care about getting it accepted. People know enough statistics to be dangerous.

But the same is true of almost any field. Unless you work in some kind of skunk works team, how many of your coworkers REALLY have a good grasp of the fundamentals in whatever profession you work in? Do you think the average CS major has any idea what an opcode is, or how to implement a binary adder (just in terms of theoretical gates, let alone actual circuitry)? How many cooks really appreciate how the presence and properties of water affects the temperature at which food cooks?

I'm sure even among experts, the average statistician is, well, average.

Re:Statistics as standalone field (0)

Anonymous Coward | about 4 months ago | (#47765319)

Don't forget that Statistics isn't one of those "Hard" math fields. A properly taught statistics course should be about 50% math and 50% ethics. This turns a lot of CS types off since they usually don't like social science.

A properly taught statistics course will also turn you in to a cynical skeptic. You'll basically never believe anything told to you by any authority every again.

Re:Statistics as standalone field (1)

Capt.Albatross (1301561) | about 4 months ago | (#47766121)

As a result [a] pure statistician is not very useful - generic analysis can be performed by software, in-depth analysis requires specific knowledge.

In-depth analysis requires a real understanding of statistics as well as of the domain. CS knowledge, at least as commonly taught, is not a substitute for for the statistics requirement.

This is not unlike complaining that assembly coding is dying. Well, yes, we now have less need to code everything that way because we have better tools.

This is not a valid analogy. HLLs automated some of the rote, mechanical aspects of implementing algorithms. They do not automate away the need for a higher-level understanding of what you are doing.

Re:Statistics as standalone field (0)

Anonymous Coward | about 4 months ago | (#47769073)

It sounds to me like you're so bad at statistics that you don't know how bad at statistics you are.

Is the benefit worth the cost? (0)

Anonymous Coward | about 4 months ago | (#47764237)

If there is a decline in quality it is probably because the quality wasn't needed and people don't want to pay for things they don't need.
Students don't want to waste their time learning something they aren't going to use when they go to work and in most cases bad or even incorrect statistics will be just as useful.
In the same way most engineering doesn't require much math but rather just an understanding of how things work. Why you have that understanding the math doesn't get much more complicated than division.

It would be great if the students knew things better, but they are limited by time and money so they will focus on the things that gives most reward to the time put in. If Prof. Matloff is worried he might want to consider teaching the students statistics for free.

Re: Is the benefit worth the cost? (0)

Anonymous Coward | about 4 months ago | (#47764421)

> In the same way most engineering doesn't require much math but rather just an understanding of how things work. Why you have that understanding the math doesn't get much more complicated than division.

Engineering stops at division? You must be joking. Engineering is more than applying d=rt everywhere, idiot.

Re: Is the benefit worth the cost? (0)

Anonymous Coward | about 4 months ago | (#47765649)

NASCAR Engineering...

Ask a math major what stats major is (-1)

Anonymous Coward | about 4 months ago | (#47764253)

Stats major is for people who couldn't hack real math, so they switch to fake math. The way stats are used today, it's a method to make data say what ever you want, not to discover patterns or trends in data. I don't think we need a separate Stats major, it is already covered by Math. Getting rid of stats would probably be better for society and get rid of misuse of stats.

Re:Ask a math major what stats major is (1)

Salem Lowe (3800579) | about 4 months ago | (#47764491)

Not true, there will ALWAYS be a need for statistics. What will politicians do if they can't lie about numbers? What will happen to all the global warming research if they can't use statistics to lie?

Re:Ask a math major what stats major is (0)

Anonymous Coward | about 4 months ago | (#47764597)

True, politicians will always need statisticians to lie for them. Alternate solution to getting rid of stats, create a Hippocratic oath for statisticians. That way, if harm is done they have to commit ritual suicide or testify against their political sponsor.

Statistics has always had difficulty with usurpers (5, Insightful)

BorisSkratchunkov (642046) | about 4 months ago | (#47764273)

Most notably psychology, economics, mathematics and beer brewing. In fact, most of the developments in stats have come about as a result of a need arising in a different discipline. Stats is inherently an applied discipline, so this is not unusual.

What is concerning is how many statistical tools, each with their own set of assumptions, have blossomed up within the past few decades. There are so many stats now that stats can no longer be an ancillary to other disciplines- it needs to be given its own space and statisticians need to be given respect for their unique expertise. There is simply too much knowledge in that domain for those in more theory-driven fields to be able to claim both expertise in the conceptual models of their fields and statistics.

Re:Statistics has always had difficulty with usurp (0)

Anonymous Coward | about 4 months ago | (#47764367)

Most notably psychology, economics, mathematics and beer brewing. In fact, most of the developments in stats have come about as a result of a need arising in a different discipline. Stats is inherently an applied discipline, so this is not unusual.

That can be said for most math.
Pure mathematicians tend to only deal with what is already known. If you start to look at the greatest advances in math and the mathematicians behind them you'll see that they most of the time came from another field and needed new math to solve their problem. Necessity is the mother of invention.

Re: Statistics has always had difficulty with usur (0)

Anonymous Coward | about 4 months ago | (#47764717)

IIRC isn't that how calculus came about?

Correlation != Causation (0)

Anonymous Coward | about 4 months ago | (#47764293)

Any time you see a summary on Slashdot with the word "statistics" repeated a few times, that observation has to apply somewhere.

Statistical Practitioners need to Modernize (5, Interesting)

wispoftow (653759) | about 4 months ago | (#47764319)

I am a researcher in medical informatics, and statistics is a huge part of my job, though I am not a classically-trained statistician.

First, I would like to offer a stark contrast between two types of statisician: 1) statisticians of the old mold who are wedded to SAS and related tools and 2) research statisticians who employ modern methods such as Bayesian statistics and rather advanced calculus. The former tend to mold all problems into what is available in the canon of SAS routines, while the latter are capable of creating custom models that suit the problem at hand.

Then, there is a new breed of scientist -- the data scientist -- who tends to use black-box machine learning methods and the classical techniques, as programs such as SAS and R have "democratized" the field. I agree with the common gripe of many traditionally-trained statisticians who object that these "data scientist" tend not to understand the statistical background of these computer codes. In fact, it is easy to download R onto one's computer and start firing data through, with little regard for the merits of the model or its results. (Not all data scientists are like this, but I'm simply stating a general observation.)

Another problem with statistics is that it can be very confusing, understanding just what things like p-values mean. After a first course in statistics, it leaves many with a bad taste -- either being terribly confusing, or rather boring. In my opinion, this is because of traditional (frequentist) statistics, which have their origins from luminaries such as Fisher and Pearson.

The "action" today is in Bayesian statistics. This formulation allows for statistical concepts to be expressed is ways that (I believe) most people can understand. But executing Bayesian statistics mandates that one understand the underlying formulation of models; in general, they are not black-box methods. Furthermore, they can be quite computationally-expensive for large data.

Statistics is suffering from perceptions of being a button-pushing, boring profession. As has happened in many other fields (e.g. computational chemistry and CFD), computer programs have democratized the field so that those who have not had years of dedicated study and training can execute statistical models. In my experience, this can be a good thing, or a very bad thing. Another issue is that there is a significant build-up of half a century of code and protocols in both industry (think big business analysis) and government agencies (think FDA).

But modern statistics is actually a hot field. Provided that one understands the background, and is willing to go the extra mile to write custom code, the rewards are endless.

Re:Statistical Practitioners need to Modernize (1)

fropenn (1116699) | about 4 months ago | (#47765609)

I would add that many disciplines are recognizing the importance of statistics and are therefore introducing applied statistics courses for [discipline X]. This causes a drop in enrollment in the pure statistics courses, thus decreasing the number of pure statistics instructors, thus decreasing the demand for individuals trained in pure statistics. In this way statistics is losing itself as a discipline and is quickly becoming specialized into various disciplines (e.g., the application of statistics for medical research).

Re:Statistical Practitioners need to Modernize (0)

Anonymous Coward | about 4 months ago | (#47767417)

Statistics is mostly thought of as an insurance company job, or working for the gov. on things such as unemployment, death rates, etc.

Re:Statistical Practitioners need to Modernize (5, Informative)

Anonymous Coward | about 4 months ago | (#47765681)

It's a funny coincidence this appeared on Slashdot, as I was just reading about this issue and discussing it with my colleagues.

I'm a statistics researcher in an applied field (university academic research) that suffers its own image problem, and my impression is that what we're witnessing in many STEM areas are problems with stereotyping in science, and marketing fads. I'm not sure that I disagree with what you're saying, but I think that there's another stereotype operating as well that cuts at the field of statistics in a second direction.

As you point out, there are the sort of applied consulting statisticians who are probably getting increased competition from "data scientists."

On the other side of the issue, though, you have complaints about theory-focused statisticians who really don't understand how to implement their developments computationally, who are also getting increased competition from "data scientists." This has been mentioned in a number of blog posts in various places, and I see as much more as the driver of "data science" as a banner than competition with consulting statisticians. E.g., CS individuals who feel they can do Hadoop and so forth, and who have had enough stats training, probably in undergrad, that they feel like they can just sort of usurp the statistics from the statisticians. They see the theory as irrelevant or something.

The problem as I see it is that individuals who identify as "data scientists" don't really understand that the theory has to come from somewhere, and they fail to appreciate the issues that come up when dealing with uncertainty. It's like everyone in the field has some undergraduate-engineering-student level understanding of statistics, and don't have to deal with thorny data collection designs, complex inferences, or replicability of findings. The sort of scenario that's motivated "data science" is essentially this: a extremely large dataset involving relatively simple classification or prediction questions about observational data where there's really no scrutiny about generalizability or the meaning of the results. This problem scenario is why they got involved instead of a statistician in the first place: because the bottleneck was the size of the dataset, not the analysis scenario.

All of the attempts to distinguish "data science" from statistics it seems to me are based on stereotypes or misunderstandings about statistics, as you point out, or on extremely short-sighted perspectives on science and math. Computational statistics has been a core part of statistics for decades (there are journals devoted to the topic), and you can find peer-reviewed articles on all sorts of computational problems in statistics (e.g., the use of GPUs in estimation problems, how to approach optimization with distributed processors, etc.). The idea that statistics is all theory, and that statisticians don't understand computational issues is naive or has a very stereotyped view of statistics (or I pity their experiences in high school and college--it sounds like they got a poor education in statistics).

This isn't to pooh-pooh the contributions of CS--it's critical. But I hate the banner of "data science"--not only is the term stupid and redundant (how can you have science without data? What other kind of science is there?), it's based on ignorant stereotypes about statistics as a field.

To me, this speaks to a longer term problem in CS, which is CS essentially discovering what's been going on in other fields and reinventing the wheel over and over again. I don't see this necessarily in CS academic departments, but I do see it where there's some interface with the business world. It's coming up now with statistics, it's come up before with social sciences and economics, it's come up with AI and neuroscience, it's come up with genomics, it comes up over and over again. It speaks to a sort of arrogance or autism in the field's culture, where they act as if their unawareness of a phenomenon means that no one has ever researched it before.

Ughh... think about statistics as the mathematics of uncertainty, and see how far you get with deemphasizing that. Damn, I hate society sometimes. I need a walk.

Re:Statistical Practitioners need to Modernize (1)

Anonymous Coward | about 4 months ago | (#47766893)

Would mod parent up if I could.

I think part of the issue is that people in software are often expected to understand and implement (code) difficult concepts in fields well outside of their domain of knowledge, simply because they are experts at coding. For example, an academic in bioinformatics explained to me that he spent months correcting code in bioinformatics software that made the most basic of errors in genetics, such as reading a strand of DNA backward. I've seen similar issues in government, where the developers either fail to understand the policies they are coding around, or willfully ignore them because they "know a better way". I wouldn't expect a programmer's understanding of some of the more difficult concepts in statistics to be much better. Part of the problem is that domain knowledge can be difficult to pick up without bungling it; another part is the hubris of some in software development. Full disclosure: this is coming from a former programmer turned scientist.

AP? (1)

Anonymous Coward | about 4 months ago | (#47764331)

What the fuck does "AP" mean?
I'm dabbling on the "AP Central" website and other but they all talk about AP courses, how to get a course labelled AP, "AP is your time well spent" but never a definition of what AP is. It's ridiculous to use such a two-letter initialism and hide its meaning like it's a secret thing for "consumers" who buy higher end education in the US.

Re:AP? (2)

LateArthurDent (1403947) | about 4 months ago | (#47764579)

It stands for "Advanced Placement." They're college-level high school courses. At the end of the year, you take the advanced placement exam, and depending on your scores and the college you attend, you can get college credits for them.

I think getting rid of an AP is a stupendously short-sighted idea. Having students take more advanced courses earlier is a great idea. If there's reason to believe the courses aren't actually as demanding as their college equivalent (and I don't think there is, based on my experience taking AP Calculus in high school and looking at what people taking Calculus in college were seeing. We covered the same material, and if anything my high school class covered more), then you can make an argument for the tests more challenging / add to the requirements of those courses. Getting rid of it is just an attempt to waste students' time and extract more money from them by forcing them to take more university courses.

Re:AP? (2, Interesting)

Anonymous Coward | about 4 months ago | (#47765417)

Getting rid of it is just an attempt to waste students' time and extract more money from them by forcing them to take more university courses.

I suspect his complaint is that in high school, AP Statistics is taught by math teachers. In college, classes are taught by professors who specialize in statistics. This goes along with his general complaint that people in other disciplines don't take the time to really understand how statistics work. Of course, the same problem exists in college statistics courses. You can take a one semester survey course or the two semester theory course. He'd prefer that everyone took the two semester course and that it was rigorously graded.

He may be right about AP Statistics though. Taking statistics in high school means that most people will have forgotten it by the time they get to advanced courses that use statistical methods. This leads to students learning statistics from the professors in those advanced courses (who are not focused on statistical rigor). Statistics is a sophomore/junior level class, where most other AP classes substitute for freshman classes.

I would tend to agree with you about the other AP classes though. There's no such thing as a "calculus professor" -- calculus is taught by a mathematics professor who is likely interested in something very different. It doesn't make much difference whether it is taught in a small high school class or a large college lecture.

Re:AP? (-1)

Anonymous Coward | about 4 months ago | (#47765979)

I wonder about, instead of giving college credit, if someone is bright enough to pass the courses in HS here in the US, why not fund their college?

A friend's two kids from Germany have their whole college career funded by the Fatherland.

A colleague of mine from China had his degree and foreign visa paid for by his country.

A former classmate from Chile had his degree paid for by his government.

Here in the US, public universities pay for room and board for foreign students anyway... why not US citizens?

Re:AP? (-1)

Anonymous Coward | about 4 months ago | (#47766395)

We had that in the US... right up until the 1980s. Once the baby boomers had all graduated and got into the workforce, they took a look at the tax rates that had paid for their college, said fuck that, and pulled the ladder up behind them. Kids nowadays either have to get in debt up to their eyeballs, or hope mommy and daddy are loaded.

The Greatest generation and every one preceding it understood the societal importance of paying things forward, and leaving your children and grandchildren more opportunities than you had. The Boomers went and turned that around. The final joke is going to be on them, though. When it comes time to retire, who are you going to sell your house and your stocks to? It won't be to the kids with $40k in student loan debt, especially when they can't even get jobs thanks to all our shit getting outsourced. Which, again, was thanks to the Boomers wanting cheap shit and damn the consequences.

All ran off to quant finance (0)

Anonymous Coward | about 4 months ago | (#47764341)

This is not a new phenomenon. Most stats grads get big checks from banks...

1984 machine (0)

Anonymous Coward | about 4 months ago | (#47764383)

You can make money from writing your own statistical software, but it's not as easy to make money as an analyst I think.
In the end, we can all get jobs sweeping the floors of the AI farms.

Because lies and statistics are interchangable (0)

Anonymous Coward | about 4 months ago | (#47764407)

If you get a degree in law, business management, or marketing you can now put statistics as an academic minor on your resume.

CS is innocent... (2)

Buchenskjoll (762354) | about 4 months ago | (#47764467)

I think the problem is that statisticians have small, unconnected habitats and overly complex mating rituals.

If They're so Smart, Then why is this Problem? (1)

dcw3 (649211) | about 4 months ago | (#47764501)

Efforts to make the field attractive to students have largely been unsuccessful."

You would think they would know which efforts work and which don't. I'm only being a bit sarcastic with the Subject line, but seriously they should be able to figure out what does and doesn't work.

This explains why statistics invades CS (0)

Anonymous Coward | about 4 months ago | (#47764507)

While statisticians complain, I have to face the quite opposite situation of having my research field swamped by people doing statistical methods for everything. And in some cases this is not reasonable. Of course, many system designers dream of just throwing pieces of code together and still being able to prove that it works (with a certain probability), but getting meaningful probabilities still requires that certain independence hypotheses are respected, and in computers true stochastic independence is difficult to obtain...

Demographic of the Current Statistician (1)

Anonymous Coward | about 4 months ago | (#47764509)

I don't dare publish this unless I am anonymous, but I must state this observation:

We are always on the lookout for new statisticians in our medical group. About 95% of our applicants are Chinese females! I had asked one of our (Chinese) scientists about this, and he said that this is because of the proliferation of MS in statistics programs that are amenable to spouses who were interested in a profession that could be attained (and makes good bacon) while their husbands were working on advanced degrees in other fields.

There are eternal complaints about how some fields are full of old white men, and this tends to squeeze out others who do not fit this profile. As a result, those fields tend to lack innovation in thought and culture.

I believe it's possible that statistics is suffering a similar issue. Perhaps the field has become homogenized, and people just aren't interested in overcoming these cultural barriers. No one wants to be the minority, e.g. no one wants to be the lone woman in the field of a bunch of IT dudes. The opposite side of the same coin is that no one wants to be the lone white man in a room full of Chinese women. (Let's keep it clean.)

Every field needs diversity, which in turn fosters content and innovation. Statistics is no exception, and the field really needs to do something about this. I think their hands are tied, as fixing this problem would be so politically incorrect, that no one would dare.

Re:Demographic of the Current Statistician (1)

ColdWetDog (752185) | about 4 months ago | (#47765169)

The opposite side of the same coin is that no one wants to be the lone white man in a room full of Chinese women.

What is the pretest probability of this being true?

Odor of Corruption (1)

drinkypoo (153816) | about 4 months ago | (#47764641)

Statistics is a dirty word today, even though modern science depends upon it. The public most commonly encounters them when they are lied about.

Re:Odor of Corruption (1)

Xtifr (1323) | about 4 months ago | (#47766513)

Indeed, which is why, when I talk to kids about math in school, one of the things I like to point out is that while statistics are, in general, rather boring, it's really important to learn enough to have at least a chance of recognizing when they're being used to lie to you. This argument gets through to a suprising number of them.

Of course it's shrinking (2)

dywolf (2673597) | about 4 months ago | (#47764647)

As far as the general public is concerned:
When it's convenient, people use numbers, real or made up, in order to disprove the other sides point and prove their own...
When it's not convenient, all statistics become questionable ("ya, but msot statistics are made up") in order to disprove the other sides point and prove their own...

The reality of the numbers don't matter. People just don't care about actual objective facts, they just want to back up their preconcieved notions to spread their stupidity. It's just like how Americans approach science in general really.

Hard to do right, easy to not notice you're wrong (4, Insightful)

DoofusOfDeath (636671) | about 4 months ago | (#47764673)

I'm not very trained in statistics, but I've read more than my fair share of academic computer science papers over the years.

Even with my limited training in statistics, I've known enough to be appalled by the errant statistical reasoning used. Or even not used. I.e., "We don't know how many times to run a program to get a 'valid' average running time, so we ran it three times. Here's the average: ..." The authors seemingly aren't just ignorant of how to get the answer; they often seem to have not thought through what questions they're trying to answer in the first place with their measurements and resulting statistics.

I think a few problems come into play here:

  • The mathematics of statistics can be hard.
  • Thinking through the meanings of statistics requires careful thought, especially for experimental design and/or system performance characterization. Many CS practitioners would prefer to not invest mental energy in this aspect of their work because they don't enjoy it; it's a distraction to what they want to do.
  • Because so many people in CS are bad at statistics, peer reviewers tend to let it slide. This helps foster a culture problem. If I'm under the wire to get a paper published and I'm near deadline, do I take an extra 20 hours to get the statistics right? Especially knowing that I'm judged by the number of published papers, and that the peer reviewers won't notice or care about poor statistical reasoning?
  • It's easy to make statistical reasoning errors without noticing it. Especially if you're not surrounded by statisticians.

Despite CS majors thinking we're so smart about mathematical issues, I think this might be one area where that confidence is delusional. I suspect most psychology majors who paid attention in their Experimental Design courses are more capable in the appropriate mathematics than are most CS majors.

Re:Hard to do right, easy to not notice you're wro (0)

Anonymous Coward | about 4 months ago | (#47765791)

How can you compare a kid running a program three times to obtain a mean to the calculus required for even the most trivial statistical problems?

Re:Hard to do right, easy to not notice you're wro (1)

Rich0 (548339) | about 4 months ago | (#47768127)

How can you compare a kid running a program three times to obtain a mean to the calculus required for even the most trivial statistical problems?

That was his whole point. The fact that many people think that the one is a substitute for the other indicates that there is a big problem in CS.

That said, trivial statistical problems usually don't require calculus to solve. I fully appreciate that commonly used statistical functions are rooted in calculus, and you need to understand it at some level to apply them properly. However, the mechanics of solving the problems usually do not require solving integrals/etc. To use a car analogy - I can use a speedometer without knowing what a derivative is.

Statistics? (1)

Rambo Tribble (1273454) | about 4 months ago | (#47764701)

There's an app for that!

Not surprised (0)

BCW2 (168187) | about 4 months ago | (#47764711)

Statistics as taught in it's current form (just like economics) came about in the 50's, they were both designed by out of work mathematicians. Now to an unemployed Math PHD in the 50's with the start of the space program and the massive military research programs, how good were these guys? When I took Stat in college I was amazed. Take one set of data and produce two diametrically opposed answers and have them both correct? Sounds like rumor, gossip, and BS to me, not science.
No wonder there are lies, damn lies, and statistics!

Re:Not surprised (3, Insightful)

u38cg (607297) | about 4 months ago | (#47765121)

>> Take one set of data and produce two diametrically opposed answers and have them both correct?

You missed the point of the lesson. The point was that you didn't have enough data to demonstrate that your model was valid. That's all.

Re:Not surprised (3, Insightful)

ColdWetDog (752185) | about 4 months ago | (#47765183)

Take one set of data and produce two diametrically opposed answers and have them both correct? Sounds like rumor, gossip, and BS to me, not science.
No wonder there are lies, damn lies, and statistics!

Somebody missed the lecture on assumptions.

There are three types of lies... (1)

Xiver (13712) | about 4 months ago | (#47764991)

"There are three types of lies: lies, damn lies, and statistics." -attribution disputed

http://en.wikipedia.org/wiki/L... [wikipedia.org]

Actually there are five types of lies (0)

Anonymous Coward | about 4 months ago | (#47766069)

In rank order:
lies damn lies statistics benchmarks {hated corporation} benchmarks

Difficult to determine what TFA is about (1)

dtjohnson (102237) | about 4 months ago | (#47765099)

Is this very poorly written article about: 1) students not choosing to pursue a career path in computer science rather than statistics... or... 2) CS people doing poor-quality statistics work... or... 3)banning the Advanced Placement "Statistics" class because students are relying too much on their "pocket calculators." We get three-articles-in-one to talk about here. At least they are all loosely related to something called "statistics."

So? (0)

Anonymous Coward | about 4 months ago | (#47765269)

So who cares?

When the number of people doing statistics gets low enough, salaries will go up and more people will start studying them.

There's also H1Bs to solve the problem.

The number of Liberal Arts majors is declining as well; if that happens where will Starbucks get there baristas?

Lies (0)

Anonymous Coward | about 4 months ago | (#47765299)

There four types of lies:
  Lies, Damned Lies, Statistics and CS Simulations.

Stats in c++ STL (0)

Anonymous Coward | about 4 months ago | (#47765551)

Participate in the c++ meetings and code up STL stats facilities make gets into next c++ standard.

Stats is hard, let's go coding (1)

RWerp (798951) | about 4 months ago | (#47766231)

Statistics done right is hard and boring. People prefer hacking to do hard and boring stuff.

Finding work in statistics (1)

tomhath (637240) | about 4 months ago | (#47766451)

I work with a couple of very good statisticians. What they do is a mystery to me, but one thing I can say for sure - a good programmer or DBA will find work much more easily than a good statistician. In large part because PHBs have no clue why they need someone with more than two semesters of probability in almost every application.

Another problem with students going into statistics in the US is that virtually all of the instructors don't speak very good English. To this day I want to say things like "probabirity", "rotatation about the ashes", and the one that confused everyone in the class - "ashama" (eventually translated to axiom).

Re:Finding work in statistics (1)

Zero__Kelvin (151819) | about 4 months ago | (#47766859)

"don't speak very good English."

Did you know that, statistically speaking, 100% of people who "don't speak very good English." don't speak English very well?

Re:Finding work in statistics (0)

Anonymous Coward | about 4 months ago | (#47767125)

"Good" in front of "English" is an adjective and perfectly acceptable. If it were behind, and thus an adverb on "speak", then you'd have a problem.

Re:Finding work in statistics (1)

Zero__Kelvin (151819) | about 4 months ago | (#47767969)

Ah my brethern! Would that it twas 'nuff to be semantically correct, but alas, 'tis not good English, for your words speaketh an untruth that furthers not the word of thy Lord!

I'm going to do the unthinkable and use a programming analogy circa 2014 on Slashdot:

if (x=1) { printf("God, like your English, is good! (though I don't program very well)"); }

Get it? It's semantically correct, but it doesn't "say" what you think it does :-)

Have you ever seen "happy" statistics? (0)

Anonymous Coward | about 4 months ago | (#47766457)

No, neither have I. They are always dreary and depressing. Nobody wants to be a statistician.

Consider being a subset of SC (1)

Karmashock (2415832) | about 4 months ago | (#47766727)

Statistics as a subset of CS isn't unreasonable given that nearly all statistics will be calculated by software.

Re:Consider being a subset of SC (1)

the eric conspiracy (20178) | about 4 months ago | (#47769005)

That's complete horseshit (along with this article). It's like saying math is a subset of CS because nearly all maths will be calculated by CS.

Stats is orthogonal to CS. You don't need one to do the other.

Having both though can give you a skill set that's quite useful.

Re:Consider being a subset of SC (1)

Karmashock (2415832) | about 4 months ago | (#47769421)

That's fine. Just throwing out an idea. Perhaps keep statistics as it is... but add a CS statistics course so that the CS students get a better background in it.

Not trying to upset apple carts here. Just trying to find reasonable solutions. :)

Who is this guy? (0)

Anonymous Coward | about 4 months ago | (#47767289)

Looking at his publications page, he hasn't published anything of note in years: http://heather.cs.ucdavis.edu/matloff/public_html/vita.html

Reading the blog post, it seems that he is woefully ignorant of the state of modern machine learning research.

Never mind the actual arguments he makes (which others addressed), I'm not really sure this guy has much authority on this matter, either in CS *or* in statistics.

I apply stats everyday (0)

Anonymous Coward | about 4 months ago | (#47767479)

The only stats I need to understand and to apply is what is the probability of me winning the powerball lottery? how many tickets can I afford to buy per attempt? and knowing that there is a high probability that gov will show up to collect a high amount of the winnings....

Statistics is awesome (1)

HalfFlat (121672) | about 4 months ago | (#47767989)

It's a shame it has such a reputation for being boring, and it is a shame that it seems to be rarely taught in an engaging way.

Statistics is the first artificial intelligence. It formalises what we know when we 'know'. It is fundamental.

It's also fairly hard to do right. But many worthwhile things are hard.

worms eye view (1)

micahraleigh (2600457) | about 4 months ago | (#47767999)

I know a lot of people who get CS gigs after school. It pays their bills. They do well.

The stats people I know are really, really rich. And there are a lot of them.

That's in Raleigh.

Better Long-Term Prospects than IT (1)

Tablizer (95088) | about 4 months ago | (#47768025)

If you like the field of statistics it seems a better long-term bet than IT. The "laws" of math are not going to change in 40 years, where-as in IT the languages, GUI's, frameworks, and Paradigm Fad of the Day will change...several times. Plus it won't give you Carpel Tunnel (unless you can't trick a grunt into data entry). You are expected to know the domain (industry) such that outsourcing is not as likely either.

Software may pay more in the short term, but career-wise, stats seems more stable.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?