Effect and cause as a clue to the meaning of science

The December 16, 2011, issue of WIRED has a piece by Jonah Lehrer called "Trials and Errors: Why Science is Failing Us" (click here to read it). Mr. Lehrer's argument seems to be that some phenomena are too complex for scientific method to be able to discover what causes them. In his conclusion he writes:

And yet, we must never forget that our causal beliefs are defined by their limitations. For too long, we’ve pretended that the old problem of causality can be cured by our shiny new knowledge. If only we devote more resources to research or dissect the system at a more fundamental level or search for ever more subtle correlations, we can discover how it all works. But a cause is not a fact, and it never will be; the things we can see will always be bracketed by what we cannot. And this is why, even when we know everything about everything, we’ll still be telling stories about why it happened. It’s mystery all the way down.

The comments following the piece do a good job of of pointing out the flaws in the reasoning by which Mr. Lehrer reaches this conclusion. However, one issue is omitted. That issue is that science is not about causes.

Science is about effects. At its simplest, an effect is a non-random relationship between two variables. Scientific experimentation investigates effects by varying one of the variables (the indendent variable) and seeing what happens to the other variable (the dependent variable). The goal is to explain the effect - that is, become more effective in predicting the dependent variable. This model can be expanded to handle large numbers of variables. For example, one of the things I do in evaluating satisfaction with a program is to investigate simultaneously the relative importance of several variables in accounting for satisfaction. What you typically find when you do this correctly is that only a few of the variables have any relationship to satisfaction. What you often find, too, is that the variables that account for their satisfaction are different from the reasons particpants report when asked why they like the program.

The methods I use are correlational, so they cannot attribute causation. What they tell you is that as one thing varies, so does another. Furthermore, the analyses of satisfaction I do are non-experimental, so I can't even be sure that the estimates of the correlations are all that exact. What I can do, though, is make a recommendation that changes be made to see if dealing with the the variables identified by the data analysis will improve satisfaction.

The same considerations apply to a lot of health research, and that consideration alone goes a long way to accounting for the examples Mr. Lehrer adduces. What health researchers do is develop their own recommendations for further research that will test whether their conclusions are correct. In fact, the supposed failure Mr. Lehrer describes is in fact a demonstration of the success of science - a hypothesis was developed from prior research to test whether a drug was effective, and the test failed to find evidence that it was effective. That failure by itself is informative - it tells us not to prescribe the drug.

One of the commenters at the link above (urgelt) goes into the issue of the adequacy of research in more detail. My post of January 5 (click here) provides another example of this type of difficulty. What is clear is that error is inherent in the process of scientific experimentation, and that the foundation of scientific method includes a recognition that error is inherent. Reports of statistical analysis of research results typically include many estimates of the error involved in the relationships estimated by the statistical techniques.

As for Mr. Lehrer's remarks about the mythical nature of causes, scientific method has long allowed explanatory variables that have no real existence (intelligence, for example, cannot be directly measured but only inferred from behaviour). Variables like this are called explanatory fictions. The reason they are allowed is that the point of science is to explain an effect, not to find out what its actual cause is. If a fictional variable can explain the effect where something tangible and real can't, so much the better. Furthermore, even a small improvement in accuracy of prediction will often produce large benefits. Obviously, something which improves accuracy only a small amount is unlikely to be a cause in any meaningful sense, but it can still play an important role in practice.

Complex systems often frustrate scientific research simply because there are so many potential effects to examine, not because scientists are naive about the nature of causes, which anyway they aren't looking for. Mr. Lehrer freely acknowledges that science has been spectacularly successful with some complex systems (the health of large populations, for example), so concluding that failures to be successful with others mean that science has failed to solve the problem of causation is not only questionable and hasty but irrelevant as well.

I am confident that the scientific research of 100 years from now will be superior to today's research. I am also confident that the reason for its superiority will not be that it has solved the problem of causation.

Website
Twitter

Research, cause, and effect © 2012, John FitzGerald

Why information overload is a myth

Everybody’s heard of information overload – a Google search I just did for information overload (in quotation marks) produced over 4 million results. In fact, though, it is data we are overloaded with, not information.

Information consists only of data that reduce uncertainty. A weather forecast is only informative if it predicts the weather accurately. If it doesn't predict the weather accurately, we could end up leaving our umbrellas at home on rainy days. Similarly, if we base corporate decisions on data that don’t predict the results we want to achieve, we could end up being embarrassed and out of pocket.

As the Schumpeter blog in the Economist said on December 31: “As communication grows ever easier, the important thing is detecting whispers of useful information in a howling hurricane of noise.” It’s that overload of noise we must fear.

How do you reduce an overload of noise?

• By not collecting data that are irrelevant to the decisions you make.
  
• By not collecting data that are nearly identical to informative data you already collect.
  
• By not collecting more data than you need.
  
• By not combining pieces of information in ways in ways which produce an uninformative total score (by weighting them, for example).

But how do you avoid doing these things? Chiefly by analysing your data with sound statistical methods. For example, you can estimate the relevance of data to a decision with methods like the correlation coefficient. You can use principal components analysis to find variables that are telling you the same story. You can use sampling theory to decide how much data you need to collect. You can use psychometric analysis to combine pieces of information into a single score effectively. The battle against uninformative data has not been won, but you can win that part of it that takes place in your office.

Website
Twitter

Why Information Overload is a Myth © 2012, John FitzGerald

Cognitive decline research: Questions the CBC didn't ask

Today's CBC news report (click here) of a study of cognitive decline is pretty standard science reporting. I'm sure that other news sources provided much the same story. Anyway, it confines itself to reporting the results the researchers reported, results which were fairly stated.

However, there is other information the CBC might have provided, but didn't. First, it doesn't provide a link to the study (or hadn't when I posted a comment asking for one). I, for one, was interested in learning what "a 3.6% decline in mental reasoning" was. Does a decline of that size have a serious effect on people's functioning?

So I looked for the link and found it (here). It's an open access article that can be downloaded free in a PDF. The article doesn't provide a quick answer to the question of how serious the declines observed are, but the authors do suggest that further attention might be paid to people whose declines are greater than the mean in the study. That suggests to me the mean declines are not that serious, although I readily admit I may be reading something into the authors' suggestion that isn't there.

What I also found, though, is that the researchers did not control for health. Since older people tend to be less healthy, were these cognitive declines due to changes in brain function or to the fatigue resulting from poor health? Information about medical risk factors was collected, but the article does not report that it was incorporated in the statistical analyses.

None of this is intended to question the adequacy of the research. Being able to carry on a rigorous study of over 10,000 people for 24 years is proof enough of the researchers' competence. What this is intended to question is the value of a news report that simply reports results without examining them. I'm sure that if the researchers had been asked about the relationship of the health information they collected to cognitive decline they could have explained it fully. I'm sure if they'd been asking questions like that the journalists would have enjoyed their jobs more, too.

Secrets of the truth cult!!

For much of their history human beings have taken part in rituals in which an authority informs other people of what is supposed to be the Truth. I call this the pulpit model of information. For centuries Europeans went to church and an authority got up in the pulpit and told them what to believe about the world (and other places).

This model was later adopted by the schools, no doubt because the schools were established by churches. Whatever the reason, schooling until recently consisted of listening to an authority tell you what to believe about the world (in universities, it still often consists of this). In school, though, you were even tested to make sure you’d learned the approved view of things.

In school you also acquire the idea that Truth is something that can be found on the printed page. Consequently we come to accept something that has been published as true, without verifying that it is.

It’s not surprising that we come to look on the truth as something that is dispensed by authorities. Consequently, we look around for people who look like authorities, and treat what they say as information. Furthermore, we treat the methods they use to come up with things to say as methods that can be used to define information. We are often wrong.

Given the track record of authorities (remember all those biological weapons that, according to authorities, Iraq was just itching to use against the West?), depending on them to tell us the truth is a questionable approach. Another problem with this approach is that there is considerable doubt as to whether we need to know the truth, anyway.

Here’s something that’s true: Churchill, Manitoba, is named for John Churchill, first governor of the Hudson’s Bay Company. That’s a fact. Despite being a fact, though, it doesn’t help me get served when I drop in to the local branch of his company.

Every day we are bombarded with truths. The newspaper tells us things like what the temperature was yesterday in Beijing and what celebrities have (or had) their birthdays today. I remember once reading in the paper that it was the late Alfred Hitchcock’s birthday and thinking “I can’t really send him a card, can I?”

Better than mere truth is information. Information is confused with many things that are not informative, though.

Facts, as we have just seen, are not necessarily informative. Unless I’ve made a bet about what the high temperature in Beijing was going to be, that fact cannot be said to inform me of anything.

Furthermore, many items of information are not factual. The idea of intelligence, for example, cannot be said to be a fact, since there is widespread disagreement about just what intelligence is. However, the concept of intelligence is informative because in speculating about it we discover useful things. We have even discovered some of the shortcomings of the idea of intelligence.

As we have also seen, authoritative statements are not necessarily informative. Another reason they're not necessarily informative is that they disagree with each other. In fact, many of them work according to decision models which encourage disagreement as a way of establishing crucial issues that need to be tested. Courts of English law, for example, require two or more highly trained professionals to argue for exactly opposite points.

People also often assume that a logically sound argument is informative. However, it need not be. We can reason as soundly as it’s possible to reason and still be wrong.

Deductive reasoning starts with a general premise or principle. It then applies that premise to a specific piece of evidence and draws a conclusion about that piece of evidence. For example, we might reason like this:

• All Canadians are British subjects. (general principle)
  
• John FitzGerald is a Canadian. (evidence)
  
• Therefore, John FitzGerald is a British subject. (conclusion)

Well, that conclusion is true. However, let’s suppose we reason like this:

• All Canadians have French first names.
  
• John FitzGerald’s first name is not French.
  
• Therefore, John FitzGerald is not a Canadian.

That conclusion is not true, although the reasoning is entirely sound. Since my first name is not French, the conclusion that I am not Canadian follows logically from the general principle that all Canadians have French first names. The problem, of course, is that the general principle is wrong. Consequently, all statements that follow logically from it are most likely to be wrong. That example is a bit artificial, but people draw sound conclusions from erroneous premises all the time.

For example, many people reasoned out thoroughly logical arguments that on January 1, 2000 the world would be thrown into chaos. I say their beliefs were serious because they acted on them. They stockpiled food, for example, they bought portable electric generators, and some even created fortified shelters to protect themselves from people who hadn’t stockpiled food or bought generators.

As we saw on January 1, 2000, though, the computers didn’t fail. Some of the premises in those thoroughly logical arguments had been unsound. Logic is a tool. Logic does not guarantee that your arguments will stand up any more than a hammer guarantees that the bookcase you build with it will stand up.

Information is often confused with consensus. The supposed existence of a consensus among scientists about global warming is supposed to imply that the consensus opinion is highly likely to be true. Well, a hundred years ago a consensus of scientists would have told you that other races were inferior to whites.

The issue of consensus about global warming seems to have been raised initially as a red herring. That is, people argued against taking action against global warming because there was no scientific consensus about what caused it.

However, consensus has nothing to do with it. At one time there was a scientific consensus that the sun revolved around the earth. That point seems to have escaped the people who are opposed to taking action against global warming, though. Now they complain that this consensus they considered so desirable is being forced on them.

What is informative about an idea is its ability to predict events. The chief value of consensus seems to be coming up with a plan that everyone, or at least everyone important, is willing to go along with. To me, that seems a lot like what lemmings do.

Information cannot be defined by its source. If an expert meteorologist says tomorrow will be sunny, clouds don’t decide to go somewhere else just because a respected source says they will. Information is defined by its effect. Information increases the probability that we will act in effective ways. If it never rains on days when the weather forecast calls for rain, you’re going to end up lugging around a useless umbrella. If it always rains on days your bunions hurt, though, your bunions are a mine of information.

The Truth Cult © 2007, John FitzGerald

Another dubious sports statistic

I believe it is against Canadian law for a televised hockey game to be completed without the announcer mentioning, somewhere amid his (sic) endless recitation of players' hometowns, that getting the first goal is all-important, since the team that gets the first goal wins such a high percentage of games.

This belief seems to have come from a study of all major league baseball games between 1966 and 1987 which found that 66% of the games were won by the team that scored first. That’s an interesting finding because in baseball the visiting team is more likely to score first (since it bats first). However, the home team was still more likely to win, so the importance of the first run was still questionable. In 1998 Tom Ruane published an article in which he showed that teams scoring the first run were less likely to win than teams who were the first to score each of the second through ninth runs. The first run, it seemed, was actually the least important run to score. How can that be?, you may be asking. How can a run associated with 66% of victories be unimportant?

The reason it’s unimportant is most likely that the winning team scores more runs than the losing team. Consequently, it’s more likely to score the first run. So even if scoring the first run has no effect on the chances of winning a game, the winning team is still more likely to score the first run.

To examine this possibility I chose data from another sport in which teams don’t alternate offensive and defensive sessions. I collected scores from 110 National Hockey League games played from November 30, 2006 to December 14, 2006. I included games settled by shootout, but gave no credit to the winning team for the goal awarded for the shootout. The team scoring the first goal won 70% of these games (77 of the 110). However, the winning team also scored 68% of the goals (439 of 649). So, if scoring the first goal did not improve a team’s chances of winning a game, you’d still expect the winning team to score the first goal in 68% of the games, or 75 games. The improvement here is all of two percentage points.

But is it an improvement? You can’t reasonably expect that teams scoring 68% of the goals will necessarily win exactly 68% of the games. Other factors have some effect on the outcome, so you’d expect them to win a number around 75. Fortunately, we can estimate the probability that:

• if scoring the first goal does not increase a team’s chances of winning and
  
• if winning teams score 68% of the goals then
• the team scoring the first goal will win 77 games.

That probability is 44%. Conventional standards of statistical signficance would reject the idea that the first goal is of any importance when the percentage is that high. However, arguing that the probability of the difference being real is still greater than 50% is entirely reasonable. But if we look at the difference that way, we still have to conclude that there is only a 56% chance that scoring the first goal increased the likelihood of winning a game, and that if it did increase the probability of winning a game, it increased it by only 2 percentage points (aka one chance in 50). Either way, that first goal doesn’t seem all that important.

I propose an alternative to the Law of the All-Important First Goal/Run. I modestly call it FitzGerald's Law: the first team to score the winning goal will win. My law has as much explanatory value as the Law of the Fatal First Goal/Run, but is logically more elegant. It also reminds me of another statistical topic which baffles me: why, in a baseball game which finishes with a score of 11-10, can the player who drove in the first run for the winning team get credit for the game-winning RBI? Hm?

Another dubious sports statistic © 1995, 2006 John FitzGerald
More articles at ActualAnalysis.com

Better living through multiple linear regression analysis

I probably say somewhere on the main ste that multiple regression analysis is overused, and indeed it is. Nevertheless, it does have valuable uses which I don't want to frighten people away from, so here's an article about one of them.

I regularly use regression analysis to clarify for a client the factors affecting satisfaction with training and rehabilitation programs the client offers. It started with a review of a program about which the client knew rhat the more enthusiastic about the program consumers were on entry, the more satisfied they were at the end. The question was whether final satisfaction or dissatisfaction with the programs was simply a self-fulfilling prophecy – did consumers say they were satisfied or dissatisfied with the programs simply to justify their initial attitudes?

The client also collected information about consumers' opinions of various characteristics of their programs. This information was not correlated with initial attitude, nor were different types of this information correlated with each other. It was therefore easy, using multiple linear regression analysis, to estimate what proportion of final satisfaction could be explained by initial attitude toward the programs, and then see if characteristics of the programs explained the remainder of the final satisfaction (the residual, as it's known in regression analysis). It turned out that characteristics of the programs were twice as important as initial attitude in determining satisfaction with the programs.

So not only did multiple linear regression analysis determine that satisfaction with the programs was not a self-fulfilling prophecy, it also estimated the relative importance of initial attitude and of the actual characteristics of the programs. The analysis was made easier by the lack of correlation between the different types of information collected, but correlated information can be analyzed with more complicated designs. The possible existence of correlation, though, is the chief reason you shouldn't try this at home. Statistical and database software make it easy to do multiple linear regression analysis, but if you don't know how to deal with correlated variables or how to identify outliers (extreme observations which distort the results), you'll often get the wrong results when you use that software.

We have since gone on to use this technique to determine whether what consumers say are the important factors in determining their satisfaction are in fact the most important. We have frequently found that a simple count of the most popular explanations is contradicted by the multiple linear regression analysis. This is not surprising, since counting explanations, even if they are valid, gives us only a very rough estimate of the importance of different factors. The multile linear regression analysis clarifies the issue.

Of course, it is also important that you use a proper hypothesis-testing design. Just turning multiple linear regression loose on a set of data is almost certain to produce a large proportion of unhelpful or misleading results.

Better Living through Multiple Linear Regression Analysis © 1999, 2011 John FitzGerald

The myth of information technology

The term information technology implies to many people that the technology to which it refers creates information, transmits it, or stores it. The technologies we group together as information technology, however, rarely perform any of these functions. They are called information technology because they use information, not because they transmit it. A cellphone, for example, converts coded electrical signals into a facsimile of a person speaking. What the person is saying, though, may be balderdash.

Information consists of data which reduce uncertainty. The technology which we refer to as information technology is blithely unaware of whether the data it deals with reduce uncertainty or not.

The data provided by "information technology" may not be informative simply because they are irrelevant. For example, if I go looking for the box score of a particular baseball game in the newspaper, the other box scores, informative as they are, simply make it more difficult for me to find the one I'm interested in. These days, though, people use their information technology to collect large amounts of information which are of no relevance to the decision they're going to make.

Then again the data may not be informative because they are not intelligible. While Turkish newspapers are informative to Turks, they are not informative to me, because I don't speak Turkish. I deal with this problem by not subscribing to Turkish newspapers. However, people often use their information technology to collect large amounts of data which they can no more interpret than I can interpret Turkish newspapers. Turning data mining software loose on the data is not guaranteed to turn it into information, either, for reasons which are discussed in other articles on the main site.

The fact is that we make information, not technology. Even those rare items of software which perform analytical functions were created by human minds. Most of what we call information technology is actually nothing more than data technology. It gives us the capacity to collect large masses of data, but it is up to us to find or define the information in it.

Few people believe everything they read in the newspaper or see in the television. Few believe that every item that appears in the newspaper or on the television is relevant to their concerns. It's time for the same discernment to be shown in dealing with databases.

We hear a lot these days about the problem of information overload. In fact, it is data we are overloaded with, not information. If we set out to collect data, we will drown in data. If we condescend, though, to use our analytical abilities, and set off in search of the data that we need, we will find that you can never be overloaded with information.