How the British Geological Survey overcame bad data management - after 165 years

I have gone on at length in this blog and on its parent website about how data aren't informative until they're organized in some useful way. The first step in organizing them is making them accessible - putting them in a database, giving them a unique identifier, and organizing them so that you can use them in the way you intended.

In 1846, John Hooker, a botanist, collected 314 slides of botanical samples for the British Geological Survey. Then he had to rush off on a trip to the Himalayas and didn't get around to entering the samples in the specimen register. In April, 2011, Howard Falcon-Lang, a paleontologist, was poking around in a cabinet in a dark corner of the BGS and found the drawers of Hooker`s slides. He pulled one out, shone his flashlight on it, and read the label "C. Darwin, Esq." (Click here for a news report)

It turned out that Hooker's slides were from Darwin's expedition on the Beagle, and that Dr. Falcon-Lang was apparently the first person in 165 years to recognize what they were. Dr. Falcon-Lang expects that examination of the the samples will contribute to contemporary science. Imagine what contemporary science would be like, though, if these examples had been examined in the 1840s and 50s.

The data most of us use aren't likely to be as significant as Darwin's, but if we can't use them they're as useless as Darwin's were for 165 years. A big problem I have run into with databases is that some data just don't get entered. People entering data omit fields they consider unimportant or too difficult to collect. Often this ends up producing huge amounts of missing data, especially when data are being entered from the field by several people, and if huge quantities of data are missing the data are useless. If you want to use the data, ensure that a complete record must be entered. If you don't use the data, don't collect them. If you don't collect unnecessary data you'll probably make fewer errors in entering the necessary data.

And don't enter summaries of data. For example, enter people's exact ages, not an age range. If you use pre-defined age ranges you may end up with all the ages clumped in one or two categories, which severely limits the analysis you can do. If you enter the exact age, you can define age ranges whose categories have roughly equal numbers of people in them, which makes it easier to find differences between the categories (click here, here, and here for more about these issues.

Similarly, instead of a test score or a rating, enter the individual test and rating items. First of all, that makes it easier to clean the data - to find erroneously recorded items or scores. More importantly, it gives you the ability to assess the adequacy of the test or rating (a PDF you can download from my website describes some things you can do with ratings; click here for the PDF).

And enter the data in a format appropriate for the type of analysis you want to do. Most statistical packages for example, want records entered as rows.

Data will only talk to you if you care for them. Be nice to your data. Only collect the ones you need, and treat the ones you need with respect.

Why information overload is a myth

Everybody’s heard of information overload – a Google search I just did for information overload (in quotation marks) produced over 4 million results. In fact, though, it is data we are overloaded with, not information.

Information consists only of data that reduce uncertainty. A weather forecast is only informative if it predicts the weather accurately. If it doesn't predict the weather accurately, we could end up leaving our umbrellas at home on rainy days. Similarly, if we base corporate decisions on data that don’t predict the results we want to achieve, we could end up being embarrassed and out of pocket.

As the Schumpeter blog in the Economist said on December 31: “As communication grows ever easier, the important thing is detecting whispers of useful information in a howling hurricane of noise.” It’s that overload of noise we must fear.

How do you reduce an overload of noise?

• By not collecting data that are irrelevant to the decisions you make.
  
• By not collecting data that are nearly identical to informative data you already collect.
  
• By not collecting more data than you need.
  
• By not combining pieces of information in ways in ways which produce an uninformative total score (by weighting them, for example).

But how do you avoid doing these things? Chiefly by analysing your data with sound statistical methods. For example, you can estimate the relevance of data to a decision with methods like the correlation coefficient. You can use principal components analysis to find variables that are telling you the same story. You can use sampling theory to decide how much data you need to collect. You can use psychometric analysis to combine pieces of information into a single score effectively. The battle against uninformative data has not been won, but you can win that part of it that takes place in your office.

Website
Twitter

Why Information Overload is a Myth © 2012, John FitzGerald

Secrets of the truth cult!!

For much of their history human beings have taken part in rituals in which an authority informs other people of what is supposed to be the Truth. I call this the pulpit model of information. For centuries Europeans went to church and an authority got up in the pulpit and told them what to believe about the world (and other places).

This model was later adopted by the schools, no doubt because the schools were established by churches. Whatever the reason, schooling until recently consisted of listening to an authority tell you what to believe about the world (in universities, it still often consists of this). In school, though, you were even tested to make sure you’d learned the approved view of things.

In school you also acquire the idea that Truth is something that can be found on the printed page. Consequently we come to accept something that has been published as true, without verifying that it is.

It’s not surprising that we come to look on the truth as something that is dispensed by authorities. Consequently, we look around for people who look like authorities, and treat what they say as information. Furthermore, we treat the methods they use to come up with things to say as methods that can be used to define information. We are often wrong.

Given the track record of authorities (remember all those biological weapons that, according to authorities, Iraq was just itching to use against the West?), depending on them to tell us the truth is a questionable approach. Another problem with this approach is that there is considerable doubt as to whether we need to know the truth, anyway.

Here’s something that’s true: Churchill, Manitoba, is named for John Churchill, first governor of the Hudson’s Bay Company. That’s a fact. Despite being a fact, though, it doesn’t help me get served when I drop in to the local branch of his company.

Every day we are bombarded with truths. The newspaper tells us things like what the temperature was yesterday in Beijing and what celebrities have (or had) their birthdays today. I remember once reading in the paper that it was the late Alfred Hitchcock’s birthday and thinking “I can’t really send him a card, can I?”

Better than mere truth is information. Information is confused with many things that are not informative, though.

Facts, as we have just seen, are not necessarily informative. Unless I’ve made a bet about what the high temperature in Beijing was going to be, that fact cannot be said to inform me of anything.

Furthermore, many items of information are not factual. The idea of intelligence, for example, cannot be said to be a fact, since there is widespread disagreement about just what intelligence is. However, the concept of intelligence is informative because in speculating about it we discover useful things. We have even discovered some of the shortcomings of the idea of intelligence.

As we have also seen, authoritative statements are not necessarily informative. Another reason they're not necessarily informative is that they disagree with each other. In fact, many of them work according to decision models which encourage disagreement as a way of establishing crucial issues that need to be tested. Courts of English law, for example, require two or more highly trained professionals to argue for exactly opposite points.

People also often assume that a logically sound argument is informative. However, it need not be. We can reason as soundly as it’s possible to reason and still be wrong.

Deductive reasoning starts with a general premise or principle. It then applies that premise to a specific piece of evidence and draws a conclusion about that piece of evidence. For example, we might reason like this:

• All Canadians are British subjects. (general principle)
  
• John FitzGerald is a Canadian. (evidence)
  
• Therefore, John FitzGerald is a British subject. (conclusion)

Well, that conclusion is true. However, let’s suppose we reason like this:

• All Canadians have French first names.
  
• John FitzGerald’s first name is not French.
  
• Therefore, John FitzGerald is not a Canadian.

That conclusion is not true, although the reasoning is entirely sound. Since my first name is not French, the conclusion that I am not Canadian follows logically from the general principle that all Canadians have French first names. The problem, of course, is that the general principle is wrong. Consequently, all statements that follow logically from it are most likely to be wrong. That example is a bit artificial, but people draw sound conclusions from erroneous premises all the time.

For example, many people reasoned out thoroughly logical arguments that on January 1, 2000 the world would be thrown into chaos. I say their beliefs were serious because they acted on them. They stockpiled food, for example, they bought portable electric generators, and some even created fortified shelters to protect themselves from people who hadn’t stockpiled food or bought generators.

As we saw on January 1, 2000, though, the computers didn’t fail. Some of the premises in those thoroughly logical arguments had been unsound. Logic is a tool. Logic does not guarantee that your arguments will stand up any more than a hammer guarantees that the bookcase you build with it will stand up.

Information is often confused with consensus. The supposed existence of a consensus among scientists about global warming is supposed to imply that the consensus opinion is highly likely to be true. Well, a hundred years ago a consensus of scientists would have told you that other races were inferior to whites.

The issue of consensus about global warming seems to have been raised initially as a red herring. That is, people argued against taking action against global warming because there was no scientific consensus about what caused it.

However, consensus has nothing to do with it. At one time there was a scientific consensus that the sun revolved around the earth. That point seems to have escaped the people who are opposed to taking action against global warming, though. Now they complain that this consensus they considered so desirable is being forced on them.

What is informative about an idea is its ability to predict events. The chief value of consensus seems to be coming up with a plan that everyone, or at least everyone important, is willing to go along with. To me, that seems a lot like what lemmings do.

Information cannot be defined by its source. If an expert meteorologist says tomorrow will be sunny, clouds don’t decide to go somewhere else just because a respected source says they will. Information is defined by its effect. Information increases the probability that we will act in effective ways. If it never rains on days when the weather forecast calls for rain, you’re going to end up lugging around a useless umbrella. If it always rains on days your bunions hurt, though, your bunions are a mine of information.

The Truth Cult © 2007, John FitzGerald

The myth of information technology

The term information technology implies to many people that the technology to which it refers creates information, transmits it, or stores it. The technologies we group together as information technology, however, rarely perform any of these functions. They are called information technology because they use information, not because they transmit it. A cellphone, for example, converts coded electrical signals into a facsimile of a person speaking. What the person is saying, though, may be balderdash.

Information consists of data which reduce uncertainty. The technology which we refer to as information technology is blithely unaware of whether the data it deals with reduce uncertainty or not.

The data provided by "information technology" may not be informative simply because they are irrelevant. For example, if I go looking for the box score of a particular baseball game in the newspaper, the other box scores, informative as they are, simply make it more difficult for me to find the one I'm interested in. These days, though, people use their information technology to collect large amounts of information which are of no relevance to the decision they're going to make.

Then again the data may not be informative because they are not intelligible. While Turkish newspapers are informative to Turks, they are not informative to me, because I don't speak Turkish. I deal with this problem by not subscribing to Turkish newspapers. However, people often use their information technology to collect large amounts of data which they can no more interpret than I can interpret Turkish newspapers. Turning data mining software loose on the data is not guaranteed to turn it into information, either, for reasons which are discussed in other articles on the main site.

The fact is that we make information, not technology. Even those rare items of software which perform analytical functions were created by human minds. Most of what we call information technology is actually nothing more than data technology. It gives us the capacity to collect large masses of data, but it is up to us to find or define the information in it.

Few people believe everything they read in the newspaper or see in the television. Few believe that every item that appears in the newspaper or on the television is relevant to their concerns. It's time for the same discernment to be shown in dealing with databases.

We hear a lot these days about the problem of information overload. In fact, it is data we are overloaded with, not information. If we set out to collect data, we will drown in data. If we condescend, though, to use our analytical abilities, and set off in search of the data that we need, we will find that you can never be overloaded with information.

Uninformation (4)

Information is not identical with experience

We often assume that because someone is experienced in a field that they are therefore well informed about it. One is particularly likely to believe this if the person involved is oneself. One might as well argue that because I take the streetcar every day that I am an expert on public transportation, or that because I watch television every day I'm an expert on television. Obviously you acquire some knowledge from your experience, but it does not necessarily constitute an understanding of your experience.

And we may simply fail to learn from our experience. Psychologists talk about the consulting room phenomenon — faced with evidence that a diagnostic test such as the Rorschach test doesn't work the way it's supposed to, some psychologists and psychiatrists will reply that they've seen it work in their consulting rooms. In fact, individual practitioners have little opportunity to establish in their practice that a test actually works. The chief criterion they can use is the success of treatment, and even a correct diagnosis may lead to unsuccessful treatment, while an incorrect one may lead to successful treatment. We can also sometimes be a little lenient in deciding how successful we’ve been.

We have seen how authorities — people with great experience in their fields — usually disagree with each other. That is, their experience has led them to contradictory conclusions, and those conclusions cannot all be informative.

We derive information from our experience — we don't just pick it up by accident. We derive it by analyzing our experience in certain ways, acting on the conclusions we’ve drawn from our analysis, and then testing the adequacy of our conclusions.

First article in the Uninformation series

Actual Analysis
Uninformation (4) © 2011, John FitzGerald

Uninformation (3)

The opinions of authorities are not necessarily informative

We often treat anything printed in an authoritative journal or asserted by an expert to be informative. Although authorities and experts do tend to be far better informed about their subjects than the average person, we still cannot assume that whatever they say is informative or even true . All you have to do to learn why we have no justification is to read what authoritative foreign journals and experts have to say about your own country. The influential journal Le monde diplomatique once published an article whose author claimed that Canada had no constitution, but rather “a collection of texts with the force of a constitution”, and that these onstitutional texts could not be challenged in lower courts . Well, the latest of this collection of texts explicitly defines it as the national constitution, and it explicitly gives all courts the power to review all matters of law, which of course includes the constitution.

Our lives are rife today with experts and expert opinions. The news media are constantly presenting experts and their opinions about every topic under the sun, the implication being that an expert=s opinion is more informative than the opinion of someone who is not an expert..

For an assertion to be informative to us, though, we have to have some idea of the likelihood that it’s true. If the expert is an expert on gardening or cooking, verifying the accuracy of what he or she says is fairly easy. If, however, the expert is an expert on politics or medicine or some other field which requires special or complicated knowledge which you do not have, you may well have no way of verifying his or her opinion. A few years ago we saw experts queuing up to predict that the stock market would rise, if not forever, at least for a long, long time to come. Certainly these experts made arguments for their positions, but usually they were adducing as evidence for their opinion facts which the ordinary person could not verify.

Another problem about expert forecasts is that the experts are rarely experts in forecasting. J. Scott Armstrong and Kesten Green have observed that the scientific forecasts we are often encouraged to believe in are too often forecasts by scientists rather than forecasts arrived at scientifically.

Another problem is that experts are not impersonal compendia of information but human beings who advocate certain disputed positions in their field. They are advocates for ideas which other experts in their fields dispute. The Western intellectual tradition is to have as many people as possible arguing about ideas. Many of these ideas have the same quality that ideas about what was going to happen on January 1, 2000 had – they are founded on data which are not fully understood.

We can hardly expect experts to be perfect. If we cannot expect them to be perfect, then we have to assess the soundness of their opinions. If we are unable to assess the soundness of their opinions, then their opinions are not informative to us. They may well be valid, but if we cannot verify that they are valid then they are not informative. At the same time as all those experts were predicting that the stock market would rise forever, some experts were predicting that the bubble was going to burst. Those experts were right, but most of us had no way of verifying that they were. Therefore, even though they were right, they were not providing us with information.

First article in the uninformation series

Next: Information is not identical with experience

Website
Uninformation (3) © 2011, John FitzGerald

Uninformation (2)

2. The logical or reasonable is not necessarily informative

People often believe that if they can construct a chain of reasoning which supports their beliefs that therefore they have demonstrated that their belief is true and informative. For example, many people reasoned out arguments which they seriously believed demonstrated that on January 1, 2000 the world would be thrown into chaos. I say their beliefs were serious because they acted on them – they stockpiled food, for example, they bought portable electric generators, and some even created fortified shelters to protect themselves from people who hadn't stockpiled food or bought generators.

As we found out, they were wrong. However, I can’t say that their conclusion was any less sound than the conclusion I and most other people drew that any disruption that might occur on January 1, 2000 would be minor. The people who drew this conclusion were sane and their reasoning from their data was sound. It was probably as sound or sounder than my own. In the end, one reason I and most other people were right and they were wrong is that we were using better data – data which were more informative. Another reason is that we were just luckier. In fact, no one fully understood all the factors one would have to assess to produce an accurate forecast of what would happen to the power grid on January 1, 2000. Furthermore, we probably weren’t aware of all the factors that would have to be considered.

If sound reasoning is based on invalid and inadequate data, it will reach invalid and inadequate conclusions. None of us is perfect – not even, as unlikely as it may seem, you or I – and we all at one time or another base logical conclusions on unsound data. And sometimes our reasoning just slips a gear, too. Even if our reasoning is perfect, none of us is omniscient, either. We can easily overlook important considerations.

That’s why the betting industry exists. If you’ve ever heard some of the explanations – often vehement ones – which horseplayers come up with to explain why the sure thing they bet in the last race ran as if he was pulling a milk wagon, you’ll know that relying too much on reason can not only cost you money but also lead you into an unjustified skepticism about the honesty and competence of one’s fellow human beings.

Obviously logic is involved in the development of information, just as facts are involved. It is not by itself informative, though. Two plus two equals four, but if the right answer is five you’re still wrong. That is why conclusions drawn from data need to be tested before they can be accepted as sound. If you think the 5-horse in the next race is going to romp, you won’t know that you’re right till the race has been run. And no matter what the weather report says, you won’t know whether it’s going to rain tomorrow or not until tomorrow arrives.

First article in the Uninformation series

Next: Information is not the statement of an authority

Actual Analysis
Uninformation (2) © 2011, John FitzGerald

Uninformation (1)

Information consists of data which establish whether or not an assertion is false. Not all data do this. One of the reasons we have difficulty becoming and staying informed is that we sometimes accept as informative things which really aren’t, or least aren’t necessarily. This is the first in a series of posts in which we’ll look at a few things which are not information.

1. Information is not synonymous with facts

People often confuse information with facts. Someone who knows a lot of facts is considered to be well informed. A fact is only informative, though, if it helps you settle a question you need to know the answer to. If someone is on trial for armed robbery, the Crown does not submit evidence that the defendant is a skilled bridge player, true as that evidence may be.

Here's a fact: Churchill, Manitoba, is named for John Churchill, first governor of the Hudson's Bay Company. That=s a fact. Despite being a fact, though, it doesn't help me answer the question “Where do I find the men’s shirts?” whenever I drop in to one of the Bay’s branches. So for me that datum is not informative, factual though it be.

Furthermore, there are plenty of items of information that are not factual. The idea of intelligence, for example, cannot be said to be a fact, since there is widespread disagreement about just what intelligence is. However, the concept of intelligence is informative because in speculating about it we discover useful things. We have even discovered some of the shortcomings of the idea of intelligence.

Information is always derived from facts, and it always helps to predict facts. However, it need not be factual itself, and something which is factual need not be informative. As someone who has spent his life filling his memory with facts whose relevance to my life is highly questionable (see note about John Churchill above), I realize that collecting trivia can be enjoyable. Until they tell you something useful, though, trivia are just trivial.

Next: The logical or reasonable is not necessarily informative

Actual Analysis
Uninformation (1) © 2011, John FitzGerald

Accuracy is not enough

Data are not necessarily information. They are informative only to the extent that they reduce uncertainty. If you want to know what programs are on television tonight, knowing yesterday's television schedule will not help you. Yesterday's schedule is full of data, but the data are no longer informative.

In psychometric terms, informative data are those which are valid – which predict events of interest to you. To be valid data must be accurate; in fact, the validity of information is limited by its accuracy. Of course, inaccurate data cannot be valid, and the maximum possible validity of accurate data is equal to the square root of its reliability coefficient.

The minimum validity of accurate data, however, is always zero. Sometimes data are not valid simply because they are distributed in a way ill-suited to the statistics which are used to assess validity; often the distribution can be modified through a mathematical transformation and validity restored. Sometimes the data are simply irrelevant or poorly defined.

In Canada a federal election campaign is under way. As usual the press commentary about it includes frequent presentation of poll results. At the moment the poll results are the unverifiable opinions of the 30% of the population that takes part in polls about what they think they'll be doing a month from now. These people probably differ significantly from people who don't take part in polls. They've probably got more time on their hands for a start, which means they're likely older, better off, and so on. That is, they are probably not even accurate estimates of unverifiable opinions. If you check the excellent Polling Report website you'll find that American polls have been dependably incompetent at predicting the results of American presidential elections, which are simple two-candidate races. In Canada, with three national parties and a big regional party they are likely to be even less effective.

Anyway, if you depend on any type of database, it should be checked regularly to ensure not only accuracy but also relevance and utility.

Accuracy is not Enough © 2001, 2011 John FitzGerald

More articles from www.ActualAnalysis.com