Transparent evaluation

One of the ways I earn my living is by evaluating service programs. People are often wary of program evaluation, since evaluation is a word with many meanings, many of these meanings negative. In research, though, evaluation has a very simple and neutral meaning. It is simply determining whether an event of interest has happened.

Program evaluation, therefore, is simply a matter of determining whether a program is doing the things that it is supposed to be doing. I prefer to look on it, in fact, as giving a program a chance to show what it can do. So, if you're going to show what a program can do, how do you go about it?

First of all you need a plan – a statement of what the program is supposed to be doing. If you're evaluating a program for the first time, the first step is likely to be the development of a program logic model. A program logic model is simply a description of the steps in the program and the decisions made once steps are completed.

Once you have the program logic model, you then determine if the program is following the model. Obviously, to do that you need records. A crucial part of any service program is a system of records which provides:

• descriptions of the services being provided to each consumer,
  
• descriptions of the goals which these services are to help
  the consumer achieve,
  
• measures of the extent to which the goals have been achieved,
  and
  
• descriptions of the decisions made as a result of the
  achievement or non-achievement of goals.

Obviously judicious examination of records like those is going to help you determine if the program logic model is being followed. If the program is not implementing the plan fully, then you can take steps to improve its chances of doing so.

The records system will also permit a thoroughgoing outcome evaluation. Accurate estimates of the program's success in achieving its ultimate goals can easily be calculated.

Furthermore, a good system of records will enable program staff or anyone else to perform the outcome evaluation by themselves. When you require an external evaluation, for example, you won't have to pay your independent consultant to develop an evaluation from the ground up. The evaluative standards will be set, and the evidence will be collected. Your consultant can spend time doing something more sophisticated and effective, such as studying specific aspects of the program that you consider important.

In short, the goal of program evaluation is to make the program transparent. If program evaluation is successful, there will be general agreement about what the goals of the program are, about the ways in which the program should be trying to achieve these goals, and about what the world should look like if the program is successful. There will also be clear standards by which anyone can reliably measure the degree of success achieved by the program. That also makes evaluation more bearable for staff, since they don't have to worry about their work being evaluated by standards of which they have not been informed.

Doing all this can be a lot of work. However, the benefits are enormous, and you need spend no more money, in either the short or long term, than you could end up spending on less productive approaches.

Transparent Evaluation © 2002, John FitzGerald
From ActualAnalysis.com

Now on Twitter

You can now follow me on Twitter: just click here.

DIY Guide

I have expanded the current newsletter into a do-it-yourself guide for tuning up your rating systems; just click here.

Actual Analysis website

Average vs. average

I have run across people who, when calculating a mean, will discard their two or three highest and two or three lowest pieces of data and calculate the mean for the rest of their data. What they want to do is protect themselves against the effects of skew, specifically the distortion of a mean by a few extreme scores.

That probably doesn't hurt, but there is a simpler and much more effective way of dealing with this problem – use the median. The median is the score that is midway between the highest and the lowest. In other words it is the true average of your set of data (the mean is an estimate of the median). So use the MEDIAN function in your spreadsheet rather than the MEAN function.

There are some exceptions to this rule, though. If you're using your data to estimate a total – the total value of donations to an organization, for example – you'd use the mean. If you want to compare two sets of data with a statistical test you would usually be better off to use the mean.

And if the SKEWNESS function in your spreadsheet provides a skewness coefficient for your set of data that is higher than -1.00 and less than 1.00 you normally don't worry about this at all.

Current newsletter

For my current newsletter click here.

Uninformation (4)

Information is not identical with experience

We often assume that because someone is experienced in a field that they are therefore well informed about it. One is particularly likely to believe this if the person involved is oneself. One might as well argue that because I take the streetcar every day that I am an expert on public transportation, or that because I watch television every day I'm an expert on television. Obviously you acquire some knowledge from your experience, but it does not necessarily constitute an understanding of your experience.

And we may simply fail to learn from our experience. Psychologists talk about the consulting room phenomenon — faced with evidence that a diagnostic test such as the Rorschach test doesn't work the way it's supposed to, some psychologists and psychiatrists will reply that they've seen it work in their consulting rooms. In fact, individual practitioners have little opportunity to establish in their practice that a test actually works. The chief criterion they can use is the success of treatment, and even a correct diagnosis may lead to unsuccessful treatment, while an incorrect one may lead to successful treatment. We can also sometimes be a little lenient in deciding how successful we’ve been.

We have seen how authorities — people with great experience in their fields — usually disagree with each other. That is, their experience has led them to contradictory conclusions, and those conclusions cannot all be informative.

We derive information from our experience — we don't just pick it up by accident. We derive it by analyzing our experience in certain ways, acting on the conclusions we’ve drawn from our analysis, and then testing the adequacy of our conclusions.

First article in the Uninformation series

Actual Analysis
Uninformation (4) © 2011, John FitzGerald

Uninformation (3)

The opinions of authorities are not necessarily informative

We often treat anything printed in an authoritative journal or asserted by an expert to be informative. Although authorities and experts do tend to be far better informed about their subjects than the average person, we still cannot assume that whatever they say is informative or even true . All you have to do to learn why we have no justification is to read what authoritative foreign journals and experts have to say about your own country. The influential journal Le monde diplomatique once published an article whose author claimed that Canada had no constitution, but rather “a collection of texts with the force of a constitution”, and that these onstitutional texts could not be challenged in lower courts . Well, the latest of this collection of texts explicitly defines it as the national constitution, and it explicitly gives all courts the power to review all matters of law, which of course includes the constitution.

Our lives are rife today with experts and expert opinions. The news media are constantly presenting experts and their opinions about every topic under the sun, the implication being that an expert=s opinion is more informative than the opinion of someone who is not an expert..

For an assertion to be informative to us, though, we have to have some idea of the likelihood that it’s true. If the expert is an expert on gardening or cooking, verifying the accuracy of what he or she says is fairly easy. If, however, the expert is an expert on politics or medicine or some other field which requires special or complicated knowledge which you do not have, you may well have no way of verifying his or her opinion. A few years ago we saw experts queuing up to predict that the stock market would rise, if not forever, at least for a long, long time to come. Certainly these experts made arguments for their positions, but usually they were adducing as evidence for their opinion facts which the ordinary person could not verify.

Another problem about expert forecasts is that the experts are rarely experts in forecasting. J. Scott Armstrong and Kesten Green have observed that the scientific forecasts we are often encouraged to believe in are too often forecasts by scientists rather than forecasts arrived at scientifically.

Another problem is that experts are not impersonal compendia of information but human beings who advocate certain disputed positions in their field. They are advocates for ideas which other experts in their fields dispute. The Western intellectual tradition is to have as many people as possible arguing about ideas. Many of these ideas have the same quality that ideas about what was going to happen on January 1, 2000 had – they are founded on data which are not fully understood.

We can hardly expect experts to be perfect. If we cannot expect them to be perfect, then we have to assess the soundness of their opinions. If we are unable to assess the soundness of their opinions, then their opinions are not informative to us. They may well be valid, but if we cannot verify that they are valid then they are not informative. At the same time as all those experts were predicting that the stock market would rise forever, some experts were predicting that the bubble was going to burst. Those experts were right, but most of us had no way of verifying that they were. Therefore, even though they were right, they were not providing us with information.

First article in the uninformation series

Next: Information is not identical with experience

Website
Uninformation (3) © 2011, John FitzGerald