or How Semantics Constrained a Field

Our discipline is stricken with too many values. I’m speaking semantically, of course. We use the word “values” to mean many things, including personal values, cultural and organizational values, criteria (or dimensions of merit), general and specific values (in terms of standards), monetary value (or worth), and in the action form as valuing (or judging). A conflation of terms hinders the discipline’s ability to be accessible others, particularly our clients and stakeholders, and unnecessarily confuses beginning evaluators, and perhaps even those who are more experienced. Although Scriven’s (1991) Evaluation Thesaurus defines several of these, the text itself is a hurdle for an evaluation newcomer and hence not serving the purpose I aim for here: clarity.

Given the fact that well-published and experienced evaluators are confusing their uses of the word “value” or some derivation of it, it is sufficient to say that additional clarifying language may bring focus to our work and at least help us understand each other a bit better. At least six different meanings of the word “values” (or some derivative of it) appear throughout our field and our lives.

Table 1. Values Taxonomy

Without specifying the type of values to which we are referring we end up making (or not) sense that sounds like this:

Evaluation: The valuing (#6) of value (#5) based on values (#3), determined by values (#4), underscored by values (#2) and influenced by many values (#1).

So when we ask a client to discuss with us the values of the evaluation, which type are we referring to? More importantly, which type do the clients think we are referring to? When we use the term “value judgment” with a group of stakeholders, it is quite possible to have multiple interpretations.

Let’s look at a specific example of how our vocabulary has caused confusion for the field of evaluation, and hence for our stakeholders:

As Scriven (2007) notes in the Values subcomponent of the Key Evaluation Checklist, many sources of values should be taken into consideration when making a criteria (#3) list. While forms #1 and #2 should be part and parcel of this list, so too must they be scrutinized with other sources, such as legal considerations. In this way, values types #1 and #2 become part of the evaluation (#6). However, contrary to the assertion of Schwandt (1997), House and others (cf. SenGupta, Hopson & Thompson-Robinson [2004]), the act of valuing #6, does not create values type #1. At best, and in a long-reaching circumstance, evaluators may cause people to reflect or rethink their personal values (#1) but this seems more to be an overstatement of the impact of evaluation. Value type #2 also doesn’t stand the logical test of being determined by value type #6. While organizations may change practices as a result of evaluations, organizational values, like cultural and personal values (#2 and #1) are deep-seated, underpinning day-to-day actions and well as overall visions and missions. In essence, it appears that an oversimplification of the values taxonomy has occurred. However, it is also possible that the authors speak to a more specific variation of values that is not made clear in their writings.

The main mistake involved in this conflation of terms seems to be more due to an overstatement of evaluation impact and an underestimation of the sources of our values (#3). Let’s look at Schwandt’s (1997) phrasing: “Assessments of value [#6] cannot help but entail making claims about what ought to be done or avoided or what is right to do… Through making such interpretations, evaluators not only inform the means by which human or social good is realized but shape our definition of the social good as well” (p. 26). Essentially, Schwandt is claiming that our criteria of merit (#3) has the potential to shape social views of good, bad, and ugly. Yet it is clear through the research in the field and reflection on individual practice that criteria are not created in a vacuum. Rather, evaluators pull together criteria from existing social norms, even if from varying sources. So rather than create new definitions of social good, we are simply reflecting what society (or some subcategorized portion thereof) has said is good.  This is precisely why we consider values (#2) when making criteria. (Still, if we take organizational values to be our criteria, we are essentially making the same mistake as many goal-based evaluations, which do not question the appropriateness of the organization’s goals prior to making determinations about the merit, worth, or significance of the organization’s efforts to achieve those goals.)

It is evident from this illustration that even when an evaluator-author holds two separate meanings in her head, when using the same word to represent those meanings, confusion can easily occur for readers, evaluation students, clients, and perhaps even the evaluator-author herself. (The same can be said for our double meaning for the word “standards” – those of the Joint Committee origin and those on which performance is judged [i.e. value type #4] can be confusing when we take a moment to educate stakeholders on what it is we do. But that’s another blog post.)

Instead of picking away at the logic of scholars, let us focus on where we can improve. The larger point for us to address is that we are struggling to be understood and it has been a self-defeating act.  As a first cut at clarity, I propose in Table 2 a sort of glossary from which we can adopt simpler, more appropriate linguistic choices and better express the logistics of our work together as evaluators.

Table 2. Revised Values Taxonomy

Perhaps unsurprisingly, there has been little manipulation of the “old” value-related terms in creating new ones. Rather, the idea is that we use other established names when we have them, and in places where we don’t, we use the appropriate adjectives to describe ourselves and our work, with the explicit purpose of reducing confusion in the field. To restate the earlier example:

Evaluation: The judging of monetary worth [among other things] based on criteria, determined by standards, underscored by cultural and organizational values and influenced by many personal values.

Ah, now don’t we all feel much better?

[ii] Stufflebeam (2007) suggests the use of widely-accepted cultural values, such as those stemming from the United Nations. He discusses this in his description of the CIPP model in Evaluation Theory, Models, and Applications, page 331.

[iii] Scriven (2007) outlines the sources of values quite well in his Key Evaluation Checklist, pages 6-10, though it would have served the field better to consistently call it “criteria of merit” or just “criteria.”

[iv] Referring to standards as specific values is becoming more common. Coryn (2007) delineates this in his dissertation, Evaluation of Researchers and Their Research, page 43.

[v] Interchanging “monetary worth” and “value” is common, but especially included in evaluation through Scriven’s Evaluation Thesaurus, pages 382-383.

[vi] For more explication, see Scriven’s Evaluation Thesaurus, page 375.