or How Semantics Constrained a Field
Our discipline is stricken with too many values. I’m speaking semantically, of course. We use the word “values” to mean many things, including personal values, cultural and organizational values, criteria (or dimensions of merit), general and specific values (in terms of standards), monetary value (or worth), and in the action form as valuing (or judging). A conflation of terms hinders the discipline’s ability to be accessible others, particularly our clients and stakeholders, and unnecessarily confuses beginning evaluators, and perhaps even those who are more experienced. Although Scriven’s (1991) Evaluation Thesaurus defines several of these, the text itself is a hurdle for an evaluation newcomer and hence not serving the purpose I aim for here: clarity.
Given the fact that well-published and experienced evaluators are confusing their uses of the word “value” or some derivation of it, it is sufficient to say that additional clarifying language may bring focus to our work and at least help us understand each other a bit better. At least six different meanings of the word “values” (or some derivative of it) appear throughout our field and our lives.
Table 1. Values Taxonomy
| Value # | Term | Contextual Meaning | Use in Evaluation Literature |
1 | Personal values | Beliefs or morals. Core understandings and dispositions held by individuals. Informed by #2, but with individual variation. | To some, hardly mentioned but as a source of bias. To others, the basis from which the remaining types of values are decided.[i] |
2 | Cultural or organizational values | Central tenets that can characterize a mass of people. Variably held by the people in the mass. UN Statement on Human Rights might represent largest mass. Religious texts would be another. | When asked for details, some evaluators fall back on a widely-agreed upon set of values, like the UN’s work. Other skinny answers, like the values of the org, are used when criteria are needed, without examining the fit or questioning the values of the org.[ii] |
3 | Criteria (general values) | Those dimensions on which an evaluand will be judged, based on its performance. What makes a “good” evaluand of a certain type. | Often conflated with #1 and #2. While #1 and #2 may appear here but are not sufficient to determine a good evaluand of a certain type.[iii] |
4 | Standards (specific values) | The distinguishment between performance levels for an evaluand. The demarcation between “good,” “fair,” and “bad” levels of performance. Specific aspects of #3. | Often conflated with #3. Can be presented in a rubric or grading scheme to show how an evaluand will be judged (#6). Again, #1 and #2 appear here.[iv] |
5 | Monetary worth | The cost of something, given its benefits or opportunities it affords. | One of the three determinants of an evaluative conclusion. Tangentially related to #1-#4 and #6, but most distinct.[v] |
6 | Judging (valuing) | The root of the word “evaluation,” determining merit, significance or #5 of an evaluand, and the ultimate distinguishing factor of our profession. | Use of some of #1-#5 to make a decision about an evaluand. Lack of clarity on which of the above terms leads to “valuing.”[vi] |
Without specifying the type of values to which we are referring we end up making (or not) sense that sounds like this:
Evaluation: The valuing (#6) of value (#5) based on values (#3), determined by values (#4), underscored by values (#2) and influenced by many values (#1).
So when we ask a client to discuss with us the values of the evaluation, which type are we referring to? More importantly, which type do the clients think we are referring to? When we use the term “value judgment” with a group of stakeholders, it is quite possible to have multiple interpretations.
Let’s look at a specific example of how our vocabulary has caused confusion for the field of evaluation, and hence for our stakeholders:
As Scriven (2007) notes in the Values subcomponent of the Key Evaluation Checklist, many sources of values should be taken into consideration when making a criteria (#3) list. While forms #1 and #2 should be part and parcel of this list, so too must they be scrutinized with other sources, such as legal considerations. In this way, values types #1 and #2 become part of the evaluation (#6). However, contrary to the assertion of Schwandt (1997), House and others (cf. SenGupta, Hopson & Thompson-Robinson [2004]), the act of valuing #6, does not create values type #1. At best, and in a long-reaching circumstance, evaluators may cause people to reflect or rethink their personal values (#1) but this seems more to be an overstatement of the impact of evaluation. Value type #2 also doesn’t stand the logical test of being determined by value type #6. While organizations may change practices as a result of evaluations, organizational values, like cultural and personal values (#2 and #1) are deep-seated, underpinning day-to-day actions and well as overall visions and missions. In essence, it appears that an oversimplification of the values taxonomy has occurred. However, it is also possible that the authors speak to a more specific variation of values that is not made clear in their writings.
The main mistake involved in this conflation of terms seems to be more due to an overstatement of evaluation impact and an underestimation of the sources of our values (#3). Let’s look at Schwandt’s (1997) phrasing: “Assessments of value [#6] cannot help but entail making claims about what ought to be done or avoided or what is right to do… Through making such interpretations, evaluators not only inform the means by which human or social good is realized but shape our definition of the social good as well” (p. 26). Essentially, Schwandt is claiming that our criteria of merit (#3) has the potential to shape social views of good, bad, and ugly. Yet it is clear through the research in the field and reflection on individual practice that criteria are not created in a vacuum. Rather, evaluators pull together criteria from existing social norms, even if from varying sources. So rather than create new definitions of social good, we are simply reflecting what society (or some subcategorized portion thereof) has said is good. This is precisely why we consider values (#2) when making criteria. (Still, if we take organizational values to be our criteria, we are essentially making the same mistake as many goal-based evaluations, which do not question the appropriateness of the organization’s goals prior to making determinations about the merit, worth, or significance of the organization’s efforts to achieve those goals.)
It is evident from this illustration that even when an evaluator-author holds two separate meanings in her head, when using the same word to represent those meanings, confusion can easily occur for readers, evaluation students, clients, and perhaps even the evaluator-author herself. (The same can be said for our double meaning for the word “standards” – those of the Joint Committee origin and those on which performance is judged [i.e. value type #4] can be confusing when we take a moment to educate stakeholders on what it is we do. But that’s another blog post.)
Instead of picking away at the logic of scholars, let us focus on where we can improve. The larger point for us to address is that we are struggling to be understood and it has been a self-defeating act. As a first cut at clarity, I propose in Table 2 a sort of glossary from which we can adopt simpler, more appropriate linguistic choices and better express the logistics of our work together as evaluators.
Table 2. Revised Values Taxonomy
| Value # | Revised Term | Old Term | Definition |
1 | Personal values | Personal values | Beliefs or morals. Core understandings and dispositions held by individuals. Informed by #2, but with individual variation. |
2 | Cultural or organizational values | Cultural or organizational values | Central tenets that can characterize a mass of people. Variably held by the people in the mass. UN Statement on Human Rights might represent largest mass. Religious texts would be another. |
3 | Criteria | Criteria (general values) | Those dimensions on which an evaluand will be judged, based on its performance. What makes a “good” evaluand of a certain type. |
4 | Standards | Standards (specific values) | The distinguishment between performance levels for an evaluand. The demarcation between “good,” “fair,” and “bad” levels of performance. Specific aspects of #3. |
5 | Monetary worth | Monetary worth | The cost of something, given its benefits or opportunities it affords. |
6 | Judging | Judging (valuing) | The root of the word “evaluation,” determining merit, significance or #5 of an evaluand, and the ultimate distinguishing factor of our profession. |
Perhaps unsurprisingly, there has been little manipulation of the “old” value-related terms in creating new ones. Rather, the idea is that we use other established names when we have them, and in places where we don’t, we use the appropriate adjectives to describe ourselves and our work, with the explicit purpose of reducing confusion in the field. To restate the earlier example:
Evaluation: The judging of monetary worth [among other things] based on criteria, determined by standards, underscored by cultural and organizational values and influenced by many personal values.
Ah, now don’t we all feel much better?
[i] This is often considered the source of “subjectivity” and seen has having little place in evaluation. See, for example, Davidson’s (2005) description of Subjective 1 (based on Scriven) in Evaluation Methodology Basics, pages 88-92.
[ii] Stufflebeam (2007) suggests the use of widely-accepted cultural values, such as those stemming from the United Nations. He discusses this in his description of the CIPP model in Evaluation Theory, Models, and Applications, page 331.
[iii] Scriven (2007) outlines the sources of values quite well in his Key Evaluation Checklist, pages 6-10, though it would have served the field better to consistently call it “criteria of merit” or just “criteria.”
[iv] Referring to standards as specific values is becoming more common. Coryn (2007) delineates this in his dissertation, Evaluation of Researchers and Their Research, page 43.
[v] Interchanging “monetary worth” and “value” is common, but especially included in evaluation through Scriven’s Evaluation Thesaurus, pages 382-383.
[vi] For more explication, see Scriven’s Evaluation Thesaurus, page 375.
