When Benchmarks Describe Rather Than Aspire

A state CISO submits a major annual survey. The instrument is from a credible third party, the questions are well-designed, and the respondent completes it carefully. The report comes back six months later. On question 16, the report logs the state's preferred operating model as federated.

The problem: the state's strategic plan calls for centralization. The CISO did not intend to log a preference for the current model. But the answer did exactly that.

This is not a mistake in the usual sense. Nobody wrote the wrong answer. But the answer is wrong.

The Sequencing Problem

The survey placed a descriptive question immediately before a normative one.

Question 15 asked: how is your state's cybersecurity function currently structured? The respondent answered honestly -- federated, because that is what it is today.

Question 16 asked: which operating model do you think will be most beneficial to your state?

The two questions look independent. They are not. The respondent just spent focused cognitive effort categorizing their current structure. That mental model is active. When question 16 arrives asking what they prefer, the path of least resistance is to stay in the same frame. The answer that just felt true still feels true. The aspirational answer, the one that requires stepping out of current-state thinking and into strategic intent, requires more deliberate effort to produce.

Most respondents do not make that shift. The result is a normative answer that mirrors a descriptive one.

Why This Matters More Than It Looks

Benchmarking data gets used in three ways that amplify this problem.

First, peer comparisons. When 73 percent of states say they prefer centralized and one state logs federated, the state looks like an outlier. It may get framed that way in presentations, in legislative briefings, and in vendor pitches. The outlier framing is not necessarily wrong from a current-state perspective. It is wrong if applied to strategic direction.

Second, trend tracking. Multi-year benchmarks show movement over time. If this year's answer is federated (current state) and next year's answer is centralized (strategic intent, if the question gets answered differently), the data looks like a preference shift when nothing actually changed. The noise is in the instrument, not the state.

Third, external records. The data sits in a third party's database. Other researchers, policy analysts, and consultants will access it. They will draw inferences from it. Nobody consulting that database will know whether the answer reflected current state or strategic aspiration.

The Anchoring Mechanism

Cognitive anchoring is well-documented. The first number, label, or category that comes to mind in a decision sequence shapes subsequent judgments. It is not a sign of carelessness. It is how human cognition handles cognitive load.

Survey instruments create anchoring conditions when they place descriptive and normative questions in close sequence without an explicit reframing prompt. The respondent does not experience a transition from "what is" to "what should be." They experience one long question set about their security program, and their mental state at question 15 carries into question 16.

The effect is stronger when the descriptive question requires genuine categorization work, which the current-state operating model question does. Federated, centralized, and decentralized are not casual labels. Correctly categorizing your current structure requires thought. That thought becomes an anchor.

What To Do With Contaminated Data

If you are reviewing benchmark results and suspect current-state contamination, start by looking at sequencing. Find the descriptive question that preceded the normative one. If the normative answer matches the descriptive answer exactly, treat the normative answer with skepticism. Verify it against strategic documents: does the formal plan, the budget request, or the publicly stated direction confirm this preference, or contradict it?

If the benchmark data is yours, correct the record where you can. Flag it in internal documentation before others cite it. Create a task to answer the question accurately in the next cycle. This is not about disputing a third party's report -- it is about protecting the integrity of your own strategic record.

If you are designing a survey instrument, separate descriptive and normative questions. A simple explicit prompt between them reduces anchoring: "The following questions ask about your preferred future direction, which may differ from your current structure." That sentence costs four words and produces materially more accurate data.

For the Research Community

This is not an edge case in government IT surveys. It is a structural vulnerability in any instrument that mixes current-state description with aspirational assessment. The two question types require different cognitive modes. Asking them in close sequence, without signaling the shift, makes contamination more likely than not.

The fix is not complex. It requires survey designers to treat descriptive-normative sequencing as a variable to control, the same way they control for social desirability bias or response fatigue. A brief section break, an explicit reframing prompt, or a deliberate reordering of the question set are all sufficient.

The alternative is benchmark data that reflects where organizations are rather than where they intend to go, used by researchers and policymakers who have no way to know the difference.