Methodology Before Cohort

A charter for a cross-state cybersecurity benchmarking project sat untouched for six weeks. It named six peer states by name. Texas, Virginia, North Carolina, Georgia, South Carolina, California. That was the list. What it did not contain was a single sentence explaining how those six were chosen.

When the project reached the phase where actual comparison would begin, the methodology gate finally got written. Five criteria: statutory or regulatory anchor, alignment to a named control catalog, public availability of the source document, size comparability with the reference state, and regional and federal-context diversity. The criteria were applied to all fifty states and ranked. The cohort that fell out of the filter was not the charter cohort. South Carolina dropped for failing the statutory-anchor test jointly with the catalog-alignment test. New York took the open slot as a Tier A state with an explicit controls catalog anchor and a Northeast perspective the original list never included.

The cohort was rebuilt in an afternoon. The lesson took longer.

The problem with inherited lists

Most comparative work starts with a list. Pick three vendors. Benchmark against five peer agencies. Evaluate four frameworks. Study six states. The list gets typed into a charter, a project plan, a procurement document, or a proposal abstract. Whoever writes the list has reasons, usually reasonable ones. Those reasons rarely get written down.

Then the analysis happens. Data gets collected. Findings emerge. Someone asks, late in the engagement, "why those six?" The author back-fits an answer. The answer sounds plausible because the author genuinely had reasons, once, in the original decision moment. But the reasons are reconstructed, not recorded, and they conform neatly to the entities already named. That is motivated reasoning wearing a methodology costume.

The back-fit also hides the cases where the list is wrong. If South Carolina was on the original list because someone thought "Southeastern state, similar size, probably relevant," and the actual methodology would have disqualified the state, nobody notices. The analysis proceeds with a cohort that does not survive its own criteria. Auditors, reviewers, and stakeholders who ask hard questions catch this later. The project explains, apologizes, and occasionally re-does the work.

The fix is cheap

Write the criteria first. Apply them to the full candidate space. Let the cohort fall out.

This takes more time upfront than typing names. The time spent is recovered many times over when someone asks the inevitable question. You open the methodology document, point to the five criteria, and walk them through how each candidate ranked. Nobody has to take your word for it. The cohort is a consequence, not an assertion.

Criteria-first analysis also kills the most common objection to any comparison: "why not include X?" In an inherited cohort, the answer is some version of "we just didn't." In a criteria-derived cohort, the answer is "X was considered and fell below the threshold on criterion N." The conversation shifts from defending the list to discussing the criteria. That is a much more productive conversation because criteria are debatable, revisable, and can be improved. Lists are just lists.

What qualifies as a criterion

Not every sentence that sounds methodological is a real criterion. The test is whether the criterion is objective enough to produce the same cohort when applied by a different analyst to the same candidate space.

Useful criteria are binary where possible (does the entity meet this threshold or not), ranked where a binary test is too coarse (top-10 by a measurable property), and explicit about what they exclude. Statements like "mature programs only" or "relevant to our context" are not criteria. They are preferences wearing methodology costumes. A real version reads: "program has published a control catalog within the last three years" or "program operates under a statutory anchor naming the same catalog used by the reference program."

Weighting matters too. If some criteria are binary-required and others are ranked-preferred, say so. An auditor who sees a 50-state rankings table with unweighted criteria will assume they are all equal, and your ranked cohort will disagree with the table. Explain the weighting before the cohort emerges.

Where this shows up

Vendor shortlists are the obvious case. Most RFP cohorts are inherited. So are proof-of-concept candidate pools, framework shortlists for regulatory adoption, peer-state benchmarking exercises, committee compositions, advisory panel selections, and literature review inclusion criteria. Academic researchers deal with this in systematic review protocols, where PRISMA exists precisely to force criteria-first inclusion decisions. The discipline is well-established in one domain and absent in several others.

The bar is low. A one-page methodology document, five or fewer criteria, applied to a documented candidate space, produces a defensible cohort. Anyone can do this. The only reason it does not happen is that typing names into a charter is faster than writing rules, and the cost of the shortcut does not become visible until someone asks the hard question.

The small reframe

If you catch yourself writing a cohort without writing the criteria first, pause. You are not doing analysis yet. You are doing vibes. The analysis starts when the criteria exist, not when the names appear.

Write the rules. Run the filter. Let the names fall out. When someone later asks why those six, you will have an answer that does not have to be reverse-engineered.