Buying checklist · 2027 planning
Synthetic research.
Ask for the evidence.
Choose evidence that fits your decision before choosing a synthetic market research vendor.
This is a procurement planning resource, not a ranked “best tools” test or a 2027 product review. Minds is a vendor included in the worksheet. No vendor has a prefilled score.
Start with the editable worksheet
Four tabs cover instructions, vendor evidence, evaluation criteria and matched numeric results. Open the Sheet and choose File → Make a copy. Keep customer data and private vendor responses in your own restricted copy.
Prefer portable files? Download the vendor evidence CSV, evaluation checklist CSV or matched results CSV. The CSVs contain no formulas; the native Sheet calculates numeric gaps.
Blank evidence means not assessed. It is not a vendor failure or a recommendation.
Define the decision first
Write down the intended audience, question, outcome metric and acceptable error before reviewing vendor results. Your test should establish whether the approach helps with that decision, beyond an example that sounds plausible.
If you have 23 synthetic answers and no human results
The comparison is still unmeasured. Obtain a matching human reference:
- Run the same questions with a suitable human sample, using the same screening, stimuli, options and presentation.
- Alternatively, use an existing human dataset when its population and questions match sufficiently. Document material mismatches and the collection date.
- Reserve questions or outcomes as a holdout. Keep reference answers out of generation and tuning.
Choose sample sizes and uncertainty analysis for your decision. Record subgroup sizes, sampling limitations and repeated-run variation; this template makes no universal sample-size promise.
Twelve checks before a purchase
| Criterion | Evidence to request | How to check |
|---|---|---|
| Decision fit | State the decision and acceptable error before testing. | Record what a wrong answer would change. |
| Human reference | Obtain human results for the same population and questions. | No matched human data means validation remains unmeasured. |
| Audience match | Document geography, screening, sample, quotas and collection dates. | Explain differences between the two samples. |
| Question match | Use identical question wording, options and presentation. | Keep concept materials and context consistent. |
| Metric definition | Define the outcome, scale, unit and acceptable gap. | Compare percentages with percentages; distinguish points from relative change. |
| Holdout | Reserve questions or outcomes that were not used for tuning. | Keep the reference answer unavailable during generation. |
| Repeatability | Repeat runs and report variation and human sampling uncertainty. | A single agreeing run does not establish reliability. |
| Segment performance | Check relevant subgroups with enough human evidence. | Report subgroup sample sizes and uncertainty. |
| Provenance | Record model/version, prompts, source data and test date. | Keep an export that another reviewer can inspect. |
| Limitations | Record unsupported uses, failures and unresolved discrepancies. | Do not replace a failed test with a general accuracy claim. |
| Data handling | Request current privacy, retention and security documentation. | Assess it against your actual data and obligations. |
| Commercial fit | Obtain current terms, usage limits and an export example. | Confirm with the vendor; this template contains no price comparisons. |
Compare the same question, audience and unit
For each outcome, record the source, date, model/run details, human result, synthetic result and uncertainty. A human share of 40% and a synthetic share of 46% differ by 6 percentage points. That is not 6% relative error or an overall accuracy score.
| Option | Human share (%) | Synthetic share (%) | Absolute gap (points) |
|---|---|---|---|
| A | 40 | 46 | 6 |
| B | 35 | 29 | 6 |
| C | 25 | 25 | 0 |
| Total share | 100 | 100 | — |
| Mean option gap | — | — | 4 |
A small average can hide a material subgroup failure. Keep individual outcomes, human uncertainty, variation and limitations visible. “No matched human baseline” remains an unresolved evidence gap.
Vendor discovery starting points
The worksheet includes 17 names and official URLs from Minds’ public category article, updated 17 October 2026. Verify current availability, capabilities and terms directly.
Minds appears first to identify the publisher. The remaining vendors are alphabetical. There are no prefilled quality scores.
Record what the evidence supports
Document what was tested, what was missing, where the approach failed, the acceptable use and the reviewer. A question-level gap is not a general accuracy claim.
The year describes a planning horizon. This resource does not claim tests conducted in 2027 or verified vendor performance. Recheck sources before a procurement decision.
Existing Minds resources include the Chrome research companion and the MCP integration reference.
Explore Minds