Mid-Market B2B SaaS AI Assistant Benchmark v0.1: Methodology and Limitations

Audit date: 2026-08-19

Status: Targeted evidence and gate repair of the completed v0 benchmark

Evaluation mode: Current official public evidence only; no authenticated tenant testing

1. Purpose of v0.1

v0.1 repairs the evidence-to-classification chain identified during the post-v0 audit without restarting the 96-product research project.

The revision:

1. preserves the original v0 classifications and atomic scores in immutable original_v0_* fields;
2. adds product-specific source records and concise evidence paraphrases for the six decisive gates;
3. re-adjudicates the designated 37-product audit set;
4. derives maturity again from the repaired gate evidence;
5. renames the former Test Runs output to Documentation Scenario Mapping;
6. separates ambiguous multi-step and proactivity metrics; and
7. updates the analytical workbook, report, summary tables, and charts.

The cohort itself is unchanged.

2. Audit scope

2.1 Products re-adjudicated

The targeted audit set contains 37 products:

• every product classified at Level 2 in v0;
• every product classified at Level 3 in v0; and
• every Level 4 product with medium confidence or a beta/limited availability flag.

This rule was fixed before the v0.1 re-adjudication.

2.2 Gates re-adjudicated

Each audited product received a fresh evidence review for:

• A — Product knowledge
• D — Account/product-state awareness
• H — Task execution
• I — Multi-step execution
• J — Proactivity
• L — Transparency/control

This produced 222 product-by-gate audit records and 49 product-specific source-evidence records.

2.3 What was not re-audited

The remaining 59 products retain their original v0 classifications and scores. Dimensions B, C, E, F, G, and K were also carried forward rather than independently rescored.

Accordingly, v0.1 is a targeted gate-audit release, not a complete second-wave evaluation of all 96 products.

3. Immutable v0 preservation

The revised data keeps the original v0 values beside the v0.1 values.

Examples include:

• original_v0_maturity_level
• original_v0_maturity_confidence
• original_v0_qualifying_assistant
• original_v0_score
• original_v0_source
• original_v0_result
• original_v0_result_summary

No original v0 value was overwritten without a versioned copy.

The Maturity Change Log identifies every classification change and the evidence-based reason for it.

4. Evidence model

4.1 Evidence hierarchy

The preferred evidence order is:

1. current official operational documentation or help-center instructions;
2. current official release notes or changelogs;
3. current official product pages;
4. current official blogs or newsroom announcements.

Operational documentation receives greater weight than broad marketing language. Marketing or newsroom evidence can support a positive finding only when the behavior is explicit enough to map to a rubric criterion.

4.2 Product-specific evidence

The Source Evidence table records:

• source title;
• official URL;
• source type;
• retrieval date;
• concise product-specific evidence paraphrase;
• availability conditions;
• limitations; and
• source-evidence ID.

The Gate Audit table then links each A, D, H, I, J, and L decision to those source-evidence IDs.

4.3 Absence versus insufficient evidence

A score of 0 is used only when the official evidence reasonably establishes that the capability is absent or that a documented boundary excludes it.

A score of 1 is used for limited, ambiguous, indirect, current-object-only, narrow-feature, or separately executed behavior.

unknown is used when current official evidence is insufficient.

5. Assistant orientation

v0.1 adds an orthogonal assistant_orientation field for audited products:

• product_operation — primarily helps users understand, configure, or operate the SaaS product;
• domain_work — primarily performs work on the customer's content, data, or domain objects;
• mixed — materially combines product-operation and domain-work assistance;
• narrow_feature — a bounded AI capability that does not qualify as a general assistant.

Orientation does not replace maturity. It prevents content Q&A, analytical work, or a narrow state-changing generator from being mistaken for product-operation knowledge.

The orientation field is assessed only for the 37-product audit set in this release.

6. Gate rules

6.1 Atomic scale

Each decisive gate uses:

• 0 — absent;
• 1 — limited, ambiguous, indirect, or narrow;
• 2 — functional;
• 3 — advanced;
• unknown — insufficient evidence.

6.2 Level 2 gate

Level 2 requires:

• a general interactive assistant; and
• A >= 2, demonstrating functional product knowledge.

Customer-content Q&A alone does not satisfy this gate. A narrow generator or specialized fixer does not satisfy it merely because it changes content or code.

6.3 Level 3 gate

Level 3 requires the Level 2 gate plus:

• D >= 2, demonstrating use of actual product/account state that materially improves the response.

Awareness limited to the current page, selected record, or selected file receives at most D=1 unless broader state is explicitly used.

6.4 Level 4 gate

Level 4 requires the Level 2 gate plus:

• functional state awareness; and
• H >= 2, demonstrating that the qualifying assistant can commit a native product-state change.

The following do not independently establish Level 4:

• a draft that the user manually copies or applies;
• a suggestion that the user executes through a separate UI control;
• an action performed by a distinct automation layer rather than the assistant;
• a third-party-only action; or
• a narrow content/code generator without a qualifying general assistant.

A user confirmation can be part of execution. Confirmation is evidence of control, not evidence that execution is absent.

6.5 Multi-step execution

Multi-step execution remains orthogonal to maturity.

• Any multi-step indication: I >= 1. This includes limited or ambiguous sequences.
• Explicit dependent-action execution: H >= 2 and I >= 2.

The second measure is the publication-grade metric.

6.6 Proactivity

Two measures are now kept separate:

• Any documented proactive signal: J >= 1.
• Material proactive behavior among qualifying assistants: J >= 2 among products that pass the Level 2 gate.

The denominator and threshold must always be stated.

6.7 Transparency/control

L measures permission enforcement, previews, confirmations, review, receipts, logs, and reversibility. It does not raise maturity by itself.

7. Documentation Scenario Mapping

The v0 Test Runs table has been renamed Documentation Scenario Mapping.

Each row now states:

• mapping_basis = documentation_only;
• live_test_performed = no; and
• product_adaptation_status = generic_template_not_executed.

The table maps documented evidence to the ten standardized scenario intents. It is not a record of prompts entered into a production tenant, captured model output, or verified action success.

Scenario 10 remains unknown for every product because unsupported, impossible, and permission-boundary behavior requires live testing.

8. Revised results

Among 94 scorable products:

• 93/94 (98.9%) have some documented meaningful user-facing AI.
• 74/94 (78.7%) qualify as Level 2 or higher.
• 69/94 (73.4%) qualify as Level 3 or higher.
• 59/94 (62.8%) qualify as Level 4.
• 47/94 (50.0%) have explicit dependent-action execution.

The maturity distribution is:

• Level 0: 1
• Level 1: 19
• Level 2: 5
• Level 3: 10
• Level 4: 59
• Unknown: 2

Thirteen of the 37 audited products changed level: five upgrades and eight downgrades.

9. Interpretation of revisions

The Level 2+ count fell because domain-content assistants and narrow features were separated from functional product-knowledge assistants.

The Level 4 count rose slightly because stronger current official evidence established assistant-committed actions for several products that v0 had classified as contextual only.

These movements can coexist: v0.1 is stricter at the product-knowledge gate and more evidence-specific at the execution gate.

Because official pages were re-retrieved on 2026-08-19, a classification change can reflect:

• correction of a v0 gate application;
• stronger evidence discovery;
• clearer official documentation; or
• product/evidence change after the original snapshot.

The change log records the assigned reason, but authenticated product history was not available to isolate these effects perfectly.

10. Vendor-family sensitivity

The main analysis remains product-weighted.

As a sensitivity check, collapsing the 94 scorable products to one observation per vendor and using each vendor's highest observed product maturity produces 80 vendor observations:

• Level 4: 46/80 (57.5%)
• Level 3+: 56/80 (70.0%)
• Level 2+: 61/80 (76.3%)

This is not a replacement estimate. It shows the effect of repeated vendor families in the product-weighted cohort.

11. Denominators

• Overall maturity prevalence uses the 94 scorable products.
• The two insufficient-evidence products remain in the 96-product cohort and are never converted to zero.
• Category prevalence uses the number of scorable products in that category.
• Atomic capability prevalence uses known scores for that dimension.
• Proactivity and multi-step metrics state their threshold and denominator explicitly.
• Orientation percentages use only the 37-product audited subset.

12. Remaining limitations

12.1 No authenticated behavior tests

The benchmark cannot verify:

• answer accuracy;
• action success;
• latency;
• permissions enforcement in practice;
• fabricated completion;
• failure recovery;
• reversibility;
• safety boundaries; or
• plan and region behavior in a specific tenant.

12.2 Public-documentation bias

Official sources may:

• overstate general availability;
• omit constraints;
• describe preview, beta, or opt-in behavior;
• lag the shipped product; or
• blur the boundary between an assistant and a separate automation system.

12.3 Partial audit coverage

Only 37 products and six decisive gates were re-adjudicated. Other product classifications and dimensions remain frozen v0 observations.

12.4 Cohort reproducibility

The balanced 96-product cohort is preserved, but v0.1 does not reconstruct the original candidate universe, exclusion log, ranking, or product-specific mid-market packaging proof.

12.5 Correlated observations

Several vendors contribute multiple products. Shared assistant platforms can create correlated product observations.

12.6 Time sensitivity

The study is a rapidly changing 2026 documentation snapshot. Retool Assist, for example, is documented as scheduled for removal after its public beta on September 30, 2026.

13. Recommended next research wave

A publication-grade v1 should:

1. preserve the v0.1 gate-audit data as the documentation baseline;
2. obtain authenticated access to a smaller stratified validation sample;
3. run product-adapted prompts rather than generic templates;
4. capture screenshots, model outputs, action receipts, and permission failures;
5. test scenario 10 directly;
6. evaluate action success, recovery, and reversibility; and
7. reconstruct a reproducible sampling frame and exclusion log.

14. File map

• products.csv — revised classifications plus immutable v0 fields.
• capability_scores.csv — all 1,152 atomic records with audited and carried-forward status.
• gate_audit.csv — 222 decisive-gate adjudications.
• source_evidence.csv — product-specific official-source records.
• documentation_scenario_mapping.csv — documentation-only scenario mapping.
• maturity_change_log.csv — all 13 classification changes.
• headline_metrics.csv — reconciled definitions and v0/v0.1 values.
• category_summary.csv — revised category distribution.
• capability_prevalence.csv — atomic prevalence with audit-scope labels.
• vendor_sensitivity.csv — vendor-collapsed sensitivity analysis.
• summary_v0.1.json — machine-readable summary.
• companion XLSX, DOCX, and PDF — presentation and analytical outputs.