Seven deep diagnostic tools — each asks targeted questions and returns a concrete technical pathway, curated resources, and a suggested engagement. Built on the same evidence-driven logic applied across two decades of AI, economics, and policy work.
AI System DesignImpact EvaluationGovernment AIStartup DiagnosticESG EvidenceData InfrastructureMethodology Reference
All Toolkits
Click any card to open. Answer the questions to receive your personalised pathway and resources.
7 Diagnostic Tools · 39 Questions Total
AI System Design Diagnostic
6 targeted questions — maps your data environment, decision type, and constraints to a concrete technical architecture and stack.
Technical
Question 1 of 6
What is the primary data source your AI system would operate on?
This determines your ingestion architecture, pre-processing pipeline, and grounding strategy.
Question 2 of 6
What decision type should the AI primarily support?
Determines whether you need classification, ranking, generation, or causal reasoning — fundamentally different architectures.
Question 3 of 6
What is the accountability requirement for AI outputs in your context?
Auditability requirements drive your inference chain architecture more than any other factor.
Question 4 of 6
What is the current state of your labelled training data?
Determines whether fine-tuning, RAG, few-shot prompting, or zero-shot is the right approach.
Question 5 of 6
Where does the AI system need to run and what are your data sovereignty constraints?
Deployment environment determines model selection, latency budget, cost, and legal compliance.
Question 6 of 6
What is your most important constraint right now?
Honest constraint identification prevents building the right system for the wrong problem.
Technical Readiness Assessment Complete
Evidence-Grounded AI Architecture
Pathway
Technical Resources & Pathways
Core Architecture Pattern
Evidence Layer (ingest → clean → extract entities/relations → structure) → Vector Store (semantic index) → LLM Reasoning Layer (retrieval-augmented, not raw text) → Auditable Output with source citations. Never expose raw documents to the LLM.
Recommended Stack
Python · SpaCy or HuggingFace Transformers (NLP/NER) · LangChain or LlamaIndex (orchestration) · Pinecone, Weaviate, or pgvector (vector store) · GPT-4o or Claude 3.5 Sonnet (reasoning) · dbt or Prefect (pipeline orchestration).
Validation Framework
Define precision/recall targets per evidence category before deployment. Human-in-the-loop review for first 500 outputs minimum. Track confidence calibration. Set explicit confidence thresholds — outputs below threshold route to human review, not automation.
Critical Design Principle
The evidence layer is the moat. LLM reasoning is commodity. Invest in entity extraction quality, relation typing, source provenance, and structured knowledge representation. This is what makes your system defensible and auditable long-term.
Key Literature
'RAG for Knowledge-Intensive NLP' (Lewis et al. 2020) · 'Constitutional AI' (Anthropic 2022) · ISO/IEC 42001 AI Management Systems · NIST AI Risk Management Framework · 'Patterns for Building LLM-Based Systems' (Anthropic 2024).
Suggested Engagement: AI System Architecture Design
6–12 week engagement covering: evidence layer architecture, model selection and evaluation, deployment environment design, validation framework, and operational monitoring. Deliverable: technical specification document and working prototype on your data.
6 questions — identifies the right causal methodology, data strategy, and software stack for your evaluation context.
Methodology
Question 1 of 6
What type of intervention or change are you evaluating?
The intervention type is the first fork in identification strategy selection.
Question 2 of 6
Do you have a credible control or comparison group?
This is the single most important question in causal identification — everything else follows from the answer.
Question 3 of 6
What is the primary outcome variable you need to measure?
Outcome type determines your measurement instrument, data requirements, and publication venue.
Question 4 of 6
What administrative or proprietary micro-data do you have access to?
Data availability shapes methodology more than theoretical preference — be honest about what exists.
Question 5 of 6
What time horizon do you need to measure effects over?
Time horizon determines panel structure, discount rates, and sample attrition risk.
Question 6 of 6
What will the evaluation findings be used for?
End use determines rigour standard, format requirements, and who reviews your methodology.
Evaluation Rigour Assessment Complete
Causal Identification Strategy
Pathway
Technical Resources & Pathways
Primary Identification Method
Difference-in-Differences (DiD) with parallel pre-trends test as the default. If treatment assignment is discontinuous, add Regression Discontinuity Design. If panel attrition is high, add Synthetic Control. If endogeneity persists, search for valid Instrumental Variable.
Data and Matching Strategy
Link administrative records at the individual or firm level using probabilistic record linkage where ID keys are absent. Construct comparison group via Propensity Score Matching or Coarsened Exact Matching. Document all exclusion decisions in a pre-analysis plan.
Validity Threat Checklist
Spillover / SUTVA violation between treated and control units · Anticipation effects before policy announcement · Differential attrition across treatment arms · Parallel trends violation (test with placebo periods) · LATE vs ATE distinction if compliance is imperfect.
Software and Implementation Stack
Stata (preferred for panel IV, DiD, event studies) · R packages: did, fixest, rdrobust, Synth, MatchIt · Python: linearmodels, econml (causal ML), DoWhy · EViews for macro time-series VAR/ARIMA · Julia for large-scale simulations.
Suggested Engagement: Impact Evaluation Design and Execution
End-to-end evaluation support: identification strategy design, data acquisition and linkage plan, pre-analysis plan drafting, statistical analysis, and policy-ready report. Typical timeline 3–9 months depending on data availability and scope.
Map to ISO/IEC 42001 AI Management System standard as your governance foundation. EU: classify under AI Act risk tiers before any procurement activity begins. Canada: align with Treasury Board Directive on Automated Decision-Making impact levels. All: document intended purpose, training data, performance benchmarks, and limitations before any operational use.
Evidence Layer Before Model
Prerequisite before any AI model deployment: define observable indicators, source each indicator to a system of record, establish a measurable baseline, and document data quality tier. AI should reason over structured evidence, not raw ministerial correspondence or case file PDFs.
Government-Grade Explainability
SHAP values and feature importance alone are legally insufficient for rights-affecting decisions. Require contrastive explanation capability: "This decision was reached because of X. It would have changed if Y." Natural language rationales must be linked to source data citations. Build the explanation layer into the architecture from the start.
Reference Frameworks and Standards
OECD AI Principles (2019, updated 2024) · EU AI Act Recitals and Annex III · Canada Treasury Board ADM Directive · NIST AI Risk Management Framework 1.0 · Alan Turing Institute "Understanding AI Ethics and Safety" · ISO/IEC 42001:2023.
Suggested Engagement: Government AI Advisory and Standards Design
Governance framework design, regulatory compliance mapping, procurement evaluation support, evidence layer architecture, and explainability standard specification. Typical 3–6 month engagement with phased deliverables aligned to procurement or policy milestones.
6 questions — diagnoses moat strength, PMF signal, technical risk, and go-to-market readiness for AI ventures at any stage.
Founders
Question 1 of 6
What is the core defensible moat of your AI product?
Moat type determines investor narrative, long-term pricing power, and competitive strategy — be brutally honest.
Question 2 of 6
What is your current evidence of product-market fit?
PMF signals determine whether you should accelerate GTM spend or continue iterating on product.
Question 3 of 6
What is the primary technical risk in your current AI system?
Honest technical risk assessment prevents scaling the wrong architecture — the most expensive mistake in AI startups.
Question 4 of 6
What is the primary buyer persona and their procurement motion?
Buyer type drives sales motion design, contract structure, pricing model, and time-to-revenue.
Question 5 of 6
What stage of company are you at right now?
Stage determines the right strategic priorities and the metrics investors will scrutinise.
Question 6 of 6
What outcome are you optimising for in the next 12 months?
Goal clarity is the most underrated strategic tool — it prevents the resource dilution that kills most startups.
Venture Readiness Assessment Complete
AI Startup Strategic Roadmap
Pathway
Technical Resources & Pathways
Moat Development Priorities
Model performance is the weakest AI moat — foundation models erode it continuously. Invest in: (1) data network effects where each user makes your system smarter, (2) workflow lock-in where switching cost is operational disruption not just contract cancellation, (3) evidence layer depth that becomes your proprietary knowledge graph over time.
PMF Measurement Framework
Weekly active usage rate (target >60%) · Sean Ellis survey: >40% "very disappointed" · Net Revenue Retention >110% (the only PMF metric VCs believe at Series A) · Reference-ability rate: >50% of customers willing to be public references · Time-to-value: how quickly does a new customer see the first output they trust?
Technical Architecture for Scale
Separate evidence layer from inference layer early — this is the most important architectural decision. Evidence layer: ingest, normalise, extract entities, store with provenance. Inference layer: retrieval, reasoning, output generation. This decoupling lets you swap foundation models (GPT → Claude → Llama) without rebuilding business logic.
Enterprise AI Sales Motion
Lead with the cost of the status quo, not the technology. Require a data audit call in discovery — reveals real integration complexity. Build a single-customer business case before building a sales deck. First 3 enterprise deals must be founder-led. Quantify ROI in dollars and hours saved, not "efficiency gains". Security review is a feature, not a blocker — build your security docs before you need them.
Investor Narrative Framework
Opening: the specific problem + why it is unsolved today (market timing). Why you: unique data position or domain depth. Evidence of pull: customer quotes with numbers, not vague validation. Architecture: why your approach is defensible, not just functional. Team: relevant domain expertise that took years to build. Avoid opening with "We use AI to..."
Suggested Engagement: AI Startup Advisory
Technical architecture review, go-to-market strategy, investor narrative development, and ongoing board advisory. Structured as monthly retainer or pre-raise intensive. Particularly suited to Series A preparation, enterprise GTM design, and acquisition positioning.
5 questions — assesses measurement methodology, Scope 3 data gaps, physical-world evidence integration, and assurance readiness.
Sustainability
Question 1 of 5
What is the primary reporting obligation driving this ESG assessment?
Reporting standard determines scope, assurance level, materiality definition, and disclosure format.
Question 2 of 5
How are your Scope 1 and Scope 2 greenhouse gas emissions currently measured?
Measurement method quality is the most scrutinised element in any ESG assurance engagement.
Question 3 of 5
Where is your biggest Scope 3 data gap?
Scope 3 (value chain) typically represents >70% of an organisation's total carbon footprint.
Question 4 of 5
What physical-world data sources are currently integrated into your ESG measurement?
Physical-world data enables evidence-based ESG versus estimate-based ESG — a critical credibility distinction.
Question 5 of 5
What is the most important credibility gap you need to close?
Credibility gap type determines whether the solution is better data, better methodology, or better assurance.
Evidence Quality Assessment Complete
ESG Evidence Architecture Upgrade Plan
Pathway
Technical Resources & Pathways
Evidence Tier Architecture
Build a three-tier evidence stack: Tier 1 (Primary) — sensor, smart meter, EO satellite data; Tier 2 (Secondary) — supplier primary data, IoT logistics, utility invoices; Tier 3 (Tertiary) — spend-based estimates only where primary data is genuinely inaccessible. Document the tier for every emission source in your inventory.
Scope 3 Data Strategy
Purchased goods (Cat 1): deploy supplier data portals with CDP or PCAF-aligned templates; set primary data coverage targets by supplier spend tier. Use-of-sold-products (Cat 11): instrument products with telemetry where feasible. Financed emissions: apply PCAF Standard, start with attribution methodology before data collection.
Earth Observation Integration
Sentinel-2 (10m resolution, free) and Landsat-9 for land use change and vegetation monitoring. Synthetic Aperture Radar (SAR) for industrial activity and deforestation proxy indicators independent of cloud cover. Google Earth Engine or Microsoft Planetary Computer for large-scale analysis. NDVI, NDWI, and burn scar indices as leading indicators.
Assurance Readiness Requirements
Data lineage documented per emission source and factor · Methodology aligned to GHG Protocol Corporate Standard or ISO 14064-1 · All material estimates flagged with confidence intervals · Recalculation policy documented · Engage assurance provider (KPMG, EY, DNV, Bureau Veritas) in planning phase, not after year-end close.
Standards, Frameworks, and Tools
GHG Protocol Corporate Standard · PCAF Standard (financed emissions) · ISSB IFRS S2 · SBTi Corporate Net-Zero Standard · TNFD (nature-related disclosure) · CDP disclosure platform · SaaS tools: Persefoni, Sweep, Watershed, or Greenly for data management and reporting automation.
Suggested Engagement: ESG Evidence Architecture and Reporting Advisory
Measurement framework design, data source audit and gap analysis, Scope 3 strategy, EO data integration for physical-world grounding, and assurance readiness preparation. Typically 8–16 weeks. Can be structured as a standalone audit or ongoing advisory.
5 questions — evaluates data architecture maturity, quality issues, governance gaps, and the path to AI-ready infrastructure.
Infrastructure
Question 1 of 5
How is your data currently stored and managed across the organisation?
Storage architecture is the foundation layer — everything else depends on getting this right first.
Question 2 of 5
What is the biggest data quality issue currently affecting your analytics and AI?
Quality problem type determines remediation strategy — they require fundamentally different interventions.
Question 3 of 5
What is the primary use case driving your data infrastructure investment?
Use case determines your latency requirements, query patterns, tooling choices, and cost model.
Question 4 of 5
What is the current state of your data governance programme?
Governance debt grows faster than technical debt — every month without it costs more to fix later.
Question 5 of 5
What cloud or infrastructure environment are you operating in?
Environment determines which managed services are available and the complexity of any migration.
Infrastructure Maturity Assessment Complete
Data Infrastructure Modernisation Roadmap
Pathway
Technical Resources & Pathways
Target Architecture Pattern
Ingest (Fivetran or Airbyte for SaaS sources, custom connectors for internal systems) → Store (cloud data warehouse + object storage for unstructured) → Transform (dbt for documented, tested, version-controlled SQL transforms) → Serve (BI layer for humans, feature store for ML models). Add data catalogue from Day 1, not as an afterthought.
Data Quality Engineering Stack
dbt tests for referential integrity, uniqueness, and null constraints at transform time. Great Expectations or dbt-expectations for statistical distribution checks. Monte Carlo or Elementary for data observability — detect silent failures before they corrupt downstream models. Anomaly detection on pipeline SLAs.
Governance Architecture Principles
Data contracts: producers commit to schema, SLA, and quality level — versioned and machine-readable. Data lineage: every transformation documented automatically (OpenLineage / Marquez). Data ownership: every domain table has a named owner with accountability. Implement all three before scaling ingestion volume — retrofitting is 5× more expensive.
AI and ML Readiness Checklist
Feature store in place (Feast, Tecton, or cloud-native) · Training/serving skew documented and monitored · Model registry with full versioning and experiment tracking (MLflow or W&B) · Data validation gates run before every training job · Ground truth labels have full provenance tracking · Retraining triggers defined and automated.
Technology Reference Stack
dbt Core (transforms) · Apache Iceberg (open table format for ACID compliance) · Databricks or Spark (large-scale distributed processing) · Feast or Tecton (feature store) · DataHub or Atlan (metadata and catalogue) · Great Expectations (quality) · Prefect or Dagster (orchestration).
Suggested Engagement: Data Infrastructure Architecture Advisory
Current-state audit, target architecture design, vendor and tooling selection, data governance framework, and phased implementation roadmap. Typically a 4–8 week assessment phase followed by optional implementation support.
7 methods — click any card for when to use it, key assumptions, software tools, and critical failure modes.
Reference
Randomised Controlled Trial
Gold standard for causal inference
Difference-in-Differences
Pre/post with comparison group
Regression Discontinuity
Sharp cutoff identification
Instrumental Variables
Endogeneity correction
Synthetic Control
Single treated unit
NLP / Text Analytics
Unstructured text evidence
EO / Geospatial Analysis
Physical-world grounding
Randomised Controlled Trial
Gold standard for causal inference
When to use
When treatment and control group assignment can be randomised — either by design (field experiment) or by discovering a natural experiment (lottery, waiting list, arbitrary cutoff). The only method that fully eliminates selection bias by design.
Key assumptions
Stable Unit Treatment Value Assumption (SUTVA) — no spillovers between units. Sufficient statistical power (calculate N before randomising). Pre-specify primary outcomes and analysis plan before unblinding.
Software and tools
Stata: randtreat, ritest for randomisation inference · R: randomizr, ri2 · Python: causalml · Power calculation: G*Power, Optimal Design
Common failure modes
Non-compliance by participants reduces estimand from ATE to LATE — use IV/2SLS to recover. Hawthorne effect: treated units behave differently because they know they are observed. Ethical constraints may prevent randomisation of high-stakes interventions.
Difference-in-Differences
Pre/post with comparison group
When to use
When a policy or intervention affects some units (treated) but not others (control), and you have data before and after the treatment for both groups. Exploits the timing of treatment rather than randomisation.
Key assumptions
Parallel trends: treated and control groups must have been moving in parallel before treatment. No anticipation: units do not change behaviour before the treatment date. Stable composition: the composition of treated and control groups does not change systematically.
Software and tools
Stata: xtdidregress, eventdd · R: fixest (preferred), did (Callaway-Sant'Anna for staggered), DiDdesign · Python: linearmodels, difference-in-differences
Common failure modes
Staggered treatment rollout violates standard DiD — use Callaway-Sant'Anna (2021) or Sun-Abraham (2021) estimators. Pre-trends test failure does not automatically invalidate — examine the magnitude of deviation.
Regression Discontinuity
Sharp cutoff identification
When to use
When treatment assignment is determined by whether a continuous running variable (score, age, income) crosses a threshold. Units just below and just above the cutoff are assumed to be comparable on all unobserved characteristics.
Key assumptions
Continuity: the density and characteristics of units must be continuous at the cutoff — no sorting or manipulation. Local treatment effect: RDD estimates LATE at the cutoff only, not average treatment effect for the full population. No compound treatment at cutoff.
Bandwidth selection critically affects estimates — use data-driven bandwidth (Imbens-Kalyanaraman). Always run McCrary density test to check for manipulation of running variable around cutoff. Donut RDD if manipulation is local to cutoff.
Instrumental Variables
Endogeneity correction
When to use
When there is endogeneity (the treatment variable is correlated with the error term) and randomisation is impossible, but a valid instrument exists. An instrument affects treatment take-up but has no direct effect on the outcome except through treatment.
Key assumptions
Relevance: instrument must be strongly correlated with treatment — F-statistic > 10 (weak instrument test). Exclusion restriction: instrument affects outcome only through the treatment variable — this is fundamentally untestable and requires theoretical justification. Independence: instrument is as-good-as-random.
Software and tools
Stata: ivreg2, ivreghdfe (with high-dimensional FE) · R: ivreg (AER package), fixest for fast IV · Python: linearmodels.iv · Classic instruments: distance to college (education), rainfall (agricultural income), judge leniency (legal outcomes)
Common failure modes
Weak instruments inflate standard errors and bias TSLS toward OLS. Use Limited Information Maximum Likelihood (LIML) when F < 10. IV only identifies LATE (effect on compliers) — carefully interpret what population this represents.
Synthetic Control
Single treated unit
When to use
When you have a single treated unit (one country, one city, one firm) and a pool of potential control units, and you want to construct a data-driven weighted combination that best approximates the counterfactual trajectory of the treated unit.
Key assumptions
Convex hull: the pre-treatment characteristics of the treated unit must lie within the convex hull of the donor pool. Good pre-treatment fit: the synthetic control must closely track the treated unit in the pre-treatment period across both the outcome and predictors.
Software and tools
R: Synth (canonical), SCtools, tidysynth · Stata: synth, synth_runner · Python: SyntheticControlMethods · Augmented SCM (Ben-Michael et al.) for bias correction when fit is imperfect
Common failure modes
Do not include too many predictors — overfitting in pre-treatment period leads to poor post-treatment extrapolation. Use permutation inference (in-space and in-time placebos) rather than standard p-values. Augmented SCM preferred when donor pool is large.
NLP / Text Analytics
Unstructured text evidence
When to use
When the primary evidence source is unstructured text — documents, emails, reports, contracts, parliamentary transcripts, news, social media. Can extract structured signals from text at scale that no human could read manually.
Key assumptions
Ground truth labels for supervised tasks require expert annotation — expensive and time-consuming to do correctly. Embedding models encode semantic meaning but lose syntactic precision for tasks like date extraction or quantity identification. LLM outputs are probabilistic, not deterministic — build verification layers.
Named entity recognition models trained on news text perform poorly on legal, scientific, or government documents — always fine-tune on domain-specific data. Embedding drift: semantic space shifts if documents from different time periods are mixed. Hallucination in LLM extraction — verify factual claims against source documents programmatically.
EO / Geospatial Analysis
Physical-world grounding
When to use
When physical-world change, activity, or state is the key evidence source and cannot be reliably self-reported. Earth observation provides independent, tamper-resistant, continuous coverage of the physical world — the best alternative to on-site inspection at scale.
Key assumptions
Spatial resolution must match the scale of the phenomenon being measured — 10m Sentinel-2 pixels cannot reliably detect individual vehicles. Temporal resolution matters for change detection — know your satellite revisit period. Atmospheric correction required for reflectance-based indices (NDVI, NDWI).
Software and tools
Google Earth Engine (browser-based, large-scale free analysis) · Microsoft Planetary Computer (free, cloud-optimised) · Python: rasterio, geopandas, pyproj, xarray · QGIS for desktop visualisation · GDAL for format conversion · Sentinel Hub or Planet API for commercial tasking
Common failure modes
Cloud cover is the primary data gap for optical sensors — use SAR (Sentinel-1) as cloud-penetrating alternative. Phenological seasonality affects vegetation indices — compare same season across years. Training ML models on EO data requires large, well-labelled datasets that are expensive to produce.