Articles

Product experimentation teams often struggle with causal inference on LLM-based features due to the opt-in trap, where users who click to opt in differ systematically from those who don't. Propensity score methods can help reweight comparison groups to approximate a randomized experiment.

Discover how to avoid p-hacking and stop early in product experiments using the mixture Sequential Probability Ratio Test (mSPRT) and sequential testing. This tutorial provides a step-by-step guide on implementing mSPRT in Python and applying it to a synthetic LLM product dataset.

Standard product experiments answer a binary question about average efficacy, but uplift modeling turns that binary into a nuanced spectrum, estimating a conditional average treatment effect for each user based on their specific features.

Product teams can use counterfactual estimation to answer retrospective questions about the effects of AI prompt engineering, despite the lack of random treatment assignment in logged production data.

When measuring model quality, standard regression approaches can be flawed due to confounding variables. This tutorial shows how to use instrumental variables to extract a clean causal estimate from LLM routing decisions in Python.

Collaborator contamination occurs when users in a control group are influenced by treated users in a collaborative workspace, leading to biased results in product experiments. Cluster randomization is a technique that can help overcome this issue by flipping the coin at the workspace level.

The Global Rollout Problem occurs when a team ships a model upgrade to the entire user base simultaneously, eliminating the control group. Synthetic control is a solution that builds a weighted combination of untreated units to estimate causal effects.

Product teams can streamline their AI prototyping process with the right tools. In this article, we compare the top AI prototyping tools, including Miro, Figma, and UXPilot, to help you choose the best fit for your project.