ForTune: Running Offline Scenarios to Estimate Impact on Business Metrics
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910801948835840 |
|---|---|
| author | Dupret, Georges Sozinov, Konstantin Gonzalez, Carmen Barcena Zacks, Ziggy Yuan, Amber Carterette, Benjamin Mai, Manuel Bansal, Shubham Lien, Gwo Liang Leo Gatash, Andrey Ojeda, Roberto Sanchis Lalmas, Mounia |
| author_facet | Dupret, Georges Sozinov, Konstantin Gonzalez, Carmen Barcena Zacks, Ziggy Yuan, Amber Carterette, Benjamin Mai, Manuel Bansal, Shubham Lien, Gwo Liang Leo Gatash, Andrey Ojeda, Roberto Sanchis Lalmas, Mounia |
| contents | Making ideal decisions as a product leader in a web-facing company is extremely difficult. In addition to navigating the ambiguity of customer satisfaction and achieving business goals, one must also pave a path forward for ones' products and services to remain relevant, desirable, and profitable. Data and experimentation to test product hypotheses are key to informing product decisions. Online controlled experiments by A/B testing may provide the best data to support such decisions with high confidence, but can be time-consuming and expensive, especially when one wants to understand impact to key business metrics such as retention or long-term value. Offline experimentation allows one to rapidly iterate and test, but often cannot provide the same level of confidence, and cannot easily shine a light on impact on business metrics. We introduce a novel, lightweight, and flexible approach to investigating hypotheses, called scenario analysis, that aims to support product leaders' decisions using data about users and estimates of business metrics. Its strengths are that it can provide guidance on trade-offs that are incurred by growing or shifting consumption, estimate trends in long-term outcomes like retention and other important business metrics, and can generate hypotheses about relationships between metrics at scale. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_00133 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ForTune: Running Offline Scenarios to Estimate Impact on Business Metrics Dupret, Georges Sozinov, Konstantin Gonzalez, Carmen Barcena Zacks, Ziggy Yuan, Amber Carterette, Benjamin Mai, Manuel Bansal, Shubham Lien, Gwo Liang Leo Gatash, Andrey Ojeda, Roberto Sanchis Lalmas, Mounia Computational Engineering, Finance, and Science Applications Making ideal decisions as a product leader in a web-facing company is extremely difficult. In addition to navigating the ambiguity of customer satisfaction and achieving business goals, one must also pave a path forward for ones' products and services to remain relevant, desirable, and profitable. Data and experimentation to test product hypotheses are key to informing product decisions. Online controlled experiments by A/B testing may provide the best data to support such decisions with high confidence, but can be time-consuming and expensive, especially when one wants to understand impact to key business metrics such as retention or long-term value. Offline experimentation allows one to rapidly iterate and test, but often cannot provide the same level of confidence, and cannot easily shine a light on impact on business metrics. We introduce a novel, lightweight, and flexible approach to investigating hypotheses, called scenario analysis, that aims to support product leaders' decisions using data about users and estimates of business metrics. Its strengths are that it can provide guidance on trade-offs that are incurred by growing or shifting consumption, estimate trends in long-term outcomes like retention and other important business metrics, and can generate hypotheses about relationships between metrics at scale. |
| title | ForTune: Running Offline Scenarios to Estimate Impact on Business Metrics |
| topic | Computational Engineering, Finance, and Science Applications |
| url | https://arxiv.org/abs/2403.00133 |