# A/B Testing Results Analysis ## A/B Testing Results Analysis After conducting an A/B test, it is important to properly analyze the results. In Ptengine Experience, you can confirm the results from the "A/B Testing Analysis" section on the experience details screen. This article explains how to read test results. > **SUCCESS** **Tip: Start analyzing your test results after data has accumulated.** > > Ptengine's A/B tests determine results based on "win rate" and "minimum test duration." > > To ensure test accuracy, be sure to set these two values. > > For more details, please check the article [here](/en/experience/campaign/create-new-2/result-analysis.md). ## **Specify Evaluation Metrics** When confirming A/B test results, the first step is to specify an appropriate evaluation metric. The evaluation metric 👉 Can be selected from goals set in the experience in advance. 🔗 Goals can also be added after the test ends, but please check [here](/en/experience/campaign/create-new-2/result-analysis.md) for the method. > **WARNING** In most cases, company conversions or KPIs are set as goals, but since multiple metrics can be set, setting micro-conversions such as CTA click rates will expand the scope of analysis. > > If the path from the test verification location to final conversion is long, setting micro-conversions is recommended. When you select an evaluation metric, the report content based on that metric is displayed. The main report contents include the following: Display UU count for each pattern Goal reaching UU count, goal rate Performance and win rate ## **Determine the Superior Pattern** After selecting an evaluation metric in the steps above, confirm the details of the test results. To determine the superior pattern, check the "performance" and "win rate" in the report. ### **What is Performance** Performance shows the difference between the baseline pattern and the test pattern for the evaluation metric. The baseline refers to the original page or the first pattern. For example, if "purchase" is set as the goal for the experience, and the goal reaching rate for the baseline pattern is 4% and the goal reaching rate for the test pattern is 5%, the performance is 25%. (Formula: 5% / 4% - 1) ### **What is Win Rate** Win rate is the probability that the relevant pattern outperforms all other patterns. It is the most important metric for determining the superior pattern. At Ptengine, win rate is calculated using [Bayesian statistics](/en/experience/campaign/create-new-2/result-analysis.md). After the minimum test duration ends, if the conditions are met, it is determined to be superior. ## **Superior Pattern Determination Logic** When all three of the following conditions are met, we determine that the relevant pattern can be judged as superior and display the test results on the report screen. **Win rate of any pattern exceeds the set win rate threshold** The win rate is set to a default of 95%. The win rate threshold can also be changed by changing "win rate" in the settings in the upper right. **The minimum test duration has elapsed** The minimum test duration is set to a default of 7 days. It can also be changed from the settings in the upper right. **Display user count for each pattern is 100 or more, and goal reaching user count is 30 or more** > **WARNING** For details on win rate and minimum test duration, please check [A/B Testing Detailed Settings](/en/experience/campaign/create-new-2/result-analysis.md). ## **Long-term Estimated Values of Evaluation Metrics** On the A/B test results screen, you can not only check the current value of the evaluation metric but also check the "long-term estimated value of the evaluation metric." Try hovering your mouse over the numerical value of "Evaluation Metric" in the A/B test results table. This graph predicts the future goal rate from current data and visually shows which pattern is superior. #### **How to Read "Long-term Estimated Values of Evaluation Metrics"** ### **How to Read the Estimated Value Distribution Table** The further right the graph is, the better the performance (however, bounce rate is a negative metric, so the reverse is true. If you select "bounce rate" as an evaluation metric, a graph on the left is superior). The smaller the overlapping area of the lines of the two patterns, the more difference in performance between the two. The sharper the graph, the higher the certainty of the estimated value. Also, the more sample data, the sharper the line. ## **In-Depth Analysis with Drill-Down Analysis** A/B test results can yield insights that lead to next actions by drilling down from various angles. > **WARNING** **Note:** By utilizing [User Identification](/en/experience/campaign/create-new-2/result-analysis.md), you can also view data from any angle such as age or industry. ## **When Unable to Determine the Superior Pattern** If you cannot determine the superior pattern even after conducting an A/B test for a long period, you can deepen your analysis and gain insights using the following methods. ### 1. Drill-Down Analysis 🕵️‍♀️ Site visitors may have diverse needs. Drill-down analysis may yield insights such as the following: Differences in effectiveness for specific segments Effects by time of day or day of week Trends by device type 💡 Tip: Even if there is no difference overall, there may be effects in specific segments. ### 2. Check Heatmaps 🔥 By checking heatmaps by pattern: Visualize where user attention is concentrated Identify areas with high interaction Intuitively understand the effects of design elements ✅ Even without a difference in superiority, useful insights can be gained from differences in user behavior. ### 3. Change Goal Settings 🎯 If the final goal is too far from the experience, consider setting intermediate metrics: Example: Testing on a product list page of an e-commerce site Final goal "Purchase" → Intermediate goals "Product Detail Page Click" "Add to Cart" 🔧 Set a goal at an earlier point in the user journey and re-validate. ### 4. Statistical Significance and Practical Judgment 📊 When sample size is sufficient and test is stable If win rate is 80% or higher, practical judgment is possible > **DANGER** **Note:** Even without statistical significance, it is possible to choose one or the other as a business decision. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://helps.ptengine.com/en/experience/campaign/create-new-2/result-analysis.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.