Content A/B testing lets you run controlled experiments on your product copy. You choose two variants of a title or description, assign them to a channel, and Vendably handles traffic splitting, performance tracking, and significance testing. When the configured confidence threshold is reached, Vendably declares a winner and can publish it automatically if you choose.
What you can test
Each test covers a single content type on a single channel:
- Two title variants for one product on one channel, or
- Two description variants for one product on one channel.
Tests are channel-specific. A title test on Google Shopping runs independently of any test on the same product for Meta or another channel.
Setting up a test
- Go to Content > A/B Tests and choose Create test.
- Select the product you want to test.
- Choose the content type: Title or Description.
- Choose the channel the test will run on.
- Select Variant A and Variant B from the existing approved variants for that product, channel, and content type.
- Set the traffic split (typically 50/50).
- Set the duration: the maximum number of days the test will run before it is automatically closed.
- Set the confidence threshold (typically 95%). This is the Bayesian probability that the leading variant is genuinely better before a winner is declared.
- Choose whether to enable auto-promote: if on, Vendably publishes the winning variant automatically when a winner is declared.
- Choose Start test.
How Vendably runs the test
Once a test is running, Vendably:
- Assigns each visitor consistently to one variant for the duration of the test, so the same visitor always sees the same variant.
- Counts impressions, clicks, and conversions per variant.
- Runs a daily Bayesian significance test against the configured confidence threshold.
- Declares a winner when one variant reaches the threshold, or closes the test at the end of the configured duration if no winner has been reached.
- If auto-promote is enabled, publishes the winning variant automatically on declaration.
Test lifecycle
A test moves through these states:
- Draft: created but not yet started. You can edit the configuration.
- Running: live and collecting data. You cannot edit a running test.
- Completed: a winner was declared or the duration elapsed. The result is kept for reference.
- Cancelled: manually stopped before completion. A cancelled test keeps its collected data.
To stop a running test, cancel it first. You cannot delete a running test.
A note on confidence
The confidence figure Vendably displays is a Bayesian probability: the probability that the declared winner is genuinely the better-performing variant, not a p-value. A test needs live performance data flowing in from the channel to reach significance. Products with lower traffic take longer to accumulate enough data for a confident result.
Troubleshooting
Test not reaching a winner: the test needs sufficient traffic and time. Low-traffic products accumulate data slowly, and the duration may elapse before a winner is declared. Consider extending the duration or accepting an inconclusive result and reviewing the raw performance data manually.
Cannot delete a test: running tests must be cancelled before they can be deleted. Cancel the test first, then delete it.
Winner not published after declaration: if auto-promote was off when the test completed, the winning variant is not published automatically. Go to the completed test, review the result, and publish the winning variant manually.
Variant not available to select: only approved variants appear in the variant selector. If the variant you want to test is not listed, confirm it has been approved in the content detail view for that product and channel.