Earlier this year, boberdoo released a revolutionary ping post bid experiment feature. Bid Experiments give boberdoo users the ability to run A/B tests on their bids for incoming pings. This gives users the opportunity to run experiments on adjusted profit margins or set different static bids in order to optimize their profits, maximize their amount of winning bids and optimize their entire ping post bidding process.
If you are still new to bid experiments, be sure to read our brief introduction. In this post, we outline a specific use case and show you exactly how it is handled within our Bid Experiments tool.
Bid Experiment Case Study - Mortgage Example
Great Mortgage Leads LLC (GML) buys and sells leads for mortgage loans with values over $100,000. The market price for such leads is currently between $15 and $25 per lead. GML has a hypothesis that leads who seek larger loans resell at a higher price. With GML's current bidding strategy, they are likely bidding too low for certain lead segments while overbidding for others.
To win more bids, GML would like to test a lower profit margin for higher value loans and a higher profit margin for lower value loans. After a quick consultation with the company’s data analyst, GML decides to set the breakpoint at $70k.
Note:
-
- Decreasing the profit margin value will result in higher bid values, hence a greater chance to win bids.
- Increasing the profit margin value will result in lower bid values, hence a lower chance to win bids, but also a lower chance to overbid.
Testing Lower Profit Margin Values For Loan Values Higher Than $70k
Setting up the experiment:
Choosing the segment to experiment with:
-
- Lead Type: Mortgage
- Loan Value: between $70,000 and $100,000
- Loan Type: Purchase
GML would like to see results as soon as possible, so when setting up the bid experiment they decide on the following:
-
- Choose a source that provides significant daily ping traffic (Source: TestMortgage)
- Allocate a majority of incoming pings for the experiment (Ping Allocation: 80%) - Note: the remaining 20% of pings are not included in the control group nor the experiment group. They simply follow the existing bid settings.
- Set the minimum sample size to a rather small amount, hoping that results will be still statistically significant (Sample size: 500 pings)
In addition to increasing their market share, GML is willing to:
-
- Give up 1/5th of their per-lead profit to increase their market share in the segment (Adjustment: -20%) - Note: the profit margin set for TestMortgage source is currently $4.
Figure 1: Bid experiment specifications and settings
Bid Experiment Case Study - The Results
Once the experiment is run to completion and the results are calculated, it is important to first look at two elements:
-
- The actual performance of each test version (See Figure 2).
- The confidence level (See Figure 3).
Taking both into account during the evaluation is critical because low confidence values indicate high fluctuations in data and high error rates. If the experiment's confidence level is low, the displayed results could be due to chance.
Figure 2: Version Performance Figure 3: Experiment Confidence Level
The performance of each version (current: Version A, adjusted: Version B) is defined by the average profit resulting from 1000 incoming pings that match the experiment segment. Figure 2 shows that while the current profit margin setting results in $1866 per 1000 incoming pings, decreasing the profit margin by 20% and, in turn, increasing the bid value, resulted in higher average profits by 17% ($2177).
Although it seems counterintuitive, decreasing the profit margin actually increased profits because it allowed for higher bids and resulted in more winning bids.
The adjusted version (B) outperformed the current version (A). However, before taking any actions and implementing the results it is important to check the confidence levels. Figure 3 shows that the experiment’s confidence level is 93%. From a statistical point of view, only confidence levels above 95% should be accepted. However, this threshold used in biochemistry and medical experiments could be lower in the lead generation industry depending on the risk-awareness of the user. Note that any value below 80% is barely better than a coin-toss in terms of the potential replication of results.
Figure 4: Additional Performance Charts
Figure 4 displays the additional charts provided to help choose between version A or B when confidence or performance levels are not convincing enough or if the user needs to drill down and check additional information such as Buying Rate and Volume, Revenue and Cost, or Daily and Hourly Averages.
Bid Experiment Case Study - The Decision
Since every manager has a different attitude regarding market share, profit goals, or risk, it is very difficult to tell a business which option is the best. Sometimes it is obvious, but sometimes it only depends on the personality of the user and the stakes involved.
In this case, a risk-averse person would choose to run the experiment again with setting the minimum sample size from 500 to 1,000-2,000. This way, the confidence level can be increased to a statistically acceptable level and give the most accurate representation of the test.
However, by looking at the average profit per ping and then cross-checking the buying rate and buying volume to put the market share into context, it is obvious that version B truly performed much better. In this case, a user more willing to accept risk would probably accept the results (even with a 93% confidence level) and implement the adjustments keeping in mind that the overall average profit might be slightly lower or higher when running over a larger sample.
Next Time: Testing Higher Profit Margin Values For Loan Values Lower Than $70k
After it's first successful test, GML wants to run its second test to see if it can improve its bidding strategy for loan values under $70k. See the setup and results of GML's second test here!
boberdoo.com has been building technology for the lead generation industry since 2001. boberdoo's Bid Experiments tool is the latest feature that is helping lead companies as well as lead buyers grow and optimize their businesses. If you operate in a ping post vertical, we strongly encourage you to explore boberdoo's Bid Experiments. Give us a call at 800-776-5646 to learn more or fill out the form below.