Earlier this year, boberdoo released a revolutionary ping post bid experiment feature. Bid Experiments give boberdoo users the ability to run A/B tests on their bids for incoming pings. This gives users the opportunity to run experiments on adjusted profit margins or set different static bids in order to optimize their profits, maximize their amount of winning bids and optimize their entire ping post bidding process.
If you are still new to bid experiments, be sure to read our brief introduction. In this post, we outline a specific use case and show you exactly how it is handled within our Bid Experiments tool.
Great Mortgage Leads LLC (GML) buys and sells leads for mortgage loans with values over $100,000. The market price for such leads is currently between $15 and $25 per lead. GML has a hypothesis that leads who seek larger loans resell at a higher price. With GML's current bidding strategy, they are likely bidding too low for certain lead segments while overbidding for others.
To win more bids, GML would like to test a lower profit margin for higher value loans and a higher profit margin for lower value loans. After a quick consultation with the company’s data analyst, GML decides to set the breakpoint at $70k.
Note:
Setting up the experiment:
Choosing the segment to experiment with:
GML would like to see results as soon as possible, so when setting up the bid experiment they decide on the following:
In addition to increasing their market share, GML is willing to:
Figure 1: Bid experiment specifications and settings
Once the experiment is run to completion and the results are calculated, it is important to first look at two elements:
Taking both into account during the evaluation is critical because low confidence values indicate high fluctuations in data and high error rates. If the experiment's confidence level is low, the displayed results could be due to chance.
Figure 2: Version Performance Figure 3: Experiment Confidence Level
The performance of each version (current: Version A, adjusted: Version B) is defined by the average profit resulting from 1000 incoming pings that match the experiment segment. Figure 2 shows that while the current profit margin setting results in $1866 per 1000 incoming pings, decreasing the profit margin by 20% and, in turn, increasing the bid value, resulted in higher average profits by 17% ($2177).
Although it seems counterintuitive, decreasing the profit margin actually increased profits because it allowed for higher bids and resulted in more winning bids.
The adjusted version (B) outperformed the current version (A). However, before taking any actions and implementing the results it is important to check the confidence levels. Figure 3 shows that the experiment’s confidence level is 93%. From a statistical point of view, only confidence levels above 95% should be accepted. However, this threshold used in biochemistry and medical experiments could be lower in the lead generation industry depending on the risk-awareness of the user. Note that any value below 80% is barely better than a coin-toss in terms of the potential replication of results.
Figure 4: Additional Performance Charts
Figure 4 displays the additional charts provided to help choose between version A or B when confidence or performance levels are not convincing enough or if the user needs to drill down and check additional information such as Buying Rate and Volume, Revenue and Cost, or Daily and Hourly Averages.
Since every manager has a different attitude regarding market share, profit goals, or risk, it is very difficult to tell a business which option is the best. Sometimes it is obvious, but sometimes it only depends on the personality of the user and the stakes involved.
In this case, a risk-averse person would choose to run the experiment again with setting the minimum sample size from 500 to 1,000-2,000. This way, the confidence level can be increased to a statistically acceptable level and give the most accurate representation of the test.
However, by looking at the average profit per ping and then cross-checking the buying rate and buying volume to put the market share into context, it is obvious that version B truly performed much better. In this case, a user more willing to accept risk would probably accept the results (even with a 93% confidence level) and implement the adjustments keeping in mind that the overall average profit might be slightly lower or higher when running over a larger sample.
After it's first successful test, GML wants to run its second test to see if it can improve its bidding strategy for loan values under $70k. See the setup and results of GML's second test here!
boberdoo.com has been building technology for the lead generation industry since 2001. boberdoo's Bid Experiments tool is the latest feature that is helping lead companies as well as lead buyers grow and optimize their businesses. If you operate in a ping post vertical, we strongly encourage you to explore boberdoo's Bid Experiments. Give us a call at 800-776-5646 to learn more or fill out the form below.