Bid Experiment Case Study (Part 2) Increasing Profit Margin

In part 1 of our bid experiment case study, we tested exactly how decreasing the profit margin for bids on high value mortgage leads would affect profits for our test company Great Mortgage Leads LLC (GML). If you have not already read part 1 of our mortgage lead bid experiment case study, please start there. If you are not familiar with boberdoo's Bid Experiments tool, please read more here.

Bid Experiment Case Study (Part 2) - Mortgage Example

In our previous bid experiment, we learned that GML can increase both their buying rate as well as their average profit per 1,000 pings by decreasing their profit margin on high value mortgage leads. Next, GML wants to run a bid experiment on low value mortgage leads to determine whether or not they are over-bidding on these pings and taking a smaller profit margin than necessary.

Note:

1. 1. Decreasing the profit margin value will result in higher bid values, hence a greater chance to win bids.
   2. Increasing the profit margin value will result in lower bid values, hence a lower chance to win bids, but also a lower chance to overbid.

Testing Higher Profit Margin Values For Loan Values Lower Than $70k

Setting up the experiment:
Choosing the segment to experiment with:

1. • Lead Type: Mortgage
   • Loan Value: between $0 and $70,000
   • Loan Type: Purchase

Since supply and demand in the lead market highly fluctuate within the day and the week, this time GML wants to make sure that timing (eg. day of the week, or time of the day) does not directly influence the outcome and would rather wait longer for fully unbiased and statistically significant results. When setting up the experiment, GML decides on the following:

1. • Choose the same source as before (Source: TestMortgage)
   • Allocate a smaller part of incoming pings to make sure the experiment will run for at least a week (Ping Allocation: 10%)
   • Set the minimum sample size to a rather high amount for statistically significant results (Sample size: 5000 pings)

To find out whether they are overbidding for Mortgage type leads with loan values below $70k, GML slightly increases their profit margin (Adjustment: +5%), hoping that the somewhat smaller bid they offer increases profits while not considerably decreasing their buying rate.

Note: The profit margin set for TestMortgage source is currently $4.

Figure 1: Bid experiment specifications and settings

Bid Experiment Case Study - The Results

Once the experiment is run to completion and the results are calculated, it is important to first look at two elements:

1. 1. The actual performance of each test version (See Figure 2).
   2. The confidence level (See Figure 3).

Remember that both the actual performance and confidence level of the experiment play crucial roles when deciding which bidding strategy works better for your chosen segment. Taking both into account during the post-experiment evaluation is critical. While low confidence values could be caused by a number of factors – high fluctuation in data, small sample size, low conversion rate, small adjustment, etc. – it certainly indicates that the results could be due to chance.

Figure 2: Version Performance                                           Figure 3: Experiment Confidence Level

The performance of each version (current: Version A, adjusted: Version B) is defined by the average profit resulting from 1000 incoming pings that match the experiment segment. Figure 2 tells us that GML’s current profit margin settings (Version A) works better than an increased profit margin (Version B) for Purchase type loans with Values up to $70k.

Keeping the profit margin at $4 resulted in $852 average profit per 1000 matched pings for GML, while increasing its profit margin by 5% dropped the average profit by approximately 10% to only $765. The main reason of running the experiment was to find out if GML overbids in this segment and whether similar buying rates could be reached with smaller bid amounts. The result of the experiment reveals that overbidding is not the case here. But remember, don’t forget to check the confidence level!

Figure 3 shows that the experiment’s confidence level is 99%. This makes sense because GML purposely ran the experiment using a large sample size of 5000 total matched pings. From a statistical point of view this, this is a desirable confidence level and it suggests that results are significant and can be fully trusted.

Figure 4 provides information on Buying Rate and Volume, Revenue and Cost, or Daily and Hourly Averages.

Figure 4: Additional Performance Charts

Increasing the profit margin by 5% resulted in a lower buying rate (21% down to 18%), and hence losses in revenue ($11k down to $8.8k). The increased profit from sold leads unfortunately did not make up for the ones lost due to lower bids and competitiveness. Interestingly, comparing the average buying rate from the previous experiment (see Figure 4 in Part 1) it is quite obvious that GML does not perform as well, in general, in lead segments with loan values lower than $70k as it does in segments with loan values above $70k. The average buying rate in lead segments with loan values higher than $70k (~55%) is more than 2.5 times higher than in lead segments with loan values lower than $70k (~20%). The smaller market share tells us that GML’s bids in lead segments with loan values lower than $70k are much less competitive.

Bid Experiment Case Study - The Decision

With a well-defined hypothesis, a straightforward initial question, “What are the chances that I am overbidding and what happens if I bid less?” and confidence level this high (99%), deciding on the value of the profit margin is simple. Since version A ($4 profit margin) outperformed the increased profit margin in version B ($4+5% profit margin), we learned that GML certainly does not overbid in the chosen segment. The average buying rate immediately dropped due to slightly lower bids, meaning that the current profit margin and existing bid strategy works better.

There could be two ways to continue from here. GML could...

1. 1. Sit back and be happy about the fact that their gut feeling and existing bid strategy worked and do nothing.
   2. Continue experimenting and further optimize their bid strategy.

Additional Experiment Options

1. • GML’s market share is quite low in the segment with a buying rate varying around 20%. Instead of increasing the profit margin, GML could try further increasing the average bid instead by lowering its profit margin. It could result in winning more bids and hence higher revenue and profit even though the margin is lower.
   • There could be additional factors that define the market value of mortgage leads. Running experiments that combine the loan value with age, state, or gender could help further break down the optimal margins.