10 Executives share what they really think about experimentation

Published on the Kameleoon website on February 26, 2024.


The relationship between experimentation teams and the C-suite is often presented as “us” and “them” — portraying executives as lacking interest and appreciation for experimentation. But is this a true reflection of today's C-suite? We decided to speak to ten company executives to get their take on experimentation and explore how they view the practice.

Is experimentation valuable, or is it a waste of time and resources? 

We know the way to most executives' hearts is to talk about revenue and the bottom line. So, do the executives we spoke to consider experimentation an overall net gain or a resource drain? And, if they see it as valuable, can they move past the money and recognize the non-financial benefits, too?

While it’s often hard to attribute and forecast revenue gains from individual experiments, there is often an impact on KPIs. Typically, gains can be categorized into two buckets: exponential gains and small incremental growth, as Michelle recognizes:

There is absolutely value in experimentation, both for optimization and innovation. Experimentation is how we can build exponential growth by testing many variables and building incremental gains on different aspects of a hypothesis. That is understanding what works and doesn’t and building upon what works best.

Michelle Killebrew

SVP, Chief Marketing Officer, Americas at NTT Ltd.

EXPERIMENTATION PROTECTS REVENUE THROUGH EFFICIENCY & BETTER DECISIONS

Several executives saw experimentation as a way to preserve resources by helping teams make better decisions and saving time from being spent on poor choices, as Charity discusses.

Experimentation is an optimal approach to resource preservation, regardless of outcome. For example, experimentation allows you to run tests to determine where budgets would be most wisely spent. Instead of going all in, testing allows you to get more value out of your dollar – whether it ends up being the “right path” or not. That’s because, even when the initiative does not turn out as expected or planned, a lesson is always learned. Think of experimentation as an investment in growth and learning.

Charity Hudnall

CMO at Vagaro

Another interesting framing shared by Joosep is how experimentation can help you “protect revenue."

Experimentation is definitely valuable. It's a key part of learning what works and doesn't work in e-commerce. Experimentation can lead to improved decision-making and more efficient use of resources. By testing different strategies or features, you can discover what best engages your customers and drives sales, ultimately protecting your revenue and providing valuable insights​​.

Joosep Seitam

Co-Founder at Icecartel

EXPERIMENTATION DRIVES INNOVATION

Several leaders see experimentation as being critical for driving innovation, whether that’s overall business innovation or specifically within product development or marketing. It’s worth noting that they see the innovations derived from experimentation as an overall gain, as Campbell says.

For me, experimentation embraces innovation in product development and marketing, which provides companies with excellent opportunities to discover new features and benefits. Any expense can be overly compensated with a properly planned and executed experiment.

Campbell Tourgis

Executive Vice President & Chief Operating Office

Andrew explains how he sees experimentation as a method for “intelligent investment."

Experimentation is at the core of any innovative business model. It allows us to test market responsiveness, gather data, and adjust strategies to maximize effectiveness. At Formen, we've found incredible value in A/B testing - examining everything from product variants to our marketing messaging. Without it, we'd be flying blind, relying on conjecture instead of concrete data. It's less about expenditure and more about intelligent investing — of time, resources, and effort.

Andrew Grella

CEO of Formen Inc.

Setting the company direction when it comes to innovation and growth is a hefty decision, which is likely why some executives see experimentation as a strategic approach to these areas, as Matt describes.

As a CTO, I strongly believe in the value of experimentation, particularly regarding A/B testing, product feature testing, and conversion optimization. I don't see it as a waste of time, especially if you know how and what you are looking to ascertain from the data. I'd more likely describe it as a strategic approach to innovation and growth. It's the simplest way to gain insights into customer preferences, behavior, and trends, allowing you to refine strategies and site layout accordingly.

Matt Aird

Chief Technology Officer at Custom Neon

On the flip side, if you aren’t using experimentation to innovate, your company risks stagnating. This innovation can be driven not only in marketing or product development but processes as well, as Tim explains.

In my view, experimentation is not a waste but an investment. It's the engine that drives innovation and improvement. Without experimentation, we risk stagnation and miss opportunities for optimization and growth. It's through trying new things, testing assumptions, and learning from both successes and failures that we can refine our products and processes.

Tim Green

COO of TeamUp

YES, BUT DATA VALIDITY DETERMINES THE VALUE

The overwhelming response is positive about the benefits of experimentation, but a couple of executives rightly caveat their views based on the quality of experimentation. Evaldas shares why this is the case with a great example.

Certainly, experiments can be valuable if they are performed by skilled individuals who can demonstrate the validity of the data. Without statistical data and knowledgeable people to analyze it, the efforts will be a waste of time. It also depends on the type of testing. If we want to test whether a discount code boosts our sales, I believe that is a waste of time because the results of the testing are easily predictable. However, if we want to test different discount options and do A/B testing, then I see the value. It is really dangerous to lose focus and experiment on everything, even on common knowledge.

Evaldas Mockus

VP of Growth at Omnisend

A/B testing isn’t the only method of experimentation, and executives must understand that this method isn’t always suitable. It’s also crucial that teams using experimentation do so with a rigorous process and the right impetus. Aleksandar explains why this is the case.

A/B testing is just one form of experimentation. But what often happens is people use A/B testing as a crutch. When you don't know your customer or can't internally agree on something, you hope that something as impartial as data will show who's right and who's wrong. That rarely happens. Tests require a stable environment, a lot of time, and a huge sample - everything startups don't have.

But if you're an established company, you can't make changes as you used to. Every big change that upsets the status quo is a risk that can affect the bottom line, so caution and formal testing is paramount. In the end, experimentation is important, but the form depends on where your company is. If you're a startup, it's better to iterate quickly and listen to feedback, and if you're an established company, you can be more data-focused.

Aleksandar Olic

VP of Marketing at CAKE.com

What’s your initial reaction to feature testing & website experimentation?

All of the executives we spoke to appreciate the value of experimentation for various reasons, but we wanted to delve a little deeper into their opinions. We asked those interviewed about their immediate thoughts when they heard phrases like ‘feature testing’ or ‘website experimentation.’ Their responses show an exciting range of perspectives and go deeper into why they, as leaders, hold certain opinions about experimentation. 

TOOLS FOR GROWTH

Several executives we spoke with focus on how web and feature experimentation are tools or processes for growth. Joosep shared his view.

It's important to think of them as tools for growth and improvement. These processes involve trying out new ideas on your website to see how changes can impact user behavior and conversions. It's about making informed decisions to enhance the user experience and optimize your website for better performance​​.

Joosep Seitam

Co-Founder at Icecartel

This view was shared by Matt, too. 

My immediate thoughts revolve around opportunities for improvement and innovation. These practices enable us to test hypotheses, iterate quickly, and ultimately enhance the user experience, increasing conversions and revenue for our business.

Matt Aird

Chief Technology Officer at Custom Neon®

A USER-CENTRIC APPROACH TO PRODUCT DEVELOPMENT & MARKETING

For marketing leaders, web or feature testing might conjure up thoughts around some of their main challenges, such as understanding the market and creating strategies that truly resonate with the right audiences. However, the benefits of experimentation go beyond just creating better products or marketing, as Charity shares.

My immediate thoughts are about the invaluable insights gained by gauging the market response to an offering. This feedback will lay the foundation for informed decision-making, as it should influence your go-to-market strategy.

Rolling out your new features to a smaller cohort is beneficial because it will uncover ways to improve them and how the market will actually implement them in real-life scenarios. It will even help you determine how to communicate the offering to your audience. This approach helps avoid a common risk companies run when heavily investing in new features that fall short, often due to the most simplistic components.

Absorbing real-time insights from the beta-testing group not only benefits your company but also the end user because, this way, the product is, in essence, built for and by them. This user-centric method helps build trust with your clients, fosters loyalty, and instantly creates a group of advocates for your brand.

Charity Hudnall

CMO at Vagaro

INTEGRAL TO PRODUCT DEVELOPMENT & CONTINUOUS IMPROVEMENT

In a world of agile product development, one-off rounds of customer research don’t cut it. Instead, constant feedback loops are needed to inform the product development process, with feature experimentation offering a method for achieving this for leaders like Tim. 

I think of opportunity. These activities are chances to learn more about our users, validate our ideas, and make data-driven decisions. They're integral to the product development cycle, helping us continually improve our offerings and deliver value to our customers.

Tim Green

COO of TeamUp

Andrew shares a similar reaction. 

Phrases like 'feature testing' and 'web experimentation' instantly evoke a sense of exciting exploration in me. They signify growth, dynamism, and a continuous discovery process. These processes are key to us — they determine the effectiveness of new products, user interfaces, or even a simple website alteration. It's about learning, reiterating, and making better decisions each time.

Andrew Grella

CEO & Founder of Formen Inc.

OVERLY MYOPIC

For leaders who have a good understanding of experimentation, terms such as web testing may be perceived as a simplified view of what experimentation practices truly encompass, as Michelle felt.

Both are important to understand engagement and customer value. However, when I hear those terms, I often feel that the testing in those areas is overly myopic.

As an over-simplified example, “web experimentation” evokes the concept of button color testing in my mind, which, of course, may not always be the case. Great “web experimentation” can deliver much more, such as buyer journey experimentation to deliver on engagement, assist in providing visitors with what they are seeking, and improve performance against business objectives. Feature testing is an absolute must before rolling product enhancements out to general availability.

Michelle Killebrew

SVP, Chief Marketing Officer, Americas at NTT Ltd

Results of experimentation 

Finally, we asked executives to share the results of their experimentation efforts, whether good or bad. Below are some first-hand examples these leaders shared, which impacted their overall view of experimentation. So what struck them as a successful result or a noteworthy failure?   

LEARNINGS THAT DRIVE DEMAND 

Experimentation can impact various metrics and improve a whole host of channels, experiences, and strategies. You can use the practice if you can form a hypothesis and apply an experimental method. What stood out for Michelle is how learnings from one test can apply to other business areas and the compounding nature of iterative tests.

Many years ago, I did the marketing for an analytics company, and so I was always using analytics to test and iterate. First, within the campaign, then applying those learnings from campaign to campaign, and then beyond into new GTM strategies as the business was acquired and integrated into larger portfolio offerings.

The learnings were significant. We learned that we could serve a multi-touch interaction to continue the customer experience while people were in consumption mode, but early tests demonstrated pop-up experiences were blocked. We tested until we got the experience right, and the demand conversion metrics went up and up with every iteration! The output of this testing methodology was so successful, that the vendor we used to deploy it productized it, I learned years later.

Michelle Killebrew

SVP, Chief Marketing Officer - Americas at NTT Ltd

ECOMMERCE CHECKOUT SUCCESS THAT GROWS REVENUE 

While many executives focus on revenue gains, it’s essential to recognize the savings from avoiding “mistakes.” For Matt, the combination of the two makes for a notable case study.

One significant success story of experimentation involves our A/B testing initiatives on the checkout process. By experimenting with different checkout flows, payment options, and page inclusions such as cart item thumbnails and customer reviews, we increased conversion rates by 22%, resulting in substantial revenue growth.

Even when experimentation doesn't yield the desired results, it still significantly informs our future strategies and prevents unnecessary mistakes.

Matt Aird

Chief Technology Officer at Custom Neon

Andrew shares a similar story that recognizes the value of both winning and losing tests to the business. 

Initially, our anti-aging creams were not seeing the expected sales. Instead of discontinuing the product line, we experimented with various marketing approaches, audience segments, and packaging designs. It wasn't until we connected the product to a demo video and specific lifestyle message that we started seeing a significant increase in sales. On the contrary, there have been instances where we discontinued certain products post-experimentation due to lackluster demand, recognizing that not all experiments lead to success. Still, each one is an opportunity to learn and evolve.

Andrew Grella

CEO & Founder of Formen Inc.

A TEST-AND-LEARN APPROACH TO PROBLEMS 

As already discussed, experimentation can take a number of different forms and apply in a host of situations. For some executives, an experimental mindset can apply to any problem or challenge, as Megan shares in this interesting example; 

Sometimes, I will set up experiments simply to invalidate potential options. This can be difficult with people variables, which are generally qualitative and not easily controlled. For example, we had a trend of employees quitting early in a certain role. Despite our very in-depth SOPs, the most common feedback for quitting was "lack of training."

To test this, we offered a full-time "trainer" to read the SOPs to the new hires and take them through their role. When this person quit, they said they were trained very well but made other excuses for quitting. Finally, we narrowed down the variables and realized it was a reason not given: lack of community. The type of people applying for this role were often extroverted, so we hypothesized that they might not succeed because of the lack of social interaction. We restructured the role to include them in a community so they weren’t working solo all day.

We need more data to conclude whether our hypothesis was correct, but that would require this person to leave the position, and so far, they're very happy and well-supported within the team.

Megan Cox

CEO of Genie Supply

WHEN EXPERIMENTATION GOES WRONG

It’s not all rosy success stories. As some of the executives mentioned already, experimentation relies on valid data and robust processes. If those aren’t in place due to a lack of tooling or knowledge gaps, it can lead to teams making poor decisions based on faulty assumptions. 

I’ve seen A/B testing on performance landing pages without tracking through the funnel, which has led to increased top-of-funnel conversion (by appealing to a broader audience) but reduced revenue conversion due to more volume to sift through and less precise value to the core target audience.

Michelle Killebrew

SVP, Chief Marketing Officer - Americas at NTT Ltd

Sometimes, it’s not the experiment itself but the lack of one which proves a valuable lesson. Evaldas shares his story.

We skipped the experimentation phase and changed our website's main copy and layout, which ultimately destroyed our product registration metrics. We noticed that companies in our niche were updating their home pages with fancy design layouts.

Without any testing, we decided to follow the competition and update our website in the same direction. After one week, it became evident that people didn't understand our proposal, and we were losing product registrations as a result.

Evaldas Mockus

VP of Growth at Omnisend

Experimentation: A valuable asset  

This article debunks the trope that C-suite executives are dismissive of experimentation. The discussions above reveal an appreciation for experimentation as a driver of growth, innovation, and informed decision-making, as well as avoiding the cost of poor decisions. 


While the financial advantages of experimentation were important, it’s clear that value is also derived from testing assumptions, learning from failures, and being user-centric. 


Finally, leaders recognized the importance of robust tooling, processes, and valid data in order to run experiments that can be relied upon. 


Thanks to the following executives for sharing their thoughts on this topic.

 

If you’d like to keep up-to-date with all the industry's top experimentation-related news, sign up for our newsletter or find industry leaders to follow in our Expert Directory. 

Info
Written by Michelle Killebrew On In Tagged , , ,