The Reformist CTO’s Guide to Impact Intelligence

Example #1: A Customer Support Chatbot

What’s the contribution of an AI customer support chatbot to limiting call volume (while maintaining customer satisfaction) in your contact center?

Figure 3: Downstream Impact of an AI Chatbot

It is not enough anymore to assume success based on mere solution delivery. Or even the number of satisfactory chatbot sessions which Figure 3 calls virtual assistant capture. That’s proximate impact. It’s what the Lean Startup mantra of build-measure-learn aims for typically. However, downstream impact in the form of call savings is what really matters in this case. In general, proximate impact might not be a reliable leading indicator of downstream impact.

A chatbot might be a small initiative in the larger scheme, but small initiatives are a good place to exercise your impact intelligence muscle.

Example #2: Regulatory Compliance AI assistant

Consider a common workflow in regulatory compliance. A compliance analyst is assigned a case. They study the case, its relevant regulations and any recent changes to them. They then apply their expertise and arrive at a recommendation. A final decision is made after subjecting the recommendation to a number of reviews and approvals depending on the importance or severity of the case. The Time to Decision might be of the order of hours, days or even weeks depending on the case and its industry sector. Slow decisions could adversely affect the business. If it turns out that the analysts are the bottleneck, then perhaps it might help to develop an AI assistant (“Regu Nerd”) to interpret and apply the ever-changing regulations. Figure 4 shows the impact network for the initiative.

Figure 4: Impact Network for an AI Interpreter of Regulations

Its proximate impact may be reported in terms of the uptake of the assistant (e.g., prompts per analyst per week), but it is more meaningful to assess the time saved by analysts while processing a case. Any real business impact would arise from an improvement in Time to Decision. That’s downstream impact, and it would only come about if the assistant were effective and if the Time to initial recommendation were indeed the bottleneck in the first place.

Example #3: Email Marketing SaaS

Consider a SaaS business that offers an email marketing solution. Their revenue depends on new subscriptions and renewals. Renewal depends on how useful the solution is to their customers, among other factors like price competitiveness. Figure 5 shows the relevant section of their impact network.

Figure 5: Impact Network for an Email Marketing SaaS

The clearest sign of customer success is how much additional revenue a customer could make through the leads generated via the use of this solution. Therefore, the product team keeps adding functionality to improve engagement with emails. For instance, they might decide to personalize the timing of email dispatch as per the recipient’s historical behavior. The implementation uses behavioral heuristics from open/click logs to identify peak engagement windows per contact. This information is fed to their campaign scheduler. What do you think is the measure of success of this feature? If you limit it to Email Open Rate or Click Through Rate you could verify with an A/B test. But that would be proximate impact only.

Leverage Points

Drawing up an impact network is a common first step. It serves as a commonly understood visual, somewhat like the ubiquitous language of domain driven design. To improve impact intelligence, leaders must address the flaws in their organization’s idea-to-impact cycle ( Figure 6). Although it is displayed here as a sequence, iteration makes it a cycle.

Any of the segments of this cycle might be weak but the first (idea selection) and the last (impact measurement & iteration) are particularly relevant for impact intelligence. A lack of rigor here leads to the vicious cycle of spray-and-pray ( Figure 1). The segments in the middle are more in the realm of execution or delivery. They contribute more to impact than to impact intelligence.

Figure 6: Leverage Points in the Idea to Impact Cycle

In systems thinking, leverage points are strategic intervention points within a system where a small shift in one element can produce significant changes in the overall system behavior. Figure 6 highlights the two leverage points for impact intelligence: idea selection and impact measurement. However, these two segments typically fall under the remit of business leaders, business relationship managers, or CPOs (Product). On the other hand, you—a tech CXO—are the one under productivity pressure resulting from poor business impact. How might you introduce rigor here?

In theory, you could try talking to the leaders responsible for idea selection and impact measurement. But if they were willing and able, they’d have likely spotted and addressed the problem themselves. The typical Classic Enterprise is not free of politics. Having this conversation in such a place might only result in polite reassurances and nudges not to worry about it as a tech CXO.

This situation is common in places that have grown Product and Engineering as separate functions with their own CXOs or senior vice presidents. Smaller or younger companies have the opportunity to avoid growing into this dysfunction. But you might be in a company that is well past this orgdesign decision.