A leader's guide to evaluating new technologies

If I can say one thing about technology over the last few years, it is that it is not boring. As much as we might like to focus on the challenges we already have in front of us, there always seems to be another new technology that we are forced to deal with. AI is just the latest. How do we separate the game-changers from the gimmicks?

In my career as a CTO, I have grappled with the emergence of cloud computing, the API economy, NoSQL, microservices, natural language processing, internet of things, containers, infrastructure as a service, and of course AI. And those were just some of the winners. The list of tech that did not live up to the hype is much longer. (I am looking at you, VR and blockchain.)

Whether you are a technology leader or in the C-suite, you simply do not have the luxury of ignoring emerging technologies. If you ignore or pass on a technology that turns out to be the real deal, you have put your team and your business behind the curve and at a competitive disadvantage. It is hard to win a race from behind. If you have embraced and invested in a technology that turns out to be a dud, you tied up massive amounts of time, labor, and capital on the wrong thing. Lost opportunity cost and lost actual cost. Neither is good.

AI is certainly the latest and loudest new kid on the block. Many leaders are unsure how to evaluate AI for their organization so that they can avoid these pitfalls. I have evaluated, rejected, or adopted a lot of technologies in my time as a CTO. This last year running my AI start-up has given me the opportunity to have conversations with many leadership teams confronting this dilemma. Should we adopt AI? Should we reject AI? Or should we sit tight and wait? How do we decide?

I want to share a tried and tested framework that I have used to clarify the new technology "should we or shouldn't we" conversation and confidently reach a decision. This analysis framework holds true whether you are evaluating AI or any other new innovation.

Ask yourself these four questions.

1. Is the technology real, or is it vapor?

Do you remember cold fusion? A groundbreaking revolution in energy production that was going to revolutionize our entire society. Except no one other than the folks who announced it could make it work. Do you remember Theranos? A revolution in healthcare testing — all the medical tests you need with a single drop of blood, delivered while you wait. Except it did not work.

Any technology you consider should obviously work, and work repeatably. You should be able to see a live demonstration. You should see it work for other businesses and for other organizations in your peer group. If you cannot see the technology deliver repeatable, demonstrable value with your own eyes, skip it. Or at least wait and see.

The good news for generative AI is that it already passes this test.

2. Does the technology solve a problem my organization actually has?

This is a big one, and where I have seen the most organizations get into trouble. You can be looking at an amazing and valid technology, and you can see it demonstrated to great effect. Awesome. But does it solve one of your organization's problems?

Blockchain for the enterprise is my favorite example of a solution looking for a problem. We saw so many people enchanted with the coolness of the blockchain, and then bend themselves into pretzels trying to find someplace, anyplace, in their business where it could be used.

The technology should have clear application to a real problem you and your business have today. If you cannot honestly point to an exact problem and then unambiguously explain how this technology helps to solve that problem, then walk away. "But it's so cool" is not a good enough reason.

This is where your evaluation of generative AI will start to diverge from other organizations and your peers. What real problems can you solve with AI? There are sweet spots where AI is the perfect solution. But is it perfect for your problem?

3. Does the technology solve my problem better than the solutions I am already using?

This is more subtle. There are so many technologies that pass the first two tests: they are very real, and they address and can solve one or more of your pressing problems. But now ask yourself, how are we solving it now? Will the new technology clearly improve the result for your specific use case?

When MongoDB, Couchbase, and Cassandra came along, many of my peers and I were enchanted by NoSQL. We wanted our slice of "big data" like the big kids. Everyone, me included, wanted to move our SQL databases to NoSQL. But guess what? For a huge percentage of use cases, good old relational databases remained, and still remain, the right choice.

Be sure you are choosing the best tool for the job, not just the newest tool for the job. Many times, the proven "legacy" tech is legacy for a reason. It works, and it has been honed over the years to do that job extremely well. Will the new technology significantly improve the result or add significantly more value? If the answer is no, save your time and money. You should pass.

(It is surprising to think about how many business problems can be effectively solved with Excel. I am only half kidding.)

Evaluating generative AI solutions gets very interesting here. There are many low-cost, high-value AI solutions that will pass this test. There are also high-cost, high-risk, high-reward AI implementations that may or may not.

4. Is the implementation worth the cost of redeploying valuable resources?

If we have gotten this far, we know your candidate technology delivers a real solution, it solves one or more of your actual problems, and it is a clear improvement on your current solution. But is it a dramatically better solution to your problem? Notice I said dramatically. If the value is close to the same or barely better than before, then the juice is not worth the squeeze.

Deploying new technology can be expensive. Not only do you have to lay out capital to purchase the new technology itself, but you will need to spend other precious resources to implement it. People, money, and time are finite resources. The value you expect to gain, whether it is added revenue or reduced costs, must solve your problem and exceed your expected spend for the labor, time, and opportunity costs to put it into production.

This is not to say that the expense of implementation is never worth it. Far from it. There can be a huge range of both costs and benefits. Your job is to be clear-eyed about the cost versus reward before you jump in with both feet.

So how did you do?

Be honest. If the new technology fails one or more of these tests, you can feel great about making the call for your business to pass and walk away. You have saved yourself and your organization lost dollars, lost time, and lost opportunities.

However, if the technology you are evaluating honestly passes all four tests, then you have a duty to implement it for your business. You require the benefits of the new tech to stay competitive and deliver maximum value and the best experience for your team and your customers. Remember that list of winners earlier? Those were clear cases of new technologies that solved huge problems and generated massive new value for those that took the plunge.

Is AI right for your org? It depends. I am an AI practitioner building AI applications, so I admit I am biased. But even I honestly think that AI is only the right solution for a fraction of what I have heard people promise AI will solve.

There is no one size fits all. The problems you need to solve, and the value of each solution, will differ widely between organizations. This framework is here to help you focus on your specific problems and your specific organization. Your mileage may vary.

What problems do you think AI will solve for your organization? And where is AI the wrong tool for the job?

Keep reading