How LSEG Manages the Total Cost of AI
Ken Beinert, Chief Technology and Operating officer at LSEG, joins J.R. Storment to explain how one of the world’s largest financial data providers plans for the total cost of AI, measures cost per task, and decides who owns the spend.
Episode summary
LSEG plans for the full AI estate, not just tokens. Ken Beinert explains cost per task, eval-driven model selection, and why the product P&L owner owns the AI spend.
Episode Highlights
LSEG budgets for RAG infrastructure, data preparation, guardrails, evals, and observability, not just tokens.
00:43Cost per task is the core unit, with evals for accuracy, quality, latency, and cost on each task.
07:42Models are chosen by in-situ evals before launch and re-evaluated as new models arrive.
02:38The product P&L owner owns the AI spend, backed by central eval and cost controls.
12:34Tokens are only part of the AI investment
LSEG plans upfront for RAG infrastructure, data cleansing, real-time guardrails, recurring evals, and observability, so it can manage cost and quality over time.
Unit economics improve while total cost can rise
Newer models and tuning based on telemetry lower the cost of a solution over time, but overall spend can still climb as customers and colleagues use it more.
Evals, not a rule book, decide model choice
Models change too fast for fixed rules. LSEG evaluates each model in situ before launch and keeps re-evaluating as new models arrive.
In AI, variability means customer experience
Cloud FinOps managed variability in consumption and demand. In AI, the model you choose changes the user experience and the value delivered, not only the cost.
The person who owns the P&L for the product also owns the AI spend. That aligns incentives and responsibility.
We agreed we can call our main files AGENTS.md. If we can do that, then maybe we can build a standard for agent runtime.
Key takeaways
- LSEG plans for the total cost of AI, including RAG infrastructure, data preparation, real-time guardrails, recurring evals, and observability, not just token spend.
- Cost per task is the core unit economic. Each typical task gets evals for accuracy, quality, latency, experience, and cost.
- Model selection runs on in-situ evals rather than a fixed rule book, and LSEG uses a mix of frontier, open-weight, smaller, and custom models.
- The product P&L owner owns the AI spend, while enterprise AI and finance jointly run central platforms for evals and cost controls.
- AI economics extends FinOps: model choice changes the customer experience, not only the cost, and running agents safely at scale may need a standard for agent runtime.
How does one of the world’s largest financial data providers manage AI economics? Ken Beinert, chief technology and operating officer at LSEG, joins J.R. Storment, executive director of the Tokenomics Foundation, to explain why LSEG plans for the total cost of AI rather than token spend alone. He describes cost per task as LSEG’s core unit economic, explains why model selection depends on in-situ evals rather than a fixed rule book, and covers how LSEG mixes frontier, open-weight, and custom models.
The conversation also covers accountability, governance, and the future of AI tokenomics. Ken lays out why the product P&L owner owns the AI spend, how governance starts at intake, and why running agents safely at scale may need a standard for agent runtime. He also explains how AI economics differs from FinOps: model choice changes the customer experience, not only the cost.
Produced by The Stack for the Tokenomics Foundation. Read The Stack’s article on this interview.
State of AI Tokenomics data: tokeneconomics.com/state-of-tokenomics/
00:00LSEG’s Three Pillars of AI Transformation
We think about our AI transformation strategy really in three pillars. The first is AI-ready data, where we’re focused on making sure that our customers have access to our trusted, high-quality data in APIs and formats that are extremely accessible to AI. The second is transformative products, where we embed AI directly into our existing products and evolve those products so that they’re truly AI native. And the third, of course, is what we call intelligent enterprise, where we’re focused on enabling our internal colleagues, our employees, with the right tools and practices to really transform how they work.
00:43Total Cost of AI: What’s in the AI Estate
When we think about economics, we often say that it’s about the total cost of AI, because models and tokens are often a quarter of the spend. How do you think about that entire AI estate? What are you including when we’re talking about AI? Is that on-prem? Is it storage and compute? Is it labor? What’s included in that equation for you?
When we think about the investment for an AI solution, I think really about it in two different phases. The first, of course, is preparation for launch of an AI solution, where we want to focus first on the problem that we’re trying to solve. Secondly, on the success metrics that we’ll use to measure whether or not we have achieved our desired outcomes. The spend, the quality, the user experience are all part of defining the problem and defining the right solution for that problem.
When we invest upfront in an AI solution, we’re not just planning for token spend. We’re also planning for other infrastructure, like RAG infrastructure. We’re planning for data cleansing and preparation to make sure that the data is ready for AI solutions. And then, of course, we’re also planning ahead for real-time guardrails, recurring evals, and comprehensive observability and telemetry so that we can manage both the cost and the quality of various solutions over time.
02:03Proving AI Value with Customer Metrics
You mentioned KPIs, and one of the survey points that came out of the State of AI Tokenomics was how much organizations struggle to prove and show value from their AI investments. How are you thinking about doing that? How do you prove and measure and show value on your AI investments?
The vast majority of the metrics that we focus on are really customer success metrics. Our goal at LSEG is to deliver great products that our customers find value in. And so measuring the value from a customer’s perspective is paramount to us.
02:38Model Routing and Evals in Situ
So how are you thinking about providing a mix of the right AI, the right model choices, to developers? One of the things you mentioned was model routing. That was also in the survey data, showing that companies were increasingly investing in building and buying different model options, and then routing and governance layers on top. Are you looking to automate that process? Is it internal education for people about which models to use? How do you ensure that they’re using the right tool for the job?
I think it is a little early to say we can put a rule book or automation in place to guide developers in selecting a model for their solution, because it is evolving so quickly. One of the factors that I think is most key to model selection is doing comprehensive evals of a model in situ for each solution before we launch it. And then, of course, continuing to monitor those evals over time and determining, as new models and new solutions become available, whether or not we get better outcomes, whether in user experience, latency, or cost optimization, from changing the models within the solutions we’re evaluating.
03:56Frontier, Open-Weight, and Small Models
How are you thinking about the portfolio mix? We’ve got frontier models that everybody’s using, we’ve got open-weight models, we’ve got SLMs. Are you using a mix of those? What does that look like?
We are using a mix of them. You can imagine quite a lot of our customer-facing solutions and our more advanced solutions do leverage frontier models. But increasingly we’re finding that many of our solutions can be solved for with non-frontier models, smaller models, in some cases custom models. I do expect that our use of models we don’t currently consider part of our solutions is going to increase. That’s clearly because the rate of advancement, the types of models, the model families that are available are just surging right now. And it seems like a trend that’s going to continue throughout this year and next.
Do you expect to see a shift in the classes you’re using, more toward open weights, or more toward frontier, or more toward those specialized models?
Well, we don’t really think about it as closed weight or open weight. Clearly we’re beginning to see open-weight models that really compete successfully with closed-weight models, both at the frontier level and at the smaller model level. And we do expect to use more of them over time.
05:23Pricing AI When Costs Keep Shifting
Let’s talk a little about how the variability of AI models is cascading into potential pricing model changes for you, or maybe just margin management. In the historic CapEx-heavy world, where you’ve got data centers, or even in cloud, there was more predictability on how much you’re going to spend. With AI usage, both for developer productivity and for mission-critical AI, there’s a lot less predictability. So how are you seeing that variability cascade into how you price to your customers?
The most important part of pricing AI-based solutions, both for our customers and for our own internal usage, is to make sure we understand the business case upfront and that we do very thorough evaluation for quality and cost ahead of time, so we understand the unit economics. The second most important part is to bake the right assumptions into those unit economics so that we have the opportunity to continue to make them successful over time.
Like the rest of the industry, what we’re seeing is that prices change very rapidly in multiple dimensions for potential AI solutions. But at the same time, we’re often seeing that unit economics for an existing solution can improve over time, especially as newer models become available or as we tune our solutions based on our own telemetry and ongoing evals of those products. Overall cost, on the other hand, certainly can increase as we see customers and colleagues using those same products even more.
07:04Tech and Finance Alignment
On overall cost, have the conversations changed with your finance partners around managing more quickly changing usage? Every week there are changes to the way models are priced. How has that tech and finance conversation shifted with all that variability?
I don’t think it has shifted that much, since the focus on unit economics, starting from customer value, and understanding the full product before we launch it have been central to the conversation between product, operations, engineering, and finance since we began these solutions.
07:42Cost Per Task as the Unit Economic
You mentioned unit economics a few times. Is that a granular unit economic, like our cost to serve this customer, or is it a bigger concept? What do the unit economics look like for you?
The unit economics really can vary based on a specific use case. But for me, the example that comes most immediately to mind is cost per task, where what we really want to understand are the typical tasks that a user uses a given AI solution for, so that we can develop the right evals for each of those tasks to measure accuracy, quality, latency, feel and experience, as well as cost.
08:24Trading Off Cost, Latency, and Quality
In AI economics, we talk a lot about cost and value, but you’ve talked about speed and latency and quality. How do you enable those trade-offs? Are there certain jobs or use cases where you heavily index toward one versus another?
Within LSEG, we have a number of use cases where we use AI as an input to our products. We have data preparation and data cleansing work that runs perhaps overnight and essentially is not latency sensitive, not part of the direct user experience. There, clearly, we have an even greater range of models that we can select from, as opposed to use cases where a customer is engaging directly in an experience with the model.
09:12Running Agents Safely at Scale
Right now, the challenges seem to be around visibility and value and managing all these different models. If we had this conversation in 12 months, what do you think the challenges will be? Where do you think this evolving AI economics landscape will end up?
Wow, this is really tough. It’s a big question, but… I clearly do not know. It would just be silly for me to attempt to answer this question, right? If we’re talking again 12 months from now, one problem that I’d love to see us as an industry solve is the challenge of running agents safely at scale.
Today, when we select an agent platform, we’re selecting solutions for identity management, entitlements, containerization, runtime infrastructure, sandboxes, and, of course, observability and telemetry. Each of the solutions we could use has its own differences, its own potential gaps, which can make it quite hard to switch between platforms, between models, and between providers. If we solve this problem at scale, it will really help the industry as a whole to deliver better AI solutions.
10:27A Standard for Agent Runtime
There are a lot of bespoke options in there, a lot of choices and cardinality, it sounds like. So are you pushing for standards? What would be the way to accomplish that?
I do think there’s significant opportunity for standardization here. After all, it was just this week or so that we agreed we can call our main files AGENTS.md. If we can do that, then maybe we can build a standard for agent runtime.
10:57AI Governance Starts at Intake
So what about AI governance? How are you thinking about it?
Well, at LSEG, we’re focused on ensuring that we use AI responsibly across the group. First and foremost, that means educating our colleagues on how to use AI responsibly: what the best practices are, what the right tools are to use, and, of course, how they should build and deliver AI solutions both to our customers and for internal use.
When it comes to governing AI, we believe it starts with intake, where we understand what use cases we want to launch before we launch them. Then, as we’re building and developing those use cases, we look at them through the right architectural lens to ensure that we’re launching what we want to launch, and that we operate it the way we want to operate it, before we do so. Using AI responsibly goes beyond developing AI solutions that are custom to LSEG. It extends to using AI solutions from partners, from vendors, and from other companies responsibly. There, we’re focused on ensuring that we select the best safe solutions for our colleagues to use.
12:13Open Source in the Mix
What about open source in that mix?
A lot of the AI solutions that we use at LSEG actually are open source. And, of course, we’re focused on vetting those open source solutions for safety, for security, and for responsible use as much as if they were a vendor product that we paid for.
12:34Who Owns AI Spend
How are you driving accountability for AI spend in the firm?
Well, first and foremost, we believe that the person who owns the P&L for the product also owns the AI spend. That aligns incentives and responsibility. Secondly, we believe in providing central platforms and controls around AI, both for evals and cost controls, as a joint venture between enterprise AI and finance. The reason for that is that while the accountability needs to remain with the P&L owner, centrally we have the opportunity to stay abreast of best practices and transformative AI solutions that can be rolled out across these use cases.
What we’re focused on is ensuring that we have holistic product strategy across the group, and that the best practices we have in building and operating AI solutions are embedded into each of these products, both when the product is built and as best practices and the technology evolve. That means sharing information and making sure all of our product owners understand the opportunities we have as a group to optimize those solutions for user experience, operational effectiveness, and certainly cost.
13:50AI Economics vs. Cloud FinOps
I see AI economics really, in some ways, as an evolution of cloud FinOps. Clearly it is more complex. The pace of change in model selection is much greater than what we see with new types of compute, storage, and network infrastructure. There’s much more variability in cost, and there are just so many different dimensions of cost in each solution: for example, token caching, different types of models, and fine-tuning of existing models.
Caching and fine-tuning are things that FinOps didn’t have to deal with; they’re more technical. That’s one of the differences we’re seeing. Tokenomics is getting into more technical areas, KV cache management being a great example, but it’s also cascading way over into changing models, margins, and pricing. So do you think those two things will be the same over time, or do you expect them to be different?
I think the practices we have in place for FinOps today, we’re going to continue to have. Those are the platforms on which we’re going to run our AI solutions. But we’re going to raise the standard for how we do financial management so that it is more real time, and we add an increased element of managing for user experience. What we find in AI, unlike compute, unlike storage, is that there’s a much greater element of variable user experience based on how we select the components of our AI tech stacks.
15:30Variability of Customer Experience
That’s really interesting, because when we talk about variability historically in FinOps, which is becoming business as usual for a lot of companies, the variability is in the technology, the consumption, the demand. The variability you’re talking about in AI economics is a variability of customer experience and outcomes. That’s maybe one of the best descriptions of the difference I’ve heard. If you carry that forward to model selection, or developers choosing which AI to use, not only will it affect their cost, which is a very FinOps concept, but choosing the right model will affect the customer value that comes out the back end.
That’s absolutely right.
16:24What’s Next for AI
Honestly, the thing that I’m most excited about is understanding the real potential of AI and putting it in place within our organization. It seems like every day there are new insights into practices that we can apply. And almost every day we’re seeing new models, not just new frontier models with capabilities beyond what we expected, but entirely new model paradigms that enable use cases that literally did not seem feasible even a week ago.
Tokens are only part of the AI investment
LSEG plans upfront for RAG infrastructure, data cleansing, real-time guardrails, recurring evals, and observability, so it can manage cost and quality over time.
Total cost of AIUnit economics improve while total cost can rise
Newer models and tuning based on telemetry lower the cost of a solution over time, but overall spend can still climb as customers and colleagues use it more.
PricingEvals, not a rule book, decide model choice
Models change too fast for fixed rules. LSEG evaluates each model in situ before launch and keeps re-evaluating as new models arrive.
Model selectionIn AI, variability means customer experience
Cloud FinOps managed variability in consumption and demand. In AI, the model you choose changes the user experience and the value delivered, not only the cost.
AI vs. FinOpsThe person who owns the P&L for the product also owns the AI spend. That aligns incentives and responsibility.
We agreed we can call our main files AGENTS.md. If we can do that, then maybe we can build a standard for agent runtime.