Notable Quotes
"We are also Customer Zero. And what that means is that we test the products, the features, the capabilities before they become available to customers."
"It's not that technology is not working, technology is doing its job, but it's really around the performance and latency. The customer expectation, the user expectation is different."
"Our users didn't want us to use that process and they just wanted to know what was the outcome of it. Just do the work that you're supposed to do and just tell me what I should do next and what's my next best action."
"There is no bad ideas, there's just ideas. And if you were to look at every single idea from the vantage point of feedback and learning, then you have not lost. You have only to gain."
"I think of CSAT as a lagging indicator, and CES is more specific to that particular job, an intent that we expect that the customer was here for."
Takeaways
Nondeterministic output needs its own evaluation layer, so ServiceNow built an LLM judge that scores against a golden data set before anything reaches a customer.
Users did not want the agentified version of the human process. They wanted the outcome and the next best action, not the steps.
The fix was a feedback loop that narrates what the AI is doing in the background, which buys the patience a long task needs.
