Digits Presented Lessons Learned from Agents in Production
Key Takeaways from AI in Production 2025: Lessons from Asheville
Last Friday, I had the opportunity to attend AI in Production 2025 in Asheville, NC - a focused gathering that brought together teams from leading tech companies including Ramp, GitHub, Adobe, and Digits. Despite its smaller size, the conference delivered outsized insights into the practical challenges of deploying AI systems at scale.
Rethinking ML Design: Beyond the Algorithm
Anne T. Griffin's presentation on ML and design challenged conventional thinking about how we build AI features. Her most provocative insight? Test model outputs directly with customers. Too often, we evaluate models in isolation, obsessing over metrics while losing sight of actual user experience.
Griffin also highlighted an intriguing paradox: "True AI companies don't call it that." She pointed to Instagram as a prime example - a platform powered by sophisticated ML systems that users simply experience as a seamless product. This observation underscores how the best AI implementations are often invisible to end users.
Her framework for approaching ML features centers on three fundamental questions:
- What problem are we solving?
- Who are we solving it for?
- How are we solving it today?
These questions might seem basic, but they're frequently overlooked in the rush to implement cutting-edge technology.
Security in the Age of LLMs: A New Frontier
David Hawthorne's presentation on OWASP security considerations for LLMs was particularly eye-opening. He walked through the OWASP Top 10 for LLMs, emphasizing a critical point: "With LLMs, security issues are chained together." Unlike traditional security vulnerabilities that can often be addressed in isolation, LLM security requires thinking about cascading risks and compound effects.
Hawthorne didn't pull punches in his critique of current security standards, particularly SOC2 compliance. He argued that existing frameworks fail to adequately address the unique security challenges posed by LLMs - a sobering reminder that our security practices haven't caught up with our technology.
His recommendation? Adopt evaluation-driven development, keeping ML considerations central throughout the development process rather than treating security as an afterthought.
The Hidden Challenge of RAG Freshness
Drishti Jain's presentation on RAG (Retrieval-Augmented Generation) freshness addressed a problem that's easy to overlook but critical for production systems. While much attention goes to building RAG systems, maintaining their freshness over time presents unique challenges.
The core issue: reindexing is prohibitively expensive, leading many teams to let their RAG systems slowly degrade. Even more concerning is what Jain called "silent embedding drift" - the gradual degradation of embedding quality that can go unnoticed until it significantly impacts performance. This resonated particularly strongly as we face similar challenges with our own embedding systems.
Beyond the Sessions: Community and Connections
The value of AI in Production extended well beyond the formal presentations. Digits sponsored the conference after party, which proved to be more than just a social gathering - it became a natural extension of the day's learning. The conversations that sparked during breaks and continued into the evening added tremendous value to the conference experience. These informal exchanges often surfaced practical insights and war stories that you don't get in formal presentations.
Networking and conversations at the AI in Production 2025 after party
Continued discussions about production AI challenges well into the evening
Looking Ahead
AI in Production 2025 may have been a smaller conference, but it delivered concentrated insights into the real challenges teams face when moving AI from prototype to production. The presentations reinforced that successful AI deployment isn't just about algorithms—it's about design, security, maintenance, and most importantly, keeping user needs at the center of everything we build.
A big thank you to the organizers for putting together such a focused and valuable event, and to Digits for fostering those invaluable post-conference connections. I'm already looking forward to what insights next year's conference will bring.
What production AI challenges are you facing? I'd love to hear about your experiences and continue the conversation from the conference.
