Key Findings from the LangChain State of AI 2024 Report
The LangChain State of AI 2024 report highlights significant shifts in the landscape of LLM application development. Nearly 30,000 users are actively utilizing LangSmith, providing valuable insights into how developers are building, testing, and iterating on their AI applications. The report’s data indicates a notable increase in the use of open-source models, driven by platforms like Ollama and Groq, alongside established providers like OpenAI.
One of the most striking trends is the rise of AI agent applications. LangSmith traces reveal a significant increase in multi-step, agentic workflows, with 21.9% of traces now involving tool calls – a substantial increase from 0.5% in 2023. This shift reflects a move towards autonomous agents capable of dynamically invoking functions and external resources, enhancing their ability to interact with systems and perform complex tasks. Developers are leveraging tool calling to manage interactions with databases and other external systems.
Developers are also building more complex workflows, with the average number of steps per trace increasing from 2.8 in 2023 to 7.7 in 2024. This growth indicates a trend towards chaining together multiple operations, such as information retrieval, processing, and result generation. Simultaneously, the average number of LLM calls per trace has grown modestly, from 1.1 to 1.4, suggesting a focus on efficiency and minimizing the cost of expensive LLM requests.
Organizations are increasingly utilizing LangSmith’s evaluation capabilities to ensure the quality of their LLM-generated responses. They are employing LLM-as-Judge evaluators to automate testing and generate user feedback loops, focusing on criteria such as relevance, correctness, exact match, and helpfulness. This approach allows developers to proactively identify and address inaccuracies or low-quality outputs, leading to more robust and reliable applications.
LangSmith Usage and Framework Trends
LangSmith traces extend beyond LangChain frameworks, with 15.7% originating from non-LangChain frameworks, reflecting a broader trend of interoperability supported by LangSmith. Python remains the dominant language for development (84.7%), while JavaScript usage is growing (15.3%). The adoption of LangGraph, the controllable agent framework, is also rising, with 43% of LangSmith organizations sending LangGraph traces, indicating the increasing use of orchestrated tasks.
