Research9 min read
Microsoft Research Podcast on AI Failures and Evaluation
Jennifer Neville discusses how standard benchmarks miss real-world AI failures in a new Microsoft Research Podcast episode.
From Microsoft Research
Loading
90 days of the enterprise, pro and team plans free, no card needed. See the plans
Blog
Daily notes on new models, LLM releases, agent frameworks and AI research, written from the sources we follow and delivered as a newsletter every day.
Get the morning and evening issues
Every new post of the day, in one email. Confirmation required.
Research9 min read
Jennifer Neville discusses how standard benchmarks miss real-world AI failures in a new Microsoft Research Podcast episode.
From Microsoft Research
Agents10 min read
AI transforms how developers work, shifting focus from implementation to defining problems, evaluating output, and making technical decisions.
From GitHub blog: AI & ML
Photo: Yan Krukau
AI2 min read
Leaders in AI discuss plans to slow AI development, but details are unclear. Some see potential risks, others see benefits for safety and regulation.
From TechCrunch AI
Agents2 min read
Jev is a new model that evaluates agents by giving typed answers instead of generating text. It is faster and costs less than traditional methods.
From LangChain blog
Research1 min read
Benchmark Radar is a new database and search engine designed to streamline the discovery of AI benchmarks and evaluations for LLMs and other AI systems. It provides a centralized resource for locating benchmark datasets, code, and score histories, aiding in informed model selection and evaluation.
From arXiv cs.AI
Research1 min read
A new framework automates the discovery of vulnerabilities in agentic AI systems through a seven-domain taxonomy and automated red teaming. Empirical validation across CrewAI and AutoGen reveals significant governance and privacy risks, highlighting the need for proactive safety measures.
From arXiv cs.AI
Research1 min read
DeepMind has piloted the first double-blind evaluation process for AI systems, aiming to improve assessment objectivity and reliability in model performance testing.
From Google DeepMind blog
Research1 min read
Microsoft Research introduced MindTopo, a benchmark designed to assess a VLM's ability to understand topological relationships like paths and knots. This tool provides a new method for evaluating and improving spatial reasoning and planning capabilities in AI models.
From Microsoft Research
Research1 min read
Amazon Science released a protocol where humans and AI models debate benchmarks together.
From Amazon Science
Posts are drafted from public feeds by models OpenSmartRoute routes to - the same router, skill and metering customers use - and always link to the original source. Corrections: support.
Archive (100)
