Hi - I answer from the OpenSmartRoute documentation: routing, the API, plans and quotas, self-hosting. Ask away, or open a support ticket if you need a person.
Grounded in the docs - follow a source before acting on it.
Databricks Adds NEAREST BY Join for Batch Vector Search - OpenSmartRoute
Databricks announced a new SQL join for batch vector search. This tool runs massive searches directly in its runtime engine. It finds the k nearest rows for every query row. The system uses exact or approximate similarity distances.
What Changed - Databricks announced a new SQL join for batch vector search
Databricks released NEAREST BY as a fresh SQL join command. This feature handles batch vector search tasks efficiently. For each query row, it finds the k nearest matching rows. Users can choose between exact or approximate distance metrics. The announcement targets organizations running large-scale data workloads.
Previously, teams often relied on external real-time endpoints for searches. Those older methods required separate infrastructure and sync pipelines. Network latency limited how many requests a system could handle. Databricks now pushes the scoring logic inside its own runtime engine. This change eliminates the need for an external vector store service.
The new join works within standard SQL queries. Engineers can write it using familiar syntax patterns. Managers see reduced operational complexity and lower licensing costs. The solution scales from millions to billions of vectors. It replaces the old VECTOR_SEARCH function that streamed rows one by one.
The Core Problem - Batch workloads differ from real-time serving needs
Vector search began as a problem for real-time chatbots or search bars. Those systems needed answers within tens of milliseconds per request. A single query embedding arrived, and the system returned top-k documents quickly. This design optimized for low latency on individual lookups.
However, many vector searches happen in massive batch jobs offline. Companies precompute neighbors rather than looking them up at request time. A payments firm matches 100 million transactions against 140 million merchant embeddings daily. A data firm enriches tens of millions of historical records every night. These tasks measure success by job completion time and cost.
Success depends on whether a batch finishes within its Service Level Agreement. Latency of a single lookup matters less than total throughput. Millions of queries against billions of vectors happen on a strict schedule. These workloads need an architecture focused on aggregate throughput. They require better performance, reliability, and cost efficiency than real-time serving.
Tiiuae released Falcon-Emirati-7B to handle Emirati Arabic dialect nuances. It scores 84.83% on the Alyah benchmark, beating larger competitors.
Existing engines often struggled with these specific batch shapes. Postgres and Snowflake use ORDER BY LIMIT patterns for distances. Batch jobs then needed complex LATERAL subqueries for every driving row. The optimizer lacked a clear pattern to distinguish KNN from ANN queries. That recognition proved fragile when query shapes drifted slightly. BigQuery offered table-valued functions, but column references remained strings.
How It Works - A fused Photon operator with custom blocked GEMM kernels
The Databricks Runtime fits these batch requirements perfectly. It is a distributed, fault-tolerant engine built on Spark and Photon. Photon acts as a vectorized native C++ query engine here. The team decided to build vector search directly into this engine. They avoided relying on separate infrastructure layers entirely.
The new join syntax handles asymmetric top-k ranking joins. It drives the left side while searching the right side. Ranking direction is explicit using BY SIMILARITY or BY DISTANCE clauses. LEFT OUTER keeps query rows that find no candidates. The BY expression accepts any orderable scalar over both sides.
APPROX and EXACT encode a clear semantic contract for users. EXACT guarantees true top-k results through exhaustive evaluation. APPROX allows the optimizer to substitute an approximate strategy like an ANN index. Creating or dropping an index never silently changes query results. Only queries specifying APPROX consent to approximation risks.
NEAREST BY parses into a logical join node first. The optimizer lowers this onto standard relational operators internally. It tags each query row with a generated unique id. Then it scores every pair of (query, base) vectors together. Finally, it keeps the k best rows per id with a grouped top-k aggregate.
This process inlines the kept rows back out to the final result set. Semantically, the feature encapsulates a cross join, a scalar scoring expression, and a grouped top-k aggregate. Because every operator is an ordinary relational one, the plan distributes and spills naturally. Correctness and fault tolerance come for free with this design.
Data Storage - The IVF index lives as an ordinary Delta table
Your Lakehouse also acts as your vector store in this new system. There is no separate system to sync or operate alongside it. The IVF vector index exists as an ordinary liquid-clustered Delta table. This structure prunes away most partitions during a read operation.
The IVF index helps score a fraction of the base vectors quickly. It applies when users run APPROX queries on large datasets. This reduces the number of distance calculations needed for each query. The same kernels used for scoring also handle the index construction efficiently.
Embeddings stay in Delta tables on the Lakehouse throughout their lifecycle. No separate vector store requires maintenance or synchronization pipelines. Teams avoid paying for a second system to operate and manage. Data consistency remains high because there is only one source of truth.
This storage model supports the elastic parallelism required for batch jobs. Workloads partition cleanly across hundreds to thousands of cores automatically. Compute scales itself up for the run and down to zero after it finishes. The engine handles fault tolerance without needing client-side retry loops.
The Join Syntax - NEAREST BY handles asymmetric top-k ranking joins
The join is asymmetric, similar to how LATERAL works in SQL. The left side drives the operation while the right side gets searched. Ranking direction is explicit through BY SIMILARITY descending or BY DISTANCE ascending. LEFT OUTER preserves query rows that have no matching candidates found.
The BY expression is pluggable for flexibility. Any orderable scalar over both sides works within this clause. Other scoring expressions can reuse the same syntax later. This makes it easy to adapt for different business logic needs.
NEAREST BY parses into a logical join node for the optimizer. It lowers onto standard relational operators during the planning phase. The rewrite tags each query row with a generated id internally. Then it scores every (query, base) pair against that id. Finally, it keeps the k best rows per id using a grouped top-k aggregate.
This process inlines the kept rows back out to the final result set. Semantically, the feature encapsulates a cross join, a scalar scoring expression, and a grouped top-k aggregate. Because every operator is an ordinary relational one, the plan distributes and spills naturally. Correctness and fault tolerance come for free with this design.
Why it matters - Single copy of data and single engine for execution
Implementing vector search natively in the runtime engine pays off from two main angles. First, there is a single copy of the data in Delta tables on the Lakehouse. Embeddings stay there without needing a separate vector store system. Second, there is a single engine for all execution tasks. Search runs in one engine that scales elastically with the workload.
Kernels are purposely built for batch query shapes in this engine. They push each core toward peak FLOPs per second. Scale-out multiplies the rest of the performance gains automatically. The engine already owns fault tolerance through automatic task retries. Memory pressure spills to disk when needed without user intervention.
There is no client-side concurrency control or rate limiting required. No retry loops need manual implementation by developers. This reduces operational overhead significantly for engineering teams. The design trades per-request latency for higher aggregate throughput. It optimizes the entire batch job completion within its SLA at reasonable cost.
How it compares - What existed before, what this changes and what stays the same
The old VECTOR_SEARCH function sent every row to an outside service. It used a network request for each single query row. This created a performance ceiling based on external endpoint size. The runtime engine acted only as a dispatcher for these requests.
This new join keeps all data inside Delta tables on the Lakehouse. Embeddings do not need a separate vector store system anymore. There is no sync pipeline required to keep two systems consistent.
The previous design could not handle massive batch cardinalities well. It struggled with hundreds of millions of query vectors against billions of base vectors. The new join scales on both sides of the operation effectively.
Existing engines like Postgres and Snowflake use ORDER BY clauses for search. They often lack patterns to distinguish between KNN and ANN queries. BigQuery uses string column references that parsers cannot validate reliably.
The NEAREST BY syntax encodes a binary relational operation clearly. It treats top-k ranking as a first-class join type. This structure is similar to the LATERAL join in SQL logic.
Questions this leaves open - What the source does not say and how a reader can check it
The article mentions scaling up to hundreds of millions of query vectors. It does not specify the exact maximum number of base vectors supported yet. Readers should test their specific cluster sizes to find the true limits.
The text states kernels push cores toward peak FLOPs per second. It does not provide a benchmark number for this arithmetic throughput. Engineers need to measure FLOPs/s on their own hardware to verify claims.
Memory pressure spills to disk automatically according to the design. The source does not define the trigger threshold for this spilling behavior. Users should monitor disk usage during runs to understand when it happens.
The IVF index prunes partitions on read but uses the same kernels. It is unclear if this pruning improves speed or just reduces scan size. Readers must compare exact versus approximate query times directly.
Fault tolerance handles worker loss and transient task failures automatically. The article does not state how long a failed job takes to recover from. Teams should measure total run time including recovery periods.
The feature integrates with Spark and Photon but no version numbers are given. Users need to check their Databricks Runtime version for compatibility. Older clusters might require upgrades to use the new syntax.
Licensing fees and infrastructure costs are mentioned as savings areas. The source does not provide specific cost reduction percentages or dollar amounts. Managers must calculate their own savings based on current pricing models.
The ARRAY column type is required for data models. It does not list supported vector dimensions or element types beyond floats. Developers should verify if complex vectors fit the schema constraints.
Service Level Agreements are referenced regarding job completion times. The article lacks specific SLA targets or acceptable latency thresholds. Engineers must define their own success criteria for batch jobs.
Index creation is suggested for approximate strategies but no cost is listed. Users need to estimate storage and maintenance costs for IVF indexes. This adds ongoing operational work beyond the initial query run.
What to do - Use the new syntax in your SQL queries today
Engineers can use the new NEAREST BY syntax in their SQL queries immediately. Start by writing a standard join with the BY SIMILARITY or BY DISTANCE clause. Test it on a small dataset before scaling to millions of vectors. Check that the results match expectations from previous external endpoint runs.
Managers should evaluate the cost savings compared to separate vector store services. Look at reduced licensing fees and lower infrastructure management costs. Consider how this fits into existing batch processing workflows like Spark jobs. The feature integrates seamlessly with current Lakehouse operations.
Compare performance metrics between the old VECTOR_SEARCH function and the new join. Measure throughput improvements on your specific hardware cluster sizes. Monitor job completion times against your Service Level Agreements closely. Watch for any changes in memory usage or disk spilling behavior during runs.
Check documentation for supported vector types and distance functions available. Review examples of how APPROX and EXACT flags affect query planning. Ensure your data models support the ARRAY column types required. Plan for potential index creation if you use approximate strategies often.
NVIDIA released version 1.0 of its AI Cluster Runtime to standardize GPU cluster configurations. The update adds signed validation evidence and a live dashboard for operators.
