By Keshav Murthy
Amazon’s announcement last summer of PartiQL, a SQL-compatible query language, further demonstrates the importance SQL still has in data management today. SQL was created and widely adopted more than 30 years ago and goes hand-in-hand with legacy relational database technology. But even with the advent of newer non-relational technologies like NoSQL, so-called “NewSQL” and the like, SQL is still the most common programming language in data management.
Couchbase is highlighted as the first vendor to support PartiQL directly, bringing this new query language to AWS developers everywhere. A partner of Amazon, Couchbase is pleased to have been part of what is arguably the most exciting innovation in query languages since the inception of SQL. PartiQL is a validation of the work Couchbase has implemented over the past several years through N1QL (“nickel”), which is our implementation of SQL for JSON. Our approach has been further standardized into SQL++ through the work we have done with Dr. Yannis Papakonstantinou, Don Chamberlin, and Professor Mike Carey. SQL++ offers a standards-based approach to extend the language implementations at Amazon and Couchbase, and offers further opportunities for other databases to join, contribute, and extend this standard.
Revolutionary Database Innovation Is Hard
Since the wide-spread adoption of SQL databases starting in the 80’s, we’ve seen the introduction of multiple non-relational database storage solutions including Object Oriented, XML, and NoSQL databases with key-value, graph, ledger, and time-series database implementations. These alternative database solutions provided innovative approaches to nested, semi-structured and unstructured data management storage problems and typically also came with their own, often proprietary, query languages. Although often extremely well suited for specific kinds of applications and data management challenges, they never had the general-purpose usability of relational databases or wide-spread adoption of SQL. Furthermore, these database alternatives exacerbated a growing database problem that was already in evidence even amongst the relational databases, namely data silofication. Data silofication tends to arise naturally when each organizational unit or application has different data management objectives, priorities and responsibilities. Data tends to get managed in different database end-points, often with different storage formats and query languages resulting in the very real present data problem of data silofication. Data silos severely limit data sharing across application data repositories, often leading to a decrease in the quality of the data stored, and prevents businesses from leveraging the true value of the data across the organization.
It’s All About Queryability
So how can you address this real and growing problem of data silofication without resorting to constantly moving data around between silos or attempting to stuff it all into a massive one-size-attempts-to-fit-all database solution, which are likely to fail due to their own shortcomings? Instead of offering yet more database formats with yet more query languages, what we needed was a unifying approach — one that could leverage 50 years of SQL technology with a simple and consistent way to query data across a variety of formats and services, including the newer database alternatives introduced in the past few decades.
Couchbase had been working on this challenge for the past several years. By taking SQL and extending it, we combined the flexibility of JSON data with the power of SQL. Using N1QL, developers now had access to an expressive, powerful, and complete SQL-based language with industry standard ANSI joins for querying, transforming, and manipulating JSON data.
Dr. Yannis Papakonstantinou and his team at UCSB were also looking for a way to devise a clean, well-founded query language that stayed very close to SQL, while having the power needed to process the nested, schema-optional and schema-less data models of modern NoSQL and BigData systems. They leveraged and further extended Couchbase’s N1QL, which subsequently resulted in the formalization of SQL++. Don Chamberlain, co-inventor of SQL, wrote an excellent tutorial “SQL++ for SQL Users” which explains how SQL++ can access JSON (nested, schema-optional) data structures.
Amazon also saw the great potential in SQL++ standard, initially in response to Amazon’s own needs to query and transform vast amounts and varieties of data across SQL, nested, and semi-structured data repositories in a variety of formats and storage engines. Their implementation of the SQL++ standard, called PartiQL, provides a simple and consistent way to query data across a variety of formats and services.
These implementations of the SQL++ standard (N1QL and PartiQL), offer application developers the best of both worlds — a way to leverage their existing SQL-based query expertise and incorporate the newest innovations in data formats, storage and services.
Looking Forward to a Bright Future
We look forward to working with Amazon and our mutual customers to solve real world data accessibility challenges thereby enabling richer, more connected applications and customer experiences. Together we are delivering the best of breed open source query languages and tools into the hands of developers and promoting the growth and adoption of SQL++, whether as N1QL or PartiQL queries.
Bio:
Keshav Murthy is a Vice President at Couchbase R&D. Previously, he was at MapR, IBM, Informix, Sybase, with more than 20 years of experience in database design & development. He lead the SQL and NoSQL R&D team at IBM Informix. He has received two President’s Club awards at Couchbase, two Outstanding Technical Achievement Awards at IBM. Keshav has a bachelor’s degree in Computer Science and Engineering from the University of Mysore, India, holds ten US patents and has three US patents pending.
