FaunaDB: A Guide for Relational Users

The structure of the relational model refers to Entities, which over the years have become synonymous with a Table in the relational lingo. Codd proposed (in Rule #1) that all data in the database be represented as values in column positions within rows of tables. Basically, only put one piece of data in each field, and do not put the same data in different fields of different tables. This process is called normalization and leads to the creation of entity relationships within the database. Normalization is essentially a data-analysis process used to find the simplest possible data structure for a given collection of information by eliminating the replication of data across tables. However, in order to build relationships between entities, there is a need to uniquely identify each record. This unique identifier for a table row is called a Primary key. Data in primary key columns are used to build relationships between two or more related entities.

It is very important to understand here that the ground rules for relational databases were set at a time when storing and retrieving information from disks was quite expensive, and thus application of normalization became really important. Normalizing data to the most granular level often leads to complicated data models that are extremely difficult to comprehend. Over the years, new data modeling techniques like the star schema and the snowflake schema have been accepted even though they don’t adhere to the principles of normalization.

Integrity mainly refers to relational integrity and constraints. It enforces that a foreign key of one relation either can hold values that are present in the primary key columns or must be null. This rule means that if one instance of an entity exists and refers to another entity occurrence, then the referenced entity must also exist. The other aspects of integrity are constraints that can enforce uniqueness in a specific column without making it a primary key. In relational databases, unique constraints are enforced using indexes. Indexes are key to data access in relational databases.

The data manipulation aspect of relational databases can be divided into two major elements:

Data manipulation operations adhere to relational algebra and consist of the set operators, such as intersection, union, minus, etc. The ability to Join by comparing data in two columns in two different tables forms the crux of data manipulation in relational databases. Joins provide the ability to access any data in the normalized entities.

The manipulation language as mentioned in Codd’s Rule #5 allows the user to interact with the data stored across various tables. Users access data through a high-level, declarative database language. Instead of writing an algorithm to obtain desired records one at a time, the application programmer is only required to specify a predicate that identifies the desired record(s) or combination of records.

The Structured Query Language or SQL has become the universal standard for this data manipulation language. SQL can be executed interactively or embedded in a programming language. The embedded mode makes SQL the database programming language for a host language, e.g., C, FORTRAN, COBOL, and so forth. SQL as a database manipulation language has become so popular that it has become synonymous with relational databases. It is easy to learn and provides a very intuitive syntax to operate on the data.

A database transaction is a unit of work performed within a database in a coherent and reliable way independent of other transactions. Transactions generally represent any change in the database. The built-in support for transactions has been one of the key reasons for widespread adoption of relational databases. A database transaction by definition needs to be atomic, consistent, isolated, and durable, i.e., ACID compliant.

We have discussed the foundations of relational databases in the above sections. However, over the past 40 years, the interpretation of the word "relational" has morphed. If we ran a survey across database developers, architects, and even administrators to ask them which relational property is the most important, the answers would be surprisingly different. While most developers would identify SQL, a database architect might choose referential integrity, and the administrator might choose the ACID properties.

Over the years, the use of relational databases for mission-critical systems has also ensured certain operational readiness. The ability to recover to any point in time or to fail-over to a standby system in case of any hardware failure are now considered table stakes in the relational world.

A mismatch between the object-oriented and the relational world.

New-generation programming languages are primarily object-oriented, where objects are connected via references and build an object hierarchy or graph. But relational databases store data in table rows and columns. To store the object hierarchy or the graph in a relational database, the object or graph has to be sliced and flattened to fit the normalized data format. This results in a complex joining of tables and often leads to performance issues in the databases.

While relational systems meet most business operations needs (because relationships are the way most business are modeled), the relational data model doesn’t fit in with every data domain. Today, more than 85% of corporate data is generated either in the web as free form text or by machines and modern day applications that need to store data in various formats. Textual unstructured data includes documents, presentations, videos, email, chat, and social media posts, while machine data includes data generated by various sensors, electric meters, IoT devices, and so on. Analyzing all of this unstructured data can be challenging with massive data volumes, many different file types, and high creation speeds. The structured data model, with the upfront definition of the table structure and referential constraints, is not suitable for storing and retrieving such unstructured data.

In the pre-internet era, the design of an application was driven by the data model. A business process was mapped to entity relationships and applications were built within the bounds of that definition. With the advent of the internet era, the focus changed to capturing data in its original free form. Thus, the rigidity of a structured model had to be given up. Applications now require flexibility so that data can be captured in its original form.

Relational databases thrived during the client-server era and clearly were not built to handle web-scale workloads. So, when the application load increased, the only way to keep up with performance needs was to scale up, i.e., move the database to a bigger machine. Many people tried horizontal scaling with shared disks across multiple machines, but that approach could not keep up with the demands of running a database across multiple data centers. The scale up approach left a lot to be desired and led to a high total cost of ownership. The operational overhead to maintain large databases is significant and often requires hiring a team of highly skilled DBAs to maintain the application backend.

Data volumes generated by applications are growing exponentially. In the last decade, the notion of a large database has changed from a few hundred GBs to hundreds of TBs. Even petabyte scale is not uncommon these days. Relational databases were designed to run on single servers and hence they are limited by design to the system resources that can be made available on the node. The relational databases did undergo improvements to cope with this increased demand by allowing active-passive architectures or by creating data shards. But eventually it came down to one single point of failure that always made the database vulnerable to failures. For example, Oracle RAC can support multiple database nodes, but it still requires a shared disk subsystem where a single corruption or crash can take down the entire application.

Application performance expectations have changed significantly over the years. Applications are expected to ingest data at high speed from multiple sources and to keep read latencies under a few milliseconds.

One of the prime requirements of a relational database is to support a declarative language for data manipulation. If we take a step back and assess the reasons behind the popularity of SQL, we will see it is primarily because of the fact that it helps to answer the "What?" and not the "How?". In other words, the end user doesn’t need to specify the access path (i.e., an Index or join order), but only provide the name of the entity (table). While in theory this sounds pretty good, in reality this is quite the opposite. For any given application, a team of performance engineers carefully study access patterns and suggest required indices to ensure agreed SLAs. While the end users may not care about the nuts and bolts of indices or materialized views, the operational overhead of maintaining such additional objects cannot be ignored. Now, as the data volumes grow and new access patterns emerge, the predictability of performance goes out the window. This has been a huge problem with relational databases, so much so that database vendors like Oracle have introduced several features just to ensure that the execution plan (access pattern) doesn’t change.

The other side of the problem has been the need to join multiple tables, even for addressing simple queries. As data volumes grow, performing efficient joins across multiple tables poses a challenge to meet business expectations.

Primarily because of these shortcomings, companies are progressively considering alternatives to legacy relational infrastructure.

One of the factors that has triggered the need for distributed databases is the adoption of multiple data centers by enterprise IT. Multiple data centers were initially introduced purely as a disaster recovery strategy, but enterprises soon realized that in order to stay competitive they could not have machines idle in a data center purely as insurance for an impending failure. Relational databases provide ways to keep a copy of the database in-sync across data centers, but they do not support active-active setups, i.e., writes across multiple data centers. NoSQL databases have been built with the premise of being distributed across multiple data centers.

The key benefits to using NoSQL to process Big Data are as follows:

A lot of the NoSQL databases are extremely difficult to manage and maintain. One of the possible reasons is that most of these databases are an outgrowth of an open source project. Open source committers are great at adding new functionality, but they seldom care about enterprise readiness. Large companies can throw developers at these issues and work around the immaturities, but smaller digital businesses cannot afford to do that. They need a solution that just works out of the box.

Typical problems are often very basic requirements for the operators. There are no easy ways to take incremental backups or straightforward ways to do point-in-time recoveries. Most of these first generation NoSQL databases simply lacked the enterprise readiness required to run the database in a corporate data center.

Security is a major concern for IT enterprise infrastructures. Security in NoSQL databases is very weak; authentication and encryption both in motion and at rest are very weakly implemented.

The design and query languages vary widely among various NoSQL database products. Depending on the type of NoSQL database chosen, the learning curve can be very steep.

NoSQL databases do not require any predefined data models or referential integrity. As a result, there is no predefined structure that can be used to join two different collections of data.

With these shortcomings in place, it is clear that modern-day applications need a database that has the best of breed features from both relational and NoSQL database worlds.