Data-driven application design

This is a compilation of online and offline study contents and other reference materials for ‘Data-driven application design’. ###Intro Technology has advanced with massive amounts of traffic, shortened development cycles, free and open source software, increased parallel processing, infrastructure as a service (IaaS), and high availability requirements. With the advancement of these technologies, data-centric applications that take into account the amount, complexity, and speed of data change, rather than simply calculation-centric applications, have become possible.

Hadoop: An open source framework for large-scale data processing, supporting distributed storage and processing.
Spark: A cluster computing framework for large-scale data processing and analysis, providing high-speed data processing.
NoSQL: A diverse form of data storage that does not follow the relational database model, with an emphasis on scalability and flexibility.
Big data: Large data sets that are difficult to process with traditional database tools and can come from many different types and sources.
Sharding: A technology that improves database performance by distributing and storing data across multiple servers. Data is stored in fragments.
Eventual Consistency: A consistency model in which all replicas are in the same state after a certain period of time in a distributed system.
ACID: stands for four properties that represent transaction properties: Atomicity, Consistency, Isolation, and Durability.
CAP Theorem: It is a theory that only two of consistency, availability, and partition tolerance can be guaranteed in a distributed system.
MapReduce: A programming model for parallel processing of large data sets, used in distributed computing environments. In this study, my goal is to focus on understanding data-centric applications and how large-scale websites and services actually work.

Part 1. Fundamentals of data systems

01. Reliable, scalable and easy-to-maintain applications

Data-centric applications include a database that stores data, a cache that remembers data once loaded and retrieves performance results without re-reading it, a search index that allows users to search or filter by keyword, Asynchronous processing requires stream processing, which sends messages to other processes, and batch processing, which periodically analyzes large amounts of data. Nowadays, the boundaries are blurring, such as Redis, a data store used as a message queue, and Apache Kafka, which guarantees persistence like a database, so the umbrella term data system is used. Additionally, as the number of tasks in an application increases, more tools are used. The application management cache layer and the full-text search server that maintains the cache or index are all tied to the application code. This means that developers are now not only application developers but also data system architects.

Elasticsearch: A distributed search and analytics engine used to quickly search and analyze large amounts of data. It is open source and supports real-time data search and analysis.
Solr: An open source search platform similar to Elasticsearch, built on Apache Lucene. It is used to index and search large amounts of data and provides high-performance search capabilities.

Reliability

Systems must continue to function correctly despite adversity, such as hardware or software defects or even human error. In other words, it must operate normally even if there is a defect, but here a defect is not the same as a failure. A failure refers to a situation where the required service cannot be provided to users and the entire system comes to a halt. It is a good idea to design fault-tolerant structures to prevent failures due to defects, and Netflix Chaos Monkey is an example of this approach.

Netflix Chaos Monkey
Chaos Monkey randomly shuts down instances providing services in case of unknown failures, and has now become a development principle for testing operational issues. Netflix says that its fault tolerance strategy is 'the best way to avoid failure is to fail continuously' and that it performs chaos engineering.
Spring Boot Chaos Monkey Library

Hardware Error The average time to failure for a hard disk has been reported to be approximately 10 to 50 years, and a storage cluster of 10,000 disks should expect one disk to die per day on average. To reduce system failure rates, redundancy could be added, making multiple devices redundant only for a few applications that require high availability.

Adding redundancy means things like disk RAID configurations, redundant power devices in servers, hot-swappable CPUs, and batteries and diesel generators for backup power in the data center. Recently, it has become easier to configure multiple devices, such as on cloud platforms, but this method is suitable when flexibility and elasticity are prioritized over the reliability of each device instance.
Systems of any size require a design that can withstand equipment failures to allow rolling upgrades rather than a single server system. Software Error In some cases, many hardware components do not fail, and in other cases, systematic errors within the system cause excessive use of shared resources by some processes, causing other components to fail. Since these cases are difficult to predict, it is necessary to continuously check them through testing, process isolation, and monitoring.
The leap second of June 30, 2012, which caused many applications to stop simultaneously due to a bug in the Linux kernel. Human Error Reliability can be increased in the following ways, such as operator configuration errors, etc. (Bugs cause decreased productivity, and site outages cause loss of sales and damage to reputation)
Abstractions such as interfaces and APIs
Provides a non-production sandbox to actual users by isolating areas where errors may occur
Unit tests and integration tests
Easy to handle corner cases that do not occur in normal operation
Provides data recalculation tools
Establishment of performance indicators and clear monitoring measures

Scalability

As the amount of data, traffic volume, and complexity in your system increases, there must be an appropriate way to handle it. This is because the load increases as the scale increases. Rather than simply whether or not it is scalable, it is necessary to ask questions such as what to do when the system grows in a certain way and the input of computational resources to handle the additional load. Load can be expressed in terms of load parameters, such as requests per second for a web server, read-to-write ratio for a database, concurrent active users in a chat room, cache hit ratio, etc. ####### Twitter example Consider Twitter’s home timeline. When a user posts a Tweet, that user's followers should be able to see the new Tweet from the user's home timeline.

When each follower opens their home timeline, they will be able to find the tweets of the people they follow and display them in chronological order.
There is also a way to use cache. If it is important to quickly display the home timeline and you do not need to query the data repeatedly, you can cache the contents of the home timeline and remember and process repeated requests.

In fact, since the volume of requests to post tweets on Twitter is much smaller than the volume of requests to read home timelines, we made it so that more work is done at the time of writing by inserting the latest tweet into the home timeline cache of each user who follows the user when a tweet is written. But what if the tweet is posted by someone with a large following, such as an influencer? At the time of Tweet creation, write requests may increase and it may be difficult to achieve the goal of delivering Tweets in a timely manner. Therefore, Twitter ultimately caches the read operations of tweets in most cases using method 2. In the case of excluding certain users from fan-out and following celebrities' tweets, we chose to provide them in the home timeline at the time of reading, again as in method 1.

####### Latency and response time Response time is the time seen in client processing to process a request, and includes network delay and queue delay in addition to the actual time. On the other hand, delay time refers to the time waiting for a request to be processed and is the time in an idle state while waiting for a service. Response time can be thought of as an average of a distribution of measurable values, but it is generally better to use percentiles. It's also a good idea to look at the median to see how long users will have to wait, or the top percentile to see how bad the outliers are. The top percentile is also called tail latency, and a 95th percentile response time of 1.5 seconds means that 95 out of 100 requests are less than 1.5 seconds.

Response time generally varies from request to request because it can be caused by context switches in background processes, network packet loss and TCP retransmission, garbage collection pauses, page faults that force reads from disk, and mechanical vibration of server lag.
Service Level Objectives (SLO), Service Level Agreement (SLA)
Head-of-line blocking: Even if subsequent requests are processed quickly by the server, the overall response time may be considered slow due to the time spent waiting for previous requests to complete.
Tail delay amplification: The effect of multiple backend calls resulting in slow calls and slower response times

####### Load response approach Factors that determine architecture include the amount of reads, the amount of writes, the amount of data to be stored, the complexity of the data, response time requirements, and access patterns.

Capacity scaling (scaling up) >> vertical scaling
Scaling out >> Horizontal scaling
Elastic systems are useful when loads are unpredictable, but passively scaling systems are simpler and have fewer operational surprises.
In the early stages of a startup or for an unproven product, iterating quickly to improve product functionality is more important than scaling to prepare for future loads.

Maintainability

You need to ensure that all users can work productively on the system.

Operability: Making it easy for the operations team to operate the system smoothly
Simplicity: Removing as much complexity as possible to make it easier for new engineers to understand
Abstraction can be used to hide implementation details to remove accidental complexity.
Evolvability: Enabling engineers to easily move on in the future

02. Data model and query language

Relational model and document model

A data model is an abstract model of the processing required for data relationships and flow. The data model tells us about how the program was written and how it approached the problem. This use of a data model allows us to hide complexity between layers. In other words, it reuses or simplifies data through abstraction, allowing application, database, and hardware developers to work efficiently. Relational databases and NoSQL A relational database is a database that has tuples and rows based on relationships. Despite object, XML, and network models, the relational model still dominates business data processing. NoSQL, reinterpreted as Not Only SQL, has the advantage of complementing relational databases in cases of high write throughput. Currently, depending on the application, a database design with multi-store persistence using relational databases and NoSQL together is widely used. Impedance mismatch between models occurs due to objects, which refers to problems when storing application code and queries in the database. ORM support also has limitations, and complex reference processing in relational databases often creates a large number of unnecessary tables. In this case, performance may also deteriorate as many joins are processed to retrieve a single result. Document-Oriented Database Document-oriented databases (Mongo, Resync, Couch, etc.) support JSON and XML, allowing for better locality than schema. Having no schema means there is no guarantee of the existence of the fields contained in the document. Document-oriented databases typically lack support for joins. This method stores records nested within a parent record. Unlike relational foreign keys, document models use document references.

Query language for data

SQL is a declarative query language, while IMS and Codasil are queries using imperative code. Imperative queries direct specific operations to be performed in a specific order, while declarative queries are often suited to parallel execution and are not limited to databases but can be used in other environments as well. Mapduless queries are a model for processing large amounts of data and are supported by some NoSQL. It is mainly used when performing read-only queries targeting many documents. It is a low-level programming model, mainly used in distributed systems. The map stage converts input data into key-value pairs, processes them in parallel, and produces intermediate results. In the reduce phase, intermediate results from the map phase are grouped and aggregated based on keys. We use a reduce function to aggregate intermediate results and produce the final result.

Graph data model

A graph-type data model refers to a data model made up of objects called vertices and edges in a model where many-to-many relationships are common. This is not limited to homogeneous data. Property Graph Model A property graph model has as its vertices a unique identifier/outflow edge set/inflow edge set/property collection, and the vertices are connected to other vertices by edges. It provides a lot of flexibility for traversing back and forth along a set of vertices. Cases with different structures or data granularity are also possible. Cipher Query Language A declarative query language for attribute graphs, built for the Neo4j graph database. Vertices are designated by ID, and edges are created in the form of (tail node id) -[:edge label]=>(head node id). Because it is a declarative language, the query optimizer automatically chooses the most efficient strategy. Triple Storage and Sparkle Triple storage is almost similar to the attribute graph model, and stores all information in the form of syntax such as (subject, predicate, object). Here, the subject of the triple is equivalent to a vertex in the graph, and the object is either a primitive data type such as a string or number, or one of the other vertices of the graph.

_:lucy a :Person; :name "Lucy"; :bornIn _:idaho._

Sparkle Query Language is a triple-store query language using the Resource Description Framework (RDF) data model. It was created before Cypher and has a similar form due to Cypher pattern matching. Data Log It is written as a predicate in an older language than Sparkle or Cypher, and rather than directly querying with select, it queries little by little by dividing it into steps.

// data definition
name(namemerica, 'North America').
type(namemerica, continent).
// rule definition
within_recursive(Location, name) :- name(Location, name).
within_recursive(Location, name) :- within(Location, Via),
within_recursive(Via, Name).
migrated(Name, BorIn, LivingIn) :- name(Person, Name),
born_in(Person, BornLoc);
...
within_recursive(LivingLoc, LivingIn).

Differences from other data models Unlike the network data model, the graph data model has the advantage that vertices are freely connected to other vertices through edges and can be quickly searched using unique IDs. The sorting point is also different; the graph model is sorted flexibly during read queries. Additionally, unlike document databases, graph databases target use cases where everything is potentially related.

03. Storage and search

Which database should I use? It is important to choose a database that uses the right data engine for your application. This is because depending on the data engine, what is suitable for traction processing or analysis differs. Log refers to successive addition records and is generally an append operation method used in most databases. The form of log includes not only human-readable form but also binary format. The database also has an index, which is organized separately from the existing data as additional metadata to facilitate search. Using an index, you can find the location faster by searching using the index without having to read all the data. However, since the index must also be updated during the write operation, it may create more overhead than necessary. Therefore, appropriate use considering trade-offs is required. Hash Index It consists of key-value pairs and is used to find the location of the value by finding the offset in the data file using a hash map or hash table. A representative example is the method used by Bitcask, which is advantageous when values are frequently updated. In the book, the key is the URL of the video, and the value is something that needs to be updated frequently, such as the number of clicks on the video. Just adding to the file can increase disk space, so use segments that store only the most recent values. Segments can also be merged into smaller compacted segments. Since segments cannot be overwritten, the old segment is deleted and a new segment is created. When implementing, you should consider the following:

The file format itself is quick and easy to use in binary format.
To delete a value associated with a key, you must add a special delete record.
When the database is restarted, the in-memory hash map is lost, so it must be restored by reading it again from the beginning and checking the offset of the latest value for each key.
A database can die even while writing records to the log.
You must consider whether to manage concurrency with multiple threads for writing or with a single sequential writing thread. Hashtables can degrade disk performance if there are too many keys, and are not efficient in that every individual key must be looked up. SS table and LSM index SS tables are also called sorted character tables, and refer to a format sorted by key. Therefore, each key must appear only once in each segment. If you want to find a specific key, you can quickly scan by checking the key offset of the index in memory and checking the segment on the disk corresponding to that offset. When new key-value pairs come in, they are managed in a memtable in in-memory in addition to the sorted segments on the existing disk. When a memtable exceeds a certain threshold, the storage engine can store it on disk as a new segment. In this case, the search is performed in the order of searching the memtable and then searching for the most recent segment on the disk. To avoid losing the memtable in memory due to database failure, you can keep a log on the disk and use it for restoration. LSM Tree refers to a storage engine based on the principles of file merging and compaction to be sorted into a Log-Structured Merge Tree. Since it is necessary to search even old segments to find keys that are not in the memtable, a Bloom filter is additionally used for optimization. Bloom filters can save unnecessary disk reads by notifying you that the key does not exist in the database. The compaction strategy of SS table can be divided into size hierarchy compaction and level compaction. Size-tier compaction merges new, smaller SS tables into relatively older SS tables, while level compaction has the advantage of using less disk space because the SS tables are split into smaller pieces and older data is moved to individual levels. B Tree The most common type of index is dividing it into fixed-size blocks or pages and reading or writing one page at a time. To find a key in the index, you start at the root and work your way up to the leaf pages in the range for which the key corresponds. The number of references to subpages is called the branching factor. To update the key value, you can find the page to which the key belongs and add the key there, or if there is no page space, you can update it by dividing the page and linking it to the parent page. The B-tree write operation is performed by keeping a log before writing in case of emergency and overwriting pages on the disk with new data. For concurrency control, the tree's data structures are protected with latches (light locks) and can be processed in the background without being affected by other threads. The B tree can be optimized by adding pointers or abbreviating keys on the page. Difference between LSM tree and B tree
In general, LSM trees are usually faster in writing, and B trees are faster in reading. This is because the LSM tree must check the data structure and SS table of each compaction step.
LSM trees can maintain higher write throughput than B trees. This is because LSM trees have relatively lower amplification and write compacted SS table files sequentially rather than overwriting multiple pages in the tree.
LSM trees have good compression ratio.
The advantage of the B tree is that each key exists in exactly one place in the index. The logging structured storage engine allows multiple copies of the same key to exist in different segments.

04. Coding and development

Read-schema (schemaless) databases do not enforce a schema, so they can contain a mix of binary data types and new data types written at different times. In large-scale systems, there are many nodes and cyclical upgrades are performed, so data type or schema changes cannot be reflected immediately. In other words, compatibility is important due to the coexistence of previous code and data types. Forward compatibility is more difficult to achieve as it requires that older code can also read the data written by the new code. Conversely, backward compatibility means that existing code and data types are known, so it is easy for new code to understand and apply them. Let's look at data encoding formats to solve this.

Data encoding format

Encoding here refers to the conversion from an in-memory representation to a byte string, and is also called serialization or marshaling. (Serializable in Java, Marshal in Ruby) The opposite is decryption (parsing, deserialization, unmarshalling). (In the book, the word serialization is expressed as encoding to avoid confusion with transaction units.) XML and JSON Standardized encodings include JSON and XML. Although it is the most common, when dealing with numbers, there is a problem that JSON does not distinguish between floating point numbers, and XML and CSV cannot distinguish between numbers and strings of numbers. Both do not support binary strings and are encoded with Base64, which increases the data size. And both support schema. There is talk about whether it is right to use binary encoding to reduce the size, but if the size is not significantly different from JSON encoding, it is not worth the risk of harming readability. Looking at binary encoding and JSON encoding using message packs (Figure 4-1.), when thinking about encoding readability (as an example only), JSON felt more appropriate. Currently, when companies use binary encoding, there is data stored in a format that indicates that it is binary encoded in a schema, and data that is simply stored in XML format according to medical standards. Protocol Buffers, Thrift, and Avro use schemas to describe binary encoding formats. Protocol Buffers and Thrift Prokotol Buffer was developed by Google, and Thrift was developed by Facebook. Both define a schema and create schema implementation classes. Using this, you can encode and decode schema records. Thrift has two binary encoding formats, and when using the compact protocol rather than the binary protocol, field names are expressed as field tags, so bytes can be further reduced and encoded to 34 bytes. The protocol butter is 33 bytes and is encoded similarly to Thrift's compact protocol. To support schema evolution, field tags cannot be changed as this may render the data unrecognizable. In terms of compatibility, backward compatibility must be treated as optional because new fields cannot be required for old data. And from the perspective of upward compatibility, only optional fields can be deleted, and required fields cannot be deleted. ~~If you think about it, the compatibility part is obvious, so you just need to understand it and move on.~~ Protocol buffers have a repeatable indicator so that when reading previous data, you only see the last element in the optional field. Thrift has a dedicated list datatype and supports nested lists. Abro It was developed because Thrift was not suitable for Hadoop. The schema is the same, but it uses a JSON-based language, there are no tag numbers, and the shortest encoding is 32 bytes. The field has no identifying information for its data type, and reading can only be decoded correctly if the code uses the exact same schema. Avro's forward compatibility means that you can have a new version of the write schema and an older version of the read schema. Conversely, backwards compatibility means that you can have a new version of your read schema and an older version of your write schema. In addition, there are drivers (ODBC, JDBC) that decrypt into in-memory data structures specialized for specific database vendors.

Dataflow mode

There is a possibility of data loss if an older application updates data recorded by a newer version of the application. Therefore, there is a way to migrate, but it does not happen often because it is expensive, and most relational databases allow the addition of schemas with null as the default value. Alternatively, you can create data dumps from your data warehouse and backups, such as snapshots. Data flow through service The data encoding used by the server and client enables compatibility between service API versions. In web services, REST is a design philosophy based on HTTP principles, and SOAP is an XML-based protocol for network API requests. SOAP is described in the WSDL language and allows clients to generate code to access remote services by calling local classes and methods. Remote procedure calls (RPC) are difficult to predict as network requests, so measures for failure handling are required.

Part 2. Distributed data

Part 3. Derivation

11. Stream processing

Let’s create a simple but well-functioning system Batch processing limits the input to a finite size in advance, but stream processing means that the size of the input is not limited and processing occurs in the order in which it is input. Because events are processed as they occur, reflection is processed quickly and progressively. Events typically contain timestamps and can be, for example, a user's action, a single line in a log, or data occurring on a device. Just as a file in batch processing can be read by multiple people, in streaming, events produced by a producer can be processed by multiple consumers.

Messaging system

By having a messaging system, multiple producer nodes can deliver messages to the same topic and multiple consumer nodes can receive messages from one topic. Publish/subscribe models use a variety of approaches. If the producer sends messages faster, it can use flow control to buffer messages in the queue or prevent further messages from being sent. So how should we handle the issue in stream processing? Many messaging systems communicate directly between producers and consumers over a network without going through intermediate nodes. In this case, message knowledge must be taken into consideration, and in case of message loss, the producer must retry or write to disk or create a replica to ensure that no messages are lost. However, you must also consider the possibility of losing message buffers for retries.

Message Broker

Message Broker There is an alternative to using a message queue, which is a type of database optimized for processing message streams. It runs as a server and connects to a broker server, where producers and consumers exchange messages. In the process, the producer does not wait for the consumer to process the message, and depending on the status of the queue, the producer may process the event after a long time has passed before the consumer. However, unlike direct stream processing, using a message broker allows you to manage data about message delivery from producers and consumers in one place. Message brokers are similar to databases, but are not suitable for storing persistent data. Additionally, the task queue is small and the consumer processes it by subscribing to specific patterns and topics, and although it does not support arbitrary queries, it notifies the client as soon as a data change occurs. If there are multiple consumers, producers can load-balance messages to parallelize message processing tasks to distribute the cost. Or, you can fan out the message to all consumers. Because a consumer may experience a failure when processing a message, it must explicitly allow the broker's message queue to be cleared through an acknowledgment**. If an acknowledgment is not received, the broker retries by sending the message to another consumer, but since that consumer already has work to process, the messages may not be processed in the order intended by the producer. By having an independent queue for each consumer, the message order can be prevented from changing independently, because the order is very important when causality between messages must be considered. Log-based message broker is a concept that originated from a combination of the advantages of a notification function with the persistence of a database and the low latency of a messaging system. Apache Kafka, Amazon Kinesis Streams, and Twitter's distributed logs work this way. When a producer sends a message, the message is added to a topic partition file, and the consumer reads the partition file in order. This method can process millions of messages per second by storing all messages and partitioning them across multiple devices, and can protect against failures by duplicating messages. Because partitions are sequential by message offset, when a consumer processes a message offset, offsets smaller than the current offset are known to have been processed. You can think of it as being used like the log sequential number method used in single leader database replication. (Prevention of omission when duplicating leader and follower data) If message processing is expensive and you want to parallelize processing on a message-by-message basis, but message order is not important, a JMS/AMQP-style message broker is suitable. If throughput is high and message processing speed is fast, but message order is important, a log-based approach is effective. Since the log in a circular buffer state has a limited size, when the buffer is full, the oldest fragments are discarded and filled with a new log, which may cause consumers to lose some messages. And if the consumer processing speed is slow, additional disk space is used, which means that the processing speed becomes very slow as the disk is used, unlike maintaining it only in memory. However, this has no effect because it is independent of the partitions handled by other consumers. Even when playing old messages, there is message offset information, so iterative processing can be done simply by changing the current offset setting only on the consumer client. This has the advantage of processing without affecting the input data, similar to batch processing.

Databases and Streams

It is also possible to apply message and stream ideas to databases. Writing the log itself to the database lets you know that changes have occurred in the data. When you update an item in the database, the index and data warehouse also need to be updated. In cases where it is difficult to dump the entire database, dual recording records changes in data in each system in the application code. This can cause race conditions to intersect requests, resulting in inconsistent records in the database and search index.

change data capture (CDC)

Change data capture is the process of observing all data changes and extracting them into a form that can be replicated to other systems. Since the change data log is applied to the index and data warehouse in order, the competition risk mentioned above can be prevented. Change data capture is another bureau-derived data system. To implement capture in this way, database triggers are used, but they are prone to failure and have a large performance overhead, so using replication log parsing is a more robust method. By performing log compaction, we remove the actual values and retain the most recent values, so when rebuilding a derived data system such as an index, there is a way to obtain a full copy by scanning all keys in offset order of the compacted log topic. (Function provided by Apache Kafka)

Event log

Event Sourcing refers to a development architecture pattern that stores all event logs. (In event sourcing, application state is maintained by applying event logs.) Event sourcing, a technique developed by the DDD community, stores event logs, but unlike change data capture, it reflects what happened at the application level and only adds log data, not recommending deletion, updates, etc. And reproducing it reconstructs the current system state, but requires the full history of events. The event itself is a fact at the time of creation and is distinct from the command, which is the user request itself. When an event occurs, the consumer cannot reject it, so when a command comes, it is synchronously validated and the event is generated by verifying the command with a serializable transaction. A series of events can be thought of as all mutable state derived from an event log, with each subset of the cache representing the latest value of each record and index taken from the log. This means that immutable events provide more information than the current state. For example, it is similar to recording all details, such as recording incorrect transaction details in a financial transaction in the ledger and refunds. This makes it possible to create a state-of-the-art event log that is simple in terms of concurrency control and atomic with one write in one place. However, depending on the situation, it may be inefficient to retrieve only the latest status using the event log recorded in this way. The concept of allowing various reading views by separating the format for writing and reading data is called Separation of Command and Query Responsibilities (CORS). When it's possible to transition data from a write-optimized event log to a read-optimized application state, the debate over normalization versus denormalization makes little sense. For read-optimized views, it makes perfect sense to denormalize the data.

Especially when the number of reads is much larger than the number of writes.
A scenario where one team of developers can focus on the complex domain model included in the write model, and another team can focus on the read model and user interface.
Scenarios where the system is expected to evolve over time and may contain multiple versions of a model, or where business rules change regularly.

Stream processing

Stream processing can be used to retrieve data from an event, record it in a database, cache, etc., and query it. It can also send the event itself directly to the user, or produce one or more output streams. When creating a derived stream like this, the piece of code is called an operator or operation. Unlike batch processing, stream input is not limited, so monitoring for failures is essential. Monitoring systems are useful in situations such as monitoring price changes in financial markets or machine malfunctions. For complex event processing, you can use CEP to search for specific event patterns. If an event matches, it emits a complex event containing the details of the event pattern. It is contrary to general database data queries in that event queries are stored for a long time and event queries are searched. Analytics allows you to aggregate bulk events and obtain statistical metrics. It is generally calculated based on a fixed time interval and uses probabilistic algorithms such as Bloom filters and hyperloglogs to provide approximate results.