How to avoid cloud-washed software?

Working with cloud-native applications entails continuously tackling and implementing solutions to cross-cutting concerns. One of these concerns that every project is bound to run into comes to deploying highly scalable, available logging, and monitoring solutions.

You might ask, “how do we do that? Is it possible to find "one size fits all" solution for such a complex and volatile problem?” You need to look no further!

Taking into account our experience based on  working with production-grade environments , we propose a generic architecture, built totally from open source components, that certainly provide you with the highly performant and maintainable workload. To put this into concrete terms, this platform is characterized by its:

•  High availability - every component is available 24/7 providing users with constant service even in the case of a system failure.
•  Resiliency -  crucial data are safe thanks to redundancy and/or backups.
•  Scalability - every component is able to be replicated on demand accordingly to the current load.
•  Performance - ability to be used in any and all environments.
•  Compatibility - easily integrated into any workflows.
•  Open source - every component is accessible to anyone with no restrictions.

To build an environment that enables users to achieve outcomes described above, we decided to look at Elastic Stack, fully open source logging solution, structured in a modular way.

Elastic stack

Each component has a specific function, allowing it to be scaled in and out as needed. Elastic stack is composed of:

•  Elasticsearch - RESTful, distributed search and analytics engine built on Apache Lucene able to index copious amount of data.
•  Logstash - server-side data processing pipeline, able to transform, filter and enrich events on the fly.
•  Kibana – a feature-rich visualization tool, able to perform advanced analysis on your data.

While all this looks perfect, you still need to be cautious while deploying your Elastic Stack cluster. Any downtime or data loss caused by incorrect capacity planning can be detrimental to your business value. This is extremely important, especially when it comes to production environments. Everything has to be carefully planned, including worst-case scenarios. Concerns that may weigh on the successful Elastic stack configuration and deployment are described below.

High availability

When planning any reliable, fault-tolerant systems, we have to distribute its critical parts across multiple, physically separated network infrastructures. It will provide redundancy and eliminate single points of failure.

Scalability

ELK architecture allows you to scale out quickly. Having good monitoring tools setup makes it easy to predict and react to any changes in the system's performance. This makes it resilient and helps you optimize the cost of maintaining the solution.

Monitoring and alerts

A monitoring tool along with a detailed set of alerting rules will save you a lot of time. It lets you easily maintain the cluster, plan many different activities in advance, and react immediately if anything bad happens to your software.

Resource optimization

In order to maximize the stack performance, you need to plan the hardware (or virtualized hardware) allocation carefully. While data nodes need efficient storage, ingesting nodes will need more computing power and memory. While planning this take into consideration the number of events you want to process and amount of data that has to be stored to avoid many problems in the future.

Proper component distribution

Make sure the components are properly distributed across the VMs. Improper setup may cause high CPU and memory usage, can introduce bottlenecks in the system and will definitely result in lower performance. Let's take Kibana and ingesting node as an example. Placing them on one VM will cause poor user experience since UI performance will be affected when more ingesting power is needed and vice-versa.

Data replication

Storing crucial data requires easy access to your data nodes. Ideally, your data should be replicated across multiple availability zones which will guarantee redundancy in case of any issues.

Architecture

Our proposed architecture consists of five types of virtual machines - Routers, elastic masters, elastic data, ingestors, and Kibana instances. This toolset simplifies scaling of components while separating their responsibilities. Each of them has a different function:

•  Elasticsearch_master - controls indexes and Elasticsearch master. Responsible for creating new indexes, rolling updates and monitoring clusters health.
•  Elasticsearch_data - stores data and retrieves it as needed. Can be run both as hot and warm storage, as well as provides redundancy on data.
•  Ingestor - exposes input endpoints for events both while transforming and enriching data stored in Elasticsearch.
•  Kibana - provides users with visualizations by querying Elasticsearch data.
•  Router - serves as a single point of entry, both for users and services producing data events.

Architecting your Elastic Stack deployment in this way allows for the simple upgrade procedure. Thanks to using a single point of entry, switching to a new version of Elastic Stack is as simple as pointing HAProxy to an upgraded cluster.

Using a clustered structure also allows for freely adding data nodes as needed when your traffic inevitably grows.

We witness how the world of IT constantly changes. Today, like never before, it is more often defined as “being THE business” rather than just “supporting the business”. In other words, the conventional application architectures and development methods are slowly becoming inadequate in this new world. Grape Up, playing a key role in the cloud migration strategy, helps Fortune 1000 companies make a smooth transition. We build apps that support the business itself, we advocate the agile methodology, and implement DevOps to optimize performance.

To clarify the idea behind cloud native technologies, we’ve put together the most important insights to help you and your team understand the essentials and benefits of Cloud Native Applications:

Microservices architecture

First and foremost, one must come to terms with the fact that the traditional application architecture means complex development, long testing and releasing new features only in a specific period. Whereas, the microservices approach is nothing like that. It deconstructs an application into many functional components. These components can be updated and deployed separately without having any impact on other parts of the application.

Depending on its functionality, every microservice can be updated as often as needed. For instance, a microservice that contains functionalities of a dynamic business offering will not affect other parts of the app that barely change at all. Thanks to this, an application can be developed without changing its fundamental architecture. Gone are the days when IT teams had to alter most of the application just to change one piece.

Operational design

One of the biggest issues that our customers face before the migration is the burden of moving new code releases into production. Along with monolithic architectures that combine the whole code into one executable, new code releases require deploying the entire application. Because production environment isn’t the same as development environment, it often becomes impossible for developers to detect potential bugs before the release. Also, testing new features without moving the whole environment to the new app version can become tricky. This, in turn, complicates releasing new code. Microservices solve this problem prefectly. Since the environment is divided, any changes in code are separated to executables. Thanks to this, updates do not change the rest of the application, which is what clients are initially concerned about.

API

One of the indisputable advantage of microservices which outweighs all traditional methods is the fact that they communicate by means of API. With that said, you can release new features step by step with a new API version simultaneously. And if any failures appear, there is also a possibility to shut off access to the new API while the previous version of your app is still operational. In the meantime, you can work on the new function.

DevOps

At Grape Up, we often work on on-site projects alongside our clients. On the first day of the project, we are introduced to multiple groups that are in charge of various parts of the app’s lifecycle such as operations, deployment, QA or development. Each of them has its own processes. This creates long gaps between tasks being handed over from one group to another. Such gaps result in ridiculously long deployment time frames which are very harmful to an IT business, especially when frequent releases are more than welcome. To efficiently get rid of these obstacles and improve the whole process, we introduce clients to DevOps.

By and large, DevOps is nothing else, but an attempt to eliminate the gaps between IT groups. It’s an engineering culture that Grape Up experts teach clients to use. If followed properly, they are able to transform manual processes to automated ones and start getting things done faster and better. The most important thing is to find the pain point in the application’s lifecycle. Let’s say that the QA department doesn’t have enough resources to test software and delays the entire process in time. A solution to this can be either to migrate testing to a cloud-based environment or put developers in charge of creating tests that analyze the code. By doing so, the QA stage can take place simultaneously with the development stage, not after it. And this is what it takes to understand DevOps.

The transition to cloud-native software development is no longer an option, it is a necessity. We hope that all the reasons mentioned above prompted you to embark on a journey called “Cloud-Native”, a promising opportunity for your company to grow in the years to come. And even if you’re still feeling hesitant, don’t think twice. Our expertise combined with your vision can be a great start into a brighter future for your enterprise .