In 2015, we launched the Amazon Elasticsearch Service (Amazon ES), a fully managed service that makes it easy for you to perform interactive log analysis, real-time application control, site search, and more.
Amazon ES has been a popular log analysis service due to its ability to consume large amounts of log data. In addition, with UltraWarm and cold storage levels, you can lower the cost of one-tenth of traditional hot storage on Amazon ES. Because Amazon ES integrates with Logstash, Amazon Kinesis Data Firehose, Amazon CloudWatch Logs, and AWS IoT, you can choose the secure data capture tool that meets your usage requirements.
Developers embrace open source software for many reasons. One of the main reasons is the freedom to use this software where and how they want. On January 21, 2021, Elastic NV announced that they would change their software licensing strategy. Following Elasticsearch version 7.10.2 and Kibana 7.10.2, they do not release new versions of Elasticsearch and Kibana under the permitted 2.0 version of the Apache license (ALv2). Instead, Elastic NV releases Elasticsearch and Kibana under the Elastic License, with the source code available under the Elastic License or Server Side Public License (SSPL). These licenses are not open source and do not give users the same freedom.
For this reason, we decided to create and maintain OpenSearch, a community-driven, open-source fork from the latest ALv2 version of Elasticsearch and Kibana. We are making a long-term investment in the OpenSearch project and recently released version 1.0.
OpenSearch provides a highly scalable system for fast access and response to large amounts of data with an integrated visualization tool, OpenSearch Dashboards, that makes it easy for users to explore their data. OpenSearch and OpenSearch Dashboards are originally derived from Elasticsearch 7.10.2 and Kibana 7.10.2. Like Elasticsearch and Apache Solr, OpenSearch is powered by the Apache Lucene Search Library.
Announces Amazon OpenSearch Service
Today, we rename the Amazon Elasticsearch Service to the Amazon OpenSearch Service because the service now supports OpenSearch 1.0. Although the name has changed, we will continue to deliver the same experiences without any negative impact on day-to-day operations, development methodology or business use.
Amazon OpenSearch Service offers a variety of open source engines to implement and run, including the currently available 19 versions of ALv2 Elasticsearch 7.10 and earlier and OpenSearch 1.0. We will continue to support and maintain the ALv2 Elasticsearch versions with security and bug fixes. We will deliver brand new features and functionality through OpenSearch and OpenSearch Dashboards. Amazon OpenSearch Service APIs will be backward compatible with existing service APIs, so there is no need to update your current client code or applications. We keep OpenSearch clients compatible with open source.
To get started, select in the AWS Management Console Create a domain. IN Step 1: Select implementation type, Choose OpenSearch 1.0 (latest).
We recommend OpenSearch 1.0 if you are implementing a new cluster and want access to the latest features and enhancements. OpenSearch 1.0 is compatible with open-source Elasticsearch 7.10 APIs and most clients.
Upgrading to OpenSearch 1.0
Amazon OpenSearch Service offers a hassle-free on-site upgrade path from existing Elasticsearch 6.x and 7.x managed clusters to OpenSearch. To upgrade a domain to OpenSearch 1.0 in the AWS Management Console, select the domain you want to upgrade, select Actions, and then select Upgrade domain.
Next, you can select a version you want to upgrade to OpenSearch 1.0 for your existing domain without creating a separate domain and migrating your data.
The upgrade process is irreversible. It cannot be paused or canceled. During an upgrade, you will not be able to make configuration changes to the domain. Before starting an upgrade, you can perform the pre-upgrade checks for issues that may block an upgrade and take a snapshot of the cluster by selecting Check upgrade eligibility.
Amazon OpenSearch Services launches the upgrade, which can take from 15 minutes to several hours to complete. To learn more, see Upgrading Elasticsearch and Service Software Updates in the Amazon OpenSearch Service Developer Guide.
OpenSearch includes the following features that were not previously available in open-source Elasticsearch.
|Advanced security||Provides encryption, authentication, authorization, and auditing features. They include integrations with Active Directory, LDAP, SAML, Kerberos, JSON web tokens and more. OpenSearch also provides fine-grained, role-based access control for indexes, documents and fields.|
|SQL query syntax||Provides the familiar SQL query syntax. Use aggregations, groups by and where clauses to examine your data. Read data such as JSON documents or CSV tables so you have the flexibility to use the format that works best for you.|
|Reporting||Schedule, export, and share reports from dashboards, saved searches, alerts, and visualizations.|
|Anomaly registration||Use machine learning anomaly detection based on the Random Cut Forest (RCF) algorithm to automatically detect anomalies when your data is ingested. Combine with alert to monitor data in near real time and automatically send notifications.|
|Index management||Define custom policies to automate routine index management tasks, e.g. Transfer and delete, apply them to indexes and index patterns and transform.|
|Performance Analyzer and RCA Framework||Query numerous cluster performance metrics and aggregations. Use PerfTop, the command line interface (CLI) to quickly view and analyze these metrics. Use the root causal analysis (RCA) framework to examine performance and reliability issues in clusters.|
|Asynchronous search||Run complex queries without worrying about query timing with Asynchronous search queries running in the background. Track the progress of the query, and retrieve partial results as they become available.|
|Track Analytics||Intake and visualize OpenTelemetry data for distributed applications. Visualize the event flow between these applications to identify performance issues.|
|Warning||Automatically monitor data and send warning messages to stakeholders. With an intuitive interface and a powerful API, you can easily configure, manage and monitor alerts. Create very specific warning conditions using OpenSearch’s full query language and scripting features.|
|k-NN search||Use machine learning to run the nearest neighbor search algorithm on billions of documents across thousands of dimensions with the same ease as running a standard OpenSearch query. Use aggregations and filter clauses to further refine similarity search operations. k-NN similar search options use cases such as product recommendations, fraud detection, image and video search, related document search and more.|
|Piped Processing Language||Provides a well-known request syntax with a comprehensive set of commands bounded by tubes (|) to query data.|
|Dashboard notebooks||Combine dashboards, visualizations, text and more to provide context and detailed explanations when analyzing data.|
OpenSearch 1.0 supports three new features not available in the existing Elasticsearch versions supported on the Amazon OpenSearch Service: Transforms, Data Streams, and Notebooks in OpenSearch Dashboards.
To interact with the OpenSearch community, we welcome pull requests through GitHub to fix bugs, improve performance and stability, or add new features. You can leave feedback in the OpenSearch Community Forum.
As of today, the Amazon Elasticsearch Service has been renamed the Amazon OpenSearch Service in all AWS regions. For more information, see the Amazon OpenSearch Service page.
You can send feedback to the AWS forum to the Amazon OpenSearch Service or through your usual AWS support contacts.