aws(学习笔记第四十四课) opensearch

aws(学习笔记第四十四课) opensearch

• 在aws上部署opensearch

学习内容：

• 部署opensearch
• 并了解什么是opensearch

---

1. 整体架构

1.1 代码链接

代码连接(opensearch-simple-domain)

1.2 整体架构

这里，

• 会构建一个aws的opensearch服务。
• 自动生成管理员的密码，并将管理员密码报错在aws secrets manager上。
• 这里，设定允许访问的source ip adress。

2. 什么是opensearch

2.1 elastic search的aws实现

Elasticsearch 是一个强大的分布式搜索和分析引擎，基于 Apache Lucene 构建，广泛应用于全文搜索、日志分析、实时数据处理等场景。以下是它的 主要功能：

• 全文搜索
• 实时搜索与分析
• 分布式架构与高可用性
• 日志与监控分析
• 结构化 & 非结构化数据处理
• 机器学习与 AI 增强
• 安全与权限管理

2.2 elastic search的数据源

Elasticsearch 可以与多种类型的数据源集成，支持从不同系统中导入数据并进行索引和搜索。以下是 Elasticsearch 主要支持的数据源类型及其集成方式：

• 关系型数据库（MySQL、PostgreSQL、Oracle 等）
• NoSQL 数据库（MongoDB、Cassandra 等）
• 日志系统（Logstash、Filebeat、Fluentd）
• 消息队列（Kafka、RabbitMQ）
• 文件系统（CSV、JSON、日志文件）
• 云服务（AWS S3、Google Cloud Storage）
• 大数据组件（Hadoop、Spark、Flink）
• API 数据（RESTful 服务、爬虫数据）

2.3 开始学习elastic search

elastic search get start

3. opensearch的cdk代码架构

3.1 cdk代码的整体架构

3.2 代码详细

3.2.1 创建对opensearch的访问限制

这里首先执行opensearch的版本，选择最新2.19。

        OPENSEARCH_VERSION = "2.19"# Add the authorized IP addresses (using CIDR format) that should# be granted access to the OpenSearch Domain.# Create an environment variable before running cdk deploy. E.g.:OPENSEARCH_ALLOWED_IP='["33.45.123.8/32"]'# allowed_ip_addresses = os.environ.get("OPENSEARCH_ALLOWED_IP", "x.x.x.x/32")allowed_ip_addresses = OPENSEARCH_ALLOWED_IP# Creating OpenSearch access policy to restrict# access to a specific list of IPs. We are allowing all# types of HTTP commands.opensearch_access_policy = cdk_iam.PolicyStatement(effect=cdk_iam.Effect.ALLOW,principals=[cdk_iam.AnyPrincipal()],actions=["es:ESHttp*"],resources=[],conditions={"IpAddress": {"aws:SourceIp": allowed_ip_addresses}})

3.2.2 创建对opensearch的访问密码

 # Generating a secret and storing it with AWS Secrets Manager.# https://aws.amazon.com/secrets-manager/# To list secret using CLI and jq, run:#   aws secretsmanager list-secrets | jq ".SecretList[].Name"# To retrieve a secret value using CLI and jq, run:#   aws secretsmanager get-secret-value --secret-id <secret-name>secret_opensearch_admin_password = cdk_sm.Secret(self, "OpenSearchDemoDomainAdminUser")

3.2.3 创建对opensearch的capacity config

# Capacity config documentation:# https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_opensearchservice/CapacityConfig.html#aws_cdk.aws_opensearchservice.CapacityConfig# Available instance types:# https://docs.aws.amazon.com/opensearch-service/latest/developerguide/supported-instance-types.htmlcapacity_config = cdk_opensearch.CapacityConfig(master_nodes=3,master_node_instance_type="t3.small.search",data_nodes=3,data_node_instance_type="t3.medium.search"   

3.2.4 创建对opensearch的ebs config

        # Available EBS options# https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_opensearchservice/EbsOptions.html#aws_cdk.aws_opensearchservice.EbsOptionsebs_config = EbsOptions(volume_size=10,volume_type=cdk_ec2.EbsDeviceVolumeType.GP3)

3.2.5 设定opensearch的enable zone awareness

        # Enabling zone awareness to allow data replication across AZ's.# https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_opensearchservice/ZoneAwarenessConfig.html#aws_cdk.aws_opensearchservice.ZoneAwarenessConfigzone_awareness_config = ZoneAwarenessConfig(availability_zone_count=3,enabled=True)

3.2.6 设定encrypt setting和用户名密码

        # Required when FGAC is enabledencryption_config = EncryptionAtRestOptions(enabled=True)# Required when FGAC is enabledopensearch_admin_user = "admin-user"advanced_security_config = AdvancedSecurityOptions(master_user_name=opensearch_admin_user,master_user_password=secret_opensearch_admin_password.secret_value)

3.2.7 开始真正创建opensearch

        # Required when FGAC is enabledencryption_config = EncryptionAtRestOptions(enabled=True)# Required when FGAC is enabledopensearch_admin_user = "admin-user"advanced_security_config = AdvancedSecurityOptions(master_user_name=opensearch_admin_user,master_user_password=secret_opensearch_admin_password.secret_value)

3.2.8 将必要的output进行输出

  cdk.CfnOutput(self,"OpenSearchDomainEndpoint", value=aos_domain.domain_endpoint)cdk.CfnOutput(self,"OpenSearchDashboardsURL", value=(aos_domain.domain_endpoint + "/_dashboards"))cdk.CfnOutput(self,"OpenSearchPasswordSecretName", value=secret_opensearch_admin_password.secret_name)cdk.CfnOutput(self,"OpenSearchAdminUser", value=opensearch_admin_user)

4 执行cdk创建opensearch

4.1 执行cdk的注意点

这里，如果直接执行如下命令创建opensearch，会报错。

cdk --require-approval never deploy

错误信息如下：

OpensearchSimpleDomainStack | 4/10 | 16:31:40 | CREATE_IN_PROGRESS   | AWS::Lambda::Function          | AWS679f53fac002430cb0da5b7982bd2287 (AWS679f53fac002430cb0da5b7982bd22872D164C4C) Resource creation Initiated
OpensearchSimpleDomainStack | 4/10 | 16:31:40 | CREATE_FAILED        | AWS::Lambda::Function          | AWS679f53fac002430cb0da5b7982bd2287 (AWS679f53fac002430cb0da5b7982bd22872D164C4C) Resource handler returned message: "The runtime parameter of nodejs14.x is no longer supported for creating or updating AWS Lambda functions. We recommend you use a supported runtime while creating or updating functions. (Service: Lambda, Status Code: 400, Request ID: c49cc9d9-f4b1-42e4-8a98-af597a4aa3f4) (SDK Attempt Count: 1)" (RequestToken: 07111475-e581-91e9-185c-ce93ab079201, HandlerErrorCode: InvalidRequest)
OpensearchSimpleDomainStack | 4/10 | 16:31:40 | CREATE_IN_PROGRESS   | AWS::Lambda::Function          | DummyLambdaRuntimeSetter (DummyLambdaRuntimeSetter4B38A37F) Resource creation Initiated
OpensearchSimpleDomainStack | 4/10 | 16:31:40 | CREATE_FAILED        | AWS::Lambda::Function          | DummyLambdaRuntimeSetter (DummyLambdaRuntimeSetter4B38A37F) Resource creation cancelled
OpensearchSimpleDomainStack | 4/10 | 16:31:40 | CREATE_FAILED        | AWS::OpenSearchService::Domain | OpensearchDemoDomain (OpensearchDemoDomainBEE1301C) Resource creation cancelled

，原因是，默认opensearch使用lambda是采用"nodejs14.x runtime，所以会报错。

4.2 执行cdk的错误回避对策(workaround)

cdk synth > template.yaml # 首先不直接部署cdk，而是将cdk输出到template.yaml
sed -i 's/nodejs14.x/nodejs18.x/g' template.yaml # 进行替换，升级到nodejs18
aws cloudformation deploy --template-file template.yaml --stack-name OpensearchSimpleDomainStack --capabilities CAPABILITY_IAM # 继续部署opensearch

4.3 创建opensearch的花费

创建了五六次opensearch，花费了$8,所以建议大家慎重测试opensearch！