DEV Community

Pawel Wolanski
Pawel Wolanski

Posted on • Edited on

Commerce Cloud logs in OpenSearch locally

Issue

Commerce Cloud offers customers to view and search logs in dedicated instance of AWS OpenSearch. Disk quota there is limited, so after some time logs are gone there, but you can find them still available dumped in your dedicated Azure Blob Storage.

You can find that on your Cloud Portal: Environments > [given environment] : Cloud Storage > logs

Image description

Unfortunately these files are hard to use as blob entries are impossible to filter by creation date and also by content.

Solution

In order to filter files I have used java code to connect to Azure Blob Storage and then filter roughly by date.

Download Azure Blob Files

In below example I wanted to filter our files which has been created from 11-Aug-2023, 5 AM UTC to 12-Aug-2023 11:59 PM UTC:


package pl.com.like.zip;

import com.azure.identity.*;
import com.azure.storage.blob.*;
import com.azure.storage.blob.models.*;
import java.io.*;


public class BlobLogFetch {

    public static void main(String[] args) throws IOException {

        var connectionString =
                "AccountName=XXXXXXXXXXX;AccountKey=YYYYYYYYYYYYY;EndpointSuffix=core.windows.net;DefaultEndpointsProtocol=https;";
        var container = "commerce-logs-separated";


        var containerClient = new BlobContainerClientBuilder().connectionString(connectionString)
                .containerName(container).buildClient();
        System.out.println(containerClient.exists());
        var from = OffsetDateTime.of(2023, 8, 11, 5, 0, 0, 0, ZoneOffset.UTC);
        var to = OffsetDateTime.of(2023, 8, 12, 23, 59, 0, 0, ZoneOffset.UTC);

        containerClient.listBlobs().stream().filter(i -> i.getProperties().getCreationTime().compareTo(from) > 0)
                .filter(item -> item.getProperties().getCreationTime().compareTo(to) < 0).forEach(i -> {
                    var blob = containerClient.getBlobClient(i.getName());
                    var path = Paths.get(i.getName()).normalize().toAbsolutePath();
// create dirs
                    path.toFile().getParentFile().mkdirs();
                    blob.downloadToFile(path.toString(), Boolean.TRUE);


Enter fullscreen mode Exit fullscreen mode

Above code will list all blob files, filter by file properties if meets date range requirements. If creation date is fine, file is downloaded.

Install OpenSearch

Then, we would like to be able to store file content where data are easily searchable.

There are few options on table, but if we have access to SAP Commerce Cloud, we naturally have used Amazon OpenSearch, so we will follow same tooling locally.

It is possible to find OpenSearch on Docker Hub searching for opensearchproject/opensearch.

I have used linked docker-compose.yml as it contains additionally Dashboard.

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-node2:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:

Enter fullscreen mode Exit fullscreen mode

To run it locally just download file and run:

docker-compose up -d
Enter fullscreen mode Exit fullscreen mode

OpenSearch Configuration

When docker compose is started, you should be able to find OpenSearch Dashboard available on http://localhost:5601/ and logging with credentials admin:admin.

Initially instance will not have any data. We need to create an index first. To do that, click on hamburger menu: OpenSearch Plugins > Index Management, there go to: Index Management > Indices and create index. On my instance I have created index called commerce.

During index creation you can set up data mapping. Below you can find mapping which I have used:

{
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "_p": {
      "type": "text"
    },
    "kubernetes": {
      "type": "object",
      "properties": {
        "annotations": {
          "type": "object",
          "properties": {
            "ccv2_cx_sap_com_build-code": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "ccv2_cx_sap_com_deployment-id": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "cni_projectcalico_org_containerID": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "cni_projectcalico_org_podIP": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "cni_projectcalico_org_podIPs": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "data-ingest_dynatrace_com_injected": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "dynakube_dynatrace_com_injected": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "fluentbit_io_parser": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "oneagent_dynatrace_com_injected": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "platform-security-configmap-hash": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            }
          }
        },
        "container_hash": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "container_image": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "container_name": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "docker_id": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "host": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "labels": {
          "type": "object",
          "properties": {
            "app_kubernetes_io_component": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "app_kubernetes_io_managed-by": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "app_kubernetes_io_name": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "app_kubernetes_io_part-of": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "ccv2_cx_sap_com_build-code": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "ccv2_cx_sap_com_deployment-id": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "ccv2_cx_sap_com_platform-aspect": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "ccv2_cx_sap_com_service-name": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "ccv2_cx_sap_com_service-version": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "controller-revision-hash": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "controller-uid": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "job-name": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "pod-template-hash": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "statefulset_kubernetes_io_pod-name": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "update_ccv2_cx_sap_com_mode": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            }
          }
        },
        "namespace_name": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "pod_id": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "pod_name": {
          "type": "text"
        }
      }
    },
    "log": {
      "type": "text"
    },
    "logs": {
      "type": "object",
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "@version": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "buildDate": {
          "type": "date"
        },
        "container": {
          "type": "object",
          "properties": {
            "image": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "name": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            }
          }
        },
        "content": {
          "type": "object",
          "properties": {
            "apiToken": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "apiUrl": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "clusterID": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "hasHost": {
              "type": "boolean"
            },
            "hostGroup": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "initialConnectRetry": {
              "type": "long"
            },
            "monitoringNodes": {
              "type": "object",
              "properties": {
                "aks-ey4c7ymiac-68098534-vmss000012": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "aks-ey4c7ymiac-68098534-vmss000013": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "aks-ey4c7ymiac-68098534-vmss000016": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "aks-ey4c7ymiac-68098534-vmss000017": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "aks-ey4c7ymiac-68098534-vmss000019": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "aks-ey4c7ymiac-68098534-vmss00001a": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                }
              }
            },
            "networkZone": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "paasToken": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "proxy": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "skipCertCheck": {
              "type": "boolean"
            },
            "tenantUUID": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "tlsCert": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "trustedCAs": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            }
          }
        },
        "contextMap": {
          "type": "object",
          "properties": {
            "CronJob": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "RemoteAddr": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "Tenant": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "audit": {
              "type": "object",
              "properties": {
                "actionId": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "actionName": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "data": {
                  "type": "object",
                  "properties": {
                    "PK": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "changeType": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "cronJob": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "cronJobResult": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "cronJobStatus": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "gyIsActive": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "gyUID": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "idVerified": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "idVerifiedAt": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "job": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "newUser": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "prevUser": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "residenceVerified": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "residenceVerifiedAt": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "typePK": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "user": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    }
                  }
                },
                "sessionId": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "user": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                }
              }
            },
            "bytesSent": {
              "type": "long"
            },
            "commitMillis": {
              "type": "long"
            },
            "connectionStatus": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "myid": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "processMillis": {
              "type": "long"
            },
            "remoteHost": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "requestLine": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "sourceClassName": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "sourceMethodName": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "statusCode": {
              "type": "long"
            }
          }
        },
        "endOfBatch": {
          "type": "boolean"
        },
        "env": {
          "type": "object",
          "properties": {
            "containers": {
              "type": "object",
              "properties": {
                "image": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "name": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                }
              }
            },
            "dataIngestInjected": {
              "type": "boolean"
            },
            "failurePolicy": {
              "type": "boolean"
            },
            "installPath": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "installVersion": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "installerFlavor": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "installerTech": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "installerUrl": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "k8BasePodName": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "k8BasePodUID": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "k8ClusterID": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "k8Namespace": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "k8NodeName": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "k8PodName": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "mode": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "oneAgentInjected": {
              "type": "boolean"
            },
            "workloadKind": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "workloadName": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            }
          }
        },
        "gitCommit": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "goVersion": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "hostTenant": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "instant": {
          "type": "object",
          "properties": {
            "epochSecond": {
              "type": "text"
            },
            "nanoOfSecond": {
              "type": "text"
            }
          }
        },
        "level": {
          "type": "text"
        },
        "level_value": {
          "type": "long"
        },
        "logger": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "loggerFqcn": {
          "type": "text"
        },
        "loggerName": {
          "type": "text"
        },
        "logger_name": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "marker": {
          "type": "object",
          "properties": {
            "name": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            }
          }
        },
        "message": {
          "type": "text"
        },
        "msg": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "origin": {
          "type": "text"
        },
        "platform": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "thread": {
          "type": "text"
        },
        "threadId": {
          "type": "integer"
        },
        "threadPriority": {
          "type": "integer"
        },
        "thread_name": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "thrown": {
          "type": "object",
          "properties": {
            "cause": {
              "type": "object",
              "properties": {
                "cause": {
                  "type": "object",
                  "properties": {
                    "cause": {
                      "type": "object",
                      "properties": {
                        "commonElementCount": {
                          "type": "long"
                        },
                        "extendedStackTrace": {
                          "type": "object",
                          "properties": {
                            "class": {
                              "fields": {
                                "keyword": {
                                  "type": "keyword",
                                  "ignore_above": 256
                                }
                              },
                              "type": "text"
                            },
                            "exact": {
                              "type": "boolean"
                            },
                            "file": {
                              "fields": {
                                "keyword": {
                                  "type": "keyword",
                                  "ignore_above": 256
                                }
                              },
                              "type": "text"
                            },
                            "line": {
                              "type": "long"
                            },
                            "location": {
                              "fields": {
                                "keyword": {
                                  "type": "keyword",
                                  "ignore_above": 256
                                }
                              },
                              "type": "text"
                            },
                            "method": {
                              "fields": {
                                "keyword": {
                                  "type": "keyword",
                                  "ignore_above": 256
                                }
                              },
                              "type": "text"
                            },
                            "version": {
                              "fields": {
                                "keyword": {
                                  "type": "keyword",
                                  "ignore_above": 256
                                }
                              },
                              "type": "text"
                            }
                          }
                        },
                        "localizedMessage": {
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          },
                          "type": "text"
                        },
                        "message": {
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          },
                          "type": "text"
                        },
                        "name": {
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          },
                          "type": "text"
                        }
                      }
                    },
                    "commonElementCount": {
                      "type": "long"
                    },
                    "extendedStackTrace": {
                      "type": "object",
                      "properties": {
                        "class": {
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          },
                          "type": "text"
                        },
                        "exact": {
                          "type": "boolean"
                        },
                        "file": {
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          },
                          "type": "text"
                        },
                        "line": {
                          "type": "long"
                        },
                        "location": {
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          },
                          "type": "text"
                        },
                        "method": {
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          },
                          "type": "text"
                        },
                        "version": {
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          },
                          "type": "text"
                        }
                      }
                    },
                    "localizedMessage": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "message": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "name": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    }
                  }
                },
                "commonElementCount": {
                  "type": "long"
                },
                "extendedStackTrace": {
                  "type": "object",
                  "properties": {
                    "class": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "exact": {
                      "type": "boolean"
                    },
                    "file": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "line": {
                      "type": "long"
                    },
                    "location": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "method": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "version": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    }
                  }
                },
                "localizedMessage": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "message": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "name": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                }
              }
            },
            "commonElementCount": {
              "type": "long"
            },
            "extendedStackTrace": {
              "type": "object",
              "properties": {
                "class": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "exact": {
                  "type": "boolean"
                },
                "file": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "line": {
                  "type": "long"
                },
                "location": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "method": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "version": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                }
              }
            },
            "localizedMessage": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "message": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "name": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            }
          }
        },
        "timeMillis": {
          "type": "long"
        },
        "ts": {
          "type": "date"
        },
        "version": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        }
      }
    },
    "record_date": {
      "type": "long"
    },
    "stream": {
      "type": "text"
    },
    "time": {
      "type": "date"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Feeding Search Index

Now is the time to upload data from local files to index. Data in blob items is already nicely prepared and stored in JSON format as 1-liner.

Additionally I have added code, which ignores default unsigned SSL certificate, which by default is used locally:


import groovy.json.JsonOutput
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.SSLSession;
import javax.net.ssl.HttpsURLConnection
import javax.net.ssl.SSLContext
import javax.net.ssl.SSLEngine
import javax.net.ssl.SSLSocketFactory
import javax.net.ssl.TrustManager
import javax.net.ssl.X509ExtendedTrustManager
import java.security.cert.CertificateException
import java.security.cert.X509Certificate
import java.util.Base64;

class UnsafeTrustManager extends X509ExtendedTrustManager {

    @Override
    void checkClientTrusted(X509Certificate[] x509Certificates, String s, Socket socket) throws CertificateException {}

    @Override
    void checkServerTrusted(X509Certificate[] x509Certificates, String s, Socket socket) throws CertificateException {}

    @Override
    void checkClientTrusted(X509Certificate[] x509Certificates, String s, SSLEngine sslEngine) throws CertificateException {}

    @Override
    void checkServerTrusted(X509Certificate[] x509Certificates, String s, SSLEngine sslEngine) throws CertificateException {}

    @Override
    void checkClientTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {}

    @Override
    void checkServerTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {}

    @Override
    X509Certificate[] getAcceptedIssuers() {
        return new X509Certificate[0]
    }

}


def handleError(httpResponseException) {
    println "Request failed with status code: ${httpResponseException.statusCode}"
    println "Error message: ${httpResponseException.message}"
}

def dir = new File('./')

// List all files in the directory
def postmanPost = new URL('https://localhost:9200/commerce/_doc')

def v = new HostnameVerifier() {
    public boolean verify(String hostname, SSLSession session) {
        return true;
    }
};

def sslContext = SSLContext.getInstance("TLS")
sslContext.init(null, new TrustManager[]{new UnsafeTrustManager()}, null)
def sslSocketFactory = sslContext.getSocketFactory()
def userCredentials = "admin:admin"
String basicAuth = Base64.getEncoder().encodeToString(userCredentials.getBytes());

HttpsURLConnection.setDefaultHostnameVerifier(v);

dir.eachFile { file ->

    if (file.name.endsWith(".log")) {
        println "Sending file ${file.name}"
        def inputFile = new File(dir, file.name);
        // Read file line by line
        inputFile.eachLine { line ->
            try {
                HttpsURLConnection conn = (HttpsURLConnection) postmanPost.openConnection();
                HttpsURLConnection.setDefaultHostnameVerifier(new HostnameVerifier() {
                    public boolean verify(String hostname, SSLSession session) {
                        return true;
                    }
                });
                conn.setSSLSocketFactory(sslSocketFactory);
                conn.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
                conn.setRequestProperty ("Authorization", "Basic ${basicAuth}");
                conn.setDoOutput(true);
                conn.setDoInput(true);
                conn.setRequestMethod("POST");
                conn.setHostnameVerifier(v);
                def os = conn.getOutputStream();
                os.write(line.trim().getBytes("UTF-8"));
                os.close();

                if (conn.getResponseCode() >= 300) {
                    println("Failed : HTTP error code : ${conn.getResponseCode()} for ${line}");
                }
            } catch(Exception e) {
                println("Error: ${e}, for line ${line}");
            }
        }
    }
}

Enter fullscreen mode Exit fullscreen mode

Run script by calling:

groovy [script-name].groovy
Enter fullscreen mode Exit fullscreen mode

After that, you will be able to see data in you Dashboard.

Top comments (0)