AEWS [3기] 4주차 - OpenTelemetry

Prometheus와 Grafana를 이용하면 시스템의 각종 메트릭을 수집하고, 시각화 할 수 있습니다.

하지만 Application 내부의 동작, 예를 들면 서비스들간의 Trace 정보, 호출한 SQL 및 실행시간, 연결된 소켓서버와의 통신, ...)에 대해서는 모티터링 및 분석에 한계가 있습니다.

위의 Tracing, Metrics, Logs를 Observability의 3대 기둥이라고 표현합니다.

Observability의 정의는 (https://opentelemetry.io/docs/what-is-opentelemetry/)

Observability is the ability to understand the internal state of a system by examining its outputs. In the context of software, this means being able to understand the internal state of a system by examining its telemetry data, which includes traces, metrics, and logs.

 

1. OpenTelemetry

OpenTelemetry는 2021년에 CNCF의 인큐베이팅 레벨로 올라선 프로젝트입니다.

프로젝트의 홈페이지의 메인페이지를 분석해 보겠습니다.

• https://opentelemetry.io/

 

위 그림에 OpenTelemetry의 많은 부분이 담겨 있습니다.

가장 중요한 것은 OpenTelemetry는 제품이 아니라 API, SDK 및 tool  이라는 것이며, 여러 모니터링 벤더들이 지키는 "표준" 이라는 것입니다.

OpenTelemetry의 간략한 역사를 알아보겠습니다.

Lightstep은 Tracing에 강점을 가지고 발전하고 있었고, Google이 주도하는 OpenCensus는 어플리케이션 분석에 필요한  Instrument에 강점을 가지고 발전하고 있었습니다.

위의 2개의 프로젝트가 가진 강점을 하나로 합쳐서 2019년에 OpenTelemetry 프로젝트로 통합하여 발전하고 있습니다.

 

이 블로그에서는 간단한 샘플웹어플리케이션을 이용하여 OpenTelemetry 를 단계적으로 알아가 보겠습니다.

 

2. "계산기" 샘플 웹어플리케이션

• “Spring Boot + JAVA 17” 환경에서 운영되는 사칙연산 계산기 웹어플리케이션
• 총 5개의 마이크로서비스로 구성
• 계산 결과를 MySQL DBMS 에 저장

 

2.1. “plus” 마이크로 서비스

QueryString으로 2개의 숫자를 전달 받은 후 덧셈 결과를 리턴하는 간단한 REST API 입니다.

 

2.2. “front” 마이크로 서비스

계산기 웹화면을 제공하고 사칙연산을 수행한 후 결과를 화면에 출력하고, MySQL DBMS에 계산 히스토리를 저장하는 웹어플리케이션입니다.

 

CALCULATOR.tar

0.35MB

 

Docker-Compose를 이용하여 "계산기" 샘플웹어플리케이션을 실행해 보겠습니다.

위의 파일을 다운받은후 압축해제하면 아래와 같은 디렉토리 구조가 나오게 됩니다.

각 마이크로서비스에는 Dockefile이 존재합니다.

FROM eclipse-temurin:17-jre ADD target/front-1.0.jar /app.jar ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.5.0/opentelemetry-javaagent.jar /opentelemetry-javaagent.jar CMD ["java", "-jar", "/app.jar"]

이미지에 추가한 opentelemetry-javaagent.jar 에 대해서는 아래에서 부가 설명합니다.

이러한 도커파일을 이용하여 아래와 같은 이미지를 생성한 후  Docker Hub 에 PUSH 된 상태입니다.

• yu3papa/calc-divide:otel
• yu3papa/calc-multiply:otel
• yu3papa/calc-minus:otel
• yu3papa/calc-plus:otel
• yu3papa/calc-front:otel

주요 작업은 front  디렐토리 하위의 docker-compose.yaml 파일을 수정하면서 진행합니다.

 

계산의 결과는 MySQL DBMS 에 저장합니다. MySQL DBMS는 컨테이너로 미리 실행해 놓겠습니다.

docker container run -d \                                   --restart=always \                                   --name=mysqldb \                                   -e MYSQL_ROOT_PASSWORD=edu \                                   -e MYSQL_DATABASE=calculator \                                   -p 3308:3306 \                                   mysql:8

 

계산기 어플리케이션 실행을 위해 docker-compose.yaml 파일을 아래와 같이 수정하고 실행합니다.

services:   front:     image: yu3papa/calc-front:otel     environment:       MYSQL_IP: 192.168.10.4  # <-- 본인 실습 IP       MYSQL_PORT: 3308       ENDPOINT_PLUS: http://plus:8081       ENDPOINT_MINUS: http://minus:8082       ENDPOINT_MULTIPLY: http://multiply:8083       ENDPOINT_DIVIDE: http://divide:8084     ports:       - 8080:8080     depends_on:       - plus       - minus       - multiply       - divide   plus:     image: yu3papa/calc-plus:otel   minus:     image: yu3papa/calc-minus:otel   multiply:     image: yu3papa/calc-multiply:otel   divide:     image: yu3papa/calc-divide:otel

 

docker compose 를 이용하여 실행하고 웹 화면에 접속하여 사칙연산을 수행해 봅니다.

[yu3papa@iworks front]$ docker compose up -d [+] Running 6/6  ✔ Network front_default       Created                                                                                                                                              0.0s  ✔ Container front-minus-1     Started                                                                                                                                              0.3s  ✔ Container front-multiply-1  Started                                                                                                                                              0.3s  ✔ Container front-plus-1      Started                                                                                                                                              0.3s  ✔ Container front-divide-1    Started                                                                                                                                              0.4s  ✔ Container front-front-1     Started                                                                                                                                              0.7s [yu3papa@iworks front]$ docker compose ps NAME               IMAGE                        COMMAND                  SERVICE    CREATED         STATUS         PORTS front-divide-1     yu3papa/calc-divide:otel     "/__cacert_entrypoin…"   divide     7 seconds ago   Up 7 seconds front-front-1      yu3papa/calc-front:otel      "/__cacert_entrypoin…"   front      7 seconds ago   Up 6 seconds   0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp front-minus-1      yu3papa/calc-minus:otel      "/__cacert_entrypoin…"   minus      7 seconds ago   Up 7 seconds front-multiply-1   yu3papa/calc-multiply:otel   "/__cacert_entrypoin…"   multiply   7 seconds ago   Up 7 seconds front-plus-1       yu3papa/calc-plus:otel       "/__cacert_entrypoin…"   plus       7 seconds ago   Up 7 seconds

 

이때 front 마이크로서비스의 로그를 확인해 보면 Spring Boot  관련 로그이외의 것은 보이지 않습니다.

[yu3papa@iworks front]$ docker compose logs front front-1  | front-1  |   .   ____          _            __ _ _ front-1  |  /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \ front-1  | ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ front-1  |  \\/  ___)| |_)| | | | | || (_| |  ) ) ) ) front-1  |   '  |____| .__|_| |_|_| |_\__, | / / / / front-1  |  =========|_|==============|___/=/_/_/_/ front-1  | front-1  |  :: Spring Boot ::                (v3.3.1) front-1  | front-1  | 2025-03-01T02:05:28.458Z  INFO 1 --- [           main] c.jadecross.calc.front.FrontApplication  : Starting FrontApplication v1.0 using Java 17.0.11 with PID 1 (/app.jar started by root in /) front-1  | 2025-03-01T02:05:28.477Z  INFO 1 --- [           main] c.jadecross.calc.front.FrontApplication  : No active profile set, falling back to 1 default profile: "default" front-1  | 2025-03-01T02:05:32.916Z  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. front-1  | 2025-03-01T02:05:33.363Z  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 337 ms. Found 1 JPA repository interface. front-1  | 2025-03-01T02:05:35.883Z  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 8080 (http) front-1  | 2025-03-01T02:05:35.898Z  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat] front-1  | 2025-03-01T02:05:35.898Z  INFO 1 --- [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.25] front-1  | 2025-03-01T02:05:35.941Z  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext front-1  | 2025-03-01T02:05:35.943Z  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 7048 ms front-1  | 2025-03-01T02:05:36.135Z  INFO 1 --- [           main] o.hibernate.jpa.internal.util.LogHelper  : HHH000204: Processing PersistenceUnitInfo [name: default] front-1  | 2025-03-01T02:05:36.231Z  INFO 1 --- [           main] org.hibernate.Version                    : HHH000412: Hibernate ORM core version 6.5.2.Final front-1  | 2025-03-01T02:05:36.288Z  INFO 1 --- [           main] o.h.c.internal.RegionFactoryInitiator    : HHH000026: Second-level cache disabled front-1  | 2025-03-01T02:05:36.762Z  INFO 1 --- [           main] o.s.o.j.p.SpringPersistenceUnitInfo      : No LoadTimeWeaver setup: ignoring JPA class transformer front-1  | 2025-03-01T02:05:36.803Z  INFO 1 --- [           main] cohttp://m.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting... front-1  | 2025-03-01T02:05:37.220Z  INFO 1 --- [           main] cohttp://m.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection cohttp://m.mysql.cj.jdbc.ConnectionImpl@156cfa20 front-1  | 2025-03-01T02:05:37.223Z  INFO 1 --- [           main] cohttp://m.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed. front-1  | 2025-03-01T02:05:38.374Z  INFO 1 --- [           main] o.h.e.t.j.p.i.JtaPlatformInitiator       : HHH000489: No JTA platform available (set 'hibernate.transaction.jta.platform' to enable JTA platform integration) front-1  | Hibernate: create table CalcHistory (id bigint not null auto_increment, calculatedAt varchar(255), expression varchar(255), result varchar(255), primary key (id)) engine=InnoDB front-1  | 2025-03-01T02:05:38.446Z  INFO 1 --- [           main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' front-1  | 2025-03-01T02:05:38.771Z  WARN 1 --- [           main] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning front-1  | 2025-03-01T02:05:38.800Z  INFO 1 --- [           main] o.s.b.a.w.s.WelcomePageHandlerMapping    : Adding welcome page template: index front-1  | 2025-03-01T02:05:39.268Z  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port 8080 (http) with context path '/' front-1  | 2025-03-01T02:05:39.285Z  INFO 1 --- [           main] c.jadecross.calc.front.FrontApplication  : Started FrontApplication in 13.38 seconds (process running for 15.465) front-1  | 2025-03-01T02:05:39.534Z  INFO 1 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet' front-1  | 2025-03-01T02:05:39.535Z  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet' front-1  | 2025-03-01T02:05:39.536Z  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 1 ms front-1  | 2025-03-01T02:05:52.220Z  INFO 1 --- [nio-8080-exec-5] c.jadecross.calc.front.CalcController    : front [START] calculate: front-1  | Hibernate: insert into CalcHistory (calculatedAt,expression,result) values (?,?,?) front-1  | 2025-03-01T02:05:52.760Z  INFO 1 --- [nio-8080-exec-5] c.jadecross.calc.front.CalcController    : front [END] calculate: 9+3-2 = 10.0

 

3. OpenTelemetry Zero-code Instrumentation

OpenTelemetry에는 각 프로그래밍 언어별로 어플리케이션 프로파일링할 수 있는 라이브러리를 제공하고 있습니다.

• https://opentelemetry.io/docs/languages/

이러한 라이브러리를 이용하여 어플리케이션 프로파일링 하는 방법은 크게 2가지로 분류됩니다.

• Code-based solutions via official APIs and SDKs for most languages
  • 개발자가 직접 코드에 Telemetry 수집 코드를 추가
• Zero-code solutions
  • 기존 코드 수정없이 Telmetry 수집 가능

이번 블로그에서는 "Zero-code solutions"으로 진행합니다. 현실적으로 개발자분들이 Telemtry 수집코드를 추가하는것은 많은 부담이 되어 반발이 심할것입니다.

자바 언어를 예로 들어보겠습니다.

Java 언어는 표준옵션으로 -javaagent 옵션을 제공합니다.

[yu3papa@iworks front]$ java -help ...(생략)...     -javaagent:<jarpath>[=<options>]                   load Java programming language agent, see java.lang.instrument

 

-javaagent 옵션은 개발자가 작성한 소스코드를 컴파일한 ByteCode 가 실행되어 Memory로 로드 될때 추가적인 코드를 인젝션하는 방식으로 Low-Level 수준의 AOP 프로그래밍입니다.

https://blog.naver.com/ukja/120041273574

 

Dockerfile에 포함되어 있던 opentelemetry-javaagent.jar 파일의 내부 구조에 Class 파일이 메모리에 적재될때 변환작업하는 부분이 있고 아래와 같은 구조를 갖습니다.

자바 프로그램을 실행할때 opentelemetry-javaagent.jar 파일을 -javaagent 옵션으로 적용하면 Telemetry Data(Trace, Metric, Log)를 수집할 수 있습니다.

java -javaagent:path/to/opentelemetry-javaagent.jar \      -Dotel.service.name=your-service-name \      -Dotel.traces.exporter=zipkin \      -jar myapp.jar

• https://opentelemetry.io/docs/zero-code/java/agent/

하지만 위와 같은 방식은 이미 빌드된 이미지에 적용할 수 가 없습니다.

그래서 OpenTelemtry에서는 약속된 환경변수를 이용하여 적용할 수 있게 합니다.

JAVA_TOOL_OPTIONS="-javaagent:path/to/opentelemetry-javaagent.jar" OTEL_SERVICE_NAME=your-service-name OTEL_TRACES_EXPORTER=zipkin java -jar myapp.jar

 

아래는 OpenTelemetry를 이용하여 수집한 Trace, Metric, Log를 어떤 모니터링 솔루션에게 보낼지 결정하는 환경변수 목록입니다.

 

3.1. Metric, Trace, Log 정보를 Console로 내보내기

Metric, Trace, Log 정보를 Console로 내보내기 위해 docker-compose.yaml 파일의 front 서비스를 아래와 같이 수정합니다.

-->  기본 Exporter가 Console 입니다.

 

전체 docker-compose.yaml

services:   front:     image: yu3papa/calc-front:otel     environment:       MYSQL_IP: 192.168.10.4       MYSQL_PORT: 3308       ENDPOINT_PLUS: http://plus:8081       ENDPOINT_MINUS: http://minus:8082       ENDPOINT_MULTIPLY: http://multiply:8083       ENDPOINT_DIVIDE: http://divide:8084       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: front       OTEL_TRACES_EXPORTER: console       OTEL_METRICS_EXPORTER: console       OTEL_LOGS_EXPORTER: console     ports:       - 8080:8080     depends_on:       - plus       - minus       - multiply       - divide   plus:     image: yu3papa/calc-plus:otel   minus:     image: yu3papa/calc-minus:otel   multiply:     image: yu3papa/calc-multiply:otel   divide:     image: yu3papa/calc-divide:otel

 

docker compose 를 다시 시작하고 간단한 사칙연산 후 front 서비스의 로그를 확인하면 javaagent 옵션이 적용된것과 [otel.javaage xxxx-xx-xx 로 시작하는 로그가 추가된것을 확인할 수 있습니다.

[yu3papa@iworks front]$ docker compose down [+] Running 6/6  ✔ Container front-front-1     Removed                                                                                                                                              0.2s  ✔ Container front-minus-1     Removed                                                                                                                                              0.2s  ✔ Container front-divide-1    Removed                                                                                                                                              0.2s  ✔ Container front-plus-1      Removed                                                                                                                                              0.2s  ✔ Container front-multiply-1  Removed                                                                                                                                              0.2s  ✔ Network front_default       Removed                                                                                                                                              0.1s [yu3papa@iworks front]$ docker compose up -d [+] Running 6/6  ✔ Network front_default       Created                                                                                                                                              0.0s  ✔ Container front-divide-1    Started                                                                                                                                              0.3s  ✔ Container front-plus-1      Started                                                                                                                                              0.4s  ✔ Container front-minus-1     Started                                                                                                                                              0.3s  ✔ Container front-multiply-1  Started                                                                                                                                              0.3s  ✔ Container front-front-1     Started                                                                                                                                              0.7s [yu3papa@iworks front]$ docker compose logs -f front front-1  | Picked up JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar front-1  | OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended front-1  | [otel.javaagent 2025-03-01 02:52:49:603 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.5.0 front-1  | front-1  |   .   ____          _            __ _ _ front-1  |  /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \ front-1  | ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ front-1  |  \\/  ___)| |_)| | | | | || (_| |  ) ) ) ) front-1  |   '  |____| .__|_| |_|_| |_\__, | / / / / front-1  |  =========|_|==============|___/=/_/_/_/ front-1  | front-1  |  :: Spring Boot ::                (v3.3.1) front-1  | front-1  | 2025-03-01T02:53:01.892Z INFO 'Starting FrontApplication v1.0 using Java 17.0.11 with PID 1 (/app.jar started by root in /)' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: cohttp://m.jadecross.calc.front.FrontApplication:] {} front-1  | 2025-03-01T02:53:01.892Z  INFO 1 --- [           main] c.jadecross.calc.front.FrontApplication  : Starting FrontApplication v1.0 using Java 17.0.11 with PID 1 (/app.jar started by root in /) front-1  | 2025-03-01T02:53:01.939Z INFO 'No active profile set, falling back to 1 default profile: "default"' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: cohttp://m.jadecross.calc.front.FrontApplication:] {} front-1  | 2025-03-01T02:53:01.939Z  INFO 1 --- [           main] c.jadecross.calc.front.FrontApplication  : No active profile set, falling back to 1 default profile: "default" front-1  | 2025-03-01T02:53:03.331Z INFO 'Bootstrapping Spring Data JPA repositories in DEFAULT mode.' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: org.springframework.data.repository.config.RepositoryConfigurationDelegate:] {} front-1  | 2025-03-01T02:53:03.331Z  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. front-1  | 2025-03-01T02:53:03.432Z INFO 'Finished Spring Data repository scanning in 83 ms. Found 1 JPA repository interface.' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: org.springframework.data.repository.config.RepositoryConfigurationDelegate:] {} front-1  | 2025-03-01T02:53:03.432Z  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 83 ms. Found 1 JPA repository interface. front-1  | 2025-03-01T02:53:04.317Z INFO 'Tomcat initialized with port 8080 (http)' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: org.springframework.boot.web.embedded.tomcat.TomcatWebServer:] {} front-1  | 2025-03-01T02:53:04.317Z  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 8080 (http) front-1  | 2025-03-01T02:53:04.35Z INFO 'Starting service [Tomcat]' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: org.apache.catalina.core.StandardService:] {} front-1  | 2025-03-01T02:53:04.351Z  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat] front-1  | 2025-03-01T02:53:04.351Z INFO 'Starting Servlet engine: [Apache Tomcat/10.1.25]' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: org.apache.catalina.core.StandardEngine:] {} front-1  | 2025-03-01T02:53:04.352Z  INFO 1 --- [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.25] front-1  | 2025-03-01T02:53:04.414Z INFO 'Initializing Spring embedded WebApplicationContext' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/]:] {} front-1  | 2025-03-01T02:53:04.415Z  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext front-1  | 2025-03-01T02:53:04.417Z INFO 'Root WebApplicationContext: initialization completed in 2301 ms' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext:] {} front-1  | 2025-03-01T02:53:04.417Z  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2301 ms front-1  | 2025-03-01T02:53:04.768Z INFO 'HHH000204: Processing PersistenceUnitInfo [name: default]' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: org.hibernate.jpa.internal.util.LogHelper:] {} front-1  | 2025-03-01T02:53:04.768Z  INFO 1 --- [           main] o.hibernate.jpa.internal.util.LogHelper  : HHH000204: Processing PersistenceUnitInfo [name: default] front-1  | 2025-03-01T02:53:04.914Z INFO 'HHH000412: Hibernate ORM core version 6.5.2.Final' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: org.hibernate.Version:] {} front-1  | 2025-03-01T02:53:04.914Z  INFO 1 --- [           main] org.hibernate.Version                    : HHH000412: Hibernate ORM core version 6.5.2.Final front-1  | 2025-03-01T02:53:04.987Z INFO 'HHH000026: Second-level cache disabled' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: org.hibernate.cache.internal.RegionFactoryInitiator:] {} front-1  | 2025-03-01T02:53:04.987Z  INFO 1 --- [           main] o.h.c.internal.RegionFactoryInitiator    : HHH000026: Second-level cache disabled front-1  | 2025-03-01T02:53:05.561Z INFO 'No LoadTimeWeaver setup: ignoring JPA class transformer' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: org.springframework.orm.jpa.persistenceunit.SpringPersistenceUnitInfo:] {} front-1  | 2025-03-01T02:53:05.561Z  INFO 1 --- [           main] o.s.o.j.p.SpringPersistenceUnitInfo      : No LoadTimeWeaver setup: ignoring JPA class transformer front-1  | 2025-03-01T02:53:05.603Z INFO 'HikariPool-1 - Starting...' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: cohttp://m.zaxxer.hikari.HikariDataSource:] {} front-1  | 2025-03-01T02:53:05.603Z  INFO 1 --- [           main] cohttp://m.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting... front-1  | 2025-03-01T02:53:06.163Z INFO 'HikariPool-1 - Added connection cohttp://m.mysql.cj.jdbc.ConnectionImpl@77227b1f' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: cohttp://m.zaxxer.hikari.pool.HikariPool:] {} front-1  | 2025-03-01T02:53:06.163Z  INFO 1 --- [           main] cohttp://m.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection cohttp://m.mysql.cj.jdbc.ConnectionImpl@77227b1f front-1  | 2025-03-01T02:53:06.18Z INFO 'HikariPool-1 - Start completed.' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: cohttp://m.zaxxer.hikari.HikariDataSource:] {} front-1  | 2025-03-01T02:53:06.180Z  INFO 1 --- [           main] cohttp://m.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed. front-1  | [otel.javaagent 2025-03-01 02:53:06:270 +0000] [main] INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'SELECT INFORMATION_SCHEMA.KEYWORDS' : c40ead15de14450e238c286742c662db f214ced31d3bfde3 CLIENT [tracer: io.opentelemetry.jdbc:2.5.0-alpha] AttributesMap{data={thread.id=1, thread.name=main, db.name=calculator, db.operation=SELECT, db.sql.table=INFORMATION_SCHEMA.KEYWORDS, server.port=3308, db.statement=SELECT WORD FROM INFORMATION_SCHEMA.KEYWORDS WHERE RESERVED=? ORDER BY WORD, db.system=mysql, db.connection_string=mysql://192.168.10.4:3308, server.address=192.168.10.4, db.user=root}, capacity=128, totalAddedValues=11} front-1  | [otel.javaagent 2025-03-01 02:53:06:351 +0000] [main] INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'SELECT calculator' : bd0d98ce39ff63104209f3616cadf491 2669b6dd9194b4e9 CLIENT [tracer: io.opentelemetry.jdbc:2.5.0-alpha] AttributesMap{data={thread.id=1, thread.name=main, db.name=calculator, db.operation=SELECT, server.port=3308, db.statement=SELECT @@character_set_database, @@sql_mode, db.system=mysql, db.connection_string=mysql://192.168.10.4:3308, server.address=192.168.10.4, db.user=root}, capacity=128, totalAddedValues=10} front-1  | 2025-03-01T02:53:07.739Z INFO 'HHH000489: No JTA platform available (set 'hibernate.transaction.jta.platform' to enable JTA platform integration)' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: org.hibernate.engine.transaction.jta.platform.internal.JtaPlatformInitiator:] {} front-1  | 2025-03-01T02:53:07.739Z  INFO 1 --- [           main] o.h.e.t.j.p.i.JtaPlatformInitiator       : HHH000489: No JTA platform available (set 'hibernate.transaction.jta.platform' to enable JTA platform integration) front-1  | 2025-03-01T02:53:07.819Z INFO 'Initialized JPA EntityManagerFactory for persistence unit 'default'' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean:] {}

 

4. OTEL - Trace (zipkin)

zipkin은 마이크로서비스간의 Trace를 추적하는 오픈소스 모니터링 솔루션입니다.

• https://zipkin.io/

OpenZipkin · A distributed tracing system

Zipkin도 컨테이너로 수행하기 위해 docker-compose.yaml 파일에 아래 내용을 추가하겠습니다.

서두에 언급하였듯이 OpenTelemetry는 제품이 아니라 표준입니다.

OpenTelemetry가 적용된 어플리케이션에서 Trace 정보를 표준에 맞게 노출해주면 표준을 지키는 어떠한 솔루션에서도 시각화화가 가능합니다.

OpenTelemetry가 수집한 Trace 정보를 Zipkin에게 보내려면 아래와 같은 환경변수를 설정합니다.

front 마이크로서비스에서 수집된 Trace를 zipkin 으로 내보내기 위해  docker-compose.yaml 파일을 아래와 같이 수정합니다.

 

전체 docker-compose.yaml

services:   front:     image: yu3papa/calc-front:otel     environment:       MYSQL_IP: 192.168.10.4       MYSQL_PORT: 3308       ENDPOINT_PLUS: http://plus:8081       ENDPOINT_MINUS: http://minus:8082       ENDPOINT_MULTIPLY: http://multiply:8083       ENDPOINT_DIVIDE: http://divide:8084       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: front       OTEL_TRACES_EXPORTER: console,zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_METRICS_EXPORTER: console       OTEL_LOGS_EXPORTER: console     ports:       - 8080:8080     depends_on:       - plus       - minus       - multiply       - divide       - zipkin   plus:     image: yu3papa/calc-plus:otel   minus:     image: yu3papa/calc-minus:otel   multiply:     image: yu3papa/calc-multiply:otel   divide:     image: yu3papa/calc-divide:otel   zipkin:     container_name: zipkin     image: openzipkin/zipkin:3     ports:       - 9411:9411

 

docker compose 를 다시 시작하고 간단한 사칙연산을 5회 수행합니다.

[yu3papa@iworks front]$ docker compose down [+] Running 6/6  ✔ Container front-front-1     Removed                                                                                                                                              0.2s  ✔ Container front-minus-1     Removed                                                                                                                                              0.2s  ✔ Container front-multiply-1  Removed                                                                                                                                              0.3s  ✔ Container front-divide-1    Removed                                                                                                                                              0.2s  ✔ Container front-plus-1      Removed                                                                                                                                              0.2s  ✔ Network front_default       Removed                                                                                                                                              0.1s [yu3papa@iworks front]$ docker compose up -d [+] Running 10/10  ✔ zipkin Pulled                                                                                                                                                                   17.6s    ✔ fe1bd6a77cbf Pull complete                                                                                                                                                     1.1s    ✔ 954be67f124b Pull complete                                                                                                                                                     1.2s    ✔ 0219dbc309bb Pull complete                                                                                                                                                     1.3s    ✔ 9212666e9465 Pull complete                                                                                                                                                    10.7s    ✔ b5137681fb36 Pull complete                                                                                                                                                    10.7s    ✔ feef8f32146e Pull complete                                                                                                                                                    10.7s    ✔ 5b24a42be3cf Pull complete                                                                                                                                                    10.7s    ✔ 740e41e36047 Pull complete                                                                                                                                                    10.7s    ✔ fa1761cde767 Pull complete                                                                                                                                                    14.4s [+] Running 7/7  ✔ Network front_default       Created                                                                                                                                              0.0s  ✔ Container front-minus-1     Started                                                                                                                                              1.0s  ✔ Container zipkin            Started                                                                                                                                              0.9s  ✔ Container front-divide-1    Started                                                                                                                                              1.0s  ✔ Container front-multiply-1  Started                                                                                                                                              1.0s  ✔ Container front-plus-1      Started                                                                                                                                              0.9s  ✔ Container front-front-1     Started                                                                                                                                              0.7s

 

Zipkin 웹-UI로 접속 후 [RUN QUERY] 를 클릭하여 수집된 Trace를 확인합니다.

• http://192.168.10.4:9411

 

조회된 Trace 중에 하나를 선택하고 [SHOW] 버튼을 클릭하여 Trace 정보를 확인하면  front 어플리케이션 로직내에서 호출된 메소드 흐름과, 실행된 SQL 도 확인이 가능합니다.

 

이제 다른 마이크로서비스에도 Trace정보를 zipkin으로 내보낼수 있도록 환경변수를 설정합니다.

services:   front:     image: yu3papa/calc-front:otel     environment:       MYSQL_IP: 192.168.10.4       MYSQL_PORT: 3308       ENDPOINT_PLUS: http://plus:8081       ENDPOINT_MINUS: http://minus:8082       ENDPOINT_MULTIPLY: http://multiply:8083       ENDPOINT_DIVIDE: http://divide:8084       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: front       OTEL_TRACES_EXPORTER: console,zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_METRICS_EXPORTER: console       OTEL_LOGS_EXPORTER: console     ports:       - 8080:8080     depends_on:       - plus       - minus       - multiply       - divide       - zipkin   plus:     image: yu3papa/calc-plus:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: plus       OTEL_TRACES_EXPORTER: zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_METRICS_EXPORTER: console       OTEL_LOGS_EXPORTER: console   minus:     image: yu3papa/calc-minus:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: minus       OTEL_TRACES_EXPORTER: zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_METRICS_EXPORTER: console       OTEL_LOGS_EXPORTER: console   multiply:     image: yu3papa/calc-multiply:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: multiply       OTEL_TRACES_EXPORTER: zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_METRICS_EXPORTER: console       OTEL_LOGS_EXPORTER: console   divide:     image: yu3papa/calc-divide:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: divide       OTEL_TRACES_EXPORTER: zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_METRICS_EXPORTER: console       OTEL_LOGS_EXPORTER: console   zipkin:     container_name: zipkin     image: openzipkin/zipkin:3     ports:       - 9411:9411

 

docker compose를 다시 시작하고, 계산기 웹어플리케이션에서 사칙연산을 모두 포함해서 1회 수행한 후 zipkin에서 Trace 정보를 확인합니다.

[yu3papa@iworks front]$ docker compose down [yu3papa@iworks front]$ docker compose up -d

 

Trace 정보를 확인해 보면 마이크로서비스간의 추적정보를 확인할 수 있습니다.

 

5. Prometheus + Grafana를 이용한 OpenTelemetry Metric 시각화

계산기 어플리케이션의 메트릭을 Prometheus에서 수집할 수 있도록 노출하고 Grafana 에서 시각화해 보겠습니다.

OpenTelemetry의 Metric을 Prometheus 에 노출하려면 아래와 같은 환경변수를 이용합니다.

 

5.1. Metric 내보내기 설정

먼저 front  어플리케이션 적용해 보겟습니다.

 

docker compose를 다시 시작하고, 

[yu3papa@iworks front]$ docker compose down [yu3papa@iworks front]$ docker compose up -d

 

계산기 웹어플리케이션의 9464 포트로 접속하면 Prometheus에서 수집할 수 있도록 Metric이 노출된것을 확인할 수 있습니다.

 

5.2. Prometheus 설정

Promethus도 컨테이너로 실행할 예정입니다. Prometheus가 계산기 웹어플리케이션의 메트릭을 수집하도록 Prometheus 설정파일을 생성합니다.

[yu3papa@iworks front]$ mkdir -p docker/prometheus [yu3papa@iworks front]$ vi docker/prometheus/prometheus.yaml

 

global:   scrape_interval: 10s   scrape_timeout: 10s   evaluation_interval: 10s scrape_configs:   - job_name: otel_java-agent_calculator     static_configs:       - targets:           - front:9464           - plus:9464           - minus:9464           - multiply:9464           - divide:9464

 

prometheus를 컨테이너로 실행하기 위해 docker-compose.yaml  파일을 수정합니다.

전체 docker-compose.yaml

services:   front:     image: yu3papa/calc-front:otel     environment:       MYSQL_IP: 192.168.10.4       MYSQL_PORT: 3308       ENDPOINT_PLUS: http://plus:8081       ENDPOINT_MINUS: http://minus:8082       ENDPOINT_MULTIPLY: http://multiply:8083       ENDPOINT_DIVIDE: http://divide:8084       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: front       OTEL_TRACES_EXPORTER: console,zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_METRICS_EXPORTER: console,prometheus       OTEL_EXPORTER_PROMETHEUS_HOST: 0.0.0.0       OTEL_EXPORTER_PROMETHEUS_PORT: 9464       OTEL_LOGS_EXPORTER: console     ports:       - 8080:8080       - 9464:9464     depends_on:       - plus       - minus       - multiply       - divide       - zipkin   plus:     image: yu3papa/calc-plus:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: plus       OTEL_TRACES_EXPORTER: zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_METRICS_EXPORTER: console       OTEL_LOGS_EXPORTER: console   minus:     image: yu3papa/calc-minus:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: minus       OTEL_TRACES_EXPORTER: zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_METRICS_EXPORTER: console       OTEL_LOGS_EXPORTER: console   multiply:     image: yu3papa/calc-multiply:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: multiply       OTEL_TRACES_EXPORTER: zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_METRICS_EXPORTER: console       OTEL_LOGS_EXPORTER: console   divide:     image: yu3papa/calc-divide:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: divide       OTEL_TRACES_EXPORTER: zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_METRICS_EXPORTER: console       OTEL_LOGS_EXPORTER: console   zipkin:     container_name: zipkin     image: openzipkin/zipkin:3     ports:       - 9411:9411   prometheus:     container_name: prometheus     image: prom/prometheus:v2.45.6     volumes:       - ./docker/prometheus/prometheus.yaml:/etc/prometheus/prometheus.yaml     command:       - --config.file=/etc/prometheus/prometheus.yaml     ports:       - 9090:9090

 

docker compose를 다시 시작하고, 

[yu3papa@iworks front]$ docker compose down [yu3papa@iworks front]$ docker compose up -d

 

9090 포트로 접속하고 Prometheus에 front 어플리케이션 Target 이 정상 등록 되었는지 확인합니다.

 

front 어플리케이션의 수집된 메트릭도 확인합니다.

 

5.3. Grafana 설정

Grafana를 실행하기전에 Prometheus를 DataSource로 등록하도록 grafana-datasources.yaml 파일을 미리 생성하겠습니다.

[yu3papa@iworks front]$ mkdir -p docker/grafana [yu3papa@iworks front]$ vi docker/grafana/grafana-datasources.yaml

apiVersion: 1 datasources:   - name: prometheus     type: prometheus     uid: prometheus     access: proxy     orgId: 1     url : http://prometheus:9090 # http://192.168.56.100:9090     basicAuth: false     isDefault: false     version: 1     editable: false     jsonData:       httpMethod: GET

 

Grafana 컨테이너 실행을 위해 docker-compose.yaml 파일을 아래와 같이 수정합니다.

 

전체 docker-compose.yaml

services:   front:     image: yu3papa/calc-front:otel     environment:       MYSQL_IP: 192.168.10.4       MYSQL_PORT: 3308       ENDPOINT_PLUS: http://plus:8081       ENDPOINT_MINUS: http://minus:8082       ENDPOINT_MULTIPLY: http://multiply:8083       ENDPOINT_DIVIDE: http://divide:8084       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: front       OTEL_TRACES_EXPORTER: console,zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_METRICS_EXPORTER: console,prometheus       OTEL_EXPORTER_PROMETHEUS_HOST: 0.0.0.0       OTEL_EXPORTER_PROMETHEUS_PORT: 9464       OTEL_LOGS_EXPORTER: console     ports:       - 8080:8080       - 9464:9464     depends_on:       - plus       - minus       - multiply       - divide       - zipkin   plus:     image: yu3papa/calc-plus:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: plus       OTEL_TRACES_EXPORTER: zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_LOGS_EXPORTER: console       OTEL_METRICS_EXPORTER: prometheus       OTEL_EXPORTER_PROMETHEUS_HOST: 0.0.0.0       OTEL_EXPORTER_PROMETHEUS_PORT: 9464   minus:     image: yu3papa/calc-minus:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: minus       OTEL_TRACES_EXPORTER: zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_LOGS_EXPORTER: console       OTEL_METRICS_EXPORTER: prometheus       OTEL_EXPORTER_PROMETHEUS_HOST: 0.0.0.0       OTEL_EXPORTER_PROMETHEUS_PORT: 9464   multiply:     image: yu3papa/calc-multiply:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: multiply       OTEL_TRACES_EXPORTER: zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_LOGS_EXPORTER: console       OTEL_METRICS_EXPORTER: prometheus       OTEL_EXPORTER_PROMETHEUS_HOST: 0.0.0.0       OTEL_EXPORTER_PROMETHEUS_PORT: 9464   divide:     image: yu3papa/calc-divide:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: divide       OTEL_TRACES_EXPORTER: zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_LOGS_EXPORTER: console       OTEL_METRICS_EXPORTER: prometheus       OTEL_EXPORTER_PROMETHEUS_HOST: 0.0.0.0       OTEL_EXPORTER_PROMETHEUS_PORT: 9464   zipkin:     container_name: zipkin     image: openzipkin/zipkin:3     ports:       - 9411:9411   prometheus:     container_name: prometheus     image: prom/prometheus:v2.45.6     volumes:       - ./docker/prometheus/prometheus.yaml:/etc/prometheus/prometheus.yaml     command:       - --config.file=/etc/prometheus/prometheus.yaml     ports:       - 9090:9090   grafana:     container_name: grafana     image: grafana/grafana:11.1.0     volumes:       - ./docker/grafana/grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml     ports:       - 3000:3000     depends_on:       - prometheus

 

docker compose를 다시 시작하고, 

[yu3papa@iworks front]$ docker compose down [yu3papa@iworks front]$ docker compose up -d

 

3000 번 포트로 접속하고 Grafana 로그인 후 Prometheus 서버가 DataSource로 등록되었는지 확인

 

아래 2개의 DashBoard를 임포트한후 관찰

 

 

6. OepnTelemetry Collector

OpenTelmetry  라이브러리가 수집한 Trace, Metric, Log 등의 포멧은 표준을 따릅니다. 그래서 표준 포멧을 지원하는 모니터링 도구는 모두 시각화 할 수가 있습니다.

이런 상황에서 모든 Telemetry 데이터를 중앙에서 수집한 후 변환하고, 또는 배치처리한 후 내보내기하면 어떨까 생각하게 되엇습니다. 이러한 요구사항으로 출발한 제품이 Otel Collector  입니다.

• https://opentelemetry.io/docs/collector/

Collector

 

 

7. OpenTelemetry + Loki + Grafana 를 이용한 Log 수집

Grafana Loki는 로그 수집 시스템입니다.  같은 Grafana Lab 에서 만든 제품인 만큼 Grafana 와 통합이 잘됩니다.

Loki를 컨테이너로 실행하기 위해 docker-compose.yaml 파일에 아래 내용을 추가합니다.

 

이제 계산기 마이크로서비스의 Log 정보를 Loki로 바로 보내지 않고 Otel Collector에서 수집후 Loki 로 내보내기 하겠습니다.

Otel Collector 설정파일을 아래와 같이 생성합니다.

[yu3papa@iworks front]$ mkdir -p docker/collector [yu3papa@iworks front]$ vi docker/collector/otel-collector.yaml

receivers:   otlp:     protocols:       http:         endpoint: 0.0.0.0:4318 exporters:   loki:     endpoint: http://loki:3100/loki/api/v1/push service:   pipelines:     logs:       receivers: [otlp]       exporters: [loki]

 

OTEL-Collector 컨테이너 실행을 위해 docker-compose.yaml 파일 수정합니다.

 

계산기 마이크로서비스들의 Log 정보를 Otel Collector로 내보내도록 환경변수 설정을 합니다.

 

전체 docker-compose.yaml

services:   front:     image: yu3papa/calc-front:otel     environment:       MYSQL_IP: 192.168.10.4       MYSQL_PORT: 3308       ENDPOINT_PLUS: http://plus:8081       ENDPOINT_MINUS: http://minus:8082       ENDPOINT_MULTIPLY: http://multiply:8083       ENDPOINT_DIVIDE: http://divide:8084       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: front       OTEL_TRACES_EXPORTER: console,zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_METRICS_EXPORTER: console,prometheus       OTEL_EXPORTER_PROMETHEUS_HOST: 0.0.0.0       OTEL_EXPORTER_PROMETHEUS_PORT: 9464       OTEL_LOGS_EXPORTER: console,otlp       OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: http://collector:4318/v1/logs     ports:       - 8080:8080       - 9464:9464     depends_on:       - plus       - minus       - multiply       - divide       - zipkin       - collector   plus:     image: yu3papa/calc-plus:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: plus       OTEL_TRACES_EXPORTER: zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_METRICS_EXPORTER: prometheus       OTEL_EXPORTER_PROMETHEUS_HOST: 0.0.0.0       OTEL_EXPORTER_PROMETHEUS_PORT: 9464       OTEL_LOGS_EXPORTER: otlp       OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: http://collector:4318/v1/logs   minus:     image: yu3papa/calc-minus:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: minus       OTEL_TRACES_EXPORTER: zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_METRICS_EXPORTER: prometheus       OTEL_EXPORTER_PROMETHEUS_HOST: 0.0.0.0       OTEL_EXPORTER_PROMETHEUS_PORT: 9464       OTEL_LOGS_EXPORTER: otlp       OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: http://collector:4318/v1/logs   multiply:     image: yu3papa/calc-multiply:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: multiply       OTEL_TRACES_EXPORTER: zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_METRICS_EXPORTER: prometheus       OTEL_EXPORTER_PROMETHEUS_HOST: 0.0.0.0       OTEL_EXPORTER_PROMETHEUS_PORT: 9464       OTEL_LOGS_EXPORTER: otlp       OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: http://collector:4318/v1/logs   divide:     image: yu3papa/calc-divide:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: divide       OTEL_TRACES_EXPORTER: zipkin       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_METRICS_EXPORTER: prometheus       OTEL_EXPORTER_PROMETHEUS_HOST: 0.0.0.0       OTEL_EXPORTER_PROMETHEUS_PORT: 9464       OTEL_LOGS_EXPORTER: otlp       OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: http://collector:4318/v1/logs   zipkin:     container_name: zipkin     image: openzipkin/zipkin:3     ports:       - 9411:9411   prometheus:     container_name: prometheus     image: prom/prometheus:v2.45.6     volumes:       - ./docker/prometheus/prometheus.yaml:/etc/prometheus/prometheus.yaml     command:       - --config.file=/etc/prometheus/prometheus.yaml     ports:       - 9090:9090   grafana:     container_name: grafana     image: grafana/grafana:11.1.0     volumes:       - ./docker/grafana/grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml     ports:       - 3000:3000     depends_on:       - prometheus       - loki   loki:     container_name: loki     image:  grafana/loki:2.9.9     command:       - --config.file=/etc/loki/local-config.yaml     ports:       - 3100   collector:     container_name: collector     image:  otel/opentelemetry-collector-contrib:0.104.0     command:       - --config=/etc/otel-contrib/otel-collecotr.yaml     volumes:       - ./docker/collector/otel-collector.yaml:/etc/otel-contrib/otel-collecotr.yaml     ports:       - 4318     depends_on:       - loki

 

docker compose를 다시 시작하고, 

[yu3papa@iworks front]$ docker compose down [yu3papa@iworks front]$ docker compose up -d

 

Grafana 에 로그인 후 loki를 DataSource로 등록합니다.

“Grafana  HOME  Explore” 메뉴에서 데이터 소스를 “Loki”를 선택하고 수집된 front어플리케이션의 Log 정보를 확인할 수 있습니다.

 

다음 실습을 위해 Grafana 에서 Loki 컨테이너를 DataSource 등록을 위한 설정 파일 수정합니다.

[yu3papa@iworks front]$ vi docker/grafana/grafana-datasources.yaml

apiVersion: 1 datasources:   - name: prometheus     type: prometheus     uid: prometheus     access: proxy     orgId: 1     url : http://prometheus:9090     basicAuth: false     isDefault: true     version: 1     editable: false     jsonData:       httpMethod: GET   - name: Loki     type: loki     uid: loki     access: proxy     orgId: 1     url: http://loki:3100     basicAuth: false     isDefault: false     version: 1     editable: false

 

9. 모든 시각화를 Grafana 로 통합하기

9.1. Grafana Tempo

Grafana Temp는 분산 Tracing의 Back-end로 사용하는 제품이며, 다양한 DataSource를 지원하고, Grafana에서 시각화할 수 있게 도와주는 솔루션입니다.

• https://grafana.com/oss/tempo/

 

Tempo 를 컨테이너로 실행하기 위한 Tempo  설정파일을 생성합니다.

[yu3papa@iworks front]$ mkdir -p docker/tempo [yu3papa@iworks front]$ vi docker/tempo/tempo.yaml

server:   http_listen_port: 3200 distributor:   receivers:     otlp:       protocols:         http: ingester:   max_block_duration: 5m compactor:   compaction:     block_retention: 1h metrics_generator:   registry:     external_labels:       source: tempo       cluster: docker-compose   storage:     path: /tmp/tempo/generator/wal     remote_write:       - url: http://prometheus:9090/api/v1/write         send_exemplars: true storage:   trace:     backend: local     wal:       path: /tmp/tempo/wal     local:       path: /tmp/tempo/blocks overrides:   metrics_generator_processors: [service-graphs, span-metrics]

 

Tempo 를 컨테이너로 실행하기 위해 docker-compose.yaml 에 내용을 추가합니다.

 

계산기 마이크로서비스의 Trace 정보를 zipkin과 Otel Collector로 내보내기 설정을 아래와 같이 합니다.

전체 docker-compose.yaml

services:   front:     image: yu3papa/calc-front:otel     environment:       MYSQL_IP: 192.168.10.4       MYSQL_PORT: 3308       ENDPOINT_PLUS: http://plus:8081       ENDPOINT_MINUS: http://minus:8082       ENDPOINT_MULTIPLY: http://multiply:8083       ENDPOINT_DIVIDE: http://divide:8084       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: front       OTEL_TRACES_EXPORTER: zipkin,otlp       OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: http://tempo:4318/v1/traces       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans       OTEL_METRICS_EXPORTER: console,prometheus       OTEL_EXPORTER_PROMETHEUS_HOST: 0.0.0.0       OTEL_EXPORTER_PROMETHEUS_PORT: 9464       OTEL_LOGS_EXPORTER: console,otlp       OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: http://collector:4318/v1/logs     ports:       - 8080:8080       - 9464:9464     depends_on:       - plus       - minus       - multiply       - divide       - zipkin       - collector       - tempo   plus:     image: yu3papa/calc-plus:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: plus       OTEL_METRICS_EXPORTER: prometheus       OTEL_EXPORTER_PROMETHEUS_HOST: 0.0.0.0       OTEL_EXPORTER_PROMETHEUS_PORT: 9464       OTEL_LOGS_EXPORTER: otlp       OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: http://collector:4318/v1/logs       OTEL_TRACES_EXPORTER: zipkin,otlp       OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: http://tempo:4318/v1/traces       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans   minus:     image: yu3papa/calc-minus:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: minus       OTEL_METRICS_EXPORTER: prometheus       OTEL_EXPORTER_PROMETHEUS_HOST: 0.0.0.0       OTEL_EXPORTER_PROMETHEUS_PORT: 9464       OTEL_LOGS_EXPORTER: otlp       OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: http://collector:4318/v1/logs       OTEL_TRACES_EXPORTER: zipkin,otlp       OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: http://tempo:4318/v1/traces       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans   multiply:     image: yu3papa/calc-multiply:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: multiply       OTEL_METRICS_EXPORTER: prometheus       OTEL_EXPORTER_PROMETHEUS_HOST: 0.0.0.0       OTEL_EXPORTER_PROMETHEUS_PORT: 9464       OTEL_LOGS_EXPORTER: otlp       OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: http://collector:4318/v1/logs       OTEL_TRACES_EXPORTER: zipkin,otlp       OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: http://tempo:4318/v1/traces       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans   divide:     image: yu3papa/calc-divide:otel     environment:       JAVA_TOOL_OPTIONS: -javaagent:/opentelemetry-javaagent.jar       OTEL_SERVICE_NAME: divide       OTEL_METRICS_EXPORTER: prometheus       OTEL_EXPORTER_PROMETHEUS_HOST: 0.0.0.0       OTEL_EXPORTER_PROMETHEUS_PORT: 9464       OTEL_LOGS_EXPORTER: otlp       OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: http://collector:4318/v1/logs       OTEL_TRACES_EXPORTER: zipkin,otlp       OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: http://tempo:4318/v1/traces       OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans   zipkin:     container_name: zipkin     image: openzipkin/zipkin:3     ports:       - 9411:9411   prometheus:     container_name: prometheus     image: prom/prometheus:v2.45.6     volumes:       - ./docker/prometheus/prometheus.yaml:/etc/prometheus/prometheus.yaml     command:       - --config.file=/etc/prometheus/prometheus.yaml     ports:       - 9090:9090   grafana:     container_name: grafana     image: grafana/grafana:11.1.0     volumes:       - ./docker/grafana/grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml     ports:       - 3000:3000     depends_on:       - prometheus       - loki   loki:     container_name: loki     image:  grafana/loki:2.9.9     command:       - --config.file=/etc/loki/local-config.yaml     ports:       - 3100   collector:     container_name: collector     image:  otel/opentelemetry-collector-contrib:0.104.0     command:       - --config=/etc/otel-contrib/otel-collecotr.yaml     volumes:       - ./docker/collector/otel-collector.yaml:/etc/otel-contrib/otel-collecotr.yaml     ports:       - 4318     depends_on:       - loki   tempo:     container_name: tempo     image:  grafana/tempo:main-7986de3     command:       - --config.file=/etc/tempo.yaml     volumes:       - ./docker/tempo/tempo.yaml:/etc/tempo.yaml     ports:       - 4317       - 4318       - 3200

 

docker compose를 다시 시작하고, 

[yu3papa@iworks front]$ docker compose down [yu3papa@iworks front]$ docker compose up -d

 

Grafana 에 로그인 후 Tempo를 DataSource로 등록합니다.

계산기 웹어플리케이션에서 간단한 사칙연산을 5회 수행하고, “Grafana ▶ HOME ▶ Explore” 메뉴에서 데이터 소스를 “Tempo”로 선택하고 수집된 front어플리케이션의 Trace 정보를 조회합니다.

 

이렇게 해서 Observability의 Signal(Trace, Metric, Log)를 Grfana로 통합하여 보았습니다.

 

발전과제로 Trace, Metric, Log 등을 상호 연관(Co-Relation)시켜 시스템 트러블슈팅 및 성능개선에 활용하면 상용 모니터링 도구와 견주어도 손색이 없는 Observability  환경을 구축할 수 있을 것입니다.