Skip to content

Commit

Permalink
Support Fetch Cilium Observability data to monitoring Cilium Service …
Browse files Browse the repository at this point in the history 
…network traffic (#12393)
  • Loading branch information
@mrproliu
mrproliu committed Jul 2, 2024
1 parent 75fc3e1 commit 2e8b4f9
Show file tree
Hide file tree
Showing 77 changed files with 7,031 additions and 186 deletions.
1 change: 1 addition & 0 deletions .licenserc.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ header:
- '**/src/main/proto/jaeger/**'
- '**/src/main/proto/mixer/**'
- '**/src/main/proto/policy/**'
- '**/src/main/proto/cilium/**'
- '**/src/main/proto/prometheus/client_model/metrics.proto'
- '**/src/main/proto/protoc-gen-swagger/**'
- '**/src/main/proto/validate/validate.proto'
Expand Down
5 changes: 5 additions & 0 deletions dist-material/release-docs/LICENSE
View file
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,8 @@ The text of each license is the standard Apache 2.0 license.
https://mvnrepository.com/artifact/com.linecorp.armeria/armeria/1.27.3 Apache-2.0
https://mvnrepository.com/artifact/com.linecorp.armeria/armeria-graphql/1.27.3 Apache-2.0
https://mvnrepository.com/artifact/com.linecorp.armeria/armeria-graphql-protocol/1.27.3 Apache-2.0
https://mvnrepository.com/artifact/com.linecorp.armeria/armeria-grpc/1.27.3 Apache-2.0
https://mvnrepository.com/artifact/com.linecorp.armeria/armeria-grpc-protocol/1.27.3 Apache-2.0
https://mvnrepository.com/artifact/com.linecorp.armeria/armeria-protobuf/1.27.3 Apache-2.0
https://mvnrepository.com/artifact/com.orbitz.consul/consul-client/1.5.3 Apache-2.0
https://mvnrepository.com/artifact/com.squareup.okhttp3/okhttp/3.14.9 Apache-2.0
Expand Down Expand Up @@ -295,6 +297,7 @@ The text of each license is the standard Apache 2.0 license.
https://mvnrepository.com/artifact/io.grpc/grpc-netty/1.63.0 Apache-2.0
https://mvnrepository.com/artifact/io.grpc/grpc-protobuf/1.63.0 Apache-2.0
https://mvnrepository.com/artifact/io.grpc/grpc-protobuf-lite/1.63.0 Apache-2.0
https://mvnrepository.com/artifact/io.grpc/grpc-services/1.61.0 Apache-2.0
https://mvnrepository.com/artifact/io.grpc/grpc-stub/1.63.0 Apache-2.0
https://mvnrepository.com/artifact/io.grpc/grpc-util/1.63.0 Apache-2.0
https://mvnrepository.com/artifact/io.micrometer/micrometer-commons/1.12.2 Apache-2.0
Expand Down Expand Up @@ -371,6 +374,7 @@ The text of each license is the standard Apache 2.0 license.
https://mvnrepository.com/artifact/org.xerial.snappy/snappy-java/1.1.8.4 Apache-2.0
https://mvnrepository.com/artifact/org.yaml/snakeyaml/2.0 Apache-2.0
https://npmjs.com/package/typescript/v/4.7.4 4.7.4 Apache-2.0
https://github.com/cilium/cilium/tree/v1.15.6/api/v1 Apache-2.0

========================================================================
BSD-2-Clause licenses
Expand Down Expand Up @@ -544,6 +548,7 @@ The text of each license is also included in licenses/LICENSE-[project].txt.
https://npmjs.com/package/nanoid/v/3.3.7 3.3.7 MIT
https://mvnrepository.com/artifact/org.checkerframework/checker-qual/3.33.0 MIT
https://mvnrepository.com/artifact/org.codehaus.mojo/animal-sniffer-annotations/1.23 MIT
https://mvnrepository.com/artifact/org.curioswitch.curiostack/protobuf-jackson/2.2.0 MIT
https://npmjs.com/package/pinia/v/2.0.28 2.0.28 MIT
https://npmjs.com/package/pinia/node_modules/vue-demi/v/0.13.11 0.13.11 MIT
https://npmjs.com/package/postcss/v/8.4.33 8.4.33 MIT
Expand Down
3 changes: 3 additions & 0 deletions dist-material/release-docs/LICENSE.tpl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ The text of each license is also included in licenses/LICENSE-[project].txt.
https://npmjs.com/package/{{ .Name }}/v/{{ .Version }} {{ .Version }} {{ .LicenseID }}
{{- end }}
{{- end }}
{{- if eq .LicenseID "Apache-2.0" }}
https://github.com/cilium/cilium/tree/v1.15.6/api/v1 Apache-2.0
{{- end }}
{{ end }}
=======================================================================
The zipkin-lens.jar dependency has more front-end dependencies in it and the front-end dependencies' licenses
Expand Down
7 changes: 6 additions & 1 deletion docs/en/changes/changes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,14 @@
* [Break Change] Update Nacos version to 2.3.2. Nacos 1.x server can't serve as cluster coordinator and configuration server.
* Support tracing trace query(SkyWalking and Zipkin) for debugging.
* Fix BanyanDB metrics query: used the wrong `Downsampling` type to find the schema.
* Support fetch cilium flow to monitoring network traffic between cilium services.
* Support `labelCount` function in the OAL engine.

#### UI

* Highlight search log keywords.
* Add Error URL in the browser log.
* Add a SolonMVC icon.
* Adding cilium icon and i18n for menu.

#### Documentation

Expand Down
5 changes: 5 additions & 0 deletions docs/en/concepts-and-designs/oal.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,11 @@ In this case, see `p99`, `p95`, `p90`, `p75`, and `p50` of all incoming requests

In this case, the p99 value of all incoming requests. The parameter is precise to a latency at p99, such as in the above case, and 120ms and 124ms are considered to produce the same response time.

- `labelCount`. The count of the label value.
> drop_reason_count = from(CiliumService.*).filter(verdict == "dropped").labelCount(dropReason);
In this case, the count of the drop reason of each Cilium service.

## Metrics name
The metrics name for storage implementor, alarm and query modules. The type inference is supported by core.

Expand Down
104 changes: 103 additions & 1 deletion docs/en/concepts-and-designs/scope-definitions.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -566,4 +566,106 @@ For `K8SEndpoint` and `K8SEndpointRelation`, they only have the following **prot
| componentId | The ID of component used in this call. | | string |
| destServiceName | The dest service name in kubernetes. | | string |
| destServiceName | The layer in kubernetes dest service. | | string |
| destEndpointName | The endpoint name detect in kubernetes dest service. | | string |
| destEndpointName | The endpoint name detect in kubernetes dest service. | | string |

### SCOPES with `Cilium` Prefix

All metrics starting with `Cilium` are derived from Cilium monitoring by Cilium Hubble.

#### Service, Service Instance and relations

For all `CiliumService`, `CiliumServiceInstance`, `CiliumServiceRelation` and `CiliumServiceInstanceRelation`, they all have the
following **L4**/**L7** metric contents.

| Name | Remarks | Group Key | Type |
|-----------------------|-----------------------------------------------------------------------------|-----------|--------|
| verdict | The metrics verdict from Flow. The value may be `forwarded` and `dropped`. | | string |
| type | The metrics type from Flow. The value may be `tcp`, `http`, `dns`, `kakfa`. | | string |
| direction | The metrics direction from Flow. The value may be `ingress` and `egress`. | | string |
| dropReason | When the verdict is `dropped`, the drop reason would be recorded. | | string |
| http.url | The URL of the HTTP request. | | string |
| http.code | The Response code of the HTTP response. | | int |
| http.protocol | The protocol of the HTTP request. | | string |
| http.method | The method of the HTTP request. | | string |
| kafka.errorCode | The error code of the Kafka request. | | int |
| kafka.errorCodeString | The error code explaination of the Kafka request. | | string |
| kafka.apiVersion | The API version of the Kafka request. | | string |
| kafka.apiKey | The API key of the Kafka request. | | string |
| kafka.correlationId | The correlation ID of the Kafka request. | | string |
| kafka.topic | The topic of the Kafka request. | | string |
| dns.domain | The domain of the DNS request. | | string |
| dns.queryType | The query type of the DNS request. | | string |
| dns.rcode | The response code of the DNS request. | | int |
| dns.recodeString | The response code explaination of the DNS request. | | string |
| dns.ttl | The TTL of the DNS request. | | int |
| dns.ipCount | The count of the IP addresses of the DNS responsed. | | int |
| duration | The duration(millisecond) of the L7 response. | | long |
| success | Is the response success of the L7 response. | | bool |

##### SCOPE `CiliumService`

| Name | Remarks | Group Key | Type |
|-------------|--------------------------------------------------------------------|-----------|--------|
| name | The service name in Cilium. | | string |
| layer | The layer in Cilium service. | | string |
| detectPoint | Where the relation is detected. The value may be client or server. | | enum |

##### SCOPE `CiliumServiceInstance`

| Name | Remarks | Group Key | Type |
|---------------------|--------------------------------------------------------------------|-----------|--------|
| serviceName | The service name in Cilium. | | string |
| serviceInstanceName | The pod name in Cilium. | | string |
| layer | The layer of Cilium service. | | string |
| detectPoint | Where the relation is detected. The value may be client or server. | | enum |

##### SCOPE `CiliumServiceRelation`

| Name | Remarks | Group Key | Type |
|-------------------|--------------------------------------------------------------------|-----------|--------|
| sourceServiceName | The source service name in Cilium. | | string |
| sourceLayer | The source layer service in Cilium. | | string |
| detectPoint | Where the relation is detected. The value may be client or server. | | enum |
| componentId | The ID of component used in this call. | | int |
| destServiceName | The dest service name in Cilium. | | string |
| destLayer | The dest layer service in Cilium. | | string |

##### SCOPE `CiliumServiceInstanceRelation`

| Name | Remarks | Group Key | Type |
|---------------------------|--------------------------------------------------------------------|-----------|--------|
| sourceServiceName | The source service name in Cilium. | | string |
| sourceServiceInstanceName | The source pod name in Cilium. | | string |
| sourceLayer | The source layer service in Cilium. | | string |
| detectPoint | Where the relation is detected. The value may be client or server. | | enum |
| componentId | The ID of component used in this call. | | int |
| destServiceName | The dest service name in Cilium. | | string |
| destServiceInstanceName | The dest pod name in Cilium. | | string |
| destLayer | The dest layer service in Cilium. | | string |

#### Endpoint and Endpoint Relation

For `CiliumEndpoint` and `CiliumEndpointRelation`, they have all the fields of **L4**/**L7** metric contents, but the `type` only would be `http`, `dns` or `kafka`.

##### SCOPE `CiliumEndpoint`

| Name | Remarks | Group Key | Type |
|--------------|---------------------------------------------------------|-----------|--------|
| serviceName | The service name in Cilium. | | string |
| layer | The layer in Cilium service. | | string |
| endpointName | The endpoint name detect in Cilium service. | | string |

##### SCOPE `CiliumEndpointRelation`

| Name | Remarks | Group Key | Type |
|--------------------|--------------------------------------------------------------------|-----------|--------|
| sourceServiceName | The source service name in Cilium. | | string |
| sourceLayer | The layer in Cilium source service. | | enum |
| sourceEndpointName | The endpoint name detect in Cilium source service. | | string |
| detectPoint | Where the relation is detected. The value may be client or server. | | enum |
| componentId | The ID of component used in this call. | | int |
| destServiceName | The dest service name in Cilium. | | string |
| destLayer | The layer in Cilium dest service. | | enum |
| destEndpointName | The endpoint name detect in Cilium dest service. | | string |


1 change: 1 addition & 0 deletions docs/en/guides/How-to-build.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -84,4 +84,5 @@ Refer to [Build docker image](../../../docker) for more details.
* `grpc-java` and `java` folders in **oap-server/exporter/target/generated-sources/protobuf**
* `grpc-java` and `java` folders in **oap-server/server-configuration/grpc-configuration-sync/target/generated-sources/protobuf**
* `grpc-java` and `java` folders in **oap-server/server-alarm-plugin/target/generated-sources/protobuf**
* `grpc-java` and `java` folders in **oap-server/server-fetcher-plugin/fetcher-proto/target/generated-sources/protobuf**
* `antlr4` folder in **oap-server/oal-grammar/target/generated-sources**
Loading

0 comments on commit 2e8b4f9

Please sign in to comment.