時系列データストリーム (TSDS) - 手動でダウンサンプリングを実行（Run downsampling manually）

手動でダウンサンプリングを実行する
前提条件
時系列データストリームを作成する
時系列データを取り込む
- Python
- Ruby
- Js
- Console
- Python
- Ruby
- Js
- Console
- Python
- Ruby
- Js
- Console
- Python
- Ruby
- Js
- Console
TSDSをダウンサンプリングする
- Python
- Ruby
- Js
- Console
- Console-Result
- Python
- Ruby
- Js
- Console
- Python
- Ruby
- Js
- Console
- Python
- Ruby
- Js
- Console
- Python
- Js
- Console
結果を表示する

手動でダウンサンプリングを実行する

推奨される方法は、インデックスライフサイクル管理 (ILM)を通じて時系列データストリーム (TSDS)をダウンサンプリングすることです。しかし、ILMを使用していない場合は、TSDSを手動でダウンサンプリングできます。このガイドでは、典型的なKubernetesクラスターの監視データを使用してその方法を示します。

手動ダウンサンプリングをテストするには、次の手順に従ってください:

前提条件

TSDSの前提条件を参照してください。
データストリームを直接ダウンサンプリングすることはできず、複数のインデックスを一度にダウンサンプリングすることもできません。1つの時系列インデックス (TSDSバックインデックス) のみをダウンサンプリングできます。
インデックスをダウンサンプリングするには、読み取り専用である必要があります。TSDS書き込みインデックスの場合、最初にロールオーバーして読み取り専用にする必要があります。
ダウンサンプリングはUTCタイムスタンプを使用します。
ダウンサンプリングには、時系列インデックスに少なくとも1つのメトリックフィールドが存在する必要があります。

時系列データストリームを作成する

まず、TSDSを作成します。簡単のために、時系列マッピングではすべてのtime_series_metricパラメータがgauge型に設定されていますが、counterやhistogramなどの他の値も使用できます。time_series_metric値は、ダウンサンプリング中に使用される統計的表現の種類を決定します。

インデックステンプレートには、静的な時系列次元のセットが含まれています: host、namespace、node、およびpod。時系列次元は、ダウンサンプリングプロセスによって変更されることはありません。

Python

resp = client.indices.put_index_template(
   name="my-data-stream-template",
   index_patterns=[
   "my-data-stream*"
   ],
   data_stream={},
   template={
   "settings": {
   "index": {
   "mode": "time_series",
   "routing_path": [
   "kubernetes.namespace",
   "kubernetes.host",
   "kubernetes.node",
   "kubernetes.pod"
   ],
   "number_of_replicas": 0,
   "number_of_shards": 2
   }
   },
   "mappings": {
   "properties": {
   "@timestamp": {
   "type": "date"
   },
   "kubernetes": {
   "properties": {
   "container": {
   "properties": {
   "cpu": {
   "properties": {
   "usage": {
   "properties": {
   "core": {
   "properties": {
   "ns": {
   "type": "long"
   }
   }
   },
   "limit": {
   "properties": {
   "pct": {
   "type": "float"
   }
   }
   },
   "nanocores": {
   "type": "long",
   "time_series_metric": "gauge"
   },
   "node": {
   "properties": {
   "pct": {
   "type": "float"
   }
   }
   }
   }
   }
   }
   },
   "memory": {
   "properties": {
   "available": {
   "properties": {
   "bytes": {
   "type": "long",
   "time_series_metric": "gauge"
   }
   }
   },
   "majorpagefaults": {
   "type": "long"
   },
   "pagefaults": {
   "type": "long",
   "time_series_metric": "gauge"
   },
   "rss": {
   "properties": {
   "bytes": {
   "type": "long",
   "time_series_metric": "gauge"
   }
   }
   },
   "usage": {
   "properties": {
   "bytes": {
   "type": "long",
   "time_series_metric": "gauge"
   },
   "limit": {
   "properties": {
   "pct": {
   "type": "float"
   }
   }
   },
   "node": {
   "properties": {
   "pct": {
   "type": "float"
   }
   }
   }
   }
   },
   "workingset": {
   "properties": {
   "bytes": {
   "type": "long",
   "time_series_metric": "gauge"
   }
   }
   }
   }
   },
   "name": {
   "type": "keyword"
   },
   "start_time": {
   "type": "date"
   }
   }
   },
   "host": {
   "type": "keyword",
   "time_series_dimension": True
   },
   "namespace": {
   "type": "keyword",
   "time_series_dimension": True
   },
   "node": {
   "type": "keyword",
   "time_series_dimension": True
   },
   "pod": {
   "type": "keyword",
   "time_series_dimension": True
   }
   }
   }
   }
   }
   },
)
print(resp)

Ruby

response = client.indices.put_index_template(
  name: 'my-data-stream-template',
  body: {
   index_patterns: [
   'my-data-stream*'
   ],
   data_stream: {},
   template: {
   settings: {
   index: {
   mode: 'time_series',
   routing_path: [
   'kubernetes.namespace',
   'kubernetes.host',
   'kubernetes.node',
   'kubernetes.pod'
   ],
   number_of_replicas: 0,
   number_of_shards: 2
   }
   },
   mappings: {
   properties: {
   "@timestamp": {
   type: 'date'
   },
   kubernetes: {
   properties: {
   container: {
   properties: {
   cpu: {
   properties: {
   usage: {
   properties: {
   core: {
   properties: {
   ns: {
   type: 'long'
   }
   }
   },
   limit: {
   properties: {
   pct: {
   type: 'float'
   }
   }
   },
   nanocores: {
   type: 'long',
   time_series_metric: 'gauge'
   },
   node: {
   properties: {
   pct: {
   type: 'float'
   }
   }
   }
   }
   }
   }
   },
   memory: {
   properties: {
   available: {
   properties: {
   bytes: {
   type: 'long',
   time_series_metric: 'gauge'
   }
   }
   },
   majorpagefaults: {
   type: 'long'
   },
   pagefaults: {
   type: 'long',
   time_series_metric: 'gauge'
   },
   rss: {
   properties: {
   bytes: {
   type: 'long',
   time_series_metric: 'gauge'
   }
   }
   },
   usage: {
   properties: {
   bytes: {
   type: 'long',
   time_series_metric: 'gauge'
   },
   limit: {
   properties: {
   pct: {
   type: 'float'
   }
   }
   },
   node: {
   properties: {
   pct: {
   type: 'float'
   }
   }
   }
   }
   },
   workingset: {
   properties: {
   bytes: {
   type: 'long',
   time_series_metric: 'gauge'
   }
   }
   }
   }
   },
   name: {
   type: 'keyword'
   },
   start_time: {
   type: 'date'
   }
   }
   },
   host: {
   type: 'keyword',
   time_series_dimension: true
   },
   namespace: {
   type: 'keyword',
   time_series_dimension: true
   },
   node: {
   type: 'keyword',
   time_series_dimension: true
   },
   pod: {
   type: 'keyword',
   time_series_dimension: true
   }
   }
   }
   }
   }
   }
  }
)
puts response

Js

const response = await client.indices.putIndexTemplate({
  name: "my-data-stream-template",
  index_patterns: ["my-data-stream*"],
  data_stream: {},
  template: {
   settings: {
   index: {
   mode: "time_series",
   routing_path: [
   "kubernetes.namespace",
   "kubernetes.host",
   "kubernetes.node",
   "kubernetes.pod",
   ],
   number_of_replicas: 0,
   number_of_shards: 2,
   },
   },
   mappings: {
   properties: {
   "@timestamp": {
   type: "date",
   },
   kubernetes: {
   properties: {
   container: {
   properties: {
   cpu: {
   properties: {
   usage: {
   properties: {
   core: {
   properties: {
   ns: {
   type: "long",
   },
   },
   },
   limit: {
   properties: {
   pct: {
   type: "float",
   },
   },
   },
   nanocores: {
   type: "long",
   time_series_metric: "gauge",
   },
   node: {
   properties: {
   pct: {
   type: "float",
   },
   },
   },
   },
   },
   },
   },
   memory: {
   properties: {
   available: {
   properties: {
   bytes: {
   type: "long",
   time_series_metric: "gauge",
   },
   },
   },
   majorpagefaults: {
   type: "long",
   },
   pagefaults: {
   type: "long",
   time_series_metric: "gauge",
   },
   rss: {
   properties: {
   bytes: {
   type: "long",
   time_series_metric: "gauge",
   },
   },
   },
   usage: {
   properties: {
   bytes: {
   type: "long",
   time_series_metric: "gauge",
   },
   limit: {
   properties: {
   pct: {
   type: "float",
   },
   },
   },
   node: {
   properties: {
   pct: {
   type: "float",
   },
   },
   },
   },
   },
   workingset: {
   properties: {
   bytes: {
   type: "long",
   time_series_metric: "gauge",
   },
   },
   },
   },
   },
   name: {
   type: "keyword",
   },
   start_time: {
   type: "date",
   },
   },
   },
   host: {
   type: "keyword",
   time_series_dimension: true,
   },
   namespace: {
   type: "keyword",
   time_series_dimension: true,
   },
   node: {
   type: "keyword",
   time_series_dimension: true,
   },
   pod: {
   type: "keyword",
   time_series_dimension: true,
   },
   },
   },
   },
   },
  },
});
console.log(response);

Console

PUT _index_template/my-data-stream-template
{
  "index_patterns": [
   "my-data-stream*"
  ],
  "data_stream": {},
  "template": {
   "settings": {
   "index": {
   "mode": "time_series",
   "routing_path": [
   "kubernetes.namespace",
   "kubernetes.host",
   "kubernetes.node",
   "kubernetes.pod"
   ],
   "number_of_replicas": 0,
   "number_of_shards": 2
   }
   },
   "mappings": {
   "properties": {
   "@timestamp": {
   "type": "date"
   },
   "kubernetes": {
   "properties": {
   "container": {
   "properties": {
   "cpu": {
   "properties": {
   "usage": {
   "properties": {
   "core": {
   "properties": {
   "ns": {
   "type": "long"
   }
   }
   },
   "limit": {
   "properties": {
   "pct": {
   "type": "float"
   }
   }
   },
   "nanocores": {
   "type": "long",
   "time_series_metric": "gauge"
   },
   "node": {
   "properties": {
   "pct": {
   "type": "float"
   }
   }
   }
   }
   }
   }
   },
   "memory": {
   "properties": {
   "available": {
   "properties": {
   "bytes": {
   "type": "long",
   "time_series_metric": "gauge"
   }
   }
   },
   "majorpagefaults": {
   "type": "long"
   },
   "pagefaults": {
   "type": "long",
   "time_series_metric": "gauge"
   },
   "rss": {
   "properties": {
   "bytes": {
   "type": "long",
   "time_series_metric": "gauge"
   }
   }
   },
   "usage": {
   "properties": {
   "bytes": {
   "type": "long",
   "time_series_metric": "gauge"
   },
   "limit": {
   "properties": {
   "pct": {
   "type": "float"
   }
   }
   },
   "node": {
   "properties": {
   "pct": {
   "type": "float"
   }
   }
   }
   }
   },
   "workingset": {
   "properties": {
   "bytes": {
   "type": "long",
   "time_series_metric": "gauge"
   }
   }
   }
   }
   },
   "name": {
   "type": "keyword"
   },
   "start_time": {
   "type": "date"
   }
   }
   },
   "host": {
   "type": "keyword",
   "time_series_dimension": true
   },
   "namespace": {
   "type": "keyword",
   "time_series_dimension": true
   },
   "node": {
   "type": "keyword",
   "time_series_dimension": true
   },
   "pod": {
   "type": "keyword",
   "time_series_dimension": true
   }
   }
   }
   }
   }
  }
}

時系列データを取り込む

時系列データストリームは最近のデータのみを受け入れるように設計されているため、この例では、データがインデックスされる際にデータを時間シフトするために取り込みパイプラインを使用します。その結果、インデックスされたデータは、過去15分の@timestampを持つことになります。

このリクエストでパイプラインを作成します:

Python

resp = client.ingest.put_pipeline(
   id="my-timestamp-pipeline",
   description="Shifts the @timestamp to the last 15 minutes",
   processors=[
   {
   "set": {
   "field": "ingest_time",
   "value": "{{_ingest.timestamp}}"
   }
   },
   {
   "script": {
   "lang": "painless",
   "source": "\n          def delta = ChronoUnit.SECONDS.between(\n            ZonedDateTime.parse(\"2022-06-21T15:49:00Z\"),\n            ZonedDateTime.parse(ctx[\"ingest_time\"])\n          );\n          ctx[\"@timestamp\"] = ZonedDateTime.parse(ctx[\"@timestamp\"]).plus(delta,ChronoUnit.SECONDS).toString();\n        "
   }
   }
   ],
)
print(resp)

Ruby

response = client.ingest.put_pipeline(
  id: 'my-timestamp-pipeline',
  body: {
   description: 'Shifts the @timestamp to the last 15 minutes',
   processors: [
   {
   set: {
   field: 'ingest_time',
   value: '{{_ingest.timestamp}}'
   }
   },
   {
   script: {
   lang: 'painless',
   source: "\n          def delta = ChronoUnit.SECONDS.between(\n            ZonedDateTime.parse(\"2022-06-21T15:49:00Z\"),\n            ZonedDateTime.parse(ctx[\"ingest_time\"])\n          );\n          ctx[\"@timestamp\"] = ZonedDateTime.parse(ctx[\"@timestamp\"]).plus(delta,ChronoUnit.SECONDS).toString();\n        "
   }
   }
   ]
  }
)
puts response

Js

const response = await client.ingest.putPipeline({
  id: "my-timestamp-pipeline",
  description: "Shifts the @timestamp to the last 15 minutes",
  processors: [
   {
   set: {
   field: "ingest_time",
   value: "{{_ingest.timestamp}}",
   },
   },
   {
   script: {
   lang: "painless",
   source:
   '\n          def delta = ChronoUnit.SECONDS.between(\n            ZonedDateTime.parse("2022-06-21T15:49:00Z"),\n            ZonedDateTime.parse(ctx["ingest_time"])\n          );\n          ctx["@timestamp"] = ZonedDateTime.parse(ctx["@timestamp"]).plus(delta,ChronoUnit.SECONDS).toString();\n        ',
   },
   },
  ],
});
console.log(response);

Console

PUT _ingest/pipeline/my-timestamp-pipeline
{
  "description": "Shifts the @timestamp to the last 15 minutes",
  "processors": [
   {
   "set": {
   "field": "ingest_time",
   "value": "{{_ingest.timestamp}}"
   }
   },
   {
   "script": {
   "lang": "painless",
   "source": """
   def delta = ChronoUnit.SECONDS.between(
   ZonedDateTime.parse("2022-06-21T15:49:00Z"),
   ZonedDateTime.parse(ctx["ingest_time"])
   );
   ctx["@timestamp"] = ZonedDateTime.parse(ctx["@timestamp"]).plus(delta,ChronoUnit.SECONDS).toString();
   """
   }
   }
  ]
}

次に、バルクAPIリクエストを使用して、TSDSを自動的に作成し、10件のドキュメントのセットをインデックスします:

Python

resp = client.bulk(
   index="my-data-stream",
   refresh=True,
   pipeline="my-timestamp-pipeline",
   operations=[
   {
   "create": {}
   },
   {
   "@timestamp": "2022-06-21T15:49:00Z",
   "kubernetes": {
   "host": "gke-apps-0",
   "node": "gke-apps-0-0",
   "pod": "gke-apps-0-0-0",
   "container": {
   "cpu": {
   "usage": {
   "nanocores": 91153,
   "core": {
   "ns": 12828317850
   },
   "node": {
   "pct": 0.0000277905
   },
   "limit": {
   "pct": 0.0000277905
   }
   }
   },
   "memory": {
   "available": {
   "bytes": 463314616
   },
   "usage": {
   "bytes": 307007078,
   "node": {
   "pct": 0.01770037710617187
   },
   "limit": {
   "pct": 0.00009923134671484496
   }
   },
   "workingset": {
   "bytes": 585236
   },
   "rss": {
   "bytes": 102728
   },
   "pagefaults": 120901,
   "majorpagefaults": 0
   },
   "start_time": "2021-03-30T07:59:06Z",
   "name": "container-name-44"
   },
   "namespace": "namespace26"
   }
   },
   {
   "create": {}
   },
   {
   "@timestamp": "2022-06-21T15:45:50Z",
   "kubernetes": {
   "host": "gke-apps-0",
   "node": "gke-apps-0-0",
   "pod": "gke-apps-0-0-0",
   "container": {
   "cpu": {
   "usage": {
   "nanocores": 124501,
   "core": {
   "ns": 12828317850
   },
   "node": {
   "pct": 0.0000277905
   },
   "limit": {
   "pct": 0.0000277905
   }
   }
   },
   "memory": {
   "available": {
   "bytes": 982546514
   },
   "usage": {
   "bytes": 360035574,
   "node": {
   "pct": 0.01770037710617187
   },
   "limit": {
   "pct": 0.00009923134671484496
   }
   },
   "workingset": {
   "bytes": 1339884
   },
   "rss": {
   "bytes": 381174
   },
   "pagefaults": 178473,
   "majorpagefaults": 0
   },
   "start_time": "2021-03-30T07:59:06Z",
   "name": "container-name-44"
   },
   "namespace": "namespace26"
   }
   },
   {
   "create": {}
   },
   {
   "@timestamp": "2022-06-21T15:44:50Z",
   "kubernetes": {
   "host": "gke-apps-0",
   "node": "gke-apps-0-0",
   "pod": "gke-apps-0-0-0",
   "container": {
   "cpu": {
   "usage": {
   "nanocores": 38907,
   "core": {
   "ns": 12828317850
   },
   "node": {
   "pct": 0.0000277905
   },
   "limit": {
   "pct": 0.0000277905
   }
   }
   },
   "memory": {
   "available": {
   "bytes": 862723768
   },
   "usage": {
   "bytes": 379572388,
   "node": {
   "pct": 0.01770037710617187
   },
   "limit": {
   "pct": 0.00009923134671484496
   }
   },
   "workingset": {
   "bytes": 431227
   },
   "rss": {
   "bytes": 386580
   },
   "pagefaults": 233166,
   "majorpagefaults": 0
   },
   "start_time": "2021-03-30T07:59:06Z",
   "name": "container-name-44"
   },
   "namespace": "namespace26"
   }
   },
   {
   "create": {}
   },
   {
   "@timestamp": "2022-06-21T15:44:40Z",
   "kubernetes": {
   "host": "gke-apps-0",
   "node": "gke-apps-0-0",
   "pod": "gke-apps-0-0-0",
   "container": {
   "cpu": {
   "usage": {
   "nanocores": 86706,
   "core": {
   "ns": 12828317850
   },
   "node": {
   "pct": 0.0000277905
   },
   "limit": {
   "pct": 0.0000277905
   }
   }
   },
   "memory": {
   "available": {
   "bytes": 567160996
   },
   "usage": {
   "bytes": 103266017,
   "node": {
   "pct": 0.01770037710617187
   },
   "limit": {
   "pct": 0.00009923134671484496
   }
   },
   "workingset": {
   "bytes": 1724908
   },
   "rss": {
   "bytes": 105431
   },
   "pagefaults": 233166,
   "majorpagefaults": 0
   },
   "start_time": "2021-03-30T07:59:06Z",
   "name": "container-name-44"
   },
   "namespace": "namespace26"
   }
   },
   {
   "create": {}
   },
   {
   "@timestamp": "2022-06-21T15:44:00Z",
   "kubernetes": {
   "host": "gke-apps-0",
   "node": "gke-apps-0-0",
   "pod": "gke-apps-0-0-0",
   "container": {
   "cpu": {
   "usage": {
   "nanocores": 150069,
   "core": {
   "ns": 12828317850
   },
   "node": {
   "pct": 0.0000277905
   },
   "limit": {
   "pct": 0.0000277905
   }
   }
   },
   "memory": {
   "available": {
   "bytes": 639054643
   },
   "usage": {
   "bytes": 265142477,
   "node": {
   "pct": 0.01770037710617187
   },
   "limit": {
   "pct": 0.00009923134671484496
   }
   },
   "workingset": {
   "bytes": 1786511
   },
   "rss": {
   "bytes": 189235
   },
   "pagefaults": 138172,
   "majorpagefaults": 0
   },
   "start_time": "2021-03-30T07:59:06Z",
   "name": "container-name-44"
   },
   "namespace": "namespace26"
   }
   },
   {
   "create": {}
   },
   {
   "@timestamp": "2022-06-21T15:42:40Z",
   "kubernetes": {
   "host": "gke-apps-0",
   "node": "gke-apps-0-0",
   "pod": "gke-apps-0-0-0",
   "container": {
   "cpu": {
   "usage": {
   "nanocores": 82260,
   "core": {
   "ns": 12828317850
   },
   "node": {
   "pct": 0.0000277905
   },
   "limit": {
   "pct": 0.0000277905
   }
   }
   },
   "memory": {
   "available": {
   "bytes": 854735585
   },
   "usage": {
   "bytes": 309798052,
   "node": {
   "pct": 0.01770037710617187
   },
   "limit": {
   "pct": 0.00009923134671484496
   }
   },
   "workingset": {
   "bytes": 924058
   },
   "rss": {
   "bytes": 110838
   },
   "pagefaults": 259073,
   "majorpagefaults": 0
   },
   "start_time": "2021-03-30T07:59:06Z",
   "name": "container-name-44"
   },
   "namespace": "namespace26"
   }
   },
   {
   "create": {}
   },
   {
   "@timestamp": "2022-06-21T15:42:10Z",
   "kubernetes": {
   "host": "gke-apps-0",
   "node": "gke-apps-0-0",
   "pod": "gke-apps-0-0-0",
   "container": {
   "cpu": {
   "usage": {
   "nanocores": 153404,
   "core": {
   "ns": 12828317850
   },
   "node": {
   "pct": 0.0000277905
   },
   "limit": {
   "pct": 0.0000277905
   }
   }
   },
   "memory": {
   "available": {
   "bytes": 279586406
   },
   "usage": {
   "bytes": 214904955,
   "node": {
   "pct": 0.01770037710617187
   },
   "limit": {
   "pct": 0.00009923134671484496
   }
   },
   "workingset": {
   "bytes": 1047265
   },
   "rss": {
   "bytes": 91914
   },
   "pagefaults": 302252,
   "majorpagefaults": 0
   },
   "start_time": "2021-03-30T07:59:06Z",
   "name": "container-name-44"
   },
   "namespace": "namespace26"
   }
   },
   {
   "create": {}
   },
   {
   "@timestamp": "2022-06-21T15:40:20Z",
   "kubernetes": {
   "host": "gke-apps-0",
   "node": "gke-apps-0-0",
   "pod": "gke-apps-0-0-0",
   "container": {
   "cpu": {
   "usage": {
   "nanocores": 125613,
   "core": {
   "ns": 12828317850
   },
   "node": {
   "pct": 0.0000277905
   },
   "limit": {
   "pct": 0.0000277905
   }
   }
   },
   "memory": {
   "available": {
   "bytes": 822782853
   },
   "usage": {
   "bytes": 100475044,
   "node": {
   "pct": 0.01770037710617187
   },
   "limit": {
   "pct": 0.00009923134671484496
   }
   },
   "workingset": {
   "bytes": 2109932
   },
   "rss": {
   "bytes": 278446
   },
   "pagefaults": 74843,
   "majorpagefaults": 0
   },
   "start_time": "2021-03-30T07:59:06Z",
   "name": "container-name-44"
   },
   "namespace": "namespace26"
   }
   },
   {
   "create": {}
   },
   {
   "@timestamp": "2022-06-21T15:40:10Z",
   "kubernetes": {
   "host": "gke-apps-0",
   "node": "gke-apps-0-0",
   "pod": "gke-apps-0-0-0",
   "container": {
   "cpu": {
   "usage": {
   "nanocores": 100046,
   "core": {
   "ns": 12828317850
   },
   "node": {
   "pct": 0.0000277905
   },
   "limit": {
   "pct": 0.0000277905
   }
   }
   },
   "memory": {
   "available": {
   "bytes": 567160996
   },
   "usage": {
   "bytes": 362826547,
   "node": {
   "pct": 0.01770037710617187
   },
   "limit": {
   "pct": 0.00009923134671484496
   }
   },
   "workingset": {
   "bytes": 1986724
   },
   "rss": {
   "bytes": 402801
   },
   "pagefaults": 296495,
   "majorpagefaults": 0
   },
   "start_time": "2021-03-30T07:59:06Z",
   "name": "container-name-44"
   },
   "namespace": "namespace26"
   }
   },
   {
   "create": {}
   },
   {
   "@timestamp": "2022-06-21T15:38:30Z",
   "kubernetes": {
   "host": "gke-apps-0",
   "node": "gke-apps-0-0",
   "pod": "gke-apps-0-0-0",
   "container": {
   "cpu": {
   "usage": {
   "nanocores": 40018,
   "core": {
   "ns": 12828317850
   },
   "node": {
   "pct": 0.0000277905
   },
   "limit": {
   "pct": 0.0000277905
   }
   }
   },
   "memory": {
   "available": {
   "bytes": 1062428344
   },
   "usage": {
   "bytes": 265142477,
   "node": {
   "pct": 0.01770037710617187
   },
   "limit": {
   "pct": 0.00009923134671484496
   }
   },
   "workingset": {
   "bytes": 2294743
   },
   "rss": {
   "bytes": 340623
   },
   "pagefaults": 224530,
   "majorpagefaults": 0
   },
   "start_time": "2021-03-30T07:59:06Z",
   "name": "container-name-44"
   },
   "namespace": "namespace26"
   }
   }
   ],
)
print(resp)

Ruby

response = client.bulk(
  index: 'my-data-stream',
  refresh: true,
  pipeline: 'my-timestamp-pipeline',
  body: [
   {
   create: {}
   },
   {
   "@timestamp": '2022-06-21T15:49:00Z',
   kubernetes: {
   host: 'gke-apps-0',
   node: 'gke-apps-0-0',
   pod: 'gke-apps-0-0-0',
   container: {
   cpu: {
   usage: {
   nanocores: 91_153,
   core: {
   ns: 12_828_317_850
   },
   node: {
   pct: 2.77905e-05
   },
   limit: {
   pct: 2.77905e-05
   }
   }
   },
   memory: {
   available: {
   bytes: 463_314_616
   },
   usage: {
   bytes: 307_007_078,
   node: {
   pct: 0.01770037710617187
   },
   limit: {
   pct: 9.923134671484496e-05
   }
   },
   workingset: {
   bytes: 585_236
   },
   rss: {
   bytes: 102_728
   },
   pagefaults: 120_901,
   majorpagefaults: 0
   },
   start_time: '2021-03-30T07:59:06Z',
   name: 'container-name-44'
   },
   namespace: 'namespace26'
   }
   },
   {
   create: {}
   },
   {
   "@timestamp": '2022-06-21T15:45:50Z',
   kubernetes: {
   host: 'gke-apps-0',
   node: 'gke-apps-0-0',
   pod: 'gke-apps-0-0-0',
   container: {
   cpu: {
   usage: {
   nanocores: 124_501,
   core: {
   ns: 12_828_317_850
   },
   node: {
   pct: 2.77905e-05
   },
   limit: {
   pct: 2.77905e-05
   }
   }
   },
   memory: {
   available: {
   bytes: 982_546_514
   },
   usage: {
   bytes: 360_035_574,
   node: {
   pct: 0.01770037710617187
   },
   limit: {
   pct: 9.923134671484496e-05
   }
   },
   workingset: {
   bytes: 1_339_884
   },
   rss: {
   bytes: 381_174
   },
   pagefaults: 178_473,
   majorpagefaults: 0
   },
   start_time: '2021-03-30T07:59:06Z',
   name: 'container-name-44'
   },
   namespace: 'namespace26'
   }
   },
   {
   create: {}
   },
   {
   "@timestamp": '2022-06-21T15:44:50Z',
   kubernetes: {
   host: 'gke-apps-0',
   node: 'gke-apps-0-0',
   pod: 'gke-apps-0-0-0',
   container: {
   cpu: {
   usage: {
   nanocores: 38_907,
   core: {
   ns: 12_828_317_850
   },
   node: {
   pct: 2.77905e-05
   },
   limit: {
   pct: 2.77905e-05
   }
   }
   },
   memory: {
   available: {
   bytes: 862_723_768
   },
   usage: {
   bytes: 379_572_388,
   node: {
   pct: 0.01770037710617187
   },
   limit: {
   pct: 9.923134671484496e-05
   }
   },
   workingset: {
   bytes: 431_227
   },
   rss: {
   bytes: 386_580
   },
   pagefaults: 233_166,
   majorpagefaults: 0
   },
   start_time: '2021-03-30T07:59:06Z',
   name: 'container-name-44'
   },
   namespace: 'namespace26'
   }
   },
   {
   create: {}
   },
   {
   "@timestamp": '2022-06-21T15:44:40Z',
   kubernetes: {
   host: 'gke-apps-0',
   node: 'gke-apps-0-0',
   pod: 'gke-apps-0-0-0',
   container: {
   cpu: {
   usage: {
   nanocores: 86_706,
   core: {
   ns: 12_828_317_850
   },
   node: {
   pct: 2.77905e-05
   },
   limit: {
   pct: 2.77905e-05
   }
   }
   },
   memory: {
   available: {
   bytes: 567_160_996
   },
   usage: {
   bytes: 103_266_017,
   node: {
   pct: 0.01770037710617187
   },
   limit: {
   pct: 9.923134671484496e-05
   }
   },
   workingset: {
   bytes: 1_724_908
   },
   rss: {
   bytes: 105_431
   },
   pagefaults: 233_166,
   majorpagefaults: 0
   },
   start_time: '2021-03-30T07:59:06Z',
   name: 'container-name-44'
   },
   namespace: 'namespace26'
   }
   },
   {
   create: {}
   },
   {
   "@timestamp": '2022-06-21T15:44:00Z',
   kubernetes: {
   host: 'gke-apps-0',
   node: 'gke-apps-0-0',
   pod: 'gke-apps-0-0-0',
   container: {
   cpu: {
   usage: {
   nanocores: 150_069,
   core: {
   ns: 12_828_317_850
   },
   node: {
   pct: 2.77905e-05
   },
   limit: {
   pct: 2.77905e-05
   }
   }
   },
   memory: {
   available: {
   bytes: 639_054_643
   },
   usage: {
   bytes: 265_142_477,
   node: {
   pct: 0.01770037710617187
   },
   limit: {
   pct: 9.923134671484496e-05
   }
   },
   workingset: {
   bytes: 1_786_511
   },
   rss: {
   bytes: 189_235
   },
   pagefaults: 138_172,
   majorpagefaults: 0
   },
   start_time: '2021-03-30T07:59:06Z',
   name: 'container-name-44'
   },
   namespace: 'namespace26'
   }
   },
   {
   create: {}
   },
   {
   "@timestamp": '2022-06-21T15:42:40Z',
   kubernetes: {
   host: 'gke-apps-0',
   node: 'gke-apps-0-0',
   pod: 'gke-apps-0-0-0',
   container: {
   cpu: {
   usage: {
   nanocores: 82_260,
   core: {
   ns: 12_828_317_850
   },
   node: {
   pct: 2.77905e-05
   },
   limit: {
   pct: 2.77905e-05
   }
   }
   },
   memory: {
   available: {
   bytes: 854_735_585
   },
   usage: {
   bytes: 309_798_052,
   node: {
   pct: 0.01770037710617187
   },
   limit: {
   pct: 9.923134671484496e-05
   }
   },
   workingset: {
   bytes: 924_058
   },
   rss: {
   bytes: 110_838
   },
   pagefaults: 259_073,
   majorpagefaults: 0
   },
   start_time: '2021-03-30T07:59:06Z',
   name: 'container-name-44'
   },
   namespace: 'namespace26'
   }
   },
   {
   create: {}
   },
   {
   "@timestamp": '2022-06-21T15:42:10Z',
   kubernetes: {
   host: 'gke-apps-0',
   node: 'gke-apps-0-0',
   pod: 'gke-apps-0-0-0',
   container: {
   cpu: {
   usage: {
   nanocores: 153_404,
   core: {
   ns: 12_828_317_850
   },
   node: {
   pct: 2.77905e-05
   },
   limit: {
   pct: 2.77905e-05
   }
   }
   },
   memory: {
   available: {
   bytes: 279_586_406
   },
   usage: {
   bytes: 214_904_955,
   node: {
   pct: 0.01770037710617187
   },
   limit: {
   pct: 9.923134671484496e-05
   }
   },
   workingset: {
   bytes: 1_047_265
   },
   rss: {
   bytes: 91_914
   },
   pagefaults: 302_252,
   majorpagefaults: 0
   },
   start_time: '2021-03-30T07:59:06Z',
   name: 'container-name-44'
   },
   namespace: 'namespace26'
   }
   },
   {
   create: {}
   },
   {
   "@timestamp": '2022-06-21T15:40:20Z',
   kubernetes: {
   host: 'gke-apps-0',
   node: 'gke-apps-0-0',
   pod: 'gke-apps-0-0-0',
   container: {
   cpu: {
   usage: {
   nanocores: 125_613,
   core: {
   ns: 12_828_317_850
   },
   node: {
   pct: 2.77905e-05
   },
   limit: {
   pct: 2.77905e-05
   }
   }
   },
   memory: {
   available: {
   bytes: 822_782_853
   },
   usage: {
   bytes: 100_475_044,
   node: {
   pct: 0.01770037710617187
   },
   limit: {
   pct: 9.923134671484496e-05
   }
   },
   workingset: {
   bytes: 2_109_932
   },
   rss: {
   bytes: 278_446
   },
   pagefaults: 74_843,
   majorpagefaults: 0
   },
   start_time: '2021-03-30T07:59:06Z',
   name: 'container-name-44'
   },
   namespace: 'namespace26'
   }
   },
   {
   create: {}
   },
   {
   "@timestamp": '2022-06-21T15:40:10Z',
   kubernetes: {
   host: 'gke-apps-0',
   node: 'gke-apps-0-0',
   pod: 'gke-apps-0-0-0',
   container: {
   cpu: {
   usage: {
   nanocores: 100_046,
   core: {
   ns: 12_828_317_850
   },
   node: {
   pct: 2.77905e-05
   },
   limit: {
   pct: 2.77905e-05
   }
   }
   },
   memory: {
   available: {
   bytes: 567_160_996
   },
   usage: {
   bytes: 362_826_547,
   node: {
   pct: 0.01770037710617187
   },
   limit: {
   pct: 9.923134671484496e-05
   }
   },
   workingset: {
   bytes: 1_986_724
   },
   rss: {
   bytes: 402_801
   },
   pagefaults: 296_495,
   majorpagefaults: 0
   },
   start_time: '2021-03-30T07:59:06Z',
   name: 'container-name-44'
   },
   namespace: 'namespace26'
   }
   },
   {
   create: {}
   },
   {
   "@timestamp": '2022-06-21T15:38:30Z',
   kubernetes: {
   host: 'gke-apps-0',
   node: 'gke-apps-0-0',
   pod: 'gke-apps-0-0-0',
   container: {
   cpu: {
   usage: {
   nanocores: 40_018,
   core: {
   ns: 12_828_317_850
   },
   node: {
   pct: 2.77905e-05
   },
   limit: {
   pct: 2.77905e-05
   }
   }
   },
   memory: {
   available: {
   bytes: 1_062_428_344
   },
   usage: {
   bytes: 265_142_477,
   node: {
   pct: 0.01770037710617187
   },
   limit: {
   pct: 9.923134671484496e-05
   }
   },
   workingset: {
   bytes: 2_294_743
   },
   rss: {
   bytes: 340_623
   },
   pagefaults: 224_530,
   majorpagefaults: 0
   },
   start_time: '2021-03-30T07:59:06Z',
   name: 'container-name-44'
   },
   namespace: 'namespace26'
   }
   }
  ]
)
puts response

Js

const response = await client.bulk({
  index: "my-data-stream",
  refresh: "true",
  pipeline: "my-timestamp-pipeline",
  operations: [
   {
   create: {},
   },
   {
   "@timestamp": "2022-06-21T15:49:00Z",
   kubernetes: {
   host: "gke-apps-0",
   node: "gke-apps-0-0",
   pod: "gke-apps-0-0-0",
   container: {
   cpu: {
   usage: {
   nanocores: 91153,
   core: {
   ns: 12828317850,
   },
   node: {
   pct: 0.0000277905,
   },
   limit: {
   pct: 0.0000277905,
   },
   },
   },
   memory: {
   available: {
   bytes: 463314616,
   },
   usage: {
   bytes: 307007078,
   node: {
   pct: 0.01770037710617187,
   },
   limit: {
   pct: 0.00009923134671484496,
   },
   },
   workingset: {
   bytes: 585236,
   },
   rss: {
   bytes: 102728,
   },
   pagefaults: 120901,
   majorpagefaults: 0,
   },
   start_time: "2021-03-30T07:59:06Z",
   name: "container-name-44",
   },
   namespace: "namespace26",
   },
   },
   {
   create: {},
   },
   {
   "@timestamp": "2022-06-21T15:45:50Z",
   kubernetes: {
   host: "gke-apps-0",
   node: "gke-apps-0-0",
   pod: "gke-apps-0-0-0",
   container: {
   cpu: {
   usage: {
   nanocores: 124501,
   core: {
   ns: 12828317850,
   },
   node: {
   pct: 0.0000277905,
   },
   limit: {
   pct: 0.0000277905,
   },
   },
   },
   memory: {
   available: {
   bytes: 982546514,
   },
   usage: {
   bytes: 360035574,
   node: {
   pct: 0.01770037710617187,
   },
   limit: {
   pct: 0.00009923134671484496,
   },
   },
   workingset: {
   bytes: 1339884,
   },
   rss: {
   bytes: 381174,
   },
   pagefaults: 178473,
   majorpagefaults: 0,
   },
   start_time: "2021-03-30T07:59:06Z",
   name: "container-name-44",
   },
   namespace: "namespace26",
   },
   },
   {
   create: {},
   },
   {
   "@timestamp": "2022-06-21T15:44:50Z",
   kubernetes: {
   host: "gke-apps-0",
   node: "gke-apps-0-0",
   pod: "gke-apps-0-0-0",
   container: {
   cpu: {
   usage: {
   nanocores: 38907,
   core: {
   ns: 12828317850,
   },
   node: {
   pct: 0.0000277905,
   },
   limit: {
   pct: 0.0000277905,
   },
   },
   },
   memory: {
   available: {
   bytes: 862723768,
   },
   usage: {
   bytes: 379572388,
   node: {
   pct: 0.01770037710617187,
   },
   limit: {
   pct: 0.00009923134671484496,
   },
   },
   workingset: {
   bytes: 431227,
   },
   rss: {
   bytes: 386580,
   },
   pagefaults: 233166,
   majorpagefaults: 0,
   },
   start_time: "2021-03-30T07:59:06Z",
   name: "container-name-44",
   },
   namespace: "namespace26",
   },
   },
   {
   create: {},
   },
   {
   "@timestamp": "2022-06-21T15:44:40Z",
   kubernetes: {
   host: "gke-apps-0",
   node: "gke-apps-0-0",
   pod: "gke-apps-0-0-0",
   container: {
   cpu: {
   usage: {
   nanocores: 86706,
   core: {
   ns: 12828317850,
   },
   node: {
   pct: 0.0000277905,
   },
   limit: {
   pct: 0.0000277905,
   },
   },
   },
   memory: {
   available: {
   bytes: 567160996,
   },
   usage: {
   bytes: 103266017,
   node: {
   pct: 0.01770037710617187,
   },
   limit: {
   pct: 0.00009923134671484496,
   },
   },
   workingset: {
   bytes: 1724908,
   },
   rss: {
   bytes: 105431,
   },
   pagefaults: 233166,
   majorpagefaults: 0,
   },
   start_time: "2021-03-30T07:59:06Z",
   name: "container-name-44",
   },
   namespace: "namespace26",
   },
   },
   {
   create: {},
   },
   {
   "@timestamp": "2022-06-21T15:44:00Z",
   kubernetes: {
   host: "gke-apps-0",
   node: "gke-apps-0-0",
   pod: "gke-apps-0-0-0",
   container: {
   cpu: {
   usage: {
   nanocores: 150069,
   core: {
   ns: 12828317850,
   },
   node: {
   pct: 0.0000277905,
   },
   limit: {
   pct: 0.0000277905,
   },
   },
   },
   memory: {
   available: {
   bytes: 639054643,
   },
   usage: {
   bytes: 265142477,
   node: {
   pct: 0.01770037710617187,
   },
   limit: {
   pct: 0.00009923134671484496,
   },
   },
   workingset: {
   bytes: 1786511,
   },
   rss: {
   bytes: 189235,
   },
   pagefaults: 138172,
   majorpagefaults: 0,
   },
   start_time: "2021-03-30T07:59:06Z",
   name: "container-name-44",
   },
   namespace: "namespace26",
   },
   },
   {
   create: {},
   },
   {
   "@timestamp": "2022-06-21T15:42:40Z",
   kubernetes: {
   host: "gke-apps-0",
   node: "gke-apps-0-0",
   pod: "gke-apps-0-0-0",
   container: {
   cpu: {
   usage: {
   nanocores: 82260,
   core: {
   ns: 12828317850,
   },
   node: {
   pct: 0.0000277905,
   },
   limit: {
   pct: 0.0000277905,
   },
   },
   },
   memory: {
   available: {
   bytes: 854735585,
   },
   usage: {
   bytes: 309798052,
   node: {
   pct: 0.01770037710617187,
   },
   limit: {
   pct: 0.00009923134671484496,
   },
   },
   workingset: {
   bytes: 924058,
   },
   rss: {
   bytes: 110838,
   },
   pagefaults: 259073,
   majorpagefaults: 0,
   },
   start_time: "2021-03-30T07:59:06Z",
   name: "container-name-44",
   },
   namespace: "namespace26",
   },
   },
   {
   create: {},
   },
   {
   "@timestamp": "2022-06-21T15:42:10Z",
   kubernetes: {
   host: "gke-apps-0",
   node: "gke-apps-0-0",
   pod: "gke-apps-0-0-0",
   container: {
   cpu: {
   usage: {
   nanocores: 153404,
   core: {
   ns: 12828317850,
   },
   node: {
   pct: 0.0000277905,
   },
   limit: {
   pct: 0.0000277905,
   },
   },
   },
   memory: {
   available: {
   bytes: 279586406,
   },
   usage: {
   bytes: 214904955,
   node: {
   pct: 0.01770037710617187,
   },
   limit: {
   pct: 0.00009923134671484496,
   },
   },
   workingset: {
   bytes: 1047265,
   },
   rss: {
   bytes: 91914,
   },
   pagefaults: 302252,
   majorpagefaults: 0,
   },
   start_time: "2021-03-30T07:59:06Z",
   name: "container-name-44",
   },
   namespace: "namespace26",
   },
   },
   {
   create: {},
   },
   {
   "@timestamp": "2022-06-21T15:40:20Z",
   kubernetes: {
   host: "gke-apps-0",
   node: "gke-apps-0-0",
   pod: "gke-apps-0-0-0",
   container: {
   cpu: {
   usage: {
   nanocores: 125613,
   core: {
   ns: 12828317850,
   },
   node: {
   pct: 0.0000277905,
   },
   limit: {
   pct: 0.0000277905,
   },
   },
   },
   memory: {
   available: {
   bytes: 822782853,
   },
   usage: {
   bytes: 100475044,
   node: {
   pct: 0.01770037710617187,
   },
   limit: {
   pct: 0.00009923134671484496,
   },
   },
   workingset: {
   bytes: 2109932,
   },
   rss: {
   bytes: 278446,
   },
   pagefaults: 74843,
   majorpagefaults: 0,
   },
   start_time: "2021-03-30T07:59:06Z",
   name: "container-name-44",
   },
   namespace: "namespace26",
   },
   },
   {
   create: {},
   },
   {
   "@timestamp": "2022-06-21T15:40:10Z",
   kubernetes: {
   host: "gke-apps-0",
   node: "gke-apps-0-0",
   pod: "gke-apps-0-0-0",
   container: {
   cpu: {
   usage: {
   nanocores: 100046,
   core: {
   ns: 12828317850,
   },
   node: {
   pct: 0.0000277905,
   },
   limit: {
   pct: 0.0000277905,
   },
   },
   },
   memory: {
   available: {
   bytes: 567160996,
   },
   usage: {
   bytes: 362826547,
   node: {
   pct: 0.01770037710617187,
   },
   limit: {
   pct: 0.00009923134671484496,
   },
   },
   workingset: {
   bytes: 1986724,
   },
   rss: {
   bytes: 402801,
   },
   pagefaults: 296495,
   majorpagefaults: 0,
   },
   start_time: "2021-03-30T07:59:06Z",
   name: "container-name-44",
   },
   namespace: "namespace26",
   },
   },
   {
   create: {},
   },
   {
   "@timestamp": "2022-06-21T15:38:30Z",
   kubernetes: {
   host: "gke-apps-0",
   node: "gke-apps-0-0",
   pod: "gke-apps-0-0-0",
   container: {
   cpu: {
   usage: {
   nanocores: 40018,
   core: {
   ns: 12828317850,
   },
   node: {
   pct: 0.0000277905,
   },
   limit: {
   pct: 0.0000277905,
   },
   },
   },
   memory: {
   available: {
   bytes: 1062428344,
   },
   usage: {
   bytes: 265142477,
   node: {
   pct: 0.01770037710617187,
   },
   limit: {
   pct: 0.00009923134671484496,
   },
   },
   workingset: {
   bytes: 2294743,
   },
   rss: {
   bytes: 340623,
   },
   pagefaults: 224530,
   majorpagefaults: 0,
   },
   start_time: "2021-03-30T07:59:06Z",
   name: "container-name-44",
   },
   namespace: "namespace26",
   },
   },
  ],
});
console.log(response);

Console

PUT /my-data-stream/_bulk?refresh&pipeline=my-timestamp-pipeline
{"create": {}}
{"@timestamp":"2022-06-21T15:49:00Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":91153,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":463314616},"usage":{"bytes":307007078,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":585236},"rss":{"bytes":102728},"pagefaults":120901,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:45:50Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":124501,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":982546514},"usage":{"bytes":360035574,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1339884},"rss":{"bytes":381174},"pagefaults":178473,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:44:50Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":38907,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":862723768},"usage":{"bytes":379572388,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":431227},"rss":{"bytes":386580},"pagefaults":233166,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:44:40Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":86706,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":567160996},"usage":{"bytes":103266017,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1724908},"rss":{"bytes":105431},"pagefaults":233166,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:44:00Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":150069,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":639054643},"usage":{"bytes":265142477,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1786511},"rss":{"bytes":189235},"pagefaults":138172,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:42:40Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":82260,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":854735585},"usage":{"bytes":309798052,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":924058},"rss":{"bytes":110838},"pagefaults":259073,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:42:10Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":153404,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":279586406},"usage":{"bytes":214904955,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1047265},"rss":{"bytes":91914},"pagefaults":302252,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:40:20Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":125613,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":822782853},"usage":{"bytes":100475044,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":2109932},"rss":{"bytes":278446},"pagefaults":74843,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:40:10Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":100046,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":567160996},"usage":{"bytes":362826547,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1986724},"rss":{"bytes":402801},"pagefaults":296495,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:38:30Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":40018,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":1062428344},"usage":{"bytes":265142477,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":2294743},"rss":{"bytes":340623},"pagefaults":224530,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}

検索APIを使用して、ドキュメントが正しくインデックスされているかどうかを確認できます:

Python

resp = client.search(
   index="my-data-stream",
)
print(resp)

Ruby

response = client.search(
  index: 'my-data-stream'
)
puts response

Js

const response = await client.search({
  index: "my-data-stream",
});
console.log(response);

Console

GET /my-data-stream/_search

データに対して次の集計を実行して、いくつかの興味深い統計を計算します:

Python

resp = client.search(
   index="my-data-stream",
   size=0,
   aggs={
   "tsid": {
   "terms": {
   "field": "_tsid"
   },
   "aggs": {
   "over_time": {
   "date_histogram": {
   "field": "@timestamp",
   "fixed_interval": "1d"
   },
   "aggs": {
   "min": {
   "min": {
   "field": "kubernetes.container.memory.usage.bytes"
   }
   },
   "max": {
   "max": {
   "field": "kubernetes.container.memory.usage.bytes"
   }
   },
   "avg": {
   "avg": {
   "field": "kubernetes.container.memory.usage.bytes"
   }
   }
   }
   }
   }
   }
   },
)
print(resp)

Ruby

response = client.search(
  index: 'my-data-stream',
  body: {
   size: 0,
   aggregations: {
   tsid: {
   terms: {
   field: '_tsid'
   },
   aggregations: {
   over_time: {
   date_histogram: {
   field: '@timestamp',
   fixed_interval: '1d'
   },
   aggregations: {
   min: {
   min: {
   field: 'kubernetes.container.memory.usage.bytes'
   }
   },
   max: {
   max: {
   field: 'kubernetes.container.memory.usage.bytes'
   }
   },
   avg: {
   avg: {
   field: 'kubernetes.container.memory.usage.bytes'
   }
   }
   }
   }
   }
   }
   }
  }
)
puts response

Js

const response = await client.search({
  index: "my-data-stream",
  size: 0,
  aggs: {
   tsid: {
   terms: {
   field: "_tsid",
   },
   aggs: {
   over_time: {
   date_histogram: {
   field: "@timestamp",
   fixed_interval: "1d",
   },
   aggs: {
   min: {
   min: {
   field: "kubernetes.container.memory.usage.bytes",
   },
   },
   max: {
   max: {
   field: "kubernetes.container.memory.usage.bytes",
   },
   },
   avg: {
   avg: {
   field: "kubernetes.container.memory.usage.bytes",
   },
   },
   },
   },
   },
   },
  },
});
console.log(response);

Console

GET /my-data-stream/_search
{
   "size": 0,
   "aggs": {
   "tsid": {
   "terms": {
   "field": "_tsid"
   },
   "aggs": {
   "over_time": {
   "date_histogram": {
   "field": "@timestamp",
   "fixed_interval": "1d"
   },
   "aggs": {
   "min": {
   "min": {
   "field": "kubernetes.container.memory.usage.bytes"
   }
   },
   "max": {
   "max": {
   "field": "kubernetes.container.memory.usage.bytes"
   }
   },
   "avg": {
   "avg": {
   "field": "kubernetes.container.memory.usage.bytes"
   }
   }
   }
   }
   }
   }
   }
}

TSDSをダウンサンプリングする

TSDSは直接ダウンサンプリングできません。その代わりに、バックインデックスをダウンサンプリングする必要があります。データストリームのバックインデックスを確認するには、次のコマンドを実行します:

Python

resp = client.indices.get_data_stream(
   name="my-data-stream",
)
print(resp)

Ruby

response = client.indices.get_data_stream(
  name: 'my-data-stream'
)
puts response

Js

const response = await client.indices.getDataStream({
  name: "my-data-stream",
});
console.log(response);

Console

GET /_data_stream/my-data-stream

これにより、次のように返されます:

Console-Result

{
  "data_streams": [
   {
   "name": "my-data-stream",
   "timestamp_field": {
   "name": "@timestamp"
   },
   "indices": [
   {
   "index_name": ".ds-my-data-stream-2023.07.26-000001",
   "index_uuid": "ltOJGmqgTVm4T-Buoe7Acg",
   "prefer_ilm": true,
   "managed_by": "Unmanaged"
   }
   ],
   "generation": 1,
   "status": "GREEN",
   "next_generation_managed_by": "Unmanaged",
   "prefer_ilm": true,
   "template": "my-data-stream-template",
   "hidden": false,
   "system": false,
   "allow_custom_routing": false,
   "replicated": false,
   "rollover_on_write": false,
   "time_series": {
   "temporal_ranges": [
   {
   "start": "2023-07-26T09:26:42.000Z",
   "end": "2023-07-26T13:26:42.000Z"
   }
   ]
   }
   }
  ]
}


	このデータストリームのバックインデックス。

バックインデックスをダウンサンプリングする前に、TSDSをロールオーバーし、古いインデックスを読み取り専用にする必要があります。

ロールオーバーAPIを使用してTSDSをロールオーバーします:

Python

resp = client.indices.rollover(
   alias="my-data-stream",
)
print(resp)

Ruby

response = client.indices.rollover(
  alias: 'my-data-stream'
)
puts response

Js

const response = await client.indices.rollover({
  alias: "my-data-stream",
});
console.log(response);

Console

POST /my-data-stream/_rollover/

応答からold_indexの名前をコピーします。次の手順では、インデックス名をold_indexのものに置き換えます。

古いインデックスは読み取り専用モードに設定する必要があります。次のリクエストを実行します:

Python

resp = client.indices.add_block(
   index=".ds-my-data-stream-2023.07.26-000001",
   block="write",
)
print(resp)

Ruby

response = client.indices.add_block(
  index: '.ds-my-data-stream-2023.07.26-000001',
  block: 'write'
)
puts response

Js

const response = await client.indices.addBlock({
  index: ".ds-my-data-stream-2023.07.26-000001",
  block: "write",
});
console.log(response);

Console

PUT /.ds-my-data-stream-2023.07.26-000001/_block/write

次に、ダウンサンプリングAPIを使用してインデックスをダウンサンプリングし、時系列間隔を1時間に設定します:

Python

resp = client.indices.downsample(
   index=".ds-my-data-stream-2023.07.26-000001",
   target_index=".ds-my-data-stream-2023.07.26-000001-downsample",
   config={
   "fixed_interval": "1h"
   },
)
print(resp)

Ruby

response = client.indices.downsample(
  index: '.ds-my-data-stream-2023.07.26-000001',
  target_index: '.ds-my-data-stream-2023.07.26-000001-downsample',
  body: {
   fixed_interval: '1h'
  }
)
puts response

Js

const response = await client.indices.downsample({
  index: ".ds-my-data-stream-2023.07.26-000001",
  target_index: ".ds-my-data-stream-2023.07.26-000001-downsample",
  config: {
   fixed_interval: "1h",
  },
});
console.log(response);

Console

POST /.ds-my-data-stream-2023.07.26-000001/_downsample/.ds-my-data-stream-2023.07.26-000001-downsample
{
  "fixed_interval": "1h"
}

今、データストリームを変更することができ、元のインデックスをダウンサンプリングされたものに置き換えることができます:

Python

resp = client.indices.modify_data_stream(
   actions=[
   {
   "remove_backing_index": {
   "data_stream": "my-data-stream",
   "index": ".ds-my-data-stream-2023.07.26-000001"
   }
   },
   {
   "add_backing_index": {
   "data_stream": "my-data-stream",
   "index": ".ds-my-data-stream-2023.07.26-000001-downsample"
   }
   }
   ],
)
print(resp)

Js

const response = await client.indices.modifyDataStream({
  actions: [
   {
   remove_backing_index: {
   data_stream: "my-data-stream",
   index: ".ds-my-data-stream-2023.07.26-000001",
   },
   },
   {
   add_backing_index: {
   data_stream: "my-data-stream",
   index: ".ds-my-data-stream-2023.07.26-000001-downsample",
   },
   },
  ],
});
console.log(response);

Console

POST _data_stream/_modify
{
  "actions": [
   {
   "remove_backing_index": {
   "data_stream": "my-data-stream",
   "index": ".ds-my-data-stream-2023.07.26-000001"
   }
   },
   {
   "add_backing_index": {
   "data_stream": "my-data-stream",
   "index": ".ds-my-data-stream-2023.07.26-000001-downsample"
   }
   }
  ]
}

古いバックインデックスを削除できます。ただし、これにより元のデータが削除されることに注意してください。将来的に元のデータが必要な場合は、インデックスを削除しないでください。

結果を表示する

以前の検索クエリを再実行します (ダウンサンプリングされたインデックスをクエリする際にはいくつかの注意点があります):

Python

resp = client.search(
   index="my-data-stream",
)
print(resp)

Ruby

response = client.search(
  index: 'my-data-stream'
)
puts response

Js

const response = await client.search({
  index: "my-data-stream",
});
console.log(response);

Console

GET /my-data-stream/_search

新しいダウンサンプリングされたバックインデックスを持つTSDSには、1つのドキュメントしか含まれていません。カウンターの場合、このドキュメントには最後の値のみが含まれます。ゲージの場合、フィールドタイプはaggregate_metric_doubleです。元のサンプルメトリックに基づいて、min、max、sum、およびvalue_countの統計が表示されます:

Console-Result

{
  "took": 2,
  "timed_out": false,
  "_shards": {
   "total": 4,
   "successful": 4,
   "skipped": 0,
   "failed": 0
  },
  "hits": {
   "total": {
   "value": 1,
   "relation": "eq"
   },
   "max_score": 1,
   "hits": [
   {
   "_index": ".ds-my-data-stream-2023.07.26-000001-downsample",
   "_id": "0eL0wC_4-45SnTNFAAABiZHbD4A",
   "_score": 1,
   "_source": {
   "@timestamp": "2023-07-26T11:00:00.000Z",
   "_doc_count": 10,
   "ingest_time": "2023-07-26T11:26:42.715Z",
   "kubernetes": {
   "container": {
   "cpu": {
   "usage": {
   "core": {
   "ns": 12828317850
   },
   "limit": {
   "pct": 0.0000277905
   },
   "nanocores": {
   "min": 38907,
   "max": 153404,
   "sum": 992677,
   "value_count": 10
   },
   "node": {
   "pct": 0.0000277905
   }
   }
   },
   "memory": {
   "available": {
   "bytes": {
   "min": 279586406,
   "max": 1062428344,
   "sum": 7101494721,
   "value_count": 10
   }
   },
   "majorpagefaults": 0,
   "pagefaults": {
   "min": 74843,
   "max": 302252,
   "sum": 2061071,
   "value_count": 10
   },
   "rss": {
   "bytes": {
   "min": 91914,
   "max": 402801,
   "sum": 2389770,
   "value_count": 10
   }
   },
   "usage": {
   "bytes": {
   "min": 100475044,
   "max": 379572388,
   "sum": 2668170609,
   "value_count": 10
   },
   "limit": {
   "pct": 0.00009923134
   },
   "node": {
   "pct": 0.017700378
   }
   },
   "workingset": {
   "bytes": {
   "min": 431227,
   "max": 2294743,
   "sum": 14230488,
   "value_count": 10
   }
   }
   },
   "name": "container-name-44",
   "start_time": "2021-03-30T07:59:06.000Z"
   },
   "host": "gke-apps-0",
   "namespace": "namespace26",
   "node": "gke-apps-0-0",
   "pod": "gke-apps-0-0-0"
   }
   }
   }
   ]
  }
}

以前の集計を再実行します。ダウンサンプリングされたTSDSは1つのドキュメントしか含まれていませんが、元のTSDSに対する以前の集計と同じ結果が返されます。

Python

resp = client.search(
   index="my-data-stream",
   size=0,
   aggs={
   "tsid": {
   "terms": {
   "field": "_tsid"
   },
   "aggs": {
   "over_time": {
   "date_histogram": {
   "field": "@timestamp",
   "fixed_interval": "1d"
   },
   "aggs": {
   "min": {
   "min": {
   "field": "kubernetes.container.memory.usage.bytes"
   }
   },
   "max": {
   "max": {
   "field": "kubernetes.container.memory.usage.bytes"
   }
   },
   "avg": {
   "avg": {
   "field": "kubernetes.container.memory.usage.bytes"
   }
   }
   }
   }
   }
   }
   },
)
print(resp)

Ruby

response = client.search(
  index: 'my-data-stream',
  body: {
   size: 0,
   aggregations: {
   tsid: {
   terms: {
   field: '_tsid'
   },
   aggregations: {
   over_time: {
   date_histogram: {
   field: '@timestamp',
   fixed_interval: '1d'
   },
   aggregations: {
   min: {
   min: {
   field: 'kubernetes.container.memory.usage.bytes'
   }
   },
   max: {
   max: {
   field: 'kubernetes.container.memory.usage.bytes'
   }
   },
   avg: {
   avg: {
   field: 'kubernetes.container.memory.usage.bytes'
   }
   }
   }
   }
   }
   }
   }
  }
)
puts response

Js

const response = await client.search({
  index: "my-data-stream",
  size: 0,
  aggs: {
   tsid: {
   terms: {
   field: "_tsid",
   },
   aggs: {
   over_time: {
   date_histogram: {
   field: "@timestamp",
   fixed_interval: "1d",
   },
   aggs: {
   min: {
   min: {
   field: "kubernetes.container.memory.usage.bytes",
   },
   },
   max: {
   max: {
   field: "kubernetes.container.memory.usage.bytes",
   },
   },
   avg: {
   avg: {
   field: "kubernetes.container.memory.usage.bytes",
   },
   },
   },
   },
   },
   },
  },
});
console.log(response);

この例は、ダウンサンプリングが時系列データの保存されるドキュメント数を劇的に減少させる方法を示しています。選択した時間の境界内で、すでにダウンサンプリングされたデータに対してもダウンサンプリングを実行することが可能で、データが古くなるにつれてストレージと関連コストをさらに削減できます。

TSDSをダウンサンプリングする推奨方法はILMを使用することです。詳細を学ぶには、ILMを使用したダウンサンプリングの実行の例を試してください。