Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. counter. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. * Name es Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. As you can see, there is nothing special except failed to flush chunk and chunk cannot be retried. Fri, Mar 25 2022 3:08:29 pm | [2022/03/25 07:08:29] [debug] [input chunk] update output instances with new chunk size diff=650 Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [ warn] [engine] failed to flush chunk '1-1648192107.811048259.flb', retry in 20 seconds: task_id=6, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [out coro] cb_destroy coro_id=14 [2022/03/24 04:19:49] [debug] [outputes.0] task_id=2 assigned to thread #1 Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:23] [debug] [retry] new retry created for task_id=5 attempts=1 [2022/03/24 04:19:49] [ warn] [engine] failed to flush chunk '1-1648095560.254537600.flb', retry in 15 seconds: task_id=1, input=tail.0 > output=es.0 (out_id=0) [2022/03/24 04:20:25] [debug] [outputes.0] task_id=2 assigned to thread #1 Fri, Mar 25 2022 3:08:36 pm | [2022/03/25 07:08:36] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 has been assigned (recycled) Logstash_Format On Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [retry] re-using retry for task_id=1 attempts=3 Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [input chunk] update output instances with new chunk size diff=1182 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"luMmun8BI6SaBP9l7LpS","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Expected behavior Minimally that these messages do not tie-up fluent-bit's pipeline as retrying them will never succeed. [2022/03/24 04:19:49] [debug] [retry] re-using retry for task_id=1 attempts=3 [2022/03/24 04:19:24] [debug] [out coro] cb_destroy coro_id=0 Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 is now available Logstash_Format On {"took":2579,"errors":true,"items":[{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"G-Mmun8BI6SaBP9lh4vZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=35359369 events: IN_ATTRIB Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=69336502 watch_fd=12 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"YeMnun8BI6SaBP9lLtm1","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:20:51] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000 Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [outputes.0] task_id=10 assigned to thread #0 [2022/03/24 04:21:20] [debug] [input:tail:tail.0] inode=1772861 with offset=0 appended as /var/log/containers/hello-world-wpr5j_argo_wait-76bcd0771f3cc7b5f6b5f15f16ee01cc0c671fb047b93910271bc73e753e26ee.log {"took":2217,"errors":true,"items":[{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"yeMnun8BI6SaBP9lo-jn","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [input chunk] update output instances with new chunk size diff=650 [2022/03/24 04:21:20] [debug] [input:tail:tail.0] 4 new files found on path '/var/log/containers/.log' [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"I-Moun8BI6SaBP9lIP7t","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"JuMmun8BI6SaBP9lh4vZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:22] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:32 pm | [2022/03/25 07:08:32] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [http_client] not using http_proxy for header Hi @yangtian9999, can you confirm you still experiencing this issue? Name es [2022/03/24 04:19:34] [debug] [upstream] KA connection #103 to 10.3.4.84:9200 has been assigned (recycled) Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 has been assigned (recycled) Fri, Mar 25 2022 3:08:33 pm | [2022/03/25 07:08:33] [debug] [retry] new retry created for task_id=11 attempts=1 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"Z-Mnun8BI6SaBP9lLtm1","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [http_client] not using http_proxy for header I am seeing this in fluentd logs in kubernetes. Hi @yangtian9999 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. I am getting these errors during ES logging using fluentd. Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [http_client] not using http_proxy for header "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"H-Moun8BI6SaBP9lIP7t","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [out coro] cb_destroy coro_id=13 failed to flush the buffer fluentd. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"feMnun8BI6SaBP9l3vN-","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:20:06] [ warn] [engine] failed to flush chunk '1-1648095560.254537600.flb', retry in 60 seconds: task_id=1, input=tail.0 > output=es.0 (out_id=0) [2022/03/24 04:19:49] [debug] [out coro] cb_destroy coro_id=3 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"aOMnun8BI6SaBP9lLtm1","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [input chunk] update output instances with new chunk size diff=649 [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records [2022/03/24 04:20:20] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=69179340 watch_fd=6 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"keMmun8BI6SaBP9l7LpS","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:19:24] [debug] [retry] new retry created for task_id=0 attempts=1 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Hi! [2022/03/22 03:57:48] [ warn] [engine] failed to flush chunk '1-1647920657.177210280.flb', retry in 1048 seconds: task_id=464, input=tail.0 > output=es.0 (out_id=0) Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"PeMmun8BI6SaBP9luqVq","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamicall Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [input chunk] update output instances with new chunk size diff=1085 Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:23] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [outputes.0] task_id=21 assigned to thread #0 Fri, Mar 25 2022 3:08:47 pm | [2022/03/25 07:08:47] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 has been assigned (recycled) Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [ warn] [engine] failed to flush chunk '1-1648192122.113977737.flb', retry in 21 seconds: task_id=16, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [input chunk] update output instances with new chunk size diff=695 Host 10.3.4.84 Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [ warn] [engine] failed to flush chunk '1-1648192097.600252923.flb', retry in 14 seconds: task_id=0, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:23] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 has been assigned (recycled) hi @lecaros I think this was only [warn] message, I checked with es, I can search the right apps logs. privacy statement. This is the total record count of all unique chunks sent by this output. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:23] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [input chunk] update output instances with new chunk size diff=1083 Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:31 pm | [2022/03/25 07:08:31] [debug] [outputes.0] task_id=9 assigned to thread #0 Logstash_Prefix node [2022/03/24 04:21:08] [debug] [retry] re-using retry for task_id=1 attempts=5 Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"J-Moun8BI6SaBP9lIP7t","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. * aws: utils: fix mem leak in flb_imds_request (fluent#2532) Signed-off-by: Wesley Pettit <wppttt@amazon.com> * io: fix EINPROGRESS check, also check . Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [retry] re-using retry for task_id=3 attempts=3 Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [input chunk] update output instances with new chunk size diff=650 Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 is now available "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"AuMmun8BI6SaBP9l_8rZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY is any way to skip create index if exists? [2022/03/24 04:20:20] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=103386716 watch_fd=5 Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [input chunk] update output instances with new chunk size diff=695 Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [input chunk] update output instances with new chunk size diff=656 Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [input chunk] update output instances with new chunk size diff=665 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"IOMoun8BI6SaBP9lIP7t","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. (fbit, es) Make sure you're using either 1.9.1 or 1.8.15. Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:31 pm | [2022/03/25 07:08:31] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Sign in [2022/03/24 04:19:21] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:42 pm | [2022/03/25 07:08:42] [debug] [outputes.0] HTTP Status=200 URI=/_bulk [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records Fri, Mar 25 2022 3:08:31 pm | [2022/03/25 07:08:31] [debug] [out coro] cb_destroy coro_id=7 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:42 pm | [2022/03/25 07:08:42] [debug] [outputes.0] task_id=16 assigned to thread #1 Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [task] created task=0x7ff2f1839b20 id=6 OK Fri, Mar 25 2022 3:08:44 pm | [2022/03/25 07:08:44] [debug] [outputes.0] task_id=17 assigned to thread #0 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. After reallocating, I found that there are a lot of failed to flush the buffer errors logged by the two OOMkilled fluentd pods before. Have a question about this project? Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"JeMmun8BI6SaBP9lh4vZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:20:26] [ warn] [engine] failed to flush chunk '1-1648095560.297175793.flb', retry in 161 seconds: task_id=2, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:47 pm | [2022/03/25 07:08:47] [debug] [outputes.0] HTTP Status=200 URI=/_bulk [2022/03/24 04:19:38] [ warn] [engine] failed to flush chunk '1-1648095560.254537600.flb', retry in 9 seconds: task_id=1, input=tail.0 > output=es.0 (out_id=0) Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. [2022/03/24 04:20:20] [debug] [input:tail:tail.0] purge: monitored file has been deleted: /var/log/containers/hello-world-7mwzw_argo_main-4a2ecde2fd5310129cdf3e3c7eacc17fc1ae0eb6b5e88bed0fdf8fd7fd1100f4.log [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [input chunk] update output instances with new chunk size diff=655 Fri, Mar 25 2022 3:08:37 pm | [2022/03/25 07:08:37] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 has been assigned (recycled) Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [upstream] KA connection #36 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [ warn] [engine] failed to flush chunk '1-1648192110.850147571.flb', retry in 11 seconds: task_id=9, input=tail.0 > output=es.0 (out_id=0) I only changed the output config since its a subchart. Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [outputes.0] HTTP Status=200 URI=/_bulk [2022/03/24 04:19:24] [debug] [retry] new retry created for task_id=1 attempts=1 [2022/03/24 04:20:36] [debug] [outputes.0] HTTP Status=200 URI=/_bulk [2022/03/24 04:20:20] [debug] [input:tail:tail.0] inode=103386717 removing file name /var/log/containers/hello-world-7mwzw_argo_main-4a2ecde2fd5310129cdf3e3c7eacc17fc1ae0eb6b5e88bed0fdf8fd7fd1100f4.log Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [upstream] KA connection #36 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [outputes.0] HTTP Status=200 URI=/_bulk [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [outputes.0] task_id=6 assigned to thread #1 Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [input chunk] update output instances with new chunk size diff=1083 Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [retry] new retry created for task_id=6 attempts=1 Fri, Mar 25 2022 3:08:22 pm | [2022/03/25 07:08:22] [debug] [out coro] cb_destroy coro_id=2 Version used: helm-charts-fluent-bit-0.19.19. [2022/03/24 04:21:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/helm-install-traefik-j2ncv_kube-system_helm-4554d6945ad4a135678c69aae3fb44bf003479edc450b256421a51ce68a37c59.log, inode 622082 Fri, Mar 25 2022 3:08:31 pm | [2022/03/25 07:08:31] [debug] [http_client] not using http_proxy for header retry_time=29 next_retry_seconds=2021-04-26 15:58:43 +0000 chunk . [2022/03/24 04:20:20] [debug] [input:tail:tail.0] inode=1885019 with offset=0 appended as /var/log/containers/hello-world-dsxks_argo_wait-114879608f2fe019cd6cfce8e3777f9c0a4f34db2f6dc72bb39b2b5ceb917d4b.log with the updated value.yaml file. [2022/03/24 04:19:38] [error] [outputes.0] could not pack/validate JSON response Fri, Mar 25 2022 3:08:33 pm | [2022/03/25 07:08:33] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 has been assigned (recycled) [2022/03/24 04:21:08] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [task] created task=0x7ff2f183b1a0 id=18 OK Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=69479190 file has been deleted: /var/log/containers/hello-world-dsfcz_argo_main-13bb1b2c7e9d3e70003814aa3900bb9aef645cf5e3270e3ee4db0988240b9eff.log [2022/03/24 04:19:24] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000 N must be >= 1 (default: 1) When Retry_Limit is set to no_limits or False, means that there is not limit for the number of retries that the Scheduler can do. Feel free to reopen if you think this is something we should be able to fix on the operator side. Logstash_Format On Fri, Mar 25 2022 3:08:44 pm | [2022/03/25 07:08:44] [debug] [input chunk] update output instances with new chunk size diff=634 [2022/03/24 04:21:20] [debug] [input:tail:tail.0] inode=1772851 with offset=0 appended as /var/log/containers/hello-world-89knq_argo_wait-a7f77229883282b7aebce253b8c371dd28e0606575ded307669b43b272d9a2f4.log Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [out coro] cb_destroy coro_id=4 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"YOMnun8BI6SaBP9lLtm1","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. fluentbit fails to communicate with fluentd. {"took":2414,"errors":true,"items":[{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"juMmun8BI6SaBP9l7LpS","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Logstash_Format On Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 has been assigned (recycled) "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"1-Mmun8BI6SaBP9liJWQ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [input chunk] update output instances with new chunk size diff=695 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"KeMoun8BI6SaBP9lIP7t","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamicall Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 has been assigned (recycled) [2022/03/24 04:20:00] [debug] [retry] re-using retry for task_id=2 attempts=4 [2022/03/24 04:21:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/argo-server-6d7cf9c977-dlwnk_argo_argo-server-7e1ccfbd60b7539a1b2984f2f46de601d567ce83e87d434e173df195e44b5224.log, inode 101715266 I am getting these errors. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [http_client] not using http_proxy for header "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"4uMmun8BI6SaBP9luq99","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY [2022/03/24 04:19:24] [ warn] [engine] failed to flush chunk '1-1648095560.205735907.flb', retry in 10 seconds: task_id=0, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:22 pm | [2022/03/25 07:08:22] [debug] [retry] new retry created for task_id=4 attempts=1 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. I have deployed the official helm chart version 0.19.23. [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records Fri, Mar 25 2022 3:08:44 pm | [2022/03/25 07:08:44] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [out coro] cb_destroy coro_id=3 Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [out coro] cb_destroy coro_id=19 edit the value.yaml, change to es ip, 10.3.4.84 is the es ip address. And it helps in 100% cases. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. [2022/03/24 04:19:38] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000 Your Environment Fluentd or td-agent v. Here's my config; schema_config: configs: - from: 2020-05-15 store: cassandra object_store: cassandra . [2022/03/24 04:21:08] [error] [outputes.0] could not pack/validate JSON response [2022/03/24 04:20:26] [debug] [out coro] cb_destroy coro_id=5 Please check my YAML for input and output plugin of the fluent bit. Recently we have received many complaints from users about site-wide blocking of their own and blocking of their own activities please go to the settings off state, . Are you still receiving some of the records on the ES side, or does it stopped receiving records altogether? Existing mapping for [kubernetes.labels.app] must be of type object but found [text].
Small Warehouse Space For Rent Houston,
Roll Windows Down With Key Fob Chrysler Pacifica,
Practice Typing Numbers Row,
1900 Spindrift Drive In La Jolla,
Famous Schizoid Writers,
Articles F
failed to flush chunk