Fluentd的output插件支持<buffer>配置项,用以缓存日志事件,提高系统性能。可在此配置项中设置buffer插件的相关参数。
<match tag.*>
@type file
# ... parameters for output plugin
<buffer>
# buffer section parameters ...
</buffer>
# <buffer> section can be configured just once
</match>
<buffer>
@type file
</buffer>
<buffer ARGUMENT_CHUNK_KEYS>
# ...
</buffer>
chunk的键值形式为逗号分隔的字符串,或者留空。
<match tag.**>
# ...
<buffer>
# ...
</buffer>
</match>
# No chunk keys: All events will be appended into the same chunk.
11:59:30 web.access {"key1":"yay","key2":100} --|
|
12:00:01 web.access {"key1":"foo","key2":200} --|---> CHUNK_A
|
12:00:25 ssh.login {"key1":"yay","key2":100} --|
<match tag.**>
# ...
<buffer tag>
# ...
</buffer>
</match>
# Tag chunk key: events will be separated per tags
11:59:30 web.access {"key1":"yay","key2":100} --|
|---> CHUNK_A
12:00:01 web.access {"key1":"foo","key2":200} --|
12:00:25 ssh.login {"key1":"yay","key2":100} ------> CHUNK_B
timekey 60: ["12:00:00", ..., "12:00:59"], ["12:01:00", ..., "12:01:59"], ...
timekey 180: ["12:00:00", ..., "12:02:59"], ["12:03:00", ..., "12:05:59"], ...
timekey 3600: ["12:00:00", ..., "12:59:59"], ["13:00:00", ..., "13:59:59"], ...
事件会按照计算出的时间范围写入不同的chunk,并将在chunk过期后进行flush。
<match tag.**>
# ...
<buffer time>
timekey 1h # chunks per hours ("3600" also available)
timekey_wait 5m # 5mins delay for flush ("300" also available)
</buffer>
</match>
# Time chunk key: events will be separated for hours (by timekey 3600)
11:59:30 web.access {"key1":"yay","key2":100} ------> CHUNK_A
12:00:01 web.access {"key1":"foo","key2":200} --|
|---> CHUNK_B
12:00:25 ssh.login {"key1":"yay","key2":100} --|
timekey_wait参数用来设定chunk的输出延迟时间,事件在chunk中停留“延迟时间”后被flush。默认值为600(10分钟)。
timekey: 3600
-------------------------------------------------------
time range for chunk | timekey_wait | actual flush time
12:00:00 - 12:59:59 | 0s | 13:00:00
12:00:00 - 12:59:59 | 60s (1m) | 13:01:00
12:00:00 - 12:59:59 | 600s (10m) | 13:10:00
上边表格展示了每小时的缓存事件在不同的timekey_wait设置情况下实际的flush时间。
<match tag.**>
# ...
<buffer key1>
# ...
</buffer>
</match>
# Chunk keys: events will be separated by values of "key1"
11:59:30 web.access {"key1":"yay","key2":100} --|---> CHUNK_A
|
12:00:01 web.access {"key1":"foo","key2":200} -)|(--> CHUNK_B
|
12:00:25 ssh.login {"key1":"yay","key2":100} --|
这里,事件将会按照key1的不同取值写入相应的chunk中。
chunk键值支持嵌套的record字段。可以参照插件的record_accessor语法来访问嵌套字段。例如:
<match tag.**>
# ...
<buffer $.nest.field> # access record['nest']['field']
# ...
</buffer>
</match>
# <buffer tag,time>
11:58:01 ssh.login {"key1":"yay","key2":100} ------> CHUNK_A
11:59:13 web.access {"key1":"yay","key2":100} --|
|---> CHUNK_B
11:59:30 web.access {"key1":"yay","key2":100} --|
12:00:01 web.access {"key1":"foo","key2":200} ------> CHUNK_C
12:00:25 ssh.login {"key1":"yay","key2":100} ------> CHUNK_D
这里,将事件按照tag+time的方式缓存到不同的chunk中。
当然,chunk键值组合不宜过多,否则会降低I/O性能,也会消耗系统资源。
chunk的键值可以设置为[],用以禁用output插件默认的chunk键配置。
<match tag.**>
# ...
<buffer []>
# ...
</buffer>
</match>
Fluentd配置文件支持占位符变量,这些变量会在运行中被替换为实际的值。比如out_file插件支持在path参数中设置占位符。
# chunk_key: tag
# ${tag} will be replaced with actual tag string
<match log.*>
@type file
path /data/${tag}/access.log #=> "/data/log.map/access.log"
<buffer tag>
# ...
</buffer>
</match>
【码字不易,若对您有帮助,希望可以收藏转发】
联系客服