Skip to content

Commit 59d02e4

Browse files
author
Yoichi Kawasaki
committed
Initial commit
1 parent daefee7 commit 59d02e4

File tree

11 files changed

+440
-1
lines changed

11 files changed

+440
-1
lines changed

‎ChangeLog.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
## 0.1.0
2+
3+
* Inital Release

‎Gemfile

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
source 'https://rubygems.org'
2+
3+
# Specify your gem's dependencies in fluent-plugin-azuresearch.gemspec
4+
gemspec

‎README.md

Lines changed: 156 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,157 @@
11
# fluent-plugin-azure-loganalytics
2-
Azure Log Analytics output plugin for Fluentd
2+
[Azure Log Analytics](https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-overview) output plugin for Fluentd. The plugin aggregates semi-structured data in real-time and writes the buffered data via HTTPS request to Azure Log Analytics.
3+
4+
![fluent-plugin-azure-loganalytics overview](https://github.com/yokawasa/fluent-plugin-azure-loganalytics/raw/master/img/Azure-LogAnalytics-Fluentd.png)
5+
6+
## Installation
7+
```
8+
$ gem install fluent-plugin-azure-loganalytics
9+
```
10+
11+
## Configuration
12+
13+
### Azure Log Analytics
14+
To start running with Log Analytics in the Microsoft Operations Management Suite (OMS), You need to create either an OMS workspace using the OMS website or Log Analytics workspace using your Azure subscription. Workspaces created either way are functionally equivalent. Here is an instruction:
15+
16+
* [Get started with Log Analytics](https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-get-started)
17+
18+
Once you have the workspace, get Workspace ID and Shared Key (either Primary Key or Secondary Key), which are needed by [Log Analytics HTTP Data Collector API](https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-data-collector-api) to post the data to Log Analytics.
19+
20+
21+
### Fluentd - fluent.conf
22+
23+
```
24+
<match azure-loganalytics.**>
25+
@type azure-loganalytics
26+
customer_id CUSTOMER_ID # Customer ID aka WorkspaceID String
27+
shared_key KEY_STRING # The primary or the secondary Connected Sources client authentication key
28+
log_type EVENT_TYPE_NAME # The name of the event type. ex) ApacheAccessLog
29+
add_time_field true
30+
time_field_name mytime
31+
time_format %s
32+
localtime true
33+
add_tag_field true
34+
tag_field_name mytag
35+
</match>
36+
```
37+
38+
* **customer\_id (required)** - Your Operations Management Suite workspace ID
39+
* **shared\_key (required)** - The primary or the secondary Connected Sources client authentication key
40+
* **log\_type (required)** - The name of the event type that is being submitted to Log Analytics
41+
* **add\_time\_field (optional)** - Default:true. This option allows to insert a time field to record
42+
* **time\_field\_name (optional)** - Default:time. This is required only when add_time_field is true
43+
* **localtime (optional)** - Default:false. Time record is inserted with UTC (Coordinated Universal Time) by default. This option allows to use local time if you set localtime true. This is valid only when add_time_field is true
44+
* **time\_format (optional)** - Default:%s. Time format for a time field to be inserted. Default format is %s, that is unix epoch time. If you want it to be more human readable, set this %Y%m%d-%H:%M:%S, for example. This is valid only when add_time_field is true.
45+
* **add\_tag\_field (optional)** - Default:false. This option allows to insert a tag field to record
46+
* **tag\_field\_name (optional)** - Default:tag. This is required only when add_time_field is true
47+
48+
49+
## Configuration examples
50+
51+
fluent-plugin-azure-loganalytics adds **time** and **tag** attributes by default if **add_time_field** and **add_tag_field** are true respectively. Below are two types of the plugin configurations - Default and All options configuration.
52+
53+
### (1) Default Configuration (No options)
54+
<u>fluent.conf</u>
55+
```
56+
<source>
57+
@type tail # input plugin
58+
path /var/log/apache2/access.log # monitoring file
59+
pos_file /tmp/fluentd_pos_file # position file
60+
format apache # format
61+
tag azure-loganalytics.access # tag
62+
</source>
63+
64+
<match azure-loganalytics.**>
65+
@type azure-loganalytics
66+
customer_id 818f7bbc-8034-4cc3-b97d-f068dd4cd658
67+
shared_key ppC5500KzCcDsOKwM1yWUvZydCuC3m+ds/2xci0byeQr1G3E0Jkygn1N0Rxx/yVBUrDE2ok3vf4ksCzvBmQXHw==(dummy)
68+
log_type ApacheAccessLog
69+
</match>
70+
```
71+
72+
### (2) Configuration with All Options
73+
<u>fluent.conf</u>
74+
```
75+
<source>
76+
@type tail # input plugin
77+
path /var/log/apache2/access.log # monitoring file
78+
pos_file /tmp/fluentd_pos_file # position file
79+
format apache # format
80+
tag azure-loganalytics.access # tag
81+
</source>
82+
83+
<match azure-loganalytics.**>
84+
@type azure-loganalytics
85+
customer_id 818f7bbc-8034-4cc3-b97d-f068dd4cd658
86+
shared_key ppC5500KzCcDsOKwM1yWUvZydCuC3m+ds/2xci0byeQr1G3E0Jkygn1N0Rxx/yVBUrDE2ok3vf4ksCzvBmQXHw==(dummy)
87+
log_type ApacheAccessLog
88+
add_time_field true
89+
time_field_name mytime
90+
time_format %s
91+
localtime true
92+
add_tag_field true
93+
tag_field_name mytag
94+
</match>
95+
```
96+
97+
## Sample inputs and expected records
98+
99+
An expected output record for sample input will be like this:
100+
101+
<u>Sample Input (apache access log)</u>
102+
```
103+
125.212.152.166 - - [17/Jan/2016:05:03:25 +0000] "GET /foo/bar/test.html HTTP/1.1" 304 179 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36"
104+
```
105+
106+
<u>Output Record</u>
107+
108+
![fluent-plugin-azure-loganalytics output image](https://github.com/yokawasa/fluent-plugin-azure-loganalytics/raw/master/img/Azure-LogAnalytics-Output-Image.png)
109+
110+
111+
## Tests
112+
### Running test code
113+
```
114+
$ git clone https://github.com/yokawasa/fluent-plugin-azure-loganalytics.git
115+
$ cd fluent-plugin-azure-loganalytics
116+
117+
# edit CONFIG params of test/plugin/test_azure_loganalytics.rb
118+
$ vi test/plugin/test_azure_loganalytics.rb
119+
120+
# run test
121+
$ rake test
122+
```
123+
124+
### Creating package, running and testing locally
125+
```
126+
$ rake build
127+
$ rake install:local
128+
129+
# running fluentd with your fluent.conf
130+
$ fluentd -c fluent.conf -vv &
131+
132+
# send test apache requests for testing plugin ( only in the case that input source is apache access log )
133+
$ ab -n 5 -c 2 http://localhost/foo/bar/test.html
134+
```
135+
136+
## Change log
137+
* [Changelog](ChangeLog.md)
138+
139+
## Links
140+
141+
* https://rubygems.org/gems/fluent-plugin-azure-loganalytics
142+
* https://rubygems.org/gems/azure-loganalytics-datacollector-api
143+
144+
## Contributing
145+
146+
Bug reports and pull requests are welcome on GitHub at https://github.com/yokawasa/fluent-plugin-azure-loganalytics.
147+
148+
## Copyright
149+
150+
<table>
151+
<tr>
152+
<td>Copyright</td><td>Copyright (c) 2016- Yoichi Kawasaki</td>
153+
</tr>
154+
<tr>
155+
<td>License</td><td>Apache License, Version 2.0</td>
156+
</tr>
157+
</table>

‎Rakefile

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
#!/usr/bin/env rake
2+
3+
require "bundler/gem_tasks"
4+
require "rake/testtask"
5+
6+
Rake::TestTask.new(:test) do |test|
7+
test.libs << 'lib' << 'test'
8+
test.pattern = 'test/**/test_*.rb'
9+
test.verbose = true
10+
end
11+
12+
task :default do
13+
system("rake -T")
14+
end
15+

‎VERSION

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
0.1.0

‎examples/fluent_1.conf

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
<source>
2+
@type tail # input plugin
3+
path /var/log/apache2/access.log # monitoring file
4+
pos_file /tmp/fluentd_pos_file # position file
5+
format apache # format
6+
tag azure-loganalytics.access # tag
7+
</source>
8+
9+
<match azure-loganalytics.**>
10+
@type azure-loganalytics
11+
customer_id CUSTOMER_ID # Customer ID aka WorkspaceID String
12+
shared_key KEY_STRING # The primary or the secondary Connected Sources client authentication key
13+
log_type EVENT_TYPE_NAME # The name of the event type. ex) ApacheAccessLog
14+
</match>

‎examples/fluent_2.conf

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
<source>
2+
@type tail # input plugin
3+
path /var/log/apache2/access.log # monitoring file
4+
pos_file /tmp/fluentd_pos_file # position file
5+
format apache # format
6+
tag azure-loganalytics.access # tag
7+
</source>
8+
9+
<match azure-loganalytics.**>
10+
@type azure-loganalytics
11+
customer_id CUSTOMER_ID # Customer ID aka WorkspaceID String
12+
shared_key KEY_STRING # The primary or the secondary Connected Sources client authentication key
13+
log_type EVENT_TYPE_NAME # The name of the event type. ex) ApacheAccessLog
14+
add_time_field true
15+
time_field_name mytime
16+
time_format %s
17+
localtime true
18+
add_tag_field true
19+
tag_field_name mytag
20+
</match>

‎fluent-plugin-azure-loganalytics.gemspec

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# coding: utf-8
2+
lib = File.expand_path('../lib', __FILE__)
3+
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4+
5+
Gem::Specification.new do |gem|
6+
gem.name = "fluent-plugin-azure-loganalytics"
7+
gem.version = File.read("VERSION").strip
8+
gem.authors = ["Yoichi Kawasaki"]
9+
gem.email = ["yoichi.kawasaki@outlook.com"]
10+
gem.summary = %q{Azure Functions output plugin for Fluentd}
11+
gem.description = gem.summary
12+
gem.homepage = "http://github.com/yokawasa/fluent-plugin-azure-loganalytics"
13+
gem.license = "Apache-2.0"
14+
gem.has_rdoc = false
15+
16+
gem.files = `git ls-files`.split("\n")
17+
gem.executables = gem.files.grep(%r{^bin/}) { |f| File.basename(f) }
18+
gem.test_files = gem.files.grep(%r{^(test|gem|features)/})
19+
gem.require_paths = ["lib"]
20+
21+
gem.add_dependency "fluentd", [">= 0.10.58", "< 2"]
22+
gem.add_dependency "rest-client"
23+
gem.add_dependency "azure-loganalytics-datacollector-api", [">= 0.1.1"]
24+
gem.add_development_dependency "bundler", "~> 1.11"
25+
gem.add_development_dependency "rake", "~> 10.0"
26+
gem.add_development_dependency "test-unit"
27+
end
28+

‎lib/fluent/plugin/out_azure_loganalytics.rb

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# -*- coding: utf-8 -*-
2+
3+
module Fluent
4+
class AzureLogAnalyticsOutput < BufferedOutput
5+
Plugin.register_output('azure-loganalytics', self)
6+
7+
def initialize
8+
super
9+
require 'msgpack'
10+
require 'time'
11+
require "azure/loganalytics/datacollectorapi/client"
12+
end
13+
14+
config_param :customer_id, :string,
15+
:desc => "Your Operations Management Suite workspace ID"
16+
config_param :shared_key, :string, :secret => true,
17+
:desc => "The primary or the secondary Connected Sources client authentication key"
18+
config_param :log_type, :string, default: nil,
19+
:desc => "The name of the event type that is being submitted to Log Analytics"
20+
config_param :add_time_field, :bool, :default => true,
21+
:desc => "This option allows to insert a time field to record"
22+
config_param :time_field_name, :string, :default => "time",
23+
:desc => "This is required only when add_time_field is true"
24+
config_param :time_format, :string, :default => "%s",
25+
:desc => "Time format for a time field to be inserted. Default format is %s, that is unix epoch time. If you want it to be more human readable, set this %Y%m%d-%H:%M:%S, for example. This is valid only when add_time_field is true."
26+
config_param :localtime, :bool, :default => false,
27+
:desc => "Time record is inserted with UTC (Coordinated Universal Time) by default. This option allows to use local time if you set localtime true. This is valid only when add_time_field is true."
28+
config_param :add_tag_field, :bool, :default => false,
29+
:desc => "This option allows to insert a tag field to record"
30+
config_param :tag_field_name, :string, :default => "tag",
31+
:desc => "This is required only when add_time_field is true"
32+
33+
def configure(conf)
34+
super
35+
raise ConfigError, 'no customer_id' if @customer_id.empty?
36+
raise ConfigError, 'no shared_key' if @shared_key.empty?
37+
raise ConfigError, 'no log_type' if @log_type.empty?
38+
if @add_time_field and @time_field_name.empty?
39+
raise ConfigError, 'time_field_name must be set if add_time_field is true'
40+
end
41+
if @add_tag_field and @tag_field_name.empty?
42+
raise ConfigError, 'tag_field_name must be set if add_tag_field is true'
43+
end
44+
@timef = TimeFormatter.new(@time_format, @localtime)
45+
end
46+
47+
def start
48+
super
49+
# start
50+
@client=Azure::Loganalytics::Datacollectorapi::Client::new(@customer_id,@shared_key)
51+
end
52+
53+
def shutdown
54+
super
55+
# destroy
56+
end
57+
58+
def format(tag, time, record)
59+
if @add_time_field
60+
record[@time_field_name] = @timef.format(time)
61+
end
62+
if @add_tag_field
63+
record[@tag_field_name] = tag
64+
end
65+
record.to_msgpack
66+
end
67+
68+
def write(chunk)
69+
records = []
70+
chunk.msgpack_each { |record|
71+
records.push(record)
72+
}
73+
begin
74+
res = @client.post_data(@log_type, records)
75+
if not Azure::Loganalytics::Datacollectorapi::Client.is_success(res)
76+
$log.fatal "DataCollector API request failure: error code: "
77+
+ "#{res.code}, data=>" + records.to_json
78+
end
79+
rescue Exception => ex
80+
$log.fatal "Exception occured in posting to DataCollector API: "
81+
+ "'#{ex}', data=>" + records.to_json
82+
end
83+
end
84+
end
85+
end

‎test/helper.rb

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
require 'rubygems'
2+
require 'bundler'
3+
4+
begin
5+
Bundler.setup(:default, :development)
6+
rescue Bundler::BundlerError => e
7+
$stderr.puts e.message
8+
$stderr.puts "Run `bundle install` to install missing gems"
9+
exit e.status_code
10+
end
11+
require 'test/unit'
12+
13+
$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
14+
$LOAD_PATH.unshift(File.dirname(__FILE__))
15+
require 'fluent/test'
16+
unless ENV.has_key?('VERBOSE')
17+
nulllogger = Object.new
18+
nulllogger.instance_eval {|obj|
19+
def method_missing(method, *args)
20+
# pass
21+
end
22+
}
23+
$log = nulllogger
24+
end
25+
26+
require 'fluent/plugin/out_azure_loganalytics'
27+
28+
class Test::Unit::TestCase
29+
end

0 commit comments

Comments
 (0)