Class: Ralf
- Inherits:
-
Object
- Object
- Ralf
- Defined in:
- lib/ralf.rb,
lib/ralf/log.rb,
lib/ralf/bucket.rb,
lib/ralf/interpolation.rb
Defined Under Namespace
Classes: Bucket, Config, Interpolation, Log, OptionParser
Constant Summary collapse
- ROOT_DEFAULT_CONFIG_FILE =
'/etc/ralf.conf'
- USER_DEFAULT_CONFIG_FILE =
'~/.ralf.conf'
- AMAZON_LOG_FORMAT =
Regexp.new('([^ ]*) ([^ ]*) \[([^\]]*)\] ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) "([^"]*)" ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) "([^"]*)" "([^"]*)"')
Instance Attribute Summary collapse
-
#config ⇒ Object
readonly
The current configuration.
-
#s3 ⇒ Object
readonly
Instance of RightAws::S3 used by Ralf to talk to Amazon S3.
Class Method Summary collapse
-
.run(options) ⇒ Object
Create instance and run with the specified parameters.
Instance Method Summary collapse
-
#initialize(options = {}) ⇒ Ralf
constructor
Create new Ralf instance.
-
#list(with_logging = false) ⇒ Object
List all buckets with the logging info.
-
#run(output_file = nil) ⇒ Object
For all buckets for all dates in the configured range download the available log files.
Constructor Details
#initialize(options = {}) ⇒ Ralf
Create new Ralf instance
40 41 42 43 44 45 46 47 48 49 50 51 52 53 |
# File 'lib/ralf.rb', line 40 def initialize( = {}) = .dup @config = load_config(.delete(:config_file)) config.merge!() config.validate! RightAws::RightAwsBaseInterface.caching = true # enable caching to speed up Bucket.s3 = RightAws::S3.new(config.aws_access_key_id, config.aws_secret_access_key, :protocol => 'http', :port => 80, :logger => Logger.new('aws' == config.debug? ? $stdout : StringIO.new) ) end |
Instance Attribute Details
#config ⇒ Object (readonly)
The current configuration.
22 23 24 |
# File 'lib/ralf.rb', line 22 def config @config end |
#s3 ⇒ Object (readonly)
Instance of RightAws::S3 used by Ralf to talk to Amazon S3.
25 26 27 |
# File 'lib/ralf.rb', line 25 def s3 @s3 end |
Class Method Details
Instance Method Details
#list(with_logging = false) ⇒ Object
List all buckets with the logging info.
97 98 99 100 101 102 103 104 105 106 |
# File 'lib/ralf.rb', line 97 def list(with_logging = false) puts "Listing buckets..." if config.debug? Bucket.each(config.buckets, with_logging) do |bucket| print "#{bucket.name}" puts bucket.logging_enabled? ? " [#{bucket.targetbucket}/#{bucket.targetprefix}]" : " [-]" end nil end |
#run(output_file = nil) ⇒ Object
For all buckets for all dates in the configured range download the available log files. After downloading, merge the log files and convert the format from Amazon Log Format to Apache Common Log Format.
58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 |
# File 'lib/ralf.rb', line 58 def run(output_file = nil) config.output_file = output_file unless output_file.nil? raise ArgumentError.new("--output-file required") if config.output_file_missing? raise ArgumentError.new("--output-file requires ':bucket' variable") if (config.buckets.nil? || config.buckets.size > 1) and !(config.output_file_format =~ /:bucket/) puts "Processing range #{config.range}" if config.debug? # iterate over all buckets Bucket.each(config.buckets) do |bucket| print "#{bucket.name}: " if config.debug? # iterate over the full range log_files = [] config.range.each do |date| dir = config.cache_dir(:bucket => bucket.name, :date => date) log_files << Ralf.download_logs(bucket, date, dir) end log_files.flatten! # determine output file names output_log = config.output_file(:date => config.range.end, :bucket => bucket.name) merged_log = output_log + ".alf" # create directory for output file File.makedirs(File.dirname(merged_log)) # merge the log files Ralf.merge(merged_log, log_files) # convert to common log format Ralf.convert_to_common_log_format(merged_log, output_log) puts "#{log_files.size} files" if config.debug? end end |