Class: Memcached

Inherits:
Object
  • Object
show all
Defined in:
lib/memcached/memcached.rb,
lib/memcached.rb,
lib/memcached/rails.rb,
lib/memcached/behaviors.rb,
lib/memcached/exceptions.rb

Overview

The Memcached client class.

Direct Known Subclasses

Rails

Defined Under Namespace

Classes: Error, Rails

Constant Summary collapse

Lib = 
Rlibmemcached
VERSION = 
File.read("#{File.dirname(__FILE__)}/../CHANGELOG")[/v([\d\.]+)\./, 1]
BEHAVIORS = 
load_constants("MEMCACHED_BEHAVIOR_")
BEHAVIOR_VALUES = 
{
  false => 0, 
  true => 1
}
HASH_VALUES = 
{}
DISTRIBUTION_VALUES = 
{}
DIRECT_VALUE_BEHAVIORS = 
[:retry_timeout, :connect_timeout, :rcv_timeout, :socket_recv_size, :poll_timeout, :socket_send_size, :server_failure_limit]
CONVERSION_FACTORS = 
{
  :rcv_timeout => 1_000_000,
  :poll_timeout => 1_000,
  :connect_timeout => 1_000
}
FLAGS = 
0x0
DEFAULTS = 
{
  :hash => :fnv1_32,
  :no_block => false,
  :distribution => :consistent_ketama,
  :ketama_weighted => true,
  :buffer_requests => false,
  :cache_lookups => true,
  :support_cas => false,
  :tcp_nodelay => false,
  :show_backtraces => false,
  :retry_timeout => 30,
  :timeout => 0.25,
  :rcv_timeout => nil,
  :poll_timeout => nil,
  :connect_timeout => 2,
  :prefix_key => nil,
  :hash_with_prefix_key => true,
  :default_ttl => 604800,
  :default_weight => 8,
  :sort_hosts => false,
  :auto_eject_hosts => true,
  :server_failure_limit => 2,
  :verify_key => true,
  :use_udp => false,
  :binary_protocol => false,
  :chunk_split_size => 1048300
}
IGNORED =

:stopdoc:

0
ERRNO_HASH = 


EXCEPTIONS =



[]


EMPTY_STRUCT =



Rlibmemcached::MemcachedSt.new




Instance Attribute Summary collapse




Class Method Summary
collapse





Instance Method Summary
collapse






Constructor Details





#initialize(servers = "localhost:11211", opts = {}) ⇒ Memcached







Create a new Memcached instance. Accepts string or array of server strings, as well an an optional configuration hash.


Memcached.new('localhost', ...) # A single server
Memcached.new(['web001:11212', 'web002:11212'], ...) # Two servers with custom ports
Memcached.new(['web001:11211:2', 'web002:11211:8'], ...) # Two servers with default ports and explicit weights



Weights only affect Ketama hashing. If you use Ketama hashing and don’t specify a weight, the client will poll each server’s stats and use its size as the weight.


Valid option parameters are:


:prefix_key



A string to prepend to every key, for namespacing. Max length is 127.


:hash



The name of a hash function to use. Possible values are: :crc, :default, :fnv1_32, :fnv1_64, :fnv1a_32, :fnv1a_64, :hsieh, :md5, and :murmur. :fnv1_32 is fast and well known, and is the default. Use :md5 for compatibility with other ketama clients.


:distribution



Either :modula, :consistent_ketama, :consistent_wheel, or :ketama. Defaults to :ketama.


:server_failure_limit



How many consecutive failures to allow before marking a host as dead. Has no effect unless :retry_timeout is also set.


:retry_timeout



How long to wait until retrying a dead server. Has no effect unless :server_failure_limit is non-zero. Defaults to 30.


:auto_eject_hosts



Whether to temporarily eject dead hosts from the pool. Defaults to true. Note that in the event of an ejection, :auto_eject_hosts will remap the entire pool unless :distribution is set to :consistent.


:cache_lookups



Whether to cache hostname lookups for the life of the instance. Defaults to true.


:support_cas



Flag CAS support in the client. Accepts true or false. Defaults to false because it imposes a slight performance penalty. Note that your server must also support CAS or you will trigger Memcached::ProtocolError exceptions.


:tcp_nodelay



Turns on the no-delay feature for connecting sockets. Accepts true or false. Performance may or may not change, depending on your system.


:no_block



Whether to use pipelining for writes. Accepts true or false.


:buffer_requests



Whether to use an internal write buffer. Accepts true or false. Calling get or closing the connection will force the buffer to flush. Note that :buffer_requests might not work well without :no_block also enabled.


:show_backtraces



Whether Memcached::NotFound and Memcached::NotStored exceptions should include backtraces. Generating backtraces is slow, so this is off by default. Turn it on to ease debugging.


:connect_timeout



How long to wait for a connection to a server. Defaults to 2 seconds. Set to 0 if you want to wait forever.


:timeout



How long to wait for a response from the server. Defaults to 0.25 seconds. Set to 0 if you want to wait forever.


:default_ttl



The ttl to use on set if no ttl is specified, in seconds. Defaults to one week. Set to 0 if you want things to never expire.


:default_weight



The weight to use if :ketama_weighted is true, but no weight is specified for a server.


:hash_with_prefix_key



Whether to include the prefix when calculating which server a key falls on. Defaults to true.


:use_udp



Use the UDP protocol to reduce connection overhead. Defaults to false.


:binary_protocol



Use the binary protocol to reduce query processing overhead. Defaults to false.


:sort_hosts



Whether to force the server list to stay sorted. This defeats consistent hashing and is rarely useful.


:verify_key



Validate keys before accepting them. Never disable this.


:chunk_split_size



Number of bytes after which items that are bigger than the 1MB memcached bucket max. size are split.




Please note that when pipelining is enabled, setter and deleter methods do not raise on errors. For example, if you try to set an invalid key with :no_block => true, it will appear to succeed. The actual setting of the key occurs after libmemcached has returned control to your program, so there is no way to backtrack and raise the exception.















84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130




# File 'lib/memcached/memcached.rb', line 84

def initialize(servers = "localhost:11211", opts = {})
  @struct = Lib::MemcachedSt.new
  Lib.memcached_create(@struct)

  # Merge option defaults and discard meaningless keys
  @options = DEFAULTS.merge(opts)
  @options.delete_if { |k,v| not DEFAULTS.keys.include? k }
  @default_ttl = options[:default_ttl]

  # Force :buffer_requests to use :no_block
  # XXX Deleting the :no_block key should also work, but libmemcached doesn't seem to set it
  # consistently
  options[:no_block] = true if options[:buffer_requests]

  # Disallow weights without ketama
  options.delete(:ketama_weighted) if options[:distribution] != :consistent_ketama

  # Legacy accessor
  options[:prefix_key] = options.delete(:namespace) if options[:namespace]

  # Disallow :sort_hosts with consistent hashing
  if options[:sort_hosts] and options[:distribution] == :consistent
    raise ArgumentError, ":sort_hosts defeats :consistent hashing"
  end

  # Read timeouts
  options[:rcv_timeout] ||= options[:timeout] || 0
  options[:poll_timeout] ||= options[:timeout] || 0

  # Set the behaviors on the struct
  set_behaviors
  set_callbacks

  # Freeze the hash
  options.freeze

  # Set the servers on the struct
  set_servers(servers)

  # Not found exceptions
  unless options[:show_backtraces]
    @not_found = NotFound.new
    @not_found.no_backtrace = true      
    @not_stored = NotStored.new
    @not_stored.no_backtrace = true      
  end
end













Instance Attribute Details






#options ⇒ Object (readonly)







:startdoc:















42
43
44




# File 'lib/memcached/memcached.rb', line 42

def options
  @options
end













Class Method Details





.load_constants(prefix, hash = {}) ⇒ Object







:stopdoc:















6
7
8
9
10
11




# File 'lib/memcached/behaviors.rb', line 6

def self.load_constants(prefix, hash = {})
  Lib.constants.grep(/^#{prefix}/).each do |const_name|
    hash[const_name[prefix.length..-1].downcase.to_sym] = Lib.const_get(const_name)
  end
  hash
end













Instance Method Details





#add(key, value, ttl = @default_ttl, marshal = true, flags = FLAGS) ⇒ Object







Add a key/value pair. Raises Memcached::NotStored if the key already exists on the server. The parameters are the same as set.















267
268
269
270
271
272
273




# File 'lib/memcached/memcached.rb', line 267

def add(key, value, ttl=@default_ttl, marshal=true, flags=FLAGS)
  value = marshal ? Marshal.dump(value) : value.to_s
  check_return_code(
    Lib.memcached_add(@struct, key, value, ttl, flags),
    key
  )
end












#append(key, value) ⇒ Object







Appends a string to a key’s value. Accepts a String key and a String value. Raises Memcached::NotFound if the key does not exist on the server.


Note that the key must be initialized to an unmarshalled string first, via set, add, or replace with marshal set to false.















310
311
312
313
314
315
316




# File 'lib/memcached/memcached.rb', line 310

def append(key, value)
  # Requires memcached 1.2.4
  check_return_code(
    Lib.memcached_append(@struct, key, value.to_s, IGNORED, IGNORED),
    key
  )
end












#big_get(key, marshal = true) ⇒ Object







EXPERIMENTAL


Gets a key’s value from the server. Accepts a single String key.


Accepts an optional marshal argument, which defaults to true, like get.








Raises:


    

  • 
(Memcached::NotFound)

    • 










406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425




# File 'lib/memcached/memcached.rb', line 406

def big_get(key, marshal=true)
  chunk_header = get(key, true)

  # A valid chunk header should respond to #chunks.
  return marshal ? chunk_header : get(key, marshal) unless chunk_header.respond_to?(:chunks)

  chunks = chunk_header.chunks
  chunk_keys = (0...chunks).map { |i| "#{key}_#{i}" }

  # Use multiget #get - this returns a hash of key/value pairs.
  chunky_hash_browns = get(chunk_keys, false)

  # If any of the chunks are missing, the entire item is considered missing.
  raise Memcached::NotFound if chunky_hash_browns.size != chunks

  # Concat all the chunked values.
  value = chunky_hash_browns.sort.map { |k, v| v }.join
  value = Marshal.load(value) if marshal
  value
end












#big_set(key, value, timeout = 0, marshal = true) ⇒ Object







EXPERIMENTAL


Sets a key/value pair that is (potentially) bigger than 1MB (i.e. the memcached limit for bucket sizes).


Accepts optional timeout and marshal arguments like set.


The value is split into chunks (each smaller than or equal in size to the chunk_split_size option) and inserted into separate buckets. The keys are of the form: #key_0, #key_1, #key_2. The bucket referred to by the given key contains a chunk “header” with a #chunks property that equals the number of chunks.


Note that values that fit within a single chunk are still “split” - the chunk header (and the single chunk) is still set.


WARNING: This method is non-atomic by nature, since we are really performing multiple sets serially.















252
253
254
255
256
257
258
259
260
261
262
263
264




# File 'lib/memcached/memcached.rb', line 252

def big_set(key, value, timeout=0, marshal=true)
  value = marshal ? Marshal.dump(value) : value.to_s

  chunk_size = options[:chunk_split_size]
  chunks = (value.size/chunk_size.to_f).ceil

  # Set the number of chunks (in a faux "header") in the bucket for the actual key.
  set(key, OpenStruct.new(:chunks => chunks), timeout, true)

  chunks.times do |chunk_num|
    set("#{key}_#{chunk_num}", value[chunk_num * chunk_size, chunk_size], timeout, false)
  end
end












#cas(key, ttl = @default_ttl, marshal = true, flags = FLAGS) ⇒ Object
Also known as:
compare_and_swap








Reads a key’s value from the server and yields it to a block. Replaces the key’s value with the result of the block as long as the key hasn’t been updated in the meantime, otherwise raises Memcached::NotStored. Accepts a String key and a block.


Also accepts an optional ttl value.


CAS stands for “compare and swap”, and avoids the need for manual key mutexing. CAS support must be enabled in Memcached.new or a Memcached::ClientError will be raised. Note that CAS may be buggy in memcached itself.








Raises:


    

  • 
(ClientError)

    • 










333
334
335
336
337
338
339
340
341
342
343
344
345




# File 'lib/memcached/memcached.rb', line 333

def cas(key, ttl=@default_ttl, marshal=true, flags=FLAGS)
  raise ClientError, "CAS not enabled for this Memcached instance" unless options[:support_cas]

  value, flags, ret = Lib.memcached_get_rvalue(@struct, key)
  check_return_code(ret, key)
  cas = @struct.result.cas

  value = Marshal.load(value) if marshal
  value = yield value
  value = Marshal.dump(value) if marshal

  check_return_code(Lib.memcached_cas(@struct, key, value, ttl, flags, cas), key)
end












#clone ⇒ Object
Also known as:
dup








Safely copy this instance. Returns a Memcached instance.


clone is useful for threading, since each thread must have its own unshared Memcached object.















169
170
171
172
173
174
175
176




# File 'lib/memcached/memcached.rb', line 169

def clone
  # FIXME Memory leak
  # memcached = super
  # struct = Lib.memcached_clone(nil, @struct)
  # memcached.instance_variable_set('@struct', struct)
  # memcached
  self.class.new(servers, options)
end












#decrement(key, offset = 1) ⇒ Object
Also known as:
decr








Decrement a key’s value. The parameters and exception behavior are the same as increment.















287
288
289
290
291




# File 'lib/memcached/memcached.rb', line 287

def decrement(key, offset=1)
  ret, value = Lib.memcached_decrement(@struct, key, offset)
  check_return_code(ret, key)
  value
end












#delete(key) ⇒ Object







Deletes a key/value pair from the server. Accepts a String key. Raises Memcached::NotFound if the key does not exist.















352
353
354
355
356
357




# File 'lib/memcached/memcached.rb', line 352

def delete(key)
  check_return_code(
    Lib.memcached_delete(@struct, key, IGNORED),
    key
  )
end












#flush ⇒ Object







Flushes all key/value pairs from all the servers.















360
361
362
363
364




# File 'lib/memcached/memcached.rb', line 360

def flush
  check_return_code(
    Lib.memcached_flush(@struct, IGNORED)
  )
end












#get(keys, marshal = true) ⇒ Object







Gets a key’s value from the server. Accepts a String key or array of String keys.


Also accepts a marshal value, which defaults to true. Set marshal to false if you want the value to be returned directly as a String. Otherwise it will be assumed to be a marshalled Ruby object and unmarshalled.


If you pass a String key, and the key does not exist on the server, Memcached::NotFound will be raised. If you pass an array of keys, memcached’s multiget mode will be used, and a hash of key/value pairs will be returned. The hash will contain only the keys that were found.


The multiget behavior is subject to change in the future; however, for multiple lookups, it is much faster than normal mode.


Note that when you rescue Memcached::NotFound exceptions, you should use a the block rescue syntax instead of the inline syntax. Block rescues are very fast, but inline rescues are very slow.















378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399




# File 'lib/memcached/memcached.rb', line 378

def get(keys, marshal=true)
  if keys.is_a? Array
    # Multi get
    ret = Lib.memcached_mget(@struct, keys);
    check_return_code(ret, keys)

    hash = {}
    keys.each do
      value, key, flags, ret = Lib.memcached_fetch_rvalue(@struct)
      break if ret == Lib::MEMCACHED_END
      check_return_code(ret, key)
      # Assign the value
      hash[key] = (marshal ? Marshal.load(value) : value)
    end
    hash
  else
    # Single get
    value, flags, ret = Lib.memcached_get_rvalue(@struct, keys)
    check_return_code(ret, keys)
    marshal ? Marshal.load(value) : value
  end    
end












#increment(key, offset = 1) ⇒ Object
Also known as:
incr








Increment a key’s value. Accepts a String key. Raises Memcached::NotFound if the key does not exist.


Also accepts an optional offset paramater, which defaults to 1. offset must be an integer.


Note that the key must be initialized to an unmarshalled integer first, via set, add, or replace with marshal set to false.















280
281
282
283
284




# File 'lib/memcached/memcached.rb', line 280

def increment(key, offset=1)
  ret, value = Lib.memcached_increment(@struct, key, offset)
  check_return_code(ret, key)
  value
end












#prepend(key, value) ⇒ Object







Prepends a string to a key’s value. The parameters and exception behavior are the same as append.















319
320
321
322
323
324
325




# File 'lib/memcached/memcached.rb', line 319

def prepend(key, value)
  # Requires memcached 1.2.4
  check_return_code(
    Lib.memcached_prepend(@struct, key, value.to_s, IGNORED, IGNORED),
    key
  )
end












#quit ⇒ Object







Disconnect from all currently connected servers















189
190
191
192




# File 'lib/memcached/memcached.rb', line 189

def quit
  Lib.memcached_quit(@struct)
  self
end












#replace(key, value, ttl = @default_ttl, marshal = true, flags = FLAGS) ⇒ Object







Replace a key/value pair. Raises Memcached::NotFound if the key does not exist on the server. The parameters are the same as set.















299
300
301
302
303
304
305




# File 'lib/memcached/memcached.rb', line 299

def replace(key, value, ttl=@default_ttl, marshal=true, flags=FLAGS)
  value = marshal ? Marshal.dump(value) : value.to_s
  check_return_code(
    Lib.memcached_replace(@struct, key, value, ttl, flags),
    key
  )
end












#reset(current_servers = nil) ⇒ Object







Reset the state of the libmemcached struct. This is useful for changing the server list at runtime.















179
180
181
182
183
184
185
186




# File 'lib/memcached/memcached.rb', line 179

def reset(current_servers = nil)
  current_servers ||= servers
  @struct = Lib::MemcachedSt.new
  Lib.memcached_create(@struct)
  set_behaviors
  set_callbacks
  set_servers(current_servers)
end












#server_by_key(key) ⇒ Object







Return the server used by a particular key.















430
431
432
433
434
435
436
437




# File 'lib/memcached/memcached.rb', line 430

def server_by_key(key)
  ret = Lib.memcached_server_by_key(@struct, key)
  if ret.is_a?(Array)
    string = inspect_server(ret.first) 
    Rlibmemcached.memcached_server_free(ret.first)
    string
  end
end












#servers ⇒ Object







Return the array of server strings used to configure this instance.















158
159
160
161
162




# File 'lib/memcached/memcached.rb', line 158

def servers
  server_structs.map do |server|
    inspect_server(server)
  end
end












#set(key, value, ttl = @default_ttl, marshal = true, flags = FLAGS) ⇒ Object







Set a key/value pair. Accepts a String key and an arbitrary Ruby object. Overwrites any existing value on the server.


Accepts an optional ttl value to specify the maximum lifetime of the key on the server. ttl can be either an integer number of seconds, or a Time elapsed time object. 0 means no ttl. Note that there is no guarantee that the key will persist as long as the ttl, but it will not persist longer.


Also accepts a marshal value, which defaults to true. Set marshal to false if you want the value to be set directly.















225
226
227
228
229
230
231
232
233
234
235




# File 'lib/memcached/memcached.rb', line 225

def set(key, value, ttl=@default_ttl, marshal=true, flags=FLAGS)
  value = marshal ? Marshal.dump(value) : value.to_s
  check_return_code(
    Lib.memcached_set(@struct, key, value, ttl, flags),
    key
  )
rescue ClientError
  # FIXME Memcached 1.2.8 occasionally rejects valid sets 
  tried = 1 and retry unless defined?(tried)
  raise
end












#set_servers(servers) ⇒ Object







Set the server list. FIXME Does not necessarily free any existing server structs.















134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155




# File 'lib/memcached/memcached.rb', line 134

def set_servers(servers)
  Array(servers).each_with_index do |server, index|
    # Socket
    if server.is_a?(String) and File.socket?(server)
      args = [@struct, server, options[:default_weight].to_i]
      check_return_code(Lib.memcached_server_add_unix_socket_with_weight(*args))
    # Network
    elsif server.is_a?(String) and server =~ /^[\w\d\.-]+(:\d{1,5}){0,2}$/
      host, port, weight = server.split(":")
      args = [@struct, host, port.to_i, (weight || options[:default_weight]).to_i]                
      if options[:use_udp] #
        check_return_code(Lib.memcached_server_add_udp_with_weight(*args))
      else
        check_return_code(Lib.memcached_server_add_with_weight(*args))
      end
    else
      raise ArgumentError, "Servers must be either in the format 'host:port[:weight]' (e.g., 'localhost:11211' or  'localhost:11211:10') for a network server, or a valid path to a Unix domain socket (e.g., /var/run/memcached)."
    end
  end
  # For inspect
  @servers = send(:servers)
end












#stats ⇒ Object







Return a Hash of statistics responses from the set of servers. Each value is an array with one entry for each server, in the same order the servers were defined.















440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474




# File 'lib/memcached/memcached.rb', line 440

def stats
  stats = Hash.new([])

  stat_struct, ret = Lib.memcached_stat(@struct, "")
  check_return_code(ret)

  keys, ret = Lib.memcached_stat_get_keys(@struct, stat_struct)
  check_return_code(ret)

  keys.each do |key|
     server_structs.size.times do |index|

       value, ret = Lib.memcached_stat_get_rvalue(
         @struct,
         Lib.memcached_select_stat_at(@struct, stat_struct, index),
         key)
       check_return_code(ret, key)

       value = case value
         when /^\d+\.\d+$/ then value.to_f
         when /^\d+$/ then value.to_i
         else value
       end

       stats[key.to_sym] += [value]
     end
  end

  Lib.memcached_stat_free(@struct, stat_struct)
  stats
rescue Memcached::SomeErrorsWereReported => _
  e = _.class.new("Error getting stats")
  e.set_backtrace(_.backtrace)
  raise e
end