Class: Air18n::Backend

Inherits:
Object
  • Object
show all
Includes:
I18n::Backend::Base
Defined in:
lib/air18n/backend.rb

Constant Summary collapse

T_LAST_LOADED_AT =

Define constants used as cache keys

'Air18n::translations_last_loaded_at_%s'
T_LAST_UPDATED_AT = 
'Air18n::translations_last_updated_at_%s'
T_DATA = 
'Air18n::translation_data_%s'
RACE_CONDITION_TTL =

This value allows one app instance to make a DB call while other still pull slightly stale data from the cache. The unit is seconds.

5
APPROX_MAX_SCREENSHOTS_PER_KEY =

We will only take 2 screenshots per Rails instance per key. This is to avoid having to moderate 200 screenshots for every new site-wide phrase. Because this is a per-Rails-thread cap, it is a lower bound for how many screenshot we will end up with per widely-used phrase.

3

Instance Attribute Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#default_text_change_observer ⇒ Object

  • translation_data: map of locale to key to translation

  • phrase_screenshots: map of phrase key to the routes context that we have screenshots for

  • default_text_change_observer: set this to an implementation of DefaultTextChangeObserver to be alerted to changes of default text, and be able to guard against flipflops.



19
20
21
 
# File 'lib/air18n/backend.rb', line 19

def default_text_change_observer
  @default_text_change_observer
end

#phrase_screenshots ⇒ Object

  • translation_data: map of locale to key to translation

  • phrase_screenshots: map of phrase key to the routes context that we have screenshots for

  • default_text_change_observer: set this to an implementation of DefaultTextChangeObserver to be alerted to changes of default text, and be able to guard against flipflops.



19
20
21
 
# File 'lib/air18n/backend.rb', line 19

def phrase_screenshots
  @phrase_screenshots
end

#translation_data ⇒ Object

  • translation_data: map of locale to key to translation

  • phrase_screenshots: map of phrase key to the routes context that we have screenshots for

  • default_text_change_observer: set this to an implementation of DefaultTextChangeObserver to be alerted to changes of default text, and be able to guard against flipflops.



19
20
21
 
# File 'lib/air18n/backend.rb', line 19

def translation_data
  @translation_data
end

Instance Method Details

#available_locales ⇒ Object 



39
40
41
42
43
44
45
46
47
48
 
# File 'lib/air18n/backend.rb', line 39

def available_locales
  if @translation_data
    @translation_data.inject([]) do |carry, (locale, translations)|
      carry << locale unless translations.empty?
      carry
    end
  else
    []
  end
end

#check_for_new_translations(locale) ⇒ Object 



123
124
125
126
127
128
129
130
131
132
133
134
135
136
 
# File 'lib/air18n/backend.rb', line 123

def check_for_new_translations(locale)
  # When TranslateController makes a new translation, it sets
  # translations_last_updated_at to the current time in the cache.
  # If we haven't reset the i18n backends since then, we take the opportunity to reset them.

  # No-op if locale is the default locale; its translations never change.
  if locale != I18n.default_locale
    translation_last_loaded_at, last_updated_at = check_last_timestamps(locale)
    if (translation_last_loaded_at.nil?) || (last_updated_at && last_updated_at.to_i >= translation_last_loaded_at.to_i)
      I18n.cache.write(T_LAST_LOADED_AT % locale, Time.now + RACE_CONDITION_TTL) if I18n.cache
      reload_translations([locale])
    end
  end
end

#check_last_timestamps(locale) ⇒ Object 



90
91
92
93
94
95
96
97
98
99
100
101
 
# File 'lib/air18n/backend.rb', line 90

def check_last_timestamps(locale)
  # Potential race condition here but the order of operations will at worst
  # cause an extra DB call.
  if I18n.cache
    translation_last_loaded_at = I18n.cache.read(T_LAST_LOADED_AT % locale)
    last_updated_at = I18n.cache.read(T_LAST_UPDATED_AT % locale)
  else
    translation_last_loaded_at = @translations_last_loaded_at[locale]
    last_updated_at ||= Time.now
  end
  [translation_last_loaded_at, last_updated_at]
end

#get_from_cache_or_reload(locale) ⇒ Object 



103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
 
# File 'lib/air18n/backend.rb', line 103

def get_from_cache_or_reload(locale)
  # This function will check the cache if available to see if there is data.
  # If it is available, read it and store it in a instance variable.
  # If it is not available, increment the last_loaded_at cache key by
  # a few seconds in the future. (This will prevent the majority of application
  # processes from doing a database lookup by serving them slightly stale data
  # from the cache. If multiple instances start up when the cache is empty, then multiple 
  # call to the database are unavoidable unless a locking and wait style is used.
  # Refer to the specs 'I18n cache' for more information.
  cache_results = nil
  cache_results = ChunkCache::get(I18n.cache, T_DATA % locale) if I18n.cache
  if !cache_results.nil?
    @translation_data ||= {}
    @translation_data[locale.to_sym] = cache_results
  else
    I18n.cache.write(T_LAST_LOADED_AT % locale, Time.now) if I18n.cache
    reload_translations([locale])
  end
end

#guess_translation(text, orig_locale, other_locale) ⇒ Object 



145
146
147
148
 
# File 'lib/air18n/backend.rb', line 145

def guess_translation(text, orig_locale, other_locale)
  @prim_and_proper ||= PrimAndProper.new
  @prim_and_proper.guess(text, orig_locale, other_locale)
end

#has_translation?(key, locale) ⇒ Boolean

If there is a translation of given key in given, or a less-specific fallback locale, returns the most specific locale that has a translation. Returns nil otherwise.

For “guessed” specific locales like British English, returns the specific locale only if there is a manually-written translation for that locale.

This is useful for knowing whether or not there is a translation for a key in a locale. It can also be used to determine whether a translation comes from a fallback locale or not.

Returns:

  • (Boolean) 


306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
 
# File 'lib/air18n/backend.rb', line 306

def has_translation?(key, locale)
  if key.blank? || !key.is_a?(String)
    return false
  end

  init_translations(locale)

  for fallback_locale in I18n.fallbacks_for(locale, :exclude_default => true)
    if @translation_data.include?(fallback_locale) && @translation_data[fallback_locale].include?(key)
      return fallback_locale
    end

    if fallback_locale == I18n.default_locale
      return fallback_locale
    end
  end

  return nil
end

#init_translations(locale) ⇒ Object 



79
80
81
82
83
84
 
# File 'lib/air18n/backend.rb', line 79

def init_translations(locale)
  reset_phrase_screenshots unless @phrase_screenshots
  if @translation_data.nil? || !@translation_data.include?(locale)
    get_from_cache_or_reload(locale)
  end
end

#lookup(locale, key, scope = [], options = {}) ⇒ Object

This method does the meat of Air18n functionality. 1) Finds the translations for specified key and locale, falling back to

appropriate locales if necessary based on I18n.fallbacks.

2) Queues up screenshot-taking jobs if we don’t have a screenshot for a

(key, options[:routes_context])

3) Populates the phrases database if key and options aren’t

already in there.

4) Updates the phrases database if options doesn’t match the

English text in the database. Alternatively, you can use
options[:default_is_low_priority] if you don't want this behavior.

5) Makes the :xx locale translations. 6) Asks PrimAndProper to make en-GB translations and other best-guess

translations if appropriate.

7) Chooses the correct translation form based on options if present.



172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
 
# File 'lib/air18n/backend.rb', line 172

def lookup(locale, key, scope = [], options = {})
  # Useful i18n logging for debugging translation lookup problems.
  # LoggingHelper.info "Lookup! key is #{key.inspect}, options are #{options.inspect}"
  # caller.each { |l| LoggingHelper.info "    " + l }

  # Sometimes translate() is called with an array of keys. We don't handle that case.
  if key.blank? || !key.is_a?(String)
    return nil
  end

  # Force locale to symbol to allow e.g.
  #   I18n.t('foo', :default => 'Foo', :locale => @current_user.preferred_locale)
  locale = locale.to_sym

  default = if options[:default] && options[:default].is_a?(String)
              options[:default]
            else
              nil
            end
  overrides_previous_default = !options[:default_is_low_priority]

  I18n.fallbacks_for(locale).each do |l|
    init_translations(l)
  end

  # Only create new screenshots while using default locale, and ignore keys
  # that come in hash format or with a wacky namespace
  if options[:routes_context] && locale == I18n.default_locale
    # Check to see if we have screenshot for this phrase/routes context combo
    unless @phrase_screenshots[key] && (@phrase_screenshots[key].include?(options[:routes_context]) || @phrase_screenshots[key].size >= APPROX_MAX_SCREENSHOTS_PER_KEY)
      I18n.phrase_needs_screenshot(options[:routes_context], {key => @translation_data[I18n.default_locale][key] || default || key})
      @phrase_screenshots[key] = (@phrase_screenshots[key] || []) << options[:routes_context]
    end
  end

  fallback_chain = I18n.fallbacks_for(locale)
  result = nil
  locale_fallen_back_to = nil
  for fallback_locale in fallback_chain
    if @translation_data.include?(fallback_locale)
      result = @translation_data[fallback_locale][key]
      locale_fallen_back_to = fallback_locale
      break if result
    end
  end

  if result && locale_fallen_back_to != locale
    # See if we can guess a translation instead of using the fallback
    # translation.
    guess = guess_translation(result, locale_fallen_back_to, locale)
    result = guess if guess
  end

  # If there was a default-language translation, check if it matches the default.
  # If the phrase is not in the phrases table, it will be created.
  if locale == I18n.default_locale && default != nil && default != "" && result != default
    # If it doesn't, we need to update the 'phrases' table so that
    # the 'value' column reflects the latest English default text.
    store_default = true
    if result.present?
      if @default_text_change_observer.present?
        @default_text_change_observer.default_text_changed(locale, key, result, default)
        store_default = @default_text_change_observer.allow_default_text_change?(locale, key, result, default)
      end
    end
    if store_default
      phrase = Phrase.where(:key => key).first
      if phrase.present?
        # A phrase with this key already exists; change its default text.
        if (overrides_previous_default && phrase.value != default) ||
          (!overrides_previous_default && phrase.value != default && phrase.value.blank?)
          phrase.value = default

          begin
            phrase.save
          rescue Exception => e
            # If many requests happen simultaneously for a page with a new
            # phrase, a "duplicate entry" exception will happen naturally.
            # And if phrase creation fails for some other reason, we will try
            # again to create it next time automatically.
          end
        end
      else
        # Create the phrase for the first time.
        p = Phrase.create do |p|
          p.key = key
          p.value = default
        end
      end

      @translation_data[locale][key] = default
    end
    result = default
  end

  # Helps debug translation loading and default setting.
  # LoggingHelper.info "Airbnb Backend looking up key #{key.inspect}. result is #{result.inspect}. Default is #{default.inspect}"

  # if a default is given, use it here
  if result.nil? && default
    result = default
  end

  unless options[:disable_xss_check]
    xss_detection = XssDetector::safe?(default, result, I18n.default_locale, locale_fallen_back_to)
    if !xss_detection[:safe]
      # Kill the translation if the result is unsafe.
      LoggingHelper.error "Killing unsafe translation! Default is #{default.inspect}, result is #{result.inspect}, reason for kill is #{xss_detection[:result]}"
      result = default
    end
  end

  # Strip whitespace from both sides. For fun?
  result = result.strip if result.present?

  # Handle pseudo-locales.
  result = PseudoLocales.translate(locale, key, result)

  # Handle smart counts.
  result = SmartCount.choose(result, locale_fallen_back_to, options[SmartCount::INTERPOLATION_VARIABLE_NAME])

  result
end

#reload_translations(locales) ⇒ Object 



50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
 
# File 'lib/air18n/backend.rb', line 50

def reload_translations(locales)
  if defined?(STATSD)
    STATSD.increment("Air18n.backend.reload_translations")
  end
  # We don't want to load translations of unused phrases, so we exclude
  # unused from the set of translations we initialize.
  translations_hash = PhraseTranslation.translations_for_locales(locales, :exclude_unused => true)

  # In development, people might not be running the counter server to keep
  # track of which phrases are used. In this ase, load all phrases, except UGC.
  if translations_hash.all? { |locale, translations| translations.empty? }
    translations_hash = PhraseTranslation.translations_for_locales(locales, :exclude_ugc => true)
  end

  num_translations = translations_hash.inject(0) { |carry, (locale, translations)| carry + translations.size }
  LoggingHelper.info "Loaded #{num_translations} translations for locales #{locales.inspect}."

  number_of_translations = 0
  translations_hash.each_pair do |locale, data|
    store_translations(locale, data)
    number_of_translations += data.size

    @translations_last_loaded_at ||= {}
    @translations_last_loaded_at[locale] = Time.now
  end

  LoggingHelper.info "Translation data size: #{Marshal.dump(@translation_data).size}"
end

#rescreenshot(routes_context) ⇒ Object 



138
139
140
141
142
143
 
# File 'lib/air18n/backend.rb', line 138

def rescreenshot(routes_context)
  @phrase_screenshots.each do |key, routes|
    routes.delete(routes_context)
  end
  nil
end

#reset_phrase_screenshots ⇒ Object 



86
87
88
 
# File 'lib/air18n/backend.rb', line 86

def reset_phrase_screenshots
  @phrase_screenshots = PhraseScreenshot.all_phrase_urls
end

#store_translations(locale, data, options = {}) ⇒ Object

Stores translations for a given locale.



30
31
32
33
34
35
36
37
 
# File 'lib/air18n/backend.rb', line 30

def store_translations locale, data, options = {}
  @translation_data ||= {}
  @translation_data[locale.to_sym] = data
  # We pass in nil for the expiry because we do not want the translations to
  # be expired. Rather we prefer to explicitily overwrite them when reload
  # translations is called.
  ChunkCache::set(I18n.cache, T_DATA % locale, data, nil) if I18n.cache
end