Class: Webclient::Response

Inherits:
Object
  • Object
show all
Defined in:
lib/webclient/webclient_response.rb,
lib/webclient/webclient_response-text.rb,
lib/webclient/webclient_response-status.rb

Overview

check - rename to HttpResponse? and use HttpErrorResponse or such - why? why not?

Defined Under Namespace

Classes: Headers, Status

Constant Summary collapse

HTML_CHARSET_RE =

regex to capture the charset from both HTML5 and HTML4 meta tags -- the modern HTML5 tag, or -- the older HTML4 <meta http-equiv="Content-Type" ...> tag <meta http-equiv="Content-Type" content="text/html; charset=windows-1252" support multi-line (m) - why? why not???

note - add the n (NOENCODING) flag The n flag forces Ruby to compile and process the regex as a raw sequence of bytes (ASCII-8BIT). This allows it to safely match against US-ASCII, ASCII-8BIT, or UTF-8 strings without throwing compatibility errors charset note - charset class was [^"' >]+ changed to more strict/simple [a-z0-9-_]+ check if other "weirdo" encoding name exist?

%r{ <meta [^>]+
    charset [ ]* = [ ]*
            ["']? (?<charset> [a-z0-9_-]+)
}ixn
HTML_CHARSET_ALIASES = 
{
  'utf8'          => 'UTF-8',
  'utfs-8'        => 'UTF-8',     ## typo in rsssf (fix otherwise or here??)

  'cp1252'        => 'Windows-1252',
  'latin1'        => 'ISO-8859-1',
  'ascii'         => 'US-ASCII',
  'binary'        => 'ASCII-8BIT'
}

Instance Method Summary collapse

Constructor Details

#initialize(response) ⇒ Response

Returns a new instance of Response.



9
10
11
 
# File 'lib/webclient/webclient_response.rb', line 9

def initialize( response )
  @response = response
end

Instance Method Details

#_decode_text(encoding: _encoding_user) ⇒ Object

todo/check: rename encoding to html/http-like charset - why? why not? or keep encoding as used for ruby's strings



64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
 
# File 'lib/webclient/webclient_response-text.rb', line 64

def _decode_text( encoding: _encoding_user )

  if encoding.nil?
    encoding = 'UTF-8'     ### use UTF-8 as fallback (default encoding)

    encoding_source = 'fallback'
  else
    encoding_source = 'user'
  end


  # note: Net::HTTP will NOT set encoding UTF-8 etc.

  # will be set to ASCII-8BIT == BINARY == Encoding Unknown; Raw Bytes Here

  ##

  ## todo/assert

  ##   make sure encoding is  ASCII-8BIT == BINARY !!!

  ##

  ## note !!!! - make sure text is always a copy (thus, use dup(licate)!!)

  ##              NOT a reference to @response.body.to_s

  ##                otherwise force_encoding

  ##                    will change the encoding "upstream"

  text = @response.body.b.dup



  ##    note  - record 7bit ascii code range (ENC_CODERANGE_7BIT) check (on "raw" blob before changing encoding)

  ##     see https://shopify.engineering/code-ranges-ruby-strings

  ##

  ##  String#ascii_only?

  ##    returns true if every character in the string has a byte value between 0 and 127.

  ##

  ### ENC_CODERANGE_7BIT:

  ##   Every single byte in the string is between 0 and 127.

  ##  If this flag is already set, ascii_only?

  ##  immediately returns true.

  ##

  ##  ENC_CODERANGE_VALID:

  ##    The string contains valid characters for its encoding (like UTF-8),

  ##   but at least one character is outside the 0–127 range

  ##   (e.g., it contains a 128+ byte).

  ## If this flag is set, it immediately returns false.

  ##

  ##

  ##              check before optional bom-removal

  @_text_ascii_only    = text.ascii_only?


  ###

  ##  note

  ## auto-check for unicode byte-order marks (BOM)s!!

  ##   and auto-strip bom!!

  ##

  ##  common BOMs to check

  ##   UTF-8: EF BB BF

  ##   UTF-16 BE: FE FF

  ##   UTF-16 LE: FF FE

  ##   UTF-32 BE: 00 00 FE FF

  ##   UTF-32 LE: FF FE 00 00


  encoding_bom =
  if text.start_with?("\x00\x00\xFE\xFF".b)
     text = text.byteslice(4..)
     'UTF-32BE'
  elsif text.start_with?("\xFF\xFE\x00\x00".b)
     text = text.byteslice(4..)
     'UTF-32LE'
  elsif text.start_with?("\xFE\xFF".b)
     text = text.byteslice(2..)
     'UTF-16BE'
  elsif text.start_with?("\xFF\xFE".b)
     text = text.byteslice(2..)
     'UTF-16LE'
  elsif text.start_with?("\xEF\xBB\xBF".b)
     text = text.byteslice(3..)
     'UTF-8'
  else
     nil   # no bom found

  end




  if encoding_bom
    puts "  [debug] auto-removing unicode >#{encoding_bom}< encoding bom (magic bytes) in response.text"

    if encoding_bom.downcase != encoding.downcase
      puts "  [debug] !!! WARN - auto-fixing response.text encoding; >#{encoding}< overridden by >#{encoding_bom}< unicode encoding bom"
      encoding = encoding_bom
    end

    encoding_source = 'bom'
  else
     ##   fix-fix-fix   check/add http content type check with charset!!

     ##

     ##  check if html content type

     ##     text/html

     ##     application/xhtml+xml

     ##          && check html meta charset in page in first 1028 bytes

     ##

     ##  note - content_type might return nil (guard with to_s!!)

     ##   maybe use/make into  html? helper like gif? pdf? or such


      if content_type.to_s.match?( %r{text/html}i ) ||
         content_type.to_s.match?( %r{application/xhtml}i )

         if (m = HTML_CHARSET_RE.match( text[0, 1028] ))
             encoding_html =  m[:charset]
             ## note - normalize encoding_html

             ##    plus fix known type errors!!!

             encoding_html = HTML_CHARSET_ALIASES[ encoding_html.downcase ] || encoding_html

             ## fix-fix-fix

             ##  validate with ruby's builtin in encoding registry!!!

             # 3. Validate against Ruby's internal encoding registry

             ## begin

             ##   Encoding.find(standard_name).name

             ## rescue ArgumentError

             ##     unknown encoding!!!

             ## end


             if encoding_html.downcase != encoding.downcase
                ## note  - change WARN to INFO

                puts "  [debug] !!! WARN - overwrite response.text encoding; >#{encoding}< overridden by >#{encoding_html}< html meta charset"
                encoding = encoding_html
             end

             encoding_source = 'html'
        end
      end
  end


##

##    note - allow "hack-y" access to "upstream" encoding used before conversion to utf-8

##             e.g. use   response._text_encoding or

##                        response._text_encoding_source  (e.g. bom|html|http|user)

      @_text_encoding        = encoding
      @_text_encoding_source = encoding_source

###

###    if encoding.start_with? utf

##           or has encoding_bom

###       do nothing

##      otherwise

##           tally all 8-bit ascii chars (above > 127)


  if encoding_bom || encoding.downcase.start_with?( 'utf' )
       @_text_8bit = nil
  else
     ## get/track 8-bit bytes (1xxxxxxx), that is, > 127 (128-255)

     bytes  = text.bytes.select { |byte| byte > 127 }

     if bytes.empty?
       @_text_8bit = nil
     else
       @_text_8bit = "#{bytes.count} - "
       ##  bytes.tally

       ## e.g.  {195=>1, 169=>1, 240=>1, 159=>1, 152=>1, 138=>1}

       ##    note - use sort (turns in array e.g. [[138,1],...])

       @_text_8bit += bytes.tally.sort.map {|ord,count| "#{ord}=>#{count}"}.join(', ')
     end
  end






  if encoding.downcase == 'utf-8'
     text = text.force_encoding( Encoding::UTF_8 )

     ## track/check code range if valid/broken

     @_text_encoding_valid = text.valid_encoding?
  else
    ## [debug] GET=http://www.football-data.co.uk/mmz4281/0405/SC0.csv

    ##    Encoding::UndefinedConversionError: "\xA0" from ASCII-8BIT to UTF-8

    ##     note:  0xA0 (160) is NBSP (non-breaking space) in Windows-1252


   ## note: assume windows encoding (for football-data.uk)

   ##   use "Windows-1252" for input and convert to utf-8

   ##

   ##    see https://www.justinweiss.com/articles/3-steps-to-fix-encoding-problems-in-ruby/

   ##    see https://en.wikipedia.org/wiki/Windows-1252

   ## txt = txt.force_encoding( 'Windows-1252' )

   ## txt = txt.encode( 'UTF-8' )

   ##   Encoding::UTF_8 => 'UTF-8'

      puts "  [debug] try converting response.text encoding from >#{encoding}< to >UTF-8<"
      text = text.force_encoding( encoding )

      ## track/check code range if valid/broken

      ##   note - check BEFORE conversion to utf-8 - why? why not?

      @_text_encoding_valid = text.valid_encoding?

      ## note be more tolerant when converting - use replace for now - why? why not?

      ##   maybe add a strict (no replace) version later

         text = text.encode(
                  Encoding::UTF_8,
                     invalid: :replace,
                     undef:   :replace,
                     replace: "�"
                  )

           errors = text.scan( "�" )
           if errors.size > 0
              puts "  [debug] !!! WARN - #{errors.size} invalid/undef character encoding error(s) replaced w/ �"
              @_text_utf8_replace = errors.size
          end
  end


 # Normalize unicode (utf-8) string to Composed (NFC)

 #    NFC (Normalization Form Canonical Composition)


=begin
  use nfkc ??
  or delegate to userland??

Pro-Tip: Watch out for Ligatures and Compatibility Issues
While NFC handles standard accents beautifully,
you might occasionally want NFKC (Normalization Form Compatibility Composition)
instead.
 pages sometimes contain legacy typographical quirks like:
 Ligatures: The characters fi or fl typed as a single glyph.
 Roman Numerals / Fractions: Characters like Ⅳ or ½.

 If you use standard NFC, those symbols remain as complex single characters.
 If you use NFKC, Ruby will break them down into standard,
 easily searchable text (fi becomes fi, Ⅳ becomes IV, and ½ becomes 1/2).
=end


 ###

 ##  todo/check -  add nfc: true|false

 ##                   to text() as option (if unicode - utf8)  - why? why not?

 ##                or text_unicode( nfc: true|false )


 ##  comment out for now - get

 ##    unicode_normalize/normalize.rb:126:in `gsub': invalid byte sequence in UTF-8

  text = text.unicode_normalize(:nfc)

  text
end

#_encoding_user ⇒ Object 



48
 
# File 'lib/webclient/webclient_response.rb', line 48

def _encoding_user()  defined?( @_encoding_user )  ?  @_encoding_user : nil;  end

#_encoding_user=(value) ⇒ Object

note - add a writeable encoding_user attribute on default (if not set by user) returns nil



47
 
# File 'lib/webclient/webclient_response.rb', line 47

def _encoding_user=( value ) @_encoding_user = value; end

#_text_8bit ⇒ Object 



48
 
# File 'lib/webclient/webclient_response-text.rb', line 48

def _text_8bit()            defined?( @_text_8bit )            ?  @_text_8bit         : nil;  end

#_text_ascii_only ⇒ Object 



47
 
# File 'lib/webclient/webclient_response-text.rb', line 47

def _text_ascii_only()      defined?( @_text_ascii_only )      ?  @_text_ascii_only   : nil;  end

#_text_encoding ⇒ Object

internal helper to get "upstream" encoding note - unicode bom will override user encoding !!! -- use _text_encoding_upstream or such - why? why not? change/rename _8bit to chars_8bit - why? why not?



43
 
# File 'lib/webclient/webclient_response-text.rb', line 43

def _text_encoding()        defined?( @_text_encoding )        ?  @_text_encoding : nil;  end

#_text_encoding_source ⇒ Object 



44
 
# File 'lib/webclient/webclient_response-text.rb', line 44

def _text_encoding_source() defined?( @_text_encoding_source ) ?  @_text_encoding_source : nil; end

#_text_encoding_valid ⇒ Object 



46
 
# File 'lib/webclient/webclient_response-text.rb', line 46

def _text_encoding_valid()  defined?( @_text_encoding_valid)   ?  @_text_encoding_valid : nil;  end

#_text_utf8_replace ⇒ Object 



49
 
# File 'lib/webclient/webclient_response-text.rb', line 49

def _text_utf8_replace()    defined?( @_text_utf8_replace )    ?  @_text_utf8_replace : nil; end

#body ⇒ Object Also known as: blob

always use t raw binary data and always use @response.body.b or body.b (binary ascii-7bit) string/buffer here !!!!



69
 
# File 'lib/webclient/webclient_response.rb', line 69

def body() @response.body.b; end

#content_length ⇒ Object 



97
 
# File 'lib/webclient/webclient_response.rb', line 97

def content_length()  @response.content_length; end

#content_type ⇒ Object

add some predefined/built-in header(s) convenience shortcuts check: change to headers or such - why? why not?



96
 
# File 'lib/webclient/webclient_response.rb', line 96

def content_type()    @response.content_type; end

#headers ⇒ Object

nested class Response::Headers



87
88
89
 
# File 'lib/webclient/webclient_response.rb', line 87

def headers
   @headers ||= Headers.new( @response )
end

#image_gif? ⇒ Boolean Also known as: gif?

Returns:

  • (Boolean) 


103
 
# File 'lib/webclient/webclient_response.rb', line 103

def image_gif?()   content_type.to_s.match?( %r{image/gif}i );    end

#image_jpg? ⇒ Boolean Also known as: image_jpeg?, jpeg?, jpg?

note - content_type might return nil, thus, use to_s (gets converted to "")

Returns:

  • (Boolean) 


101
 
# File 'lib/webclient/webclient_response.rb', line 101

def image_jpg?()   content_type.to_s.match?( %r{image/jpeg}i );   end

#image_png? ⇒ Boolean Also known as: png?

Returns:

  • (Boolean) 


102
 
# File 'lib/webclient/webclient_response.rb', line 102

def image_png?()   content_type.to_s.match?( %r{image/png}i );    end

#json ⇒ Object

convenience helper; returns parsed json data; note: always assume utf-8 (text) encoding cache returned (parsed) json value - why? why not? add :symbolize_keys option - why? why not?



58
59
60
 
# File 'lib/webclient/webclient_response.rb', line 58

def json
    @json ||= JSON.parse( text )
end

#nok? ⇒ Boolean

Returns:

  • (Boolean) 


29
 
# File 'lib/webclient/webclient_response.rb', line 29

def nok?()         status.nok?; end

#ok? ⇒ Boolean

Returns:

  • (Boolean) 


28
 
# File 'lib/webclient/webclient_response.rb', line 28

def ok?()          status.ok?; end

#raw ⇒ Object

todo - find a better name for underlying object - instead of raw use ?? note - raw used by python requests too use for streaming and such - why? why not?



16
 
# File 'lib/webclient/webclient_response.rb', line 16

def raw() @response; end

#status ⇒ Object

response status methods



22
23
24
 
# File 'lib/webclient/webclient_response.rb', line 22

def status
  @status ||= Status.new( @response.code, message: @response.message )
end

#status_code ⇒ Object

add "flat" shortcuts - keep - why? why not?



27
 
# File 'lib/webclient/webclient_response.rb', line 27

def status_code()  status.to_i; end

#text(encoding: _encoding_user) ⇒ Object

cache (returned) decoded text - why? why not?



51
52
53
 
# File 'lib/webclient/webclient_response.rb', line 51

def text( encoding: _encoding_user )
    @text ||= _decode_text( encoding: encoding )
end

#version ⇒ Object Also known as: http_version

keep http_version on Response - why? why not? only really 1.0 and 1.1 check if value is a string?



37
 
# File 'lib/webclient/webclient_response.rb', line 37

def version() @response.http_version; end