Class: Nukitori::HtmlPreprocessor

Inherits:
Object
  • Object
show all
Defined in:
lib/nukitori/html_preprocessor.rb

Overview

Preprocesses HTML to reduce token size for LLM

Class Method Summary collapse

Class Method Details

.process(html) ⇒ String

Returns Cleaned HTML.

Parameters:

  • html (String, Nokogiri::HTML::Document) —

    HTML string or Nokogiri document

Returns:

  • (String) —

    Cleaned HTML



8
9
10
11
12
13
14
15
16
17
18
19
 
# File 'lib/nukitori/html_preprocessor.rb', line 8

def self.process(html)
  doc = html.is_a?(Nokogiri::HTML::Document) ? html.dup : Nokogiri::HTML(html)

  # Remove non-content elements
  doc.css('script, style, noscript, svg, path, meta, link, head').remove

  # Remove style attributes
  doc.css('*').each { |node| node.remove_attribute('style') }

  # Collapse whitespace
  doc.to_html.gsub(/\s+/, ' ')
end