Class: Pikuri::Extractors::Documents

Inherits:
Object
  • Object
show all
Defined in:
lib/pikuri/extractors/documents.rb

Overview

Document extractor for the Pikuri::Extractor registry: DOCX / ODT / XLSX / legacy XLS / PPTX / EPUB / RTF / PDF → Markdown, by piping the document bytes through pandoc (ODF, RTF, EPUB, DOCX), markitdown (OOXML spreadsheet / presentation), or pdftotext (PDF), selected per format.

Prefers a one-shot, networkless docker container (IMAGE) — bytes in via stdin, Markdown out via stdout, no volume mounts — and falls back to host pandoc / markitdown / pdftotext CLIs on the same stdin→stdout contract when docker is absent. Requiring the gem defines this class + the shared DOCUMENTS instance but registers nothing; a host opts in with DOCUMENTS.register.

The cross-cutting why — the container's security + reproducibility rationale, the PDF-arm trade-off against pikuri-pdf, the deliberately-unsupported formats (ODS/ODP, OCR/audio), and why paging re-converts each time — lives in pikuri-extractors/DESIGN.md.

Format detection

#matches? claims content by content-type (CONTENT_TYPES) or byte sniff (see #sniff). #extract re-sniffs (the registry duck type doesn't pass content_type to extract); when the sniff is blind — legacy XLS, an OLE2 container whose discriminating directory sits past the sample — bytes go to markitdown with no hint and its own detection takes over. Consequence: a local .xls (no transport content-type, sniff blind) isn't claimed at all and keeps today's binary refusal. Ordering edge: this instance sits after core's HTML, so a PDF served under a lying text/html header goes to HTML (pikuri-pdf front-inserts and wins that) — accepted, rare.

Constant Summary collapse

LOGGER = 
Pikuri.logger_for('Extractors')
IMAGE = 
"pikuri-internal-extractors:#{Pikuri::VERSION}"
DOCKER_DIR = 
File.expand_path('../../../docker', __dir__)
CONVERT_TIMEOUT = 
'300s'
AUTO = 
'auto'
PDF = 
'pdf'
CONTENT_TYPES = 
{
  'application/vnd.oasis.opendocument.text' => 'odt',
  'application/rtf' => 'rtf',
  'text/rtf' => 'rtf',
  'application/epub+zip' => 'epub',
  'application/pdf' => PDF,
  'application/vnd.openxmlformats-officedocument.wordprocessingml.document' => 'docx',
  'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet' => 'xlsx',
  'application/vnd.ms-excel' => 'xls',
  'application/vnd.openxmlformats-officedocument.presentationml.presentation' => 'pptx'
}.freeze
HOST_CONVERTERS = 
{
  'odt'  => i[pandoc],
  'rtf'  => i[pandoc],
  'epub' => i[pandoc markitdown],
  'docx' => i[pandoc markitdown],
  'xlsx' => i[markitdown],
  'xls'  => i[markitdown],
  'pptx' => i[markitdown],
  PDF    => i[pdftotext],
  AUTO   => i[markitdown]
}.freeze
ZIP_MAGIC = 
"PK\x03\x04".b
VERSION_PROBE_FLAGS = 
{ 'pdftotext' => '-v' }.freeze

Instance Method Summary collapse

Instance Method Details

#ensure_image! ⇒ void

This method returns an undefined value.

Build the converter image now if it isn't present — for host scripts that prefer paying the one-time build (pip install + apt, minutes) at boot rather than mid-conversation. Entirely optional: #extract builds lazily on first use otherwise.

Raises:

  • (Pikuri::Extractor::Error) —

    when docker is unavailable or the build fails.



188
189
190
191
192
193
 
# File 'lib/pikuri/extractors/documents.rb', line 188

def ensure_image!
  raise Pikuri::Extractor::Error, '`docker` is unavailable; cannot build the converter image' unless docker?

  image_ready!
  nil
end

#extract(io) ⇒ String

Convert the whole document behind io to one Markdown String. PDFs come back as one +"--- Page N ---"+-headed block per text-carrying page (see PDF); a fully scanned PDF extracts to the empty String — same contract as pikuri-pdf's extractor.

Raises:

  • (Pikuri::Extractor::Error) —

    when no converter is available, the conversion exits non-zero, or it times out.



139
140
141
142
143
 
# File 'lib/pikuri/extractors/documents.rb', line 139

def extract(io)
  with_converted(io) do |file, format|
    format == PDF ? pdf_page_lines(file).to_a.join("\n") : file.read
  end
end

#extract_lines(io) ⇒ Enumerator<String>

Same content as #extract, as a stream of +chomp+ed lines off the converter's stdout Tempfile. The full conversion still runs up front (fired on first consumption), but neither the document nor the Markdown ever materialises as one String. The enumerator owns the Tempfile and deletes it when iteration ends.

Raises:

  • (Pikuri::Extractor::Error) —

    as for #extract, raised on first consumption.



156
157
158
159
160
161
162
163
164
165
166
 
# File 'lib/pikuri/extractors/documents.rb', line 156

def extract_lines(io)
  Enumerator.new do |yielder|
    with_converted(io) do |file, format|
      if format == PDF
        pdf_page_lines(file).each { |line| yielder << line }
      else
        file.each_line { |line| yielder << line.chomp }
      end
    end
  end
end

#kind ⇒ Symbol 



114
115
116
 
# File 'lib/pikuri/extractors/documents.rb', line 114

def kind
  :document
end

#matches?(sample:, content_type:) ⇒ Boolean

Claim content this extractor can convert: a recognised content-type, or a positive byte sniff (see "Format detection" in the class docs).



126
127
128
 
# File 'lib/pikuri/extractors/documents.rb', line 126

def matches?(sample:, content_type:)
  CONTENT_TYPES.key?(content_type) || !sniff(sample).nil?
end

#register ⇒ Documents

Plug this extractor into Pikuri::Extractor.registry, before the terminal Passthrough entry but after core's HTML and pikuri-pdf's front-inserted PDF (both keep winning their formats). Idempotent — a second call is a no-op.



174
175
176
177
178
 
# File 'lib/pikuri/extractors/documents.rb', line 174

def register
  registry = Pikuri::Extractor.registry
  registry.insert(-2, self) unless registry.include?(self)
  self
end