Module: Pikuri::Extractors::PDF

Defined in:
lib/pikuri/extractors/pdf.rb

Overview

PDF → text extractor over the pdf-reader gem: each page emits a "--- Page N ---" marker line then its text, blocks joined by single newlines. The markers carry page provenance downstream (a cited answer, a vectordb hit's page number). A page with no extractable text contributes nothing — no marker — so a fully scanned PDF extracts to the empty String, a deliberate silent skip callers detect by length. No OCR here: pdf-reader gives clean text from digitally-generated PDFs, nothing from scans.

Why a separate gem

Split from pikuri-core to keep pdf-reader's five-gem dependency tail (Ascii85, afm, hashery, ruby-rc4, ttfunk) out of the core audit tree — five gems for one format nothing else in core uses. Hosts opt in with one PDF.register call. Being in-process, it pages lazily (PDF.extract_lines parses pages on demand) — a property a subprocess converter structurally cannot have. (The full this-gem-vs-pikuri-extractors PDF trade-off is in pikuri-extractors/DESIGN.md.)

Registration is explicit

Requiring the gem registers nothing; a host opts in with Pikuri::Extractors::PDF.register, which front-inserts into the registry (unlike pikuri-extractors' before-terminal insert) because the %PDF- magic sniff is the registry's strongest signal — it must beat +HTML+'s content-type match so a PDF under a lying header still extracts, and it never misfires on text.

Class Method Summary collapse

Class Method Details

.extract(io) ⇒ String

Render the PDF behind io as plain text, one +"--- Page N ---"+-headed block per page that carries text. Defined as extract_lines.to_a.join so the two duck-type shapes cannot drift apart.

Parameters:

  • io (IO, StringIO) —

    seekable IO positioned at the start of the PDF bytes.

Returns:

  • (String) —

    concatenated page blocks; possibly empty when the PDF carries no extractable text (scanned image, empty document).

Raises:

  • (Pikuri::Extractor::Error) —

    when pdf-reader refuses the document.



72
73
74
 
# File 'lib/pikuri/extractors/pdf.rb', line 72

def self.extract(io)
  extract_lines(io).to_a.join("\n")
end

.extract_lines(io) ⇒ Enumerator<String>

The lazy line stream behind extract: a marker line per text-carrying page, then that page's lines. pdf-reader parses a page's content stream only when Page#text is called, so a consumer that stops early (the Pikuri::Extractor.extract_paged window) never pays for the pages past its window.

The three typed pdf-reader parse failures — broken xrefs (PDF::Reader::MalformedPDFError), bad page refs (PDF::Reader::InvalidPageError), encrypted/XFA (PDF::Reader::UnsupportedFeatureError) — are document properties the LLM can react to, so they re-raise as Pikuri::Extractor::Error, from inside the enumerator: at consumption time, which for a broken xref means the first next. Other pdf-reader errors are bugs and crash loud.

Parameters:

  • io (IO, StringIO) —

    seekable IO positioned at the start of the PDF bytes; must remain open while the enumerator is consumed.

Returns:

  • (Enumerator<String>) —

    chomped lines, produced page-by-page.

Raises:

  • (Pikuri::Extractor::Error) —

    when pdf-reader refuses the document (raised on consumption).



99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
 
# File 'lib/pikuri/extractors/pdf.rb', line 99

def self.extract_lines(io)
  Enumerator.new do |lines|
    ::PDF::Reader.new(io).pages.each_with_index do |page, idx|
      text = page.text.strip
      next if text.empty?

      lines << "--- Page #{idx + 1} ---"
      text.split("\n").each { |line| lines << line }
    end
  rescue ::PDF::Reader::MalformedPDFError,
         ::PDF::Reader::InvalidPageError,
         ::PDF::Reader::UnsupportedFeatureError => e
    raise Pikuri::Extractor::Error,
          "PDF rendering failed: #{e.class.name.split('::').last}: #{e.message}"
  end
end

.kind ⇒ Symbol

Returns Pikuri::Extractor::Page#kind tag.

Returns:

  • (Symbol) —

    Pikuri::Extractor::Page#kind tag.



48
49
50
 
# File 'lib/pikuri/extractors/pdf.rb', line 48

def self.kind
  :pdf
end

.matches?(sample:, content_type:) ⇒ Boolean

Parameters:

  • sample (String) —

    leading bytes of the content.

  • content_type (String, nil) —

    normalized content-type, when the transport supplies one.

Returns:

  • (Boolean) 


56
57
58
 
# File 'lib/pikuri/extractors/pdf.rb', line 56

def self.matches?(sample:, content_type:)
  content_type == 'application/pdf' || sample.start_with?(FileType::PDF_MAGIC)
end

.register ⇒ Module

Insert this extractor at the front of Pikuri::Extractor.registry (see "Registration is explicit" above for why the front). Idempotent.

Returns:

  • (Module) —

    self, for one-line wiring in host scripts.



41
42
43
44
45
 
# File 'lib/pikuri/extractors/pdf.rb', line 41

def self.register
  registry = Pikuri::Extractor.registry
  registry.unshift(self) unless registry.include?(self)
  self
end