Class: LlmsTxt::BulkTransformer

Inherits:
Object
  • Object
show all
Defined in:
lib/llms_txt/bulk_transformer.rb

Overview

Bulk transforms multiple markdown files to be AI-friendly

Processes all markdown files in a directory recursively, creating LLM-friendly versions alongside the originals. Supports exclusion patterns and maintains directory structure.

Examples:

Transform all files in a directory

transformer = LlmsTxt::BulkTransformer.new('./docs',
  base_url: 'https://myproject.io',
  suffix: '.llm'
)
transformer.transform_all

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(docs_path, options = {}) ⇒ BulkTransformer

Initialize a new bulk transformer

Parameters:

  • docs_path (String) —

    path to documentation directory

  • options (Hash) (defaults to: {}) —

    transformation options

Options Hash (options):

  • :base_url (String) —

    base URL for expanding relative links

  • :convert_urls (Boolean) —

    convert HTML URLs to markdown format

  • :suffix (String) —

    suffix for transformed files (default: ‘.llm’)

  • :excludes (Array<String>) —

    glob patterns for files to exclude

  • :verbose (Boolean) —

    enable verbose output



35
36
37
38
39
40
41
 
# File 'lib/llms_txt/bulk_transformer.rb', line 35

def initialize(docs_path, options = {})
  @docs_path = docs_path
  @options = {
    suffix: '.llm',
    excludes: []
  }.merge(options)
end

Instance Attribute Details

#docs_path ⇒ String (readonly)

Returns path to documentation directory.

Returns:

  • (String) —

    path to documentation directory



21
22
23
 
# File 'lib/llms_txt/bulk_transformer.rb', line 21

def docs_path
  @docs_path
end

#options ⇒ Hash (readonly)

Returns transformation options.

Returns:

  • (Hash) —

    transformation options



24
25
26
 
# File 'lib/llms_txt/bulk_transformer.rb', line 24

def options
  @options
end

Instance Method Details

#transform_all ⇒ Array<String>

Transform all markdown files in the directory

Recursively finds all markdown files, applies transformations, and saves LLM-friendly versions with the specified suffix.

Returns:

  • (Array<String>) —

    paths of transformed files

Raises:



49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
 
# File 'lib/llms_txt/bulk_transformer.rb', line 49

def transform_all
  raise Errors::GenerationError, "Directory not found: #{docs_path}" unless File.directory?(docs_path)

  markdown_files = find_markdown_files
  transformed_files = []

  markdown_files.each do |file_path|
    next if should_exclude?(file_path)

    puts "Transforming #{file_path}..." if options[:verbose]

    transformed_content = transform_file(file_path)
    output_path = generate_output_path(file_path)

    # Ensure output directory exists
    FileUtils.mkdir_p(File.dirname(output_path))

    File.write(output_path, transformed_content)
    transformed_files << output_path

    puts "  → #{output_path}" if options[:verbose]
  end

  transformed_files
end