Class: Broadlistening::Pipeline

Inherits:
Object
  • Object
show all
Defined in:
lib/broadlistening/pipeline.rb

Overview

Orchestrates the execution of the broadlistening pipeline.

The Pipeline is responsible for:

  • Coordinating step execution order
  • Managing execution status and locking
  • Handling incremental execution (skip unchanged steps)
  • Emitting instrumentation events

Examples:

Basic usage

pipeline = Pipeline.new(api_key: "...", cluster_nums: [5, 15])
result = pipeline.run(comments, output_dir: "/path/to/output")

Force re-run all steps

pipeline.run(comments, output_dir: "/path/to/output", force: true)

Run only a specific step

pipeline.run(comments, output_dir: "/path/to/output", only: :clustering)

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(config, spec_loader: nil) ⇒ Pipeline

Returns a new instance of Pipeline.



26
27
28
29
 
# File 'lib/broadlistening/pipeline.rb', line 26

def initialize(config, spec_loader: nil)
  @config = config.is_a?(Config) ? config : Config.new(config)
  @spec_loader = spec_loader || SpecLoader.default
end

Instance Attribute Details

#config ⇒ Object (readonly)

Returns the value of attribute config.



24
25
26
 
# File 'lib/broadlistening/pipeline.rb', line 24

def config
  @config
end

#spec_loader ⇒ Object (readonly)

Returns the value of attribute spec_loader.



24
25
26
 
# File 'lib/broadlistening/pipeline.rb', line 24

def spec_loader
  @spec_loader
end

Instance Method Details

#run(comments, output_dir:, force: false, only: nil) ⇒ Hash

Run the pipeline with incremental execution support

Parameters:

  • comments (Array) —

    Array of comments to process

  • output_dir (String) —

    Directory for output files and status tracking

  • force (Boolean) (defaults to: false) —

    Force re-run all steps

  • only (Symbol, nil) (defaults to: nil) —

    Run only the specified step

Returns:

  • (Hash) —

    The result of the pipeline



38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
 
# File 'lib/broadlistening/pipeline.rb', line 38

def run(comments, output_dir:, force: false, only: nil)
  output_path = Pathname.new(output_dir)
  status = Status.new(output_path)

  raise Error, "Pipeline is locked. Another process may be running." if status.locked?

  context = Context.load_from_dir(output_path)
  context.output_dir = output_path

  # Normalize comments if not already loaded
  context.comments = normalize_comments(comments) if context.comments.empty?

  planner = Planner.new(
    config: @config,
    status: status,
    output_dir: output_path,
    spec_loader: @spec_loader
  )
  plan = planner.create_plan(force: force, only: only)

  status.start_pipeline(plan)

  execute_pipeline(plan, status, planner, context, output_path)

  status.complete_pipeline
  context.result
rescue StandardError => e
  status&.error_pipeline(e)
  raise
end