Module: Chat

Extended by:
Annotation
Defined in:
lib/scout/llm/chat/parse.rb,
lib/scout/llm/chat/prompt.rb,
lib/scout/llm/chat/process.rb,
lib/scout/llm/chat/agent_meta.rb,
lib/scout/llm/chat/annotation.rb,
lib/scout/llm/chat/provenance.rb,
lib/scout/llm/chat/tool_calls.rb,
lib/scout/llm/chat/process/meta.rb,
lib/scout/llm/chat/process/clear.rb,
lib/scout/llm/chat/process/files.rb,
lib/scout/llm/chat/process/tools.rb,
lib/scout/llm/chat/process/options.rb,
lib/scout/llm/chat/prompt/shorten_tools.rb,
lib/scout/llm/chat/prompt/shorten_tools_epoch.rb

Constant Summary collapse

REGISTERED_STRATEGIES = 
{}
DEFAULT_CONTEXT_STRATEGY = 
%w(shorten_tools_epoch)
DEFAULT_SHORT_STRING_LENGTH = 
200
DEFAULT_SHORT_JSON_LENGTH = 
2000
PROVENANCE_RELATIONS = 
%i[job dependency log result agent_job].freeze
DIRECT_LOG_CHAT_GLOBS =

ScoutCoder: DUAL-LAYOUT note. Two on-disk layouts must be globbed for a job's own chat logs and for a saved chat's .files sidecar:

new     <x>.files/<name>.chat                        (agent.chat, worker.chat, ...)
new     <x>.files/<name>.society/**/*.chat           (society tree; nested
      societies keep the plain 'society' basename deeper down)
legacy  <x>.files/log/**/*.chat                      (written by older
      scout-ai; read-only compatibility, never migrated)

Exactly these three families may be swept: resets/, other *.files subdirectories and second-order .files trees stay invisible. Files matched by more than one pattern are deduplicated and results are sorted so the traversal order is deterministic.

['*.chat', '*.society/**/*.chat', 'log/**/*.chat'].freeze
TOKEN_KEYS =

Canonical short keys for per-inference token fields written into meta messages. Every key has implicit _s (session) and _c (chat) cumulative variants computed by update_meta.

pt  - prompt / input tokens
ct  - completion / output tokens
tt  - total tokens
cct - cached (cache-hit) input tokens
cwt - cache-write input tokens
rt  - reasoning tokens
%w[pt ct tt cct cwt rt].freeze
CUMULATIVE_KEYS =

Keys that carry cumulative totals across chat requests. Used by Chat.meta to restore the last checkpoint.

TOKEN_KEYS.map { |k| "#{k}_c" }.freeze
USAGE_FIELD_MAP =

Map known provider field names to the canonical short keys. Each entry is [field_path, short_key] where field_path is an array of keys suitable for IndiferentHash.dig.

{
  # prompt / input
  %w[prompt_tokens]                  => 'pt',
  %w[input_tokens]                   => 'pt',
  # completion / output
  %w[completion_tokens]              => 'ct',
  %w[output_tokens]                  => 'ct',
  # total
  %w[total_tokens]                   => 'tt',
  # cache-hit (GLM prompt_tokens_details, OpenAI input_tokens_details)
  %w[prompt_tokens_details cached_tokens]   => 'cct',
  %w[input_tokens_details cached_tokens]    => 'cct',
  # cache-write (OpenAI input_tokens_details)
  %w[input_tokens_details cache_write_tokens] => 'cwt',
  # reasoning (GLM completion_tokens_details, OpenAI output_tokens_details)
  %w[completion_tokens_details reasoning_tokens] => 'rt',
  %w[output_tokens_details reasoning_tokens]    => 'rt',
  # Anthropic flat fields
  %w[cache_read_input_tokens]        => 'cct',
  %w[cache_creation_input_tokens]    => 'cwt',
}.freeze
DEFAULT_FULL_TOOL_CALLS =

--- shorten_tools strategy configuration ---

0
DEFAULT_FULL_TOOL_OUTPUTS = 
10
DEFAULT_MAX_TOOL_CALLS = 
40
DEFAULT_MAX_TOOL_OUTPUTS = 
DEFAULT_MAX_TOOL_CALLS
DEFAULT_MAX_TOOL_CHARS = 
100_000
DEFAULT_EPOCH_TOOL_CALL_THRESHOLD =

Total tool-call count at or below which no compaction happens.

50
DEFAULT_EPOCH_FULL_TOOL_CALLS =

Number of most-recent tool calls to keep at full fidelity.

20
DEFAULT_EPOCH_COMPACTED_TOOL_CALLS =

Number of tool calls (before the full window) to compact (truncate).

80
DEFAULT_EPOCH_SIZE =

How many new tool calls are allowed before the compaction boundary advances. Within a single epoch window the compacted prefix is frozen, maximising KV-cache / prompt-cache hits for consecutive inferences.

20

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.agent_meta_error_message(reason, chat_path: nil, output_address: nil, call_id: nil, tool_name: nil, reference: nil) ⇒ Object

Render the human readable message for a malformed or unusable agent_meta receipt. This is a plain module function, not a dedicated error class: the raised exception is ScoutException and the structured facts (chat path, output address, call id, tool name, raw entry, reference) always travel in the on_error callback reference Hash built by Chat.agent_meta_error_reference.



7
8
9
10
11
12
13
14
15
16
17
 
# File 'lib/scout/llm/chat/agent_meta.rb', line 7

def self.agent_meta_error_message(reason, chat_path: nil, output_address: nil, call_id: nil, tool_name: nil, reference: nil)
  message = +"agent_meta receipt #{reason}"
  message << " in chat #{chat_path}" if chat_path
  message << " at output #{Array(output_address).inspect}" unless output_address.nil?
  details = []
  details << "call #{call_id}" if call_id
  details << "tool #{tool_name}" if tool_name
  message << " (#{details * ', '})" unless details.empty?
  message << " reference #{reference}" if reference
  message
end

.agent_meta_error_reference(reason:, source:, output_address: nil, evidence_address: nil, call_id: nil, tool_name: nil, agent_meta_index: nil, raw_entry: nil, reference: nil) ⇒ Object

Uniform reference Hash handed to on_error for agent_meta provenance problems. Keys are always present; unavailable facts are nil. It retains the original reference/raw entry, the reason, the output and evidence addresses, the receipt entry index, the call id and tool name when known, and the chat path.



249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
 
# File 'lib/scout/llm/chat/agent_meta.rb', line 249

def self.agent_meta_error_reference(reason:, source:, output_address: nil, evidence_address: nil,
                                    call_id: nil, tool_name: nil, agent_meta_index: nil,
                                    raw_entry: nil, reference: nil)
  {
    reason: reason,
    source: source ? source.to_s : nil,
    output_address: output_address,
    evidence_address: evidence_address,
    call_id: call_id,
    tool_name: tool_name,
    agent_meta_index: agent_meta_index,
    raw_entry: raw_entry,
    reference: reference
  }
end

.agent_meta_evidence(chat, source: nil, warnings: nil) ⇒ Object

One Hash per valid agent_meta entry found across the paired tool outputs of the chat. Pairing is delegated to Chat.tool_calls; raw text is never scanned with regular expressions.

Record shape:

{
origin: :agent_meta,
meta: <IndiferentHash of meta fields>,
source: <String path or nil>,
output_address: <as returned by Chat.tool_calls>,
evidence_address: <output_address + [:agent_meta, index]> (legacy
                  entries; current-format entries use [:meta, index]),
call_id: ...,
tool_name: ...,
agent_meta_index: ...,
raw_message: {role: "meta", content: "..."} (legacy; nil for
             current-format entries, which are born deserialized)
}

Malformed data is skipped. When the caller supplies an Array through the warnings keyword, each malformed item appends one warning Hash:

{
origin: :agent_meta,
reason: :not_an_array | :not_a_hash | :invalid_role |
        :invalid_content | :unparseable_meta | :empty_meta,
source:, output_address:, evidence_address:,
call_id:, tool_name:,
agent_meta_index: (nil when the whole receipt value is malformed),
evidence_address: (nil when the index is unknown),
raw_entry: <the malformed item as found>
}

:empty_meta applies to current-format entries (deserialized field Hash with no fields). Malformed receipt data is never silently reinterpreted as provenance.



86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
 
# File 'lib/scout/llm/chat/agent_meta.rb', line 86

def self.agent_meta_evidence(chat, source: nil, warnings: nil)
  tool_calls(chat, source: source).flat_map do |call|
    output_info = call[:output_info]
    next [] unless Hash === output_info

    # Key presence, not truthiness: an explicit `meta: false` or
    # `agent_meta: nil` is a present-but-malformed receipt and must warn,
    # while an absent key simply has no receipt at all.
    has_meta = output_info.key?('meta') || output_info.key?(:meta)
    has_agent_meta = output_info.key?('agent_meta') || output_info.key?(:agent_meta)
    next [] unless has_meta || has_agent_meta

    # Current writer emits exactly one of the two; when both are present
    # the current `meta` key wins and the legacy one is ignored.
    format = has_meta ? :meta : :agent_meta
    agent_meta = output_info['meta']
    agent_meta = output_info[:meta] if agent_meta.nil? && output_info.key?(:meta)
    agent_meta = output_info['agent_meta'] if format == :agent_meta && agent_meta.nil?
    agent_meta = output_info[:agent_meta] if format == :agent_meta && agent_meta.nil?

    add_warning = lambda do |reason, index, raw_entry|
      return unless Array === warnings
      output_address = call[:output_address]
      evidence_address = if index.nil?
                           nil
                         elsif Array === output_address
                           output_address + [format, index]
                         else
                           [output_address, format, index]
                         end
      warnings << {
        origin: :agent_meta,
        reason: reason,
        source: source ? source.to_s : nil,
        output_address: output_address,
        evidence_address: evidence_address,
        call_id: call[:call_id],
        tool_name: call[:name],
        agent_meta_index: index,
        raw_entry: raw_entry
      }
      nil
    end

    unless Array === agent_meta
      add_warning.call(:not_an_array, nil, agent_meta)
      next []
    end

    agent_meta.each_with_index.collect do |entry, index|
      unless Hash === entry
        add_warning.call(:not_a_hash, index, entry)
        next nil
      end

      meta, raw_message =
        if format == :meta
          # Current format: entries are already-deserialized field Hashes.
          # No role/content wrapper and no re-parse; an entry with no
          # fields carries no evidence and is warned about.
          fields = IndiferentHash.setup(entry.dup)
          [fields, nil]
        else
          # Legacy format: serialized {role: 'meta', content: 'k=v ...'}
          # meta messages.
          role = entry['role'] || entry[:role]
          content = entry['content'] || entry[:content]

          if role.to_s != 'meta'
            add_warning.call(:invalid_role, index, entry)
            next nil
          end

          unless String === content
            add_warning.call(:invalid_content, index, entry)
            next nil
          end

          [parse_meta(content), { role: role, content: content }]
        end

      if meta.empty?
        reason = format == :meta ? :empty_meta : :unparseable_meta
        add_warning.call(reason, index, entry)
        next nil
      end

      output_address = call[:output_address]
      # Without a source, Chat.tool_calls reports a bare message index; the
      # receipt address keeps the key-mirroring suffix so it stays truthful
      # to the persisted envelope ([:meta, index] for current data,
      # [:agent_meta, index] for legacy data).
      evidence_address = if Array === output_address
                           output_address + [format, index]
                         else
                           [output_address, format, index]
                         end
      {
        origin: :agent_meta,
        meta: meta,
        source: source ? source.to_s : nil,
        output_address: output_address,
        evidence_address: evidence_address,
        call_id: call[:call_id],
        tool_name: call[:name],
        agent_meta_index: index,
        raw_message: raw_message
      }
    end.compact
  end
end

.agent_meta_job_references(chat, source: nil, warnings: nil) ⇒ Object

agent_meta entries that carry a producer job reference (job=). Each returned record is the full evidence record with the reference added at the top level as job:, so both the reference value and its evidence location stay together. Job projection metas are producer references, not token events.



238
239
240
241
242
 
# File 'lib/scout/llm/chat/agent_meta.rb', line 238

def self.agent_meta_job_references(chat, source: nil, warnings: nil)
  agent_meta_evidence(chat, source: source, warnings: warnings)
    .select { |record| record[:meta][:job] }
    .collect { |record| record.merge(job: record[:meta][:job]) }
end

.allow_job(job) ⇒ Object 



17
18
19
20
21
 
# File 'lib/scout/llm/chat/process/tools.rb', line 17

def self.allow_job(job)
  allow_path(job.path)
  allow_path(job.info_file)
  allow_path(job.files_dir)
end

.allow_path(path) ⇒ Object 



3
4
5
6
7
8
 
# File 'lib/scout/llm/chat/process/tools.rb', line 3

def self.allow_path(path)
  Thread.current['allowed_paths'] ||= []
  return if Thread.current['allowed_paths'].include?(path)
  Log.medium "Allow #{path}"
  Thread.current['allowed_paths'] << path
end

.allow_read_job(job) ⇒ Object 



23
24
25
26
27
 
# File 'lib/scout/llm/chat/process/tools.rb', line 23

def self.allow_read_job(job)
  allow_read_path(job.path)
  allow_read_path(job.info_file)
  allow_read_path(job.files_dir)
end

.allow_read_path(path) ⇒ Object 



10
11
12
13
14
15
 
# File 'lib/scout/llm/chat/process/tools.rb', line 10

def self.allow_read_path(path)
  Thread.current['allowed_read_paths'] ||= []
  return if Thread.current['allowed_read_paths'].include?(path)
  Log.medium "Allow read #{path}"
  Thread.current['allowed_read_paths'] << path
end

.associations(messages, kb = nil) ⇒ Object 



234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
 
# File 'lib/scout/llm/chat/process/tools.rb', line 234

def self.associations(messages, kb = nil)
  tool_definitions = {}
  new = messages.collect do |message|
    role = message[:role]
    if role == 'association'
      name, path, *_ = content_tokens(message)
      kb ||= KnowledgeBase.new Scout.var.Agent.Chat.knowledge_base

      options = IndiferentHash.parse_options(message[:content])
      options[:fields] = options[:fields].split(/,\s*/) if String === options[:fields]
      options[:type] = options[:type].to_sym if String === options[:type]

      kb.register name, Path.setup(path), options

      tool_definitions.merge!(LLM.knowledge_base_tool_definition( kb, [name]))
      next
    elsif role == 'clear_associations'
      tool_definitions = {}
    else
      message
    end
  end.compact.flatten
  messages.replace new
  tool_definitions
end

.build_protected_positions(messages, total_tool_outputs) ⇒ Object

Walk the message array forward and build a Set of reverse positions (1-based from the end, matching tool_ids.length in the main reverse walk) for the most-recent instance of every repeated (name, arguments) pair. Positions in this set should never be dropped.



321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
 
# File 'lib/scout/llm/chat/prompt/shorten_tools_epoch.rb', line 321

def self.build_protected_positions(messages, total_tool_outputs)
  protected = Set.new

  fwd_pos = 0
  pending_call_keys = {}  # id → dedup_key (set by function_call, consumed by output)
  key_positions = Hash.new { |h, k| h[k] = [] }

  messages.each do |msg|
    case msg[:role].to_sym
    when :function_call
      json = msg[:content]
      next unless json
      tool_call = JSON.parse json rescue nil
      next unless tool_call
      name      = tool_call['name']
      arguments = tool_call['arguments']
      id        = tool_call['id']
      pending_call_keys[id] = Chat.epoch_dedup_key(name, arguments)
    when :function_call_output
      json = msg[:content]
      next unless json
      tool_call = JSON.parse json rescue nil
      next unless tool_call
      fwd_pos += 1
      id   = tool_call['id']
      key  = pending_call_keys.delete(id)
      next unless key
      key_positions[key] << fwd_pos
    end
  end

  key_positions.each do |key, positions|
    next if positions.length <= 1
    most_recent_fwd = positions.max
    reverse_pos     = total_tool_outputs - most_recent_fwd + 1
    protected << reverse_pos
  end

  unless protected.empty?
    Log.low "Epoch: protecting #{protected.length} repeated call(s) from dropping"
  end

  protected
end

.clean(messages, role = ['skip', 'previous_response_id']) ⇒ Object 



32
33
34
35
36
37
38
39
40
41
42
 
# File 'lib/scout/llm/chat/process/clear.rb', line 32

def self.clean(messages, role = ['skip', 'previous_response_id'])
  role = role.collect{|r| r.to_s } if Array == role
  messages.reject do |message|
    ((String === message[:content]) && message[:content].empty?) ||
      if Array === role
        role.include?(message[:role].to_s)
      else
        message[:role].to_s == role.to_s
      end
  end
end

.clear(messages, role = 'clear') ⇒ Object 



2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
 
# File 'lib/scout/llm/chat/process/clear.rb', line 2

def self.clear(messages, role = 'clear')
  new = []

  clear_tools = false
  clean_roles = []
  messages.reverse.each do |message|
    if message[:role].to_s == role.to_s
      break
    elsif message[:role].to_s == 'clear_tools' 
      clear_tools = message['content'].to_s != 'false'
    elsif message[:role].to_s == 'function_call' ||
      message[:role].to_s == 'function_call_output' 
      new << message unless clear_tools
    elsif message[:role].to_s == 'clean_role' || 
      message[:role].to_s == 'clear_role'
      clean_roles << message[:content].strip
    else
      new << message
    end
  end

  new = Chat.setup new.reverse

  clean_roles.each do |role|
    new = self.clean(new, role)
  end
  
  new
end

.config(chat) ⇒ Object 



2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
 
# File 'lib/scout/llm/chat/process/options.rb', line 2

def self.config(chat)
  new = []

  chat.select do |info|
    if Hash === info
      role = info[:role].to_s
      if role.to_s == 'config'
        key, value, *tokens = info[:content].split(" ")
        Scout::Config.set({key => value}, *tokens)
        next
      end
    end
    new << info
  end

  chat.replace new
end

.content_tokens(message) ⇒ Object 



10
11
12
 
# File 'lib/scout/llm/chat/process.rb', line 10

def self.content_tokens(message)
  Shellwords.split(message[:content].strip)
end

.direct_chat_sidecar_files(path) ⇒ Object

Return only chat logs owned by this persisted chat's .files sidecar. The save mechanism writes saved agent conversations into the chat's .files dir exactly like a job does, so a chat with a sidecar is scanned the same way a job is. This method is deliberately not recursive; recursion belongs to traverse_provenance.

ScoutCoder: DUAL-LAYOUT globbing (see DIRECT_LOG_CHAT_GLOBS) plus the root-copy exclusion. The save mechanism writes a full copy of the ROOT conversation at the TOP LEVEL of the files dir in BOTH layouts:

new     <path>.files/<name>.chat   (agent.chat, worker.chat, ...)
legacy  <path>.files/log/agent.chat

Those top-level copies must be excluded here or the chat would get a self-edge duplicating the root node. Note the asymmetry with direct_job_chat_files: a JOB's own top-level agent.chat IS traversed (renderers hide it), a CHAT's root copy is NOT. Only TOP-LEVEL *.chat files are excluded: society chats under .society/ (new) and under log/society/ (legacy) are independent conversations and are included.



68
69
70
71
72
73
74
75
76
 
# File 'lib/scout/llm/chat/provenance.rb', line 68

def self.direct_chat_sidecar_files(path)
  files_dir = path.to_s + '.files'
  return [] unless File.directory?(files_dir)
  top_level = Dir.glob(File.join(files_dir, '*.chat')).collect { |file| File.expand_path(file) }
  root_copy_legacy = File.expand_path(File.join(files_dir, 'log', 'agent.chat'))
  direct_log_chat_glob(files_dir).reject do |file|
    top_level.include?(file.to_s) || file.to_s == root_copy_legacy
  end
end

.direct_entries(chat_list) ⇒ Object

Select trace entries that carry direct token counts (not job projections).



345
346
347
348
349
350
351
 
# File 'lib/scout/llm/chat/process/meta.rb', line 345

def self.direct_entries(chat_list)
  trace_chats(chat_list).select do |entry|
    meta = entry[:meta]
    next false if meta[:job]
    TOKEN_KEYS.any? { |name| meta.include?(name) }
  end
end

.direct_job_chat_files(job) ⇒ Object

ScoutCoder: DUAL-LAYOUT globbing (see DIRECT_LOG_CHAT_GLOBS): a job's own chat logs live at .files/.chat (new), under .files/.society/** (new society tree) or under the legacy .files/log/** (written by older scout-ai; read-only compatibility, never migrated). This method is deliberately not recursive; recursion belongs to traverse_provenance.



44
45
46
47
 
# File 'lib/scout/llm/chat/provenance.rb', line 44

def self.direct_job_chat_files(job)
  job = Step.load(job) unless Step === job
  direct_log_chat_glob(job.files_dir)
end

.direct_log_chat_glob(files_dir) ⇒ Object

Glob DIRECT_LOG_CHAT_GLOBS under files_dir and return a de-duplicated, sorted list of existing chat files (Path objects). Both new and legacy layouts are returned together; callers that must exclude the root copy of the root conversation (Chat.direct_chat_sidecar_files) filter afterwards.



26
27
28
29
30
31
32
33
34
35
36
 
# File 'lib/scout/llm/chat/provenance.rb', line 26

def self.direct_log_chat_glob(files_dir)
  files_dir = files_dir.to_s
  return [] unless File.directory?(files_dir)
  DIRECT_LOG_CHAT_GLOBS
    .flat_map { |pattern| Dir.glob(File.join(files_dir, pattern)) }
    .collect { |file| File.expand_path(file) }
    .select { |file| File.file?(file) }
    .uniq
    .sort
    .collect { |file| Path.setup(file) }
end

.epoch_compacted_tool_calls ⇒ Object 



33
34
35
36
 
# File 'lib/scout/llm/chat/prompt/shorten_tools_epoch.rb', line 33

def self.epoch_compacted_tool_calls
  @@epoch_compacted_tool_calls ||= Scout::Config.get(:epoch_compacted_tool_calls, :prompt, :context,
    env: 'EPOCH_COMPACTED_TOOL_CALLS', default: DEFAULT_EPOCH_COMPACTED_TOOL_CALLS)
end

.epoch_dedup_key(name, arguments) ⇒ Object

Build a canonical deduplication key from a tool call's name + arguments. The model-generated id is deliberately excluded because it is not stable across inferences.



61
62
63
64
65
66
 
# File 'lib/scout/llm/chat/prompt/shorten_tools_epoch.rb', line 61

def self.epoch_dedup_key(name, arguments)
  normalized = epoch_sort_value(arguments || {})
  "#{name}\x00#{JSON.generate(normalized)}"
rescue
  "#{name}\x00#{arguments.inspect}"
end

.epoch_full_tool_calls ⇒ Object 



28
29
30
31
 
# File 'lib/scout/llm/chat/prompt/shorten_tools_epoch.rb', line 28

def self.epoch_full_tool_calls
  @@epoch_full_tool_calls ||= Scout::Config.get(:epoch_full_tool_calls, :prompt, :context,
    env: 'EPOCH_FULL_TOOL_CALLS', default: DEFAULT_EPOCH_FULL_TOOL_CALLS)
end

.epoch_size ⇒ Object 



38
39
40
41
 
# File 'lib/scout/llm/chat/prompt/shorten_tools_epoch.rb', line 38

def self.epoch_size
  @@epoch_size ||= Scout::Config.get(:epoch_size, :prompt, :context,
    env: 'EPOCH_SIZE', default: DEFAULT_EPOCH_SIZE)
end

.epoch_sort_value(obj) ⇒ Object

Recursively normalise a value so that hashes with differently-ordered or symbol/string keys produce the same canonical form.



47
48
49
50
51
52
53
54
55
56
 
# File 'lib/scout/llm/chat/prompt/shorten_tools_epoch.rb', line 47

def self.epoch_sort_value(obj)
  case obj
  when Hash
    obj.transform_keys(&:to_s).sort.to_h.transform_values { |v| epoch_sort_value(v) }
  when Array
    obj.map { |v| epoch_sort_value(v) }
  else
    obj
  end
end

.epoch_tool_call_threshold ⇒ Object

--- shorten_tools_epoch strategy configuration accessors ---



23
24
25
26
 
# File 'lib/scout/llm/chat/prompt/shorten_tools_epoch.rb', line 23

def self.epoch_tool_call_threshold
  @@epoch_tool_call_threshold ||= Scout::Config.get(:epoch_tool_call_threshold, :prompt, :context,
    env: 'EPOCH_TOOL_CALL_THRESHOLD', default: DEFAULT_EPOCH_TOOL_CALL_THRESHOLD)
end

.files(messages, original = nil, caller_lib_dir = Path.caller_lib_dir(nil, 'chats')) ⇒ Object 



62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
 
# File 'lib/scout/llm/chat/process/files.rb', line 62

def self.files(messages, original = nil, caller_lib_dir = Path.caller_lib_dir(nil, 'chats'))
  messages.collect do |message|
    if message[:role] == 'file' || message[:role] == 'directory'
      file = message[:content].to_s.strip
      found_file = find_file(file, original, caller_lib_dir)
      raise "File not found: #{file}" if found_file.nil?

      target = found_file

      if message[:role] == 'directory'
        Path.setup target
        target.glob('**/*').
          reject{|file|
            Open.directory?(file)
          }.collect{|file|
            files([{role: 'file', content: file}])
          }
      else
        new = Chat.tag :file, Open.read(target), file
        {role: 'user', content: new}
      end
    elsif message[:role] == 'pdf' || message[:role] == 'image'
      file = message[:content].to_s.strip
      found_file = find_file(file, original, caller_lib_dir)
      raise "File not found: #{file}" if found_file.nil?

      message[:content] = found_file
      message
    elsif message[:role] == 'step'
      step = message[:content].to_s.strip
      meta = Chat.meta(messages.dup)
      job_path = meta['job']
      job = Step.load job_path
      job = job.step(step) unless job.task_name == step
      {role: 'assistant', content: job.load.answer  }
    elsif message[:role] == 'allow_path'
      path = message[:content].to_s.strip
      Chat.allow_path(path)
      nil
    elsif message[:role] == 'allow_read_path'
      path = message[:content].to_s.strip
      Chat.allow_read_path(path)
      nil
    else
      message
    end
  end.flatten.compact
end

.find_file(file, original = nil, caller_lib_dir = Path.caller_lib_dir(nil, 'chats')) ⇒ Object 



17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
 
# File 'lib/scout/llm/chat/process/files.rb', line 17

def self.find_file(file, original = nil, caller_lib_dir = Path.caller_lib_dir(nil, 'chats'))
  path = Scout.chats[file]
  original = original.find if Path === original
  if original
    relative = File.join(File.dirname(original), file)
    relative_lib = File.join(caller_lib_dir, file) if caller_lib_dir
  end

  if relative && Open.exist?(relative)
    relative
  elsif relative_lib && Open.exist?(relative_lib)
    relative_lib
  elsif Open.exist?(file)
    file
  elsif Open.remote?(file)
    file
  elsif path.exists?
    path
  end
end

.find_role(messages, role) ⇒ Object 



18
19
20
 
# File 'lib/scout/llm/chat/process.rb', line 18

def self.find_role(messages, role)
  messages.select{|m| m[:role].to_sym == role.to_sym }
end

.follow(intro, coda) ⇒ Object 



115
116
117
118
119
120
121
122
123
124
125
126
127
 
# File 'lib/scout/llm/chat/annotation.rb', line 115

def self.follow(intro, coda)
  options = Chat.options(coda.dup)
  previous_response_id = options[:previous_response_id] if options

  if previous_response_id
    new = intro + Chat.clear(coda, :previous_response_id)
    new.unshift({role: :previous_response_id, content: previous_response_id})
  else
    new = intro + coda
  end

  Chat.setup new
end

.full_tool_calls ⇒ Object 



11
12
13
 
# File 'lib/scout/llm/chat/prompt/shorten_tools.rb', line 11

def self.full_tool_calls
  @@full_tool_calls ||= Scout::Config.get(:full_tool_calls, :prompt, :context, env: 'FULL_TOOL_CALLS')
end

.full_tool_outputs ⇒ Object 



15
16
17
 
# File 'lib/scout/llm/chat/prompt/shorten_tools.rb', line 15

def self.full_tool_outputs
  @@full_tool_outputs ||= Scout::Config.get(:full_tool_outputs, :prompt, :context, env: 'FULL_TOOL_OUTPUTS')
end

.imports(messages, original = nil, caller_lib_dir = Path.caller_lib_dir(nil, 'chats')) ⇒ Object 



38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
 
# File 'lib/scout/llm/chat/process/files.rb', line 38

def self.imports(messages, original = nil, caller_lib_dir = Path.caller_lib_dir(nil, 'chats'))
  messages.collect do |message|
    if message[:role] == 'import' || message[:role] == 'continue' || message[:role] == 'last'
      file = message[:content].to_s.strip
      found_file = find_file(file, original, caller_lib_dir)
      raise "Import not found: #{file}" if found_file.nil?

      new = LLM.messages Open.read(found_file)

      new = if message[:role] == 'continue'
              [new.reject{|msg| msg[:content].nil? || msg[:content].strip.empty? }.last]
            elsif message[:role] == 'last'
              [LLM.purge(new).reject{|msg| msg[:content].empty?}.last]
            else
              LLM.purge(new)
            end

      LLM.chat new, found_file
    else
      message
    end
  end.flatten
end

.indiferent(messages) ⇒ Object 



14
15
16
 
# File 'lib/scout/llm/chat/process.rb', line 14

def self.indiferent(messages)
  messages.collect{|msg| IndiferentHash.setup msg }
end

.job_agent_chat_files(job) ⇒ Object 



191
192
193
 
# File 'lib/scout/llm/chat/process/meta.rb', line 191

def self.job_agent_chat_files(job)
  direct_job_chat_files(job)
end

.job_chat_files(job, seen = Set.new) ⇒ Object

Return the result and logged chats for a job and all its dependencies. A job is visited only once, so shared dependencies and accidental cycles do not duplicate evidence or recurse forever.



198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
 
# File 'lib/scout/llm/chat/process/meta.rb', line 198

def self.job_chat_files(job, seen = Set.new)
  job = Step.load(job) unless Step === job
  key = File.expand_path(job.path.to_s)
  return [] if seen.include?(key)
  seen << key

  chats = []
  chats << job.path if job.done? && job.type.to_s == 'chat'

  chats.concat job_agent_chat_files(job)

  job.dependencies.each do |dependency|
    chats.concat(job_chat_files(dependency, seen))
  end

  chats.collect(&:to_s).uniq
rescue
  []
end

.job_result_chat_file(job) ⇒ Object

Return the persisted result as a chat file when the Step result type is chat. A job and its result chat may have the same path; traversal identities therefore always include the node kind.



81
82
83
84
85
 
# File 'lib/scout/llm/chat/provenance.rb', line 81

def self.job_result_chat_file(job)
  job = Step.load(job) unless Step === job
  return nil unless job.type.to_s == 'chat' && File.file?(job.path.to_s)
  Path.setup(File.expand_path(job.path.to_s))
end

.jobs(messages, original = nil) ⇒ Object 



82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
 
# File 'lib/scout/llm/chat/process/tools.rb', line 82

def self.jobs(messages, original = nil)
  messages.collect do |message|
    if message[:role] == 'job' || message[:role] == 'inline_job'
      file = message[:content].strip

      step = Step.load file

      id = Log.truncate_string(step.short_path.sub('Default_', ''), 40)
      id = id.gsub('/','-')

      if message[:role] == 'inline_job'
        path = step.path
        path = path.find if Path === path
        {role: 'file', content: step.path}
      else

        function_name = step.full_task_name.sub('#', '-')
        function_name = step.task_name
        tool_call = {
          name: function_name,
          arguments: step.provided_inputs,
          id: id,
        }

        content = if step.done?
                    Open.read(step.path)
                  elsif step.streaming?
                    step.join
                    step.load
                  elsif step.error?
                    Log.warn "Error in job #{step.path}"
                    e = step.exception
                    if ENV['SCOUT_CHAT_JOB_EXCEPTION'] == 'true'
                      {exception: e.message, stack: e.backtrace }.to_json
                    else
                      raise e
                    end
                  end

        tool_output = {
          id: id,
          content: content
        }

        [
          {role: 'function_call', content: tool_call.to_json},
          {role: 'function_call_output', content: tool_output.to_json},
        ]
      end
    else
      message
    end
  end.flatten
end

.load(file) ⇒ Object

Read a persisted chat without compiling it. Provenance inspection must not execute task, job, file, or import roles again.



187
188
189
 
# File 'lib/scout/llm/chat/process/meta.rb', line 187

def self.load(file)
  Chat.setup(Chat.parse(Open.read(file)))
end

.load_job_reference(reference) ⇒ Object

Resolve a job reference from a chat meta message into a Step. References like "Planned/ask/Default_abc.chat" are relative workflow paths. Step.load may resolve them to a Scout-specific directory that does not contain the actual job data, so we fall back to checking Rbbt.var.jobs and Scout.var.jobs.



107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
 
# File 'lib/scout/llm/chat/provenance.rb', line 107

def self.load_job_reference(reference)
  return reference if Step === reference
  ref_str = reference.to_s

  step = Step.load(ref_str)
  return step if File.exist?(step.path.to_s) || File.exist?(step.path.to_s + '.info')

  # Step.load resolves relative workflow paths (e.g. Planned/ask/Default_xyz.chat)
  # via Path.find, which may point to a directory that does not contain the
  # actual job data (e.g. ~/.scout/ instead of ~/.rbbt/var/jobs/). Try the
  # standard Rbbt workflow storage location as a fallback.
  rbbt_candidate = File.expand_path(File.join('~/.rbbt/var/jobs', ref_str))
  return Step.load(rbbt_candidate) if File.exist?(rbbt_candidate) || File.exist?(rbbt_candidate + '.info')

  step
end

.load_workflow(workflow) ⇒ Object 



30
31
32
33
34
35
36
37
38
39
40
41
 
# File 'lib/scout/llm/chat/process/tools.rb', line 30

def self.load_workflow(workflow)
  workflow = begin
               Kernel.const_get workflow
             rescue
               if Scout.chats.Agent[workflow]['workflow.rb'].exists?
                 Workflow.require_workflow_file Scout.chats.Agent[workflow]['workflow.rb'].find
                 Workflow.main || Workflow.workflows.last
               else
                 Workflow.require_workflow(workflow)
               end
             end
end

.max_tool_calls ⇒ Object 



19
20
21
 
# File 'lib/scout/llm/chat/prompt/shorten_tools.rb', line 19

def self.max_tool_calls
  @@max_tool_calls ||= Scout::Config.get(:max_tool_calls, :prompt, :context, env: 'MAX_TOOL_CALLS')
end

.max_tool_chars ⇒ Object 



27
28
29
 
# File 'lib/scout/llm/chat/prompt/shorten_tools.rb', line 27

def self.max_tool_chars
  @@max_tool_chars ||= Scout::Config.get(:max_tool_chars, :prompt, :context, env: 'MAX_TOOL_CHARS')
end

.max_tool_outputs ⇒ Object 



23
24
25
 
# File 'lib/scout/llm/chat/prompt/shorten_tools.rb', line 23

def self.max_tool_outputs
  @@max_tool_outputs ||= Scout::Config.get(:max_tool_outputs, :prompt, :context, env: 'MAX_TOOL_OUTPUTS', default: max_tool_calls)
end

.meta(messages) ⇒ Object

Meta messages are local bookkeeping and are not sent to the provider. The last direct inference checkpoint supplies the linear chat total for the next request; job metadata deliberately contributes no token counts.



138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
 
# File 'lib/scout/llm/chat/process/meta.rb', line 138

def self.meta(messages)
  meta_messages = []
  messages.reject! do |message|
    match = message[:role].to_s == 'meta'
    meta_messages << message if match
    match
  end
  return nil if meta_messages.empty?

  metas = meta_messages.collect { |message| parse_meta(message[:content]) }
  current = IndiferentHash.setup(metas.last.dup)
  checkpoint = metas.reverse.find do |meta|
    CUMULATIVE_KEYS.any? { |name| meta.include?(name) }
  end
  if checkpoint
    CUMULATIVE_KEYS.each do |name|
      current[name] = checkpoint[name] if checkpoint.include?(name)
    end
  end
  current
end

.meta_evidence(chat, source: nil, warnings: nil) ⇒ Object

All meta evidence for a chat, with a shared origin field:

* :chat_meta   - normal persisted `meta:` messages of this chat
               (meta, source, meta_address in [source, index] form,
               message: the raw message);
* :agent_meta  - receipt entries from function_call_output envelopes,
               appended after the local records (see
               Chat.agent_meta_evidence).

This is an analysis view only. Chat#meta, chat.role_messages(:meta), and Chat.token_totals([chat]) keep describing the local Chat and do not absorb delegated receipts.

Validation asymmetry, intentional: ordinary persisted meta: messages are historical data with no enforced schema, so they stay permissive here (a record is included even when Chat.parse_meta yields an empty Hash) exactly as Chat.trace_chats and Chat.token_totals have always treated them. A receipt instead has a narrow, generated schema, so malformed receipt entries are skipped with a warning instead of being silently trusted.



217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
 
# File 'lib/scout/llm/chat/agent_meta.rb', line 217

def self.meta_evidence(chat, source: nil, warnings: nil)
  local = chat.each_with_index.select do |message, _index|
    message[:role].to_s == 'meta'
  end.collect do |message, index|
    {
      origin: :chat_meta,
      meta: parse_meta(message[:content].to_s),
      source: source ? source.to_s : nil,
      meta_address: source ? [source.to_s, index] : index,
      message: message
    }
  end

  local + agent_meta_evidence(chat, source: source, warnings: warnings)
end

.normalize_usage(usage) ⇒ Object

Normalise a provider usage hash into a flat hash keyed by TOKEN_KEYS. Handles OpenAI Chat API (prompt_tokens/completion_tokens), Responses API (input_tokens/output_tokens), GLM (prompt_tokens/completion_tokens), and Anthropic (cache_read_input_tokens/cache_creation_input_tokens).

Missing fields are simply omitted from the result.



51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
 
# File 'lib/scout/llm/chat/process/meta.rb', line 51

def self.normalize_usage(usage)
  return {} if usage.nil? || usage.empty?

  IndiferentHash.setup(usage) unless usage.respond_to?(:dig)

  result = {}
  USAGE_FIELD_MAP.each do |path, short_key|
    next if result.include?(short_key) # first match wins
    value = IndiferentHash.dig(usage, *path)
    result[short_key] = value.to_i if value
  end

  # Compute total if not provided but both prompt and completion are present
  if !result.include?('tt')
    pt = result['pt']
    ct = result['ct']
    result['tt'] = pt.to_i + ct.to_i if pt && ct
  end

  result
end

.options(chat) ⇒ Object 



20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
 
# File 'lib/scout/llm/chat/process/options.rb', line 20

def self.options(chat)
  options = IndiferentHash.setup({})
  sticky_options = IndiferentHash.setup({})
  new = []

  # Most options reset after an assistant reply, but not previous_response_id
  chat.each do |info|
    if Hash === info
      role = info[:role].to_s
      if %w(endpoint model backend agent).include? role.to_s
        sticky_options[role] = info[:content]
        next
      elsif %w(persist).include? role.to_s
        options[role] = info[:content]
        next
      elsif %w(previous_response_id).include? role.to_s
        sticky_options[role] = info[:content]
      elsif %w(format).include? role.to_s
        format = info[:content]
        if Path.is_filename?(format)
          file = find_file(format)
          if file
            format = Open.json(file)
          end
        end
        options[role] = format
        next
      end

      if role.to_s == 'option'
        key, _, value = info[:content].partition(" ")
        options[key] = value
        next
      end

      if role.to_s == 'sticky_option'
        key, _, value = info[:content].partition(" ")
        sticky_options[key] = value
        next
      end

      if role == 'assistant'
        options.clear
      end
    end
    new << info
  end
  chat.replace new
  options = sticky_options.merge options
  options.delete_if{|k,v| v.nil? || v == 'nil' }
  options
end

.parse(text, role = nil) ⇒ Object 



10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
 
# File 'lib/scout/llm/chat/parse.rb', line 10

def self.parse(text, role = nil)
  default_role = "user"

  messages = []
  current_role = role || default_role
  current_content = ""
  in_protected_block = false
  protected_block_type = nil
  protected_stack = []

  role = default_role if role.nil?

  file_lines = text.split("\n")

  file_lines.each do |line|
    stripped = line.strip

    # Detect protected blocks
    if stripped.start_with?("```")
      if in_protected_block
        if protected_block_type == :ticks
          in_protected_block = false
          protected_block_type = nil
          current_content << "\n" << line unless line.strip.empty?
        end
      else
        in_protected_block = true
        protected_block_type = :ticks
        current_content << "\n" << line unless line.strip.empty?
      end
      next
    elsif stripped.end_with?("]]") && in_protected_block && protected_block_type == :square
      in_protected_block = false
      protected_block_type = nil
      line = line.sub("]]", "")
      current_content << "\n" << line unless line.strip.empty?
      next
    elsif stripped.start_with?("[[")
      in_protected_block = true
      protected_block_type = :square
      line = line.sub("[[", "")
      current_content << "\n" << line unless line.strip.empty?
      next
    elsif stripped.end_with?("]]") && in_protected_block && protected_block_type == :square
      in_protected_block = false
      protected_block_type = nil
      line = line.sub("]]", "")
      current_content << "\n" << line unless line.strip.empty?
      next
    elsif stripped.match(/^[^\s]* :-- .* {{{/)
      in_protected_block = true
      protected_block_type = :square
      line = line.sub(/^[^\s]* :-- (.*) {{{.*/, '<cmd_output cmd="\1">')
      current_content << "\n" << line unless line.strip.empty?
      next
    elsif stripped.match(/^.* :--.* }}}/) && in_protected_block && protected_block_type == :square
      in_protected_block = false
      protected_block_type = nil
      line = line.sub(/^.* :-- .* }}}.*/, "</cmd_output>")
      current_content << "\n" << line unless line.strip.empty?
      next
    elsif in_protected_block

      if protected_block_type == :xml
        if stripped =~ %r{</(\w+)>}
          closing_tag = $1
          if protected_stack.last == closing_tag
            protected_stack.pop
          end
          if protected_stack.empty?
            in_protected_block = false
            protected_block_type = nil
          end
        end
      end
      current_content << "\n" << line
      next
    end

    # XML-style tag handling (protected content)
    if stripped =~ /^<(\w+)(\s+[^>]*)?>/ && (tag = $1) && text =~ %r{</#{$1}>}
      protected_stack.push(tag)
      in_protected_block = true
      current_content << "\n" << line
      protected_block_type = :xml
      next
    end

    # Match a new message header
    if line =~ /^([a-z0-9_]+):(.*)$/
      role = $1
      inline_content = $2.strip

      current_content = current_content.strip if current_content
      # Save current message if any
      if current_content && ! current_content.empty?
        messages << { role: current_role, content: current_content }
      elsif messages.empty?
        #messages << { role: current_role, content: '' }
      end

      if inline_content.empty?
        # Block message
        current_role = role
        current_content = ""
      else
        # Inline message + next block is default role
        messages << { role: role, content: inline_content } if inline_content && ! inline_content.empty?
        current_role = 'user' if role == 'previous_response_id'
        current_role = 'user' if role == 'agent'
        current_content = ""
      end
    elsif line =~ /^\\([a-z0-9_]+):(.*)$/
      if current_content.nil?
        current_content = line[1..-1]
      else
        current_content += "\n" + line[1..-1]
      end
    else
      if current_content.nil?
        current_content = line
      else
        current_content += "\n" + line
      end
    end
  end

  # Final message
  messages << { role: current_role || default_role, content: current_content.strip } if current_content && ! current_content.empty?

  messages
end

.parse_json(text) ⇒ Object 



2
3
4
5
6
7
8
 
# File 'lib/scout/llm/chat/parse.rb', line 2

def self.parse_json(text)
  return nil if text.nil? || text.empty?
  re = /.*\`\`\`json\n(.*)\`\`\`\n?.*/sm
  re = Regexp.new(re.source.encode(text.encoding), re.options)
  text = text.gsub(re, '\1') if text.include?('```json')
  JSON.parse text
end

.parse_meta(str) ⇒ Object

Parse a serialized meta string back into an IndiferentHash.

Each token is key=value where value may be:

* A double-quoted string (used when the value contains '='):
  key="some text with = inside"
Backslash escapes inside quotes are unescaped.
* An unquoted bare value that may contain spaces but not '=':
  key=some text here
The value boundary is detected by a lookahead for the next
' key=' pattern or end-of-string.

ScoutCoder: when the quoted value contains inner double quotes that were not escaped during serialization (e.g. reasoning text that embeds file paths like ["/path"]), the quoted-value alternative must not terminate at the first inner quote. The inner quote is only treated as the closing quote when it is followed by a new key= boundary or end-of-string. This is achieved with a negative lookahead inside the character class: "(?!\s+[^\s=]+=|\s*\z)



113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
 
# File 'lib/scout/llm/chat/process/meta.rb', line 113

def self.parse_meta(str)
  str = str.to_s
  meta = IndiferentHash.setup({})
  return meta if str.empty?

  str.scan(/([^\s=]+)=("(?:[^"\\]|\\.|"(?!\s+[^\s=]+=|\s*\z))*"|.*?)(?=\s+[^\s=]+=|\s*\z)/m).each do |key, raw|
    value = if raw.start_with?('"') && raw.end_with?('"')
      raw[1..-2].gsub(/\\(.)/) { $1 }
    else
      raw
    end

    meta[key] = case value
                when /^-?\d+$/ then value.to_i
                when /^-?\d+\.\d+$/ then value.to_f
                else value
                end
  end

  meta
end

.parse_tool_message(message) ⇒ Object 



4
5
6
7
8
 
# File 'lib/scout/llm/chat/tool_calls.rb', line 4

def self.parse_tool_message(message)
  JSON.parse(message[:content].to_s)
rescue JSON::ParserError, TypeError
  nil
end

.prepare_prompt(prompt, prompt_strategies = nil) ⇒ Object

--- Prompt strategy dispatcher ---



29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
 
# File 'lib/scout/llm/chat/prompt.rb', line 29

def self.prepare_prompt(prompt, prompt_strategies = nil)
  return prompt_strategies.call(prompt) if Proc === prompt_strategies
  prompt_strategies = DEFAULT_CONTEXT_STRATEGY if prompt_strategies.nil?
  prompt_strategies = prompt_strategies.split(',') if String === prompt_strategies
  prompt_strategies.each do |strategy|
    prompt = case strategy
             when 'shorten_tools'
               Chat.shorten_tools(prompt)
             when 'shorten_tools_epoch'
               Chat.shorten_tools_epoch(prompt)
             when 'none'
               prompt
             else
               strategy_proc = REGISTERED_STRATEGIES[strategy]
               strategy_proc.call(prompt)
             end
  end
  return prompt
end

.print(chat) ⇒ Object 



143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
 
# File 'lib/scout/llm/chat/parse.rb', line 143

def self.print(chat)
  return chat if String  === chat
  "\n" + chat.collect do |message|
    IndiferentHash.setup message
    case message[:content]
    when Hash, Array
      message[:role].to_s + ":\n\n" + message[:content].to_json
    when nil, ''
      message[:role].to_s + ":"
    else
      if %w(option previous_response_id function_call function_call_output meta).include? message[:role].to_s
        message[:role].to_s + ": " + message[:content].to_s
      else
        re = Regexp.new(/^([a-z]+:)(\s)/ms)
        re = Regexp.new(re.source.encode(message[:content].to_s.encoding), re.options)
        message[:role].to_s + ":\n\n" +
          message[:content].to_s.gsub(re, '\\\\\1\2')
      end
    end
  end * "\n\n"
end

.print_brief(chat, expand = []) ⇒ Object 



165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
 
# File 'lib/scout/llm/chat/parse.rb', line 165

def self.print_brief(chat, expand = [])
  return chat if String  === chat
  expand = expand.collect{|role| role.to_s }

  "\n" + chat.collect do |message|
    message = IndiferentHash.setup message
    role, content = message.values_at :role, :content
    role = role.to_s

    str = case message[:content]
          when Hash, Array
            message[:content].to_json
          when nil, ''
            ''
          else
            message[:content].to_s
          end

    next role, "\n\n" + str if expand.include?(role)

    [role, Log.fingerprint(str)[1..-2]]
  end.compact.collect do |role,str|
    "#{role}: #{str}"
  end * "\n\n"
end

.print_tokens(tokens) ⇒ Object

Human-readable token summary suitable for one-line CLI output.



365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
 
# File 'lib/scout/llm/chat/process/meta.rb', line 365

def self.print_tokens(tokens)
  tokens = tokens.transform_keys(&:to_sym) if Hash === tokens
  parts = []
  parts << "prompt=#{Misc.human_number(tokens[:pt])}" if tokens[:pt]
  parts << "completion=#{Misc.human_number(tokens[:ct])}" if tokens[:ct]
  parts << "total=#{Misc.human_number(tokens[:tt])}" if tokens[:tt]
  if tokens[:cct] && tokens[:cct].to_i > 0
    parts << "cached=#{Misc.human_number(tokens[:cct])}"
  end
  if tokens[:cwt] && tokens[:cwt].to_i > 0
    parts << "cache_write=#{Misc.human_number(tokens[:cwt])}"
  end
  if tokens[:rt] && tokens[:rt].to_i > 0
    parts << "reasoning=#{Misc.human_number(tokens[:rt])}"
  end
  parts * ' '
end

.project(job, messages) ⇒ Object

Project a chat-task response from the job that produced it. The response gets exactly one producer marker () at its beginning, and the per-inference meta messages are kept inline, adjacent to the function calls they account for, so a projected chat carries the same token provenance as the original agent log.

The marker and the inference metas are intentionally separate messages: direct_entries and token_totals exclude any meta carrying job, so merging the marker into an inference meta would drop that inference from direct accounting. Keeping both means the same inference can be seen twice in a parent chat (projected copy + agent log); trace_indices dedups those copies by inference_id.

reas (reasoning summaries) are dropped from projected copies unless Scout::Config key chat.project.keep_reas is truthy: they dominate the size of a projected chat while token attribution and deduplication only need inference_id and the token fields.

The projection is idempotent: re-projecting an already projected response of the same job yields the same segments and token totals, without duplicating the marker or any inference meta.

ScoutCoder: re-projection is not a corner case. LLM::Agent#ask feeds every consumed dependency job chat back through Chat.project, so a chat that has already been projected once is projected again when its parent answer is consumed. The seen_inference guard below is what keeps that path from emitting the same inference twice.



410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
 
# File 'lib/scout/llm/chat/process/meta.rb', line 410

def self.project(job, messages)
  return [] if Array(messages).empty?
  keep_reas = %w(true TRUE True T 1).include?(Scout::Config.get(:keep_reas, :project, :chat, env: 'CHAT_PROJECT_KEEP_REAS').to_s)

  seen_inference = {}
  projected = Array(messages).collect do |message|
    next message.dup unless message[:role].to_s == 'meta'

    meta = parse_meta(message[:content])
    if meta[:job]
      next nil
    end

    identity = meta[:inference_id] || message[:content]
    next nil if seen_inference.key?(identity)
    seen_inference[identity] = true

    if keep_reas
      message.dup
    else
      { role: :meta, content: serialize_meta(meta.except(:reas)) }
    end
  end.compact

  return [] if projected.empty?
  [{ role: :meta, content: serialize_meta(job: job.to_s) }] + projected
end

.provenance(chat_file, prov = {}) ⇒ Object

Compatibility collector for the former chat-to-chat provenance Hash. New code should use traverse_provenance or provenance_edges, which retain job nodes and relation types.



393
394
395
396
397
398
399
400
401
402
 
# File 'lib/scout/llm/chat/provenance.rb', line 393

def self.provenance(chat_file, prov = {})
  traverse_provenance(chat_file) do |kind, object, parent_kind, parent, _relation, _first|
    next unless parent && kind == :chat
    parent_path = provenance_path(parent_kind, parent)
    prov[parent_path] ||= []
    path = provenance_path(kind, object)
    prov[parent_path] << path unless prov[parent_path].include?(path)
  end
  prov
end

.provenance_chat_files(root, **options) ⇒ Object 



374
375
376
377
378
379
380
 
# File 'lib/scout/llm/chat/provenance.rb', line 374

def self.provenance_chat_files(root, **options)
  files = []
  traverse_provenance(root, **options) do |kind, object, _pk, _parent, _relation, first|
    files << provenance_path(kind, object) if first && kind == :chat
  end
  files
end

.provenance_edges(root, **options) ⇒ Object

Flat structural edges suitable for JSON reports and renderers. Objects are intentionally retained as native values; presentation code can shorten or serialize paths as needed.

:agent_job edges carry the agent_meta receipt record that produced them under detail: (call id, tool name, output address, receipt address, job reference). Two distinct receipts pointing at the same job therefore remain two distinguishable edges instead of collapsing into one; ordinary edges keep detail: nil.



349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
 
# File 'lib/scout/llm/chat/provenance.rb', line 349

def self.provenance_edges(root, **options)
  edges = []
  traverse_provenance(root, **options) do |kind, object, parent_kind, parent, relation, _first, detail|
    next unless parent
    edge = {
      from_kind: parent_kind,
      from: parent,
      relation: relation,
      to_kind: kind,
      to: object,
      detail: detail
    }
    same_edge = lambda do |other|
      other[:relation] == relation &&
        provenance_key(other[:from_kind], other[:from]) == provenance_key(parent_kind, parent) &&
        provenance_key(other[:to_kind], other[:to]) == provenance_key(kind, object) &&
        (relation != :agent_job ||
         (detail.nil? && other[:detail].nil?) ||
         (detail && other[:detail] && detail[:evidence_address] == other[:detail][:evidence_address]))
    end
    edges << edge unless edges.any?(&same_edge)
  end
  edges
end

.provenance_error(on_error, error, kind, object, relation, reference) ⇒ Object 



98
99
100
101
 
# File 'lib/scout/llm/chat/provenance.rb', line 98

def self.provenance_error(on_error, error, kind, object, relation, reference)
  raise error unless on_error
  on_error.call(error, kind, object, relation, reference)
end

.provenance_jobs(root, **options) ⇒ Object 



382
383
384
385
386
387
388
 
# File 'lib/scout/llm/chat/provenance.rb', line 382

def self.provenance_jobs(root, **options)
  jobs = []
  traverse_provenance(root, **options) do |kind, object, _pk, _parent, _relation, first|
    jobs << object if first && kind == :job
  end
  jobs
end

.provenance_key(kind, object) ⇒ Object 



94
95
96
 
# File 'lib/scout/llm/chat/provenance.rb', line 94

def self.provenance_key(kind, object)
  [kind.to_sym, provenance_path(kind, object)]
end

.provenance_path(kind, object) ⇒ Object 



87
88
89
90
91
92
 
# File 'lib/scout/llm/chat/provenance.rb', line 87

def self.provenance_path(kind, object)
  path = File.expand_path(kind.to_sym == :job ? object.path.to_s : object.to_s)
  File.realpath(path)
rescue SystemCallError
  path
end

.provenance_token_events(root, warnings: nil, strict: false, **traversal_options) ⇒ Object

One event Hash per deduplicated direct inference event reachable from root. A job= projection meta is never a token event, and only evidence carrying at least one TOKEN_KEYS field becomes an event.

Warning routing: when the caller supplies a warnings Array, two kinds of problems are reported into it instead of raising:

* traversal-stage agent_meta problems (malformed receipts and unresolved
job references) arrive as the agent_meta_error_reference Hash plus
`message:`; the traversal keeps going so one bad receipt never hides
the rest of the provenance;
* identity_conflict records (see below).

Unrelated traversal errors keep strict semantics: they raise, both with and without a warnings Array. Without the Array, agent_meta problems raise exactly like Chat.traverse_provenance strict mode.

Identity conflicts never raise here; pass strict: true to make them raise instead of being reported (see below).

Event shape:

{
inference_id: <String or nil>,
identity: <the Array used for grouping, e.g. [:inference_id, "w1"]>,
deduplication: :inference_id | :provider_response_id |
               :legacy_lineage | :receipt_unresolved,
meta: <canonical parsed meta>,
tokens: {pt:, ct:, tt:, cct:, cwt:, rt:} (symbols, zero-filled,
        canonical evidence only),
evidence: [
  {origin: :chat_meta, source:, meta_address:, meta:, call_id: nil,
   tool_name: nil},
  {origin: :agent_meta, source:, evidence_address:, output_address:,
   agent_meta_index:, call_id:, tool_name:, meta:}
],
conflict: true|false,
incomplete_evidence: true|false
}

evidence holds EVERY persisted location of the event (the same inference copied into several saved chats/logs appears once per chat plus once per receipt). Only the canonical evidence supplies meta and tokens.

Identity (grouping) rules, in priority order:

* meta[:inference_id]                    -> [:inference_id, id]
* meta[:provider_response_id] otherwise  -> [:provider_response_id, id]
* chat-side legacy (neither)             -> [:lineage, lineage_id], the
existing global lineage dedup of Chat.trace_chat_sources is preserved
* receipt-side legacy (neither)          -> [:receipt, evidence_address];
there is no exact rule, so a receipt legacy meta is never merged with
anything (documented possible overcount for legacy data)

Identity conflicts: when evidence sharing an identity disagree on any TOKEN_KEYS value (compared as integers, missing counts as 0) or on provider_response_id (two DIFFERENT non-empty values), the event keeps every evidence record, counts only the canonical one, sets conflict: true, and - when a warnings Array was supplied - appends one Hash per conflicting event:

{reason: :identity_conflict, inference_id:, identity:,
fields: <Array of disputed field symbols>,
values: <Hash field => the raw meta values in evidence order>,
evidence: <the event evidence records>}

A missing provider_response_id in one record and a present one in another is INCOMPLETE EVIDENCE, not a conflict: one copy simply carries richer metadata (e.g. across a metadata-format migration). Such events set incomplete_evidence: true, are counted normally, and are never warned about as conflicts. The same leniency applies to optional detailed usage fields (cct/cwt/rt): only the immutable core (inference identity, pt/ct/tt, non-empty provider_response_id) can conflict.

A conflicting event is counted once from its canonical evidence, but that number is NOT authoritative: reports must label any total containing conflicts as unresolved (see Chat.provenance_token_totals conflicts:). Pass strict: true to raise ScoutException on the first conflict instead.

Checkpoint fields (*_c, *_s) are never read or summed here.



499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
 
# File 'lib/scout/llm/chat/provenance.rb', line 499

def self.provenance_token_events(root, warnings: nil, strict: false, **traversal_options)
  # Route traversal-stage agent_meta problems into the caller's warnings
  # Array instead of letting them raise.  Other traversal errors stay strict
  # (raise), and an explicitly supplied on_error keeps being called.
  if Array === warnings
    caller_on_error = traversal_options[:on_error]
    traversal_options = traversal_options.merge(
      on_error: lambda do |error, kind, object, relation, reference|
        if relation == :agent_job && reference.is_a?(Hash) && reference[:reason] &&
           error.message.to_s.include?('agent_meta receipt')
          warnings << reference.merge(message: error.message)
          caller_on_error.call(error, kind, object, relation, reference) if caller_on_error
        elsif caller_on_error
          caller_on_error.call(error, kind, object, relation, reference)
        else
          raise error
        end
      end
    )
  end

  files = provenance_chat_files(root, **traversal_options)
  sources = {}
  files.each { |file| sources[file] = Chat.load(file) }

  evidences = []

  # Chat-side direct events.  `deduplicate: false` keeps one entry per
  # persisted location: an inference copied into several saved chats/logs
  # (or projected into several places) yields several evidence records, one
  # per address.  Grouping by identity happens exactly once, below, so the
  # resulting event can list every location.
  trace_chat_sources(sources, false).each do |entry|
    meta = entry[:meta]
    next if meta[:job]
    next unless TOKEN_KEYS.any? { |key| meta.include?(key) }
    address = entry[:meta_address]
    evidences << {
      origin: :chat_meta,
      discovery: evidences.length,
      source: address ? address[0] : nil,
      meta_address: address,
      meta: meta,
      call_id: nil,
      tool_name: nil,
      inference_id: entry[:inference_id],
      lineage_id: entry[:lineage_id]
    }
  end

  # Receipt-side direct events.  Malformed receipts are reported through a
  # private buffer and merged into the caller's warnings Array only when the
  # traversal stage has not already reported the same problem (it reports
  # with the richer reference + message shape whenever :agent_job is
  # followed); without an Array, agent_meta_evidence skips them silently.
  receipt_problems = Array === warnings ? [] : nil
  sources.each do |path, chat|
    agent_meta_evidence(chat, source: path, warnings: receipt_problems).each do |record|
      meta = record[:meta]
      next if meta[:job]
      next unless TOKEN_KEYS.any? { |key| meta.include?(key) }
      evidences << {
        origin: :agent_meta,
        discovery: evidences.length,
        source: path,
        evidence_address: record[:evidence_address],
        output_address: record[:output_address],
        agent_meta_index: record[:agent_meta_index],
        meta: meta,
        call_id: record[:call_id],
        tool_name: record[:tool_name],
        inference_id: meta[:inference_id],
        lineage_id: nil
      }
    end
  end

  if receipt_problems && !receipt_problems.empty?
    reported = Set.new(warnings.collect do |warning|
      next if warning[:reason] == :identity_conflict
      [warning[:reason], warning[:output_address], warning[:agent_meta_index]]
    end)
    receipt_problems.each do |problem|
      key = [problem[:reason], problem[:output_address], problem[:agent_meta_index]]
      warnings << problem unless reported.include?(key)
    end
  end

  origin_rank = { chat_meta: 0, agent_meta: 1 }
  events = []
  by_identity = {}

  evidences.each do |evidence|
    meta = evidence[:meta]
    if evidence[:inference_id]
      identity = [:inference_id, evidence[:inference_id]]
      deduplication = :inference_id
    elsif meta[:provider_response_id]
      identity = [:provider_response_id, meta[:provider_response_id]]
      deduplication = :provider_response_id
    elsif evidence[:origin] == :chat_meta
      identity = [:lineage, evidence[:lineage_id]]
      deduplication = :legacy_lineage
    else
      identity = [:receipt, evidence[:evidence_address]]
      deduplication = :receipt_unresolved
    end

    event = by_identity[identity]
    unless event
      event = { identity: identity, deduplication: deduplication, _evidence: [] }
      by_identity[identity] = event
      events << event
    end
    event[:_evidence] << evidence
  end

  events.each do |event|
    ordered = event.delete(:_evidence).sort_by do |evidence|
      [origin_rank[evidence[:origin]], evidence[:discovery]]
    end
    canonical = ordered.first

    event[:evidence] = ordered.collect do |evidence|
      record = {
        origin: evidence[:origin],
        source: evidence[:source],
        meta: evidence[:meta],
        call_id: evidence[:call_id],
        tool_name: evidence[:tool_name]
      }
      if evidence[:origin] == :chat_meta
        record[:meta_address] = evidence[:meta_address]
      else
        record[:evidence_address] = evidence[:evidence_address]
        record[:output_address] = evidence[:output_address]
        record[:agent_meta_index] = evidence[:agent_meta_index]
      end
      record
    end

    event[:inference_id] = canonical[:inference_id]
    event[:meta] = canonical[:meta]
    event[:tokens] = TOKEN_KEYS.each_with_object({}) do |key, hash|
      hash[key.to_sym] = canonical[:meta][key].to_i
    end

    # Immutable core only: pt/ct/tt and a genuinely different non-empty
    # provider_response_id.  Optional detail fields (cct, cwt, rt) and a
    # missing-vs-present provider_response_id are incomplete evidence, not
    # conflicts.
    core_fields = %i[pt ct tt]
    fields = []
    values = {}
    core_fields.each do |key|
      next unless ordered.collect { |evidence| evidence[:meta][key].to_i }.uniq.length > 1
      fields << key
      values[key] = ordered.collect { |evidence| evidence[:meta][key] }
    end
    provider_ids = ordered.collect { |evidence| evidence[:meta][:provider_response_id].to_s }
                               .reject { |value| value.strip.empty? }.uniq
    if provider_ids.length > 1
      fields << :provider_response_id
      values[:provider_response_id] = ordered.collect { |evidence| evidence[:meta][:provider_response_id] }
    end

    # Incomplete evidence: exactly one non-empty provider_response_id value
    # while at least one copy lacks it (metadata-format migration), so there
    # is nothing to disagree about.
    present_ids = ordered.collect { |evidence| evidence[:meta][:provider_response_id].to_s }
                          .reject { |value| value.strip.empty? }
    if present_ids.uniq.length <= 1 &&
       present_ids.length != ordered.length && !present_ids.empty?
      event[:incomplete_evidence] = true
    end

    next unless fields.any?
    event[:conflict] = true
    if strict
      raise ScoutException, "agent_meta identity conflict on #{event[:identity] * '='}: disputed #{fields * ','}"
    end
    next unless Array === warnings
    warnings << {
      reason: :identity_conflict,
      inference_id: event[:inference_id],
      identity: event[:identity],
      fields: fields,
      values: values,
      evidence: event[:evidence]
    }
  end

  # Reorder keys readably; conflict defaults to false.
  events.collect do |event|
    {
      inference_id: event[:inference_id],
      identity: event[:identity],
      deduplication: event[:deduplication],
      meta: event[:meta],
      tokens: event[:tokens],
      evidence: event[:evidence],
      conflict: event[:conflict] || false,
      incomplete_evidence: event[:incomplete_evidence] || false
    }
  end
end

.provenance_token_totals(root, scope: :deduplicated_total, warnings: nil, conflicts: nil, **traversal_options) ⇒ Object

Aggregate token totals over deduplicated provenance events. The result is symbol keyed and zero initialized exactly like Chat.token_totals: ct:, tt:, cct:, cwt:, rt:.

Scopes. These are EVIDENCE COVERAGE descriptions, not a partition of cost: an event represented both in a saved child log and in a receipt belongs to :chat_evidence AND to :receipt_evidence, so the two are not additive and must never be summed together. Only :deduplicated_total and :receipt_only are safe to compare additively.

* :deduplicated_total - every event counted once (default);
* :chat_evidence      - events with at least one :chat_meta evidence
                      (physically stored in a chat/log file);
* :receipt_evidence   - events with at least one :agent_meta evidence
                      (embedded in a function_call_output receipt);
* :receipt_only       - events with receipt evidence and NO saved-chat
                      evidence (the disjoint delegated contribution).

Conflict handling: a conflicting event still contributes its canonical tokens, so a total containing conflicts is a best-effort view, not an authoritative cost. The conflicts: keyword makes that explicit in machine readable form; with strict: true conflicts raise instead (the keyword is forwarded to provenance_token_events).



729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
 
# File 'lib/scout/llm/chat/provenance.rb', line 729

def self.provenance_token_totals(root, scope: :deduplicated_total, warnings: nil,
                                 conflicts: nil, **traversal_options)
  events = provenance_token_events(root, warnings: warnings, **traversal_options)

  selected = case scope.to_sym
             when :deduplicated_total
               events
             when :chat_evidence
               events.select { |event| event[:evidence].any? { |e| e[:origin] == :chat_meta } }
             when :receipt_evidence
               events.select { |event| event[:evidence].any? { |e| e[:origin] == :agent_meta } }
             when :receipt_only
               events.select do |event|
                 event[:evidence].any? { |e| e[:origin] == :agent_meta } &&
                   event[:evidence].none? { |e| e[:origin] == :chat_meta }
               end
             else
               raise ParameterException, "Unknown token scope: #{scope}"
             end

  totals = TOKEN_KEYS.each_with_object({}) { |key, hash| hash[key.to_sym] = 0 }
  selected.each do |event|
    TOKEN_KEYS.each { |key| totals[key.to_sym] += event[:tokens][key.to_sym] }
  end

  if conflicts.is_a?(Hash)
    conflicting = events.select { |event| event[:conflict] }
    incomplete = events.select { |event| event[:incomplete_evidence] }
    conflicts[:events] = conflicting.length
    conflicts[:incomplete_evidence_events] = incomplete.length
    conflicts[:authoritative] = conflicting.empty?
  end

  totals
end

.pull(chat, role = :previous_response_id) ⇒ Object 



52
53
54
55
56
57
58
59
60
61
62
63
64
 
# File 'lib/scout/llm/chat/process/clear.rb', line 52

def self.pull(chat, role = :previous_response_id)
  last = nil
  chat.reject! do |msg|
    msg = IndiferentHash.setup msg.dup

    match = msg[:role].to_s == role.to_s
    last = msg if match
    match
  end

  return nil if last.nil?
  IndiferentHash.setup last.dup
end

.purge(chat, role = :previous_response_id) ⇒ Object 



44
45
46
47
48
49
50
 
# File 'lib/scout/llm/chat/process/clear.rb', line 44

def self.purge(chat, role = :previous_response_id)
  chat.reject do |msg|
    msg = IndiferentHash.setup msg.dup

    msg[:role].to_s == role.to_s
  end
end

.report_agent_meta_problems(chat, object, on_error = nil) ⇒ Object

Diagnostics for agent_meta receipt problems found while expanding one chat node: malformed receipts collected by Chat.agent_meta_job_references. Provenance is only ever extracted from parsed function_call_output JSON Hashes carrying an explicit receipt key (current meta, legacy agent_meta); raw output text is never inspected, so output content that merely mentions "agent_meta" cannot produce a warning. Each malformed record becomes one Chat.provenance_error call with relation :agent_job and kind/object :chat + the chat path, so strict mode (no on_error) raises ScoutException while warning mode keeps going. Returns the valid job references so the caller can enqueue them afterwards.



150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
 
# File 'lib/scout/llm/chat/provenance.rb', line 150

def self.report_agent_meta_problems(chat, object, on_error = nil)
  warnings = []
  references = agent_meta_job_references(chat, source: object, warnings: warnings)

  warnings.each do |warning|
    error = ScoutException.new(
      agent_meta_error_message(warning[:reason],
                               chat_path: object.to_s,
                               output_address: warning[:output_address],
                               call_id: warning[:call_id],
                               tool_name: warning[:tool_name],
                               reference: warning[:reference] || warning[:raw_entry])
    )
    reference = agent_meta_error_reference(reason: warning[:reason],
                                           source: warning[:source] || object.to_s,
                                           output_address: warning[:output_address],
                                           evidence_address: warning[:evidence_address],
                                           call_id: warning[:call_id],
                                           tool_name: warning[:tool_name],
                                           agent_meta_index: warning[:agent_meta_index],
                                           raw_entry: warning[:raw_entry],
                                           reference: warning[:reference])
    provenance_error(on_error, error, :chat, object, :agent_job, reference)
  end

  references
end

.resolve_job_reference(reference) ⇒ Object

Resolve one job reference into a usable Step. Returns [step, nil] when the job data is loadable (path or .info sidecar exists) and [nil, step] when the reference resolves to a directory without job data, keeping the would-be step for diagnostics. Used by the :agent_job relation so unresolved receipt references are reported instead of enqueued.



129
130
131
132
133
134
135
136
137
 
# File 'lib/scout/llm/chat/provenance.rb', line 129

def self.resolve_job_reference(reference)
  step = load_job_reference(reference)
  path = step.path.to_s
  if File.exist?(path) || File.exist?(path + '.info')
    [step, nil]
  else
    [nil, step]
  end
end

.serialize_meta(meta) ⇒ Object

Serialize a meta hash into a single space-separated string of key=value pairs. Values that contain '=' are wrapped in double quotes so that the '=' inside them is not mistaken for a key/value delimiter during parsing. Backslashes and double quotes inside quoted values are escaped.

Keys are sorted by value string length (ascending) so that the longest free-text value appears last; this preserves backward compatibility with the unquoted parser path where the final value extends to end-of-string.



81
82
83
84
85
86
87
88
89
90
91
92
93
 
# File 'lib/scout/llm/chat/process/meta.rb', line 81

def self.serialize_meta(meta)
  keys = meta.keys.sort_by { |key| String === meta[key] ? meta[key].length : 0 }
  keys.collect do |key|
    value     = meta[key]
    str_value = value.to_s
    if str_value.include?('=')
      escaped = str_value.gsub('\\') { '\\\\' }.gsub('"') { '\\"' }
      %Q(#{key}="#{escaped}")
    else
      "#{key}=#{str_value}"
    end
  end * ' '
end

.shorten_string(string, size = DEFAULT_SHORT_STRING_LENGTH, step: nil) ⇒ Object

--- Shared utility used by all strategies ---



13
14
15
16
17
18
19
20
21
22
23
24
25
 
# File 'lib/scout/llm/chat/prompt.rb', line 13

def self.shorten_string(string, size = DEFAULT_SHORT_STRING_LENGTH, step: nil)
  new = Log.truncate_string(string, size)
  if new.length < string.length
    new = ['CONTEXT-COMPACTED', 'Historical account compacted for context efficiency; it does not faithfully represent what happened',  "Original content length: #{string.length}"]
    if step
      new << "Full output found in job: #{step}"
    end
    new << 'Do not execute, copy or otherwise use this string as-is.'
    new << "Preview: <<#{new}>>"
    new = "[#{new * ' - '}]"
  end
  new
end

.shorten_tools(messages) ⇒ Object

Walks the message array in reverse (newest-first) and applies a tiered truncation/dropping policy to function_call and function_call_output messages. The most recent tool interactions get priority for full retention; older ones are progressively degraded.

This is the default strategy.



41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
 
# File 'lib/scout/llm/chat/prompt/shorten_tools.rb', line 41

def self.shorten_tools(messages)
  tool_ids = []
  tool_chars = 0
  user_messages = 1
  assistant_messages = 0

  full_tool_calls = self.full_tool_calls || DEFAULT_FULL_TOOL_CALLS
  full_tool_outputs = self.full_tool_outputs || DEFAULT_FULL_TOOL_OUTPUTS
  max_tool_calls = self.max_tool_calls || DEFAULT_MAX_TOOL_CALLS
  max_tool_outputs = self.max_tool_outputs || DEFAULT_MAX_TOOL_OUTPUTS
  max_tool_chars = self.max_tool_chars || DEFAULT_MAX_TOOL_CHARS

  full_tool_calls = full_tool_calls.to_i
  full_tool_outputs = full_tool_outputs.to_i
  max_tool_calls = max_tool_calls.to_i
  max_tool_outputs = max_tool_outputs.to_i
  max_tool_chars = max_tool_chars.to_i

  messages.reverse.collect do |msg|
    case msg[:role].to_sym
    when :function_call
      json = msg[:content]
      next msg unless json

      tool_call = JSON.parse json
      name, arguments, id = tool_call.values_at 'name', 'arguments', 'id'

      if tool_ids.length < full_tool_calls || user_messages == 0 || tool_chars < max_tool_chars
        tool_chars += json.length
        msg
      elsif tool_ids.length > max_tool_calls
        Log.medium "Skipped tool call #{id} #{name} #{json.length}"
        next
      else
        new_arguments = {}
        arguments.each do |k,v|
          new_arguments[k] = String === v ? shorten_string(v) : v
        end if arguments

        next msg if arguments.values == new_arguments.values

        tool_call['arguments'] = new_arguments
        json = tool_call.to_json
        tool_chars += json.length
        Log.medium "Truncated tool call #{id} #{name} #{msg[:content].length} to #{json.length}"
        msg = msg.dup
        msg[:content] = json
        msg
      end
    when :function_call_output
      json = msg[:content]
      next msg unless json

      tool_call = JSON.parse json
      name, content, id = tool_call.values_at 'name', 'content', 'id'
      tool_ids << id

      if tool_ids.length < full_tool_outputs || user_messages == 0 || tool_chars < max_tool_chars
        tool_chars += json.length
        msg
      elsif tool_ids.length > max_tool_outputs
        Log.medium "Skipped tool output #{id} #{name} #{json.length}"
        next 
      else
        tool_call['content'] = shorten_string(content, DEFAULT_SHORT_STRING_LENGTH*2)
        next msg if content == tool_call['content']
        json = tool_call.to_json
        tool_chars += json.length
        Log.medium "Truncated tool output #{id} #{name} #{msg[:content].length} to #{json.length}"
        msg = msg.dup
        msg[:content] = json
        msg
      end
    when :user
      user_messages += 1
      msg
    when :assistant
      assistant_messages += 1
      msg
    else
      msg
    end
  end.compact.reverse
end

.shorten_tools_epoch(messages) ⇒ Object

Cache-friendly variant of shorten_tools. Instead of recomputing the compactation boundary on every single inference (which constantly shifts the prefix and defeats KV-cache / prompt-cache), this strategy divides the conversation into epochs.

Within an epoch window of epoch_size tool calls the compaction boundary is frozen. This means that the compacted prefix (the messages up to and including the compacted region) is byte-for-byte identical for every inference inside that window.

Layout (newest at the bottom)

[ dropped ]        tool calls older than (compacted + full) → removed
[ compacted ]      up to +epoch_compacted_tool_calls+ tool calls, compacted
[ full-recent ]    +epoch_full_tool_calls+ tool calls at full fidelity
[ full-new ]       any tool calls that arrived after the epoch boundary

The compacted and full-recent regions are pinned relative to the epoch boundary, not the live tool-call count, so their content stays stable until the boundary advances.

Epoch boundary calculation

overflow  = total_tool_calls - threshold        # how many beyond threshold
epoch_idx = overflow > 0 ? (overflow - 1) / epoch_size : 0
pinned_total = threshold + (epoch_idx * epoch_size)

The pinned_total determines where the full-recent region starts. Any tool calls beyond pinned_total are treated as "new" and kept at full fidelity (they are the live portion of the prompt that changes each turn).

Example (threshold=100, full=10, compacted=40, epoch_size=10)

100 calls → compaction starts: keep 10 full, compact 40, drop 50
101 calls → pinned_total = 100, keep 10 full-recent + 1 full-new
105 calls → same pinned_total = 100, keep 10 full-recent + 5 full-new
110 calls → pinned_total = 100, keep 10 full-recent + 10 full-new
111 calls → pinned_total = 110, keep 10 full-recent + 1 full-new

Repeated-call protection

When the agent loses visibility of a previous tool call (because it was dropped or heavily compacted) it may re-issue the identical call, creating infinite retry loops. To prevent this the strategy detects repeated calls — matched by (name, arguments) excluding the unstable model-generated id — and ensures the most recent instance of every repeated call is never dropped: it is compacted instead. Older duplicate instances are dropped or compacted normally.



120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
 
# File 'lib/scout/llm/chat/prompt/shorten_tools_epoch.rb', line 120

def self.shorten_tools_epoch(messages)
  threshold = (self.epoch_tool_call_threshold || DEFAULT_EPOCH_TOOL_CALL_THRESHOLD).to_i
  full      = (self.epoch_full_tool_calls      || DEFAULT_EPOCH_FULL_TOOL_CALLS).to_i
  compacted = (self.epoch_compacted_tool_calls || DEFAULT_EPOCH_COMPACTED_TOOL_CALLS).to_i
  epoch_sz  = (self.epoch_size                 || DEFAULT_EPOCH_SIZE).to_i

  # ---- count tool outputs in the message array ----
  total_tool_outputs = messages.count { |m| m[:role].to_sym == :function_call_output }

  return messages if total_tool_outputs <= threshold
  return messages if full == 0 && compacted == 0
  return messages if epoch_sz <= 0

  # ---- compute the pinned epoch boundary ----
  overflow     = total_tool_outputs - threshold
  epoch_idx    = overflow > 0 ? (overflow - 1) / epoch_sz : 0
  pinned_total = threshold + (epoch_idx * epoch_sz)

  # Number of tool calls that are "new" (arrived after the epoch boundary)
  new_calls = total_tool_outputs - pinned_total

  # From the end, the regions are:
  #   [1 .. new_calls]                              → full-new (keep unchanged)
  #   [new_calls+1 .. new_calls+full]                → full-recent (keep unchanged)
  #   [new_calls+full+1 .. new_calls+full+compacted] → compacted
  #   everything older                              → dropped

  keep_full_count = new_calls + full
  truncate_to     = keep_full_count + compacted

  # ---- detect repeated tool calls to protect from dropping ----
  # Walk forward to identify the most-recent instance of each repeated
  # (name, arguments) pair.  Those reverse positions are added to
  # +protected_positions+ so that the main reverse walk compacts them
  # instead of dropping, preventing the agent from re-issuing a call it
  # no longer remembers.
  protected_positions = build_protected_positions(messages, total_tool_outputs)

  # tool_ids counter mirrors the original shorten_tools: incremented BEFORE
  # the check for function_call_output, so that a function_call and its
  # paired output see the same counter value.
  tool_ids = []

  kept_messages = []
  dropped_count = 0
  compacted_count = 0

  # Walk in reverse so we can apply the position-based policy.
  messages.reverse.each do |msg|
    case msg[:role].to_sym
    when :function_call_output
      json = msg[:content]
      if json.nil?
        kept_messages << msg
        next
      end

      tool_call = JSON.parse json rescue nil
      unless tool_call
        kept_messages << msg
        next
      end

      name, content, id, step = tool_call.values_at 'name', 'content', 'id', 'step'
      tool_ids << id   # increment BEFORE check (mirrors original shorten_tools)

      if tool_ids.length <= keep_full_count || protected_positions.include?(tool_ids.length)
        # full-new or full-recent (or protected repeated call) → keep unchanged
        kept_messages << msg
      elsif tool_ids.length <= truncate_to
        # compacted region → truncate the content
        new_content = shorten_string(content.to_s, DEFAULT_SHORT_STRING_LENGTH * 2, step: step)
        if new_content != content
          tool_call['content'] = new_content
          new_json = tool_call.to_json
          Log.low "Epoch: truncated tool output #{id} #{name} #{json.length} to #{new_json.length}"
          new_msg = msg.dup
          new_msg[:content] = new_json
          new_msg[:compacted] = true
          kept_messages << new_msg
          compacted_count += 1
        else
          kept_messages << msg
        end
      else
        # beyond compacted → drop
        Log.low "Epoch: dropped tool output #{id} #{name} #{json.length}"
        dropped_count += 1
      end

    when :function_call
      json = msg[:content]
      if json.nil?
        kept_messages << msg
        next
      end

      tool_call = JSON.parse json rescue nil
      unless tool_call
        kept_messages << msg
        next
      end

      name, arguments, id = tool_call.values_at 'name', 'arguments', 'id'

      # tool_ids already includes the paired output (processed before us
      # in reverse), so we use the same boundary check.
      if tool_ids.length <= keep_full_count || protected_positions.include?(tool_ids.length)
        # full (or protected repeated call) → keep unchanged
        kept_messages << msg
      elsif tool_ids.length <= truncate_to
        # compacted → truncate arguments
        if arguments && arguments.any?
          new_arguments = {}
          arguments.each do |k, v|
            new_arguments[k] = String === v ? shorten_string(v) : v
          end
          if arguments.values != new_arguments.values
            tool_call['arguments'] = new_arguments
            new_json = tool_call.to_json
            Log.low "Epoch: truncated tool call #{id} #{name} #{json.length} to #{new_json.length}"
            new_msg = msg.dup
            new_msg[:content] = new_json
            new_msg[:compacted] = true
            kept_messages << new_msg
            compacted_count += 1
          else
            kept_messages << msg
          end
        else
          kept_messages << msg
        end
      else
        # beyond compacted → drop
        Log.low "Epoch: dropped tool call #{id} #{name} #{json.length}"
        dropped_count += 1
      end
    else
      # assistant, system, meta, etc. → always keep
      kept_messages << msg
    end
  end

  kept_messages = kept_messages.reverse

	if dropped_count > 0 || compacted_count > 0
    Log.medium "Epoch strategy: pinned_total=#{pinned_total} new_calls=#{new_calls} " \
      "full=#{full} compacted=#{compacted} truncated=#{compacted_count} dropped=#{dropped_count} " \
      "protected=#{protected_positions.length}"

 compaction_message = {
role: :user,
content: <<~TEXT.chomp
 === Context Management ===

    To fit within the model context window, this conversation has been
    compacted: some tool calls arguments and tool call outputs are shown with
    compacted content, and some have been hidden entirely. Compacted content
    is show as '[CONTEXT-COMPACTED ...]'. Don't try to reconstruct what
    happened involving tools that have been compacted.

    Remember that the real tool call did not suffer any compactation at the
    time it was issued, it's only compacted in the current context.

    Compacted: #{compacted_count}
    Removed: #{dropped_count}
    
    Earlier tool results may no longer be present in the visible conversation
    history. If information appears to be missing, it may have been removed
    during context compaction rather than never existing. The absence of an
    earlier tool result in the current conversation does not necessarily mean
    that the tool has not already been executed. And care must be had accessing
    older tool calls which may be compacted.

    Repeated tool calls with the same arguments will be flagged and the very
    last instance will be protected from removal or compactation.
      TEXT
 }

    index = kept_messages.index do |msg|
      [:function_call, :function_call_output].include?(msg[:role].to_sym)
    end

    if index
      kept_messages.insert(index + 1, compaction_message)
    else
      kept_messages << compaction_message
    end
  end

  Chat.setup(kept_messages)
end

.tag(tag, content, name = nil) ⇒ Object 



2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
 
# File 'lib/scout/llm/chat/process/files.rb', line 2

def self.tag(tag, content, name = nil)
  if name
    <<-EOF.strip
<#{tag} name="#{name}">
#{content}
</#{tag}>
    EOF
  else
    <<-EOF.strip
<#{tag}>
#{content}
</#{tag}>
    EOF
  end
end

.tasks(messages, original = nil) ⇒ Object 



43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
 
# File 'lib/scout/llm/chat/process/tools.rb', line 43

def self.tasks(messages, original = nil)
  jobs =  []
  new = messages.collect do |message|
    if message[:role] == 'task' || message[:role] == 'inline_task' || message[:role] == 'exec_task'
      info = message[:content].strip

      workflow, task  = info.split(" ").values_at 0, 1

      options = IndiferentHash.parse_options info
      jobname = options.delete :jobname

      if String === workflow
        workflow = Chat.load_workflow workflow
      end

      job = workflow.job(task, jobname, options)

      jobs << job unless message[:role] == 'exec_task'

      if message[:role] == 'exec_task'
        result = job.exec
        result = result.to_s if TSV === result
        result = result.to_json unless String === result
        {role: 'user', content: result}
      elsif message[:role] == 'inline_task'
        {role: 'inline_job', content: job.path.find}
      else
        {role: 'job', content: job.path.find}
      end
    else
      message
    end
  end.flatten

  Workflow.produce(jobs) if jobs.any?

  new
end

.timestamp ⇒ Object 



772
773
774
 
# File 'lib/scout/llm/chat/provenance.rb', line 772

def self.timestamp
  Time.now.utc.iso8601(3)
end

.token_totals(chat_list) ⇒ Object

Sum direct token fields across a set of chats. Returns a hash keyed by symbol for every key in TOKEN_KEYS.



355
356
357
358
359
360
361
362
 
# File 'lib/scout/llm/chat/process/meta.rb', line 355

def self.token_totals(chat_list)
  totals = TOKEN_KEYS.each_with_object({}) { |k, h| h[k.to_sym] = 0 }
  direct_entries(chat_list).each do |entry|
    meta = entry[:meta]
    TOKEN_KEYS.each { |name| totals[name.to_sym] += meta[name].to_i }
  end
  totals
end

.tokens(root, **options) ⇒ Object

Provenance token aggregate. Receipt (agent_meta) child usage is now included through the event collector, so child inference paid inside a parent tool call is no longer invisible when the child chat was not saved.



768
769
770
 
# File 'lib/scout/llm/chat/provenance.rb', line 768

def self.tokens(root, **options)
  provenance_token_totals(root, **options)
end

.tool_call_status(call) ⇒ Object

Common, deliberately separate interpretation of a paired tool output. Missing output is unknown, JSON exceptions fail, and command-style JSON results fail when exit_status is non-zero.



57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
 
# File 'lib/scout/llm/chat/tool_calls.rb', line 57

def self.tool_call_status(call)
  return { success: nil, reason: :missing_output } unless call[:output_info]

  content = call[:output]
  parsed = begin
             JSON.parse(content.to_s)
           rescue JSON::ParserError, TypeError
             nil
           end

  if Hash === parsed && parsed['exception']
    { success: false, reason: :exception, exception: parsed['exception'] }
  elsif Hash === parsed && parsed.include?('exit_status')
    status = parsed['exit_status'].to_i
    { success: status == 0, reason: :exit_status, exit_status: status }
  else
    { success: true, reason: :output }
  end
end

.tool_calls(chat, source: nil) ⇒ Object

Pair tool calls with their outputs within one persisted chat. This reports structural facts only; use tool_call_status for the common interpretation of exception and exit-status outputs.



13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
 
# File 'lib/scout/llm/chat/tool_calls.rb', line 13

def self.tool_calls(chat, source: nil)
  calls = []
  outputs = Hash.new { |hash, key| hash[key] = [] }

  chat.each_with_index do |message, index|
    role = message[:role].to_s
    next unless %w[function_call mcp_call function_call_output].include?(role)
    info = parse_tool_message(message)
    next unless Hash === info

    call_id = info['id'] || info['call_id'] || info['tool_call_id']
    address = source ? [source.to_s, index] : index

    if role == 'function_call_output'
      outputs[call_id] << { address: address, index: index, message: message, info: info } if call_id
    else
      calls << {
        call_id: call_id,
        name: info['name'] || info.dig('function', 'name'),
        arguments: info['arguments'] || info.dig('function', 'arguments'),
        role: role.to_sym,
        call_address: address,
        call_index: index,
        call_message: message,
        call_info: info
      }
    end
  end

  calls.collect do |call|
    output = call[:call_id] && outputs[call[:call_id]].shift
    call.merge(
      output_address: output && output[:address],
      output_index: output && output[:index],
      output_message: output && output[:message],
      output_info: output && output[:info],
      output: output && output[:info]['content']
    )
  end
end

.tools(messages) ⇒ Object 



137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
 
# File 'lib/scout/llm/chat/process/tools.rb', line 137

def self.tools(messages)
  tool_definitions = IndiferentHash.setup({})
  introduced_workflows = []
  new = messages.collect do |message|
    role = message[:role]
    if role == 'mcp'
      url, *tools = content_tokens(message)

      if url == 'stdio'
        command = tools.shift
        mcp_tool_definitions = LLM.mcp_tools(url, command: command, url: nil, type: :stdio)
      else
        mcp_tool_definitions = LLM.mcp_tools(url)
      end

      if tools.any?
        tools.each do |tool|
          tool_definitions[tool] = mcp_tool_definitions[tool]
        end
      else
        tool_definitions.merge!(mcp_tool_definitions)
      end
      next
    elsif role == 'tool'
      workflow_name, task_name, *inputs = content_tokens(message)
      inputs = nil if inputs.empty?
      inputs = [] if inputs == ['none'] || inputs == ['noinputs']
      if Open.remote? workflow_name
        require 'rbbt'
        require 'scout/offsite/ssh'
        require 'rbbt/workflow/remote_workflow'
        workflow = RemoteWorkflow.new workflow_name
      else
        workflow = Chat.load_workflow workflow_name
      end

      if task_name
        definition = LLM.task_tool_definition workflow, task_name, inputs
        tool_definitions[task_name] = [workflow, definition]
      else
        tool_definitions.merge!(LLM.workflow_tools(workflow))
      end
      next
    elsif role == 'introduce'
      workflow_name = message[:content].to_s
      next if introduced_workflows.include? workflow_name
      introduced_workflows << workflow_name
      workflow = begin
                   Kernel.const_get workflow_name
                 rescue
                 end
      if Open.remote? workflow_name
        require 'rbbt'
        require 'scout/offsite/ssh'
        require 'rbbt/workflow/remote_workflow'
        workflow = RemoteWorkflow.new workflow_name
      else
        workflow = Workflow.require_workflow workflow_name
      end unless workflow

      raise "Workflow not found #{workflow_name}" if workflow.nil?

      next if workflow.documentation.empty?
      content = <<-EOF
You have access to tools from workflow '#{workflow.name}'. 
Below is the documentation of the workflow:

# #{workflow.documentation[:title]}

#{workflow.documentation[:description]}
      EOF

      {role: :user, content: content}
    elsif role == 'kb'
      knowledge_base_name, *databases = content_tokens(message)
      databases = nil if databases.empty?

      knowledge_base = KnowledgeBase.load knowledge_base_name

      if knowledge_base.all_databases.empty?
        agent = LLM.load_agent knowledge_base_name
        knowledge_base = agent.knowledge_base if agent.knowledge_base && agent.knowledge_base.all_databases.any?
      end

      knowledge_base_definition = LLM.knowledge_base_tool_definition(knowledge_base, databases)
      tool_definitions.merge!(knowledge_base_definition)
      next
    elsif role == 'clear_tools'
      tool_definitions = {}
    else
      message
    end
  end.compact.flatten
  messages.replace new
  tool_definitions
end

.trace_chat_sources(sources, deduplicate = true) ⇒ Object

Trace chats while preserving the persisted address of every meta and covered message. Sources may be a Hash of path => Chat or an Array of [path, Chat] pairs. The optional second argument keeps one entry per persisted location instead of collapsing copies (see Chat.trace_indices); it is positional because sources is naturally a brace-less Hash at call sites, which Ruby 3 would otherwise turn into keywords.



339
340
341
342
 
# File 'lib/scout/llm/chat/process/meta.rb', line 339

def self.trace_chat_sources(sources, deduplicate = true)
  pairs = sources.to_a
  trace_indices(pairs.collect { |source, chat| chat.message_index(source: source) }, deduplicate: deduplicate)
end

.trace_chats(chats) ⇒ Object 



329
330
331
 
# File 'lib/scout/llm/chat/process/meta.rb', line 329

def self.trace_chats(chats)
  trace_indices(chats.collect(&:message_index))
end

.trace_indices(indices, deduplicate: true) ⇒ Object

A meta starts a response segment. The segment continues until another meta, a new user/system turn, or the end of the chat. Consecutive and final metas with no covered messages remain as orphan records. Trace one or more message indexes into segment entries. By default entries are globally deduplicated across every supplied index (an inference copied into two chats yields one entry), which is the semantics Chat.token_totals has always had. Pass deduplicate: false when the caller needs every persisted evidence location preserved, e.g. Chat.provenance_token_events, which performs its own identity grouping so an event can list all of its evidence records.



280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
 
# File 'lib/scout/llm/chat/process/meta.rb', line 280

def self.trace_indices(indices, deduplicate: true)
  seen = Set.new
  trace = []
  add = lambda do |pending|
    return if pending.nil?
    inference_id = pending[:meta][:inference_id]
    deduplication = inference_id ? :inference_id : :legacy_lineage
    dedup_key = inference_id ? [:inference_id, inference_id] : [:lineage, pending[:id]]
    return if deduplicate && seen.include?(dedup_key)
    seen << dedup_key
    trace << {
      id: pending[:id],
      lineage_id: pending[:id],
      inference_id: inference_id,
      deduplication: deduplication,
      meta_address: pending[:address],
      meta: IndiferentHash.setup(pending[:meta].except(:reas)),
      messages: pending[:messages],
      message_addresses: pending[:message_addresses],
      orphan: pending[:messages].empty?
    }
  end

  indices.each do |index|
    pending = nil
    index.each do |info|
      case info[:role]
      when :meta
        add.call(pending)
        pending = {
          id: info[:id], meta: info[:meta], address: info[:address],
          messages: [], message_addresses: []
        }
      when :user, :system
        add.call(pending)
        pending = nil
      else
        if pending
          pending[:messages] << info[:id]
          pending[:message_addresses] << info[:address] if info[:address]
        end
      end
    end
    add.call(pending)
  end

  trace
end

.traverse_provenance(root, root_type: nil, follow: :all, on_error: nil, &block) ⇒ Object

Traverse the heterogeneous provenance graph using native Path and Step values. The block receives:

kind, object, parent_kind, parent, relation, first_visit, detail

kind is :chat or :job. Chat objects are persisted file Paths; jobs are Steps. The root has nil parent/relation. Every structural edge is yielded, including an edge to a node visited through another branch; first_visit is false in that case and the node is not expanded again.

detail is nil for every ordinary edge. For an :agent_job edge it is the agent_meta receipt record (Chat.agent_meta_job_references entry) that produced the edge, so the originating function output address, receipt address, call id and tool name stay auditable without re-parsing the parent chat. Blocks accepting only the first six arguments keep working; the trailing detail is only passed when the block can receive it (or accepts any number of arguments).

Relations describe root-outward discovery, not diagram arrow direction: chat -> producer job (:job), chat -> delegated-agent producer job (:agent_job, from agent_meta receipts), job -> dependency (:dependency), job -> log chat (:log), job -> result chat (:result), and chat -> log chat of its own .files sidecar (:log, saved agent conversations, excluding the root copy at .files/log/agent.chat).

Imported and continued chats are a chat-compilation concern, not a provenance concern. They are resolved during Chat.parse and their content is already inlined in the persisted chat file. Provenance traversal therefore never follows import, continue, or last references.

Raises:



207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
 
# File 'lib/scout/llm/chat/provenance.rb', line 207

def self.traverse_provenance(root, root_type: nil, follow: :all, on_error: nil, &block)
  return enum_for(__method__, root, root_type: root_type, follow: follow, on_error: on_error) unless block
  # Lambda blocks have strict arity; keep six-argument callbacks compatible
  # by only yielding the trailing detail when the block can receive it.
  detail_arity = lambda do
    return true if block.arity == -1
    block.arity >= 7
  end

  relations = follow == :all ? PROVENANCE_RELATIONS : Array(follow).collect(&:to_sym)
  unknown = relations - PROVENANCE_RELATIONS
  raise ParameterException, "Unknown provenance relations: #{unknown * ', '}" if unknown.any?

  root_kind = root_type && root_type.to_sym
  root_kind ||= Step === root ? :job : :chat
  root_object = if root_kind == :job
                  Step === root ? root : Step.load(root)
                else
                  path = File.expand_path(root.to_s)
                  raise ParameterException, "Chat not found: #{root}" unless File.file?(path)
                  Path.setup(path)
                end

  queue = [[root_kind, root_object, nil, nil, nil]]
  seen = Set.new

  until queue.empty?
    kind, object, parent_kind, parent, relation, detail = queue.shift
    key = provenance_key(kind, object)
    first_visit = !seen.include?(key)
    if detail_arity.call
      block.call(kind, object, parent_kind, parent, relation, first_visit, detail)
    else
      block.call(kind, object, parent_kind, parent, relation, first_visit)
    end
    next unless first_visit
    seen << key

    begin
      if kind == :chat
        chat = Chat.load(object)

        if relations.include?(:job)
          chat.jobs.each do |reference|
            begin
              job = load_job_reference(reference)
              queue << [:job, job, :chat, object, :job]
            rescue StandardError => error
              provenance_error(on_error, error, :chat, object, :job, reference)
            end
          end
        end

        if relations.include?(:agent_job)
          # Malformed receipts are diagnostics, never provenance; they are
          # reported through provenance_error and skipped.
          references = begin
                         report_agent_meta_problems(chat, object, on_error)
                       rescue StandardError => error
                         provenance_error(on_error, error, :chat, object, :agent_job, object)
                         []
                       end

          references.each do |record|
            reference = record[:job]
            begin
              job, _unresolved = resolve_job_reference(reference)
              if job
                # The receipt record travels with the edge as its detail so
                # the traversal keeps the originating output address, call
                # id and receipt index.
                queue << [:job, job, :chat, object, :agent_job, record]
              else
                error = ScoutException.new(
                  agent_meta_error_message(:unresolved_job_reference,
                                           chat_path: object.to_s,
                                           output_address: record[:output_address],
                                           call_id: record[:call_id],
                                           tool_name: record[:tool_name],
                                           reference: reference)
                )
                provenance_error(on_error, error, :chat, object, :agent_job,
                                 agent_meta_error_reference(reason: :unresolved_job_reference,
                                                            source: object.to_s,
                                                            output_address: record[:output_address],
                                                            evidence_address: record[:evidence_address],
                                                            call_id: record[:call_id],
                                                            tool_name: record[:tool_name],
                                                            reference: reference))
              end
            rescue StandardError => error
              provenance_error(on_error, error, :chat, object, :agent_job,
                               agent_meta_error_reference(reason: :unresolved_job_reference,
                                                          source: object.to_s,
                                                          reference: reference,
                                                          call_id: record[:call_id],
                                                          tool_name: record[:tool_name]))
            end
          end
        end

        if relations.include?(:log)
          # A saved chat owns its .files sidecar logs (society
          # conversations) just like a job owns its job logs.
          direct_chat_sidecar_files(object).each do |file|
            queue << [:chat, file, :chat, object, :log]
          end
        end
      else
        if relations.include?(:dependency)
          object.dependencies.each do |dependency|
            queue << [:job, dependency, :job, object, :dependency]
          end
        end

        if relations.include?(:log)
          direct_job_chat_files(object).each do |file|
            queue << [:chat, file, :job, object, :log]
          end
        end

        if relations.include?(:result) && (file = job_result_chat_file(object))
          queue << [:chat, file, :job, object, :result]
        end
      end
    rescue StandardError => error
      provenance_error(on_error, error, kind, object, relation, object)
    end
  end

  nil
end

Instance Method Details

#add_meta(key, value) ⇒ Object 



160
161
162
163
164
165
166
167
168
169
 
# File 'lib/scout/llm/chat/process/meta.rb', line 160

def add_meta(key, value)
  meta_msg = role_messages(:meta).last
  meta = meta_msg ? Chat.parse_meta(meta_msg[:content]) : {}
  meta[key] = value
  if meta_msg
    meta_msg[:content] = Chat.serialize_meta(meta)
  else
    message :meta, Chat.serialize_meta(meta)
  end
end

#answer ⇒ Object 



216
217
218
 
# File 'lib/scout/llm/chat/annotation.rb', line 216

def answer
  final[:content]
end

#append(coda) ⇒ Object 



129
130
131
132
133
134
 
# File 'lib/scout/llm/chat/annotation.rb', line 129

def append(coda)
  coda = [coda] if Hash === coda
  new = Chat.follow(self.dup, coda)
  self.replace new
  self
end

#ask(options = {}) ⇒ Object 



99
100
101
 
# File 'lib/scout/llm/chat/annotation.rb', line 99

def ask(options = {})
  LLM.ask(LLM.chat(self), options)
end

#assistant(content) ⇒ Object 



18
19
20
 
# File 'lib/scout/llm/chat/annotation.rb', line 18

def assistant(content)
  message(:assistant, content)
end

#association(name, path, options = {}) ⇒ Object 



88
89
90
91
92
 
# File 'lib/scout/llm/chat/annotation.rb', line 88

def association(name, path, options = {})
  options_str = IndiferentHash.print_options options
  content = [name, path, options_str]*" "
  message(:association, name)
end

#branch ⇒ Object 



178
179
180
 
# File 'lib/scout/llm/chat/annotation.rb', line 178

def branch
  self.annotate self.dup
end

#chat(options = {}) ⇒ Object 



103
104
105
106
107
108
109
110
111
112
113
 
# File 'lib/scout/llm/chat/annotation.rb', line 103

def chat(options = {})
  options = IndiferentHash.add_defaults options, return_messages: true
  response = ask(options)
  if Array === response
    self.concat(response)
    answer
  else
    self.push({role: :assistant, content: response})
    response
  end
end

#continue(file) ⇒ Object 



47
48
49
 
# File 'lib/scout/llm/chat/annotation.rb', line 47

def continue(file)
  message(:continue, file)
end

#create_image(file) ⇒ Object

Image



250
251
252
253
 
# File 'lib/scout/llm/chat/annotation.rb', line 250

def create_image(file, ...)
  base64_image = LLM.image(LLM.chat(self), ...)
  Open.write(file, Base64.decode64(base64_image), mode: 'wb')
end

#directory(directory) ⇒ Object 



43
44
45
 
# File 'lib/scout/llm/chat/annotation.rb', line 43

def directory(directory)
  message(:directory, directory)
end

#endpoint(value) ⇒ Object 



186
187
188
 
# File 'lib/scout/llm/chat/annotation.rb', line 186

def endpoint(value)
  option :endpoint, value
end

#exec_task(workflow, task_name, inputs = {}) ⇒ Object 



66
67
68
69
70
 
# File 'lib/scout/llm/chat/annotation.rb', line 66

def exec_task(workflow, task_name, inputs = {})
  input_str = IndiferentHash.print_options inputs
  content = [workflow, task_name, input_str]*" "
  message(:exec_task, content)
end

#file(file) ⇒ Object 



30
31
32
 
# File 'lib/scout/llm/chat/annotation.rb', line 30

def file(file)
  message(:file, file)
end

#final ⇒ Object 



204
205
206
 
# File 'lib/scout/llm/chat/annotation.rb', line 204

def final
  LLM.purge(self).last
end

#follow(coda) ⇒ Object 



136
137
138
139
140
141
 
# File 'lib/scout/llm/chat/annotation.rb', line 136

def follow(coda)
  coda = [coda] if Hash === coda
  new = Chat.follow(self.dup, coda)
  self.replace new
  self
end

#format(format) ⇒ Object 



51
52
53
 
# File 'lib/scout/llm/chat/annotation.rb', line 51

def format(format)
  message(:format, format)
end

#image(file) ⇒ Object 



194
195
196
 
# File 'lib/scout/llm/chat/annotation.rb', line 194

def image(file)
  self.message :image, file
end

#import(file) ⇒ Object 



22
23
24
 
# File 'lib/scout/llm/chat/annotation.rb', line 22

def import(file)
  message(:import, file)
end

#import_last(file) ⇒ Object 



26
27
28
 
# File 'lib/scout/llm/chat/annotation.rb', line 26

def import_last(file)
  message(:last, file)
end

#inline_job(step) ⇒ Object 



83
84
85
 
# File 'lib/scout/llm/chat/annotation.rb', line 83

def inline_job(step)
  message(:inline_job, step.path)
end

#inline_task(workflow, task_name, inputs = {}) ⇒ Object 



73
74
75
76
77
 
# File 'lib/scout/llm/chat/annotation.rb', line 73

def inline_task(workflow, task_name, inputs = {})
  input_str = IndiferentHash.print_options inputs
  content = [workflow, task_name, input_str]*" "
  message(:inline_task, content)
end

#introduce(workflow) ⇒ Object 



34
35
36
 
# File 'lib/scout/llm/chat/annotation.rb', line 34

def introduce(workflow)
  message(:introduce, workflow)
end

#job(step) ⇒ Object 



79
80
81
 
# File 'lib/scout/llm/chat/annotation.rb', line 79

def job(step)
  message(:job, step.path)
end

#job_agent_chat_files ⇒ Object

ScoutCoder: DUAL-LAYOUT filter (see Chat::DIRECT_LOG_CHAT_GLOBS). Agent chats are the NEW layout files living under the job's own files dir (.files/.chat and .files/.society/**, which no longer have a log/ component) plus the LEGACY .files/log/ tree written by older scout-ai. Other jobs' second-order .files trees are excluded by requiring the file to be under THIS job's files dir.



228
229
230
231
232
233
234
235
 
# File 'lib/scout/llm/chat/process/meta.rb', line 228

def job_agent_chat_files
  jobs.flat_map do |job|
    job_files = job.path.to_s + '.files'
    Chat.provenance_chat_files(job, root_type: :job).select do |file|
      file.include?('.files/log/') || file.start_with?(job_files + '/')
    end
  end.uniq
end

#job_agent_chats ⇒ Object 



241
242
243
 
# File 'lib/scout/llm/chat/process/meta.rb', line 241

def job_agent_chats
  job_agent_chat_files.collect { |file| Chat.load(file) }
end

#job_chat_files ⇒ Object 



218
219
220
 
# File 'lib/scout/llm/chat/process/meta.rb', line 218

def job_chat_files
  jobs.flat_map { |job| Chat.job_chat_files(job) }.uniq
end

#job_chats ⇒ Object 



237
238
239
 
# File 'lib/scout/llm/chat/process/meta.rb', line 237

def job_chats
  job_chat_files.collect { |file| Chat.load(file) }
end

#job_paths ⇒ Object Also known as: jobs 



177
178
179
180
181
 
# File 'lib/scout/llm/chat/process/meta.rb', line 177

def job_paths
  role_messages(:meta).collect do |message|
    Path.setup(Chat.parse_meta(message[:content])[:job])
  end.compact.uniq
end

#json(*args, only_ask: false, **kwargs) ⇒ Object 



156
157
158
159
160
161
162
163
164
165
 
# File 'lib/scout/llm/chat/annotation.rb', line 156

def json(*args, only_ask: false, **kwargs)
  self.format :json
  output = only_ask ? ask(*args, **kwargs) : chat(*args, **kwargs)
  obj = Chat.parse_json output
  if (Hash === obj) and obj.keys == ['content']
    obj['content']
  else
    obj
  end
end

#json_format(fornat, *args, only_ask: false, **kwargs) ⇒ Object 



167
168
169
170
171
172
173
174
175
176
 
# File 'lib/scout/llm/chat/annotation.rb', line 167

def json_format(fornat, *args, only_ask: false, **kwargs)
  self.format format
  output = only_ask ? ask(*args, **kwargs) : chat(*args, **kwargs)
  obj = Chat.parse_json output
  if (Hash === obj) and obj.keys == ['content']
    obj['content']
  else
    obj
  end
end

#last_job ⇒ Object

Return the last job reference from the meta messages in this chat.



264
265
266
267
268
269
 
# File 'lib/scout/llm/chat/annotation.rb', line 264

def last_job
  meta_msg = self.reverse.find{|info| info[:role].to_s == "meta" }
  return nil if meta_msg.nil?
  meta = Chat.parse_meta(meta_msg[:content])
  meta[:job]
end

#message(role, content) ⇒ Object 



6
7
8
 
# File 'lib/scout/llm/chat/annotation.rb', line 6

def message(role, content)
  self.append({role: role.to_s, content: content})
end

#message_index(source: nil) ⇒ Object

A lineage id identifies a message in its non-meta conversational history. Meta is deliberately excluded from that history: it starts a response segment but is not provider input.



248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
 
# File 'lib/scout/llm/chat/process/meta.rb', line 248

def message_index(source: nil)
  previous = nil
  each_with_index.collect do |message, position|
    role = message[:role].to_s
    content = message[:content].to_s
    id = Misc.digest([previous, role, content])
    info = {
      id: id,
      role: role.to_sym,
      prev: previous,
      fingerprint: Log.truncate_string(content)
    }
    info[:address] = [source.to_s, position] if source
    if role == 'meta'
      info[:meta] = Chat.parse_meta(content)
    else
      previous = id
    end
    info
  end
end

#meta ⇒ Object 



171
172
173
174
175
 
# File 'lib/scout/llm/chat/process/meta.rb', line 171

def meta
  meta_msg = role_messages(:meta).last
  return {} if meta_msg.nil?
  Chat.parse_meta(meta_msg[:content])
end

#model(value) ⇒ Object 



190
191
192
 
# File 'lib/scout/llm/chat/annotation.rb', line 190

def model(value)
  option :model, value
end

#option(name, value) ⇒ Object 



182
183
184
 
# File 'lib/scout/llm/chat/annotation.rb', line 182

def option(name, value)
  self.message 'option', [name, value] * " "
end

#pdf(file) ⇒ Object 



38
39
40
 
# File 'lib/scout/llm/chat/annotation.rb', line 38

def pdf(file)
  message(:pdf, file)
end

#prepend(intro) ⇒ Object 



143
144
145
146
147
148
 
# File 'lib/scout/llm/chat/annotation.rb', line 143

def prepend(intro)
  into = [intro] if Hash === intro
  new = Chat.follow(intro, self.dup)
  self.replace new
  self
end

#print ⇒ Object

Reporting



200
201
202
 
# File 'lib/scout/llm/chat/annotation.rb', line 200

def print
  LLM.print LLM.chat(self)
end

#purge ⇒ Object 



208
209
210
 
# File 'lib/scout/llm/chat/annotation.rb', line 208

def purge
  Chat.setup(LLM.purge(self))
end

#remove_role(role = :clear) ⇒ Object 



150
151
152
153
154
 
# File 'lib/scout/llm/chat/annotation.rb', line 150

def remove_role(role=:clear)
  messages = self.select{|m| m[:role].to_s == role.to_s }
  self.reject!{|m| m[:role].to_s == role.to_s }
  messages
end

#role_messages(role, &block) ⇒ Object 



255
256
257
258
259
260
261
 
# File 'lib/scout/llm/chat/annotation.rb', line 255

def role_messages(role, &block)
  if Symbol === role
    self.select{|msg| msg[:role].to_sym == role }
  else
    self.select{|msg| msg[:role].to_s == role }
  end
end

#save(path, force = true) ⇒ Object

Write and save



222
223
224
225
226
227
228
229
 
# File 'lib/scout/llm/chat/annotation.rb', line 222

def save(path, force = true)
  path = path.to_s if Symbol === path
  if not (Open.exists?(path) || Path === path || Path.located?(path))
    path = Scout.chats.find[path]
  end
  return if Open.exists?(path) && ! force
  Open.write path, LLM.print(self)
end

#shed ⇒ Object 



212
213
214
 
# File 'lib/scout/llm/chat/annotation.rb', line 212

def shed
  self.annotate [final]
end

#system(content) ⇒ Object 



14
15
16
 
# File 'lib/scout/llm/chat/annotation.rb', line 14

def system(content)
  message(:system, content)
end

#tag(content, name = nil, tag = :file, role = :user) ⇒ Object 



94
95
96
 
# File 'lib/scout/llm/chat/annotation.rb', line 94

def tag(content, name=nil, tag=:file, role=:user)
  self.message role, Chat.tag(tag, content, name)
end

#task(workflow, task_name, inputs = {}) ⇒ Object 



60
61
62
63
64
 
# File 'lib/scout/llm/chat/annotation.rb', line 60

def task(workflow, task_name, inputs = {})
  input_str = IndiferentHash.print_options inputs
  content = [workflow, task_name, input_str]*" "
  message(:task, content)
end

#tool(*parts) ⇒ Object 



55
56
57
58
 
# File 'lib/scout/llm/chat/annotation.rb', line 55

def tool(*parts)
  content = parts * "\n"
  message(:tool, content)
end

#tooling ⇒ Object 



260
261
262
 
# File 'lib/scout/llm/chat/process/tools.rb', line 260

def tooling
  self.select{|msg| [:introduce, :tool, :mcp, :kb].include?(msg[:role].to_sym) }
end

#user(content) ⇒ Object 



10
11
12
 
# File 'lib/scout/llm/chat/annotation.rb', line 10

def user(content)
  message(:user, content)
end

#write(path, force = true) ⇒ Object 



231
232
233
234
235
236
237
238
 
# File 'lib/scout/llm/chat/annotation.rb', line 231

def write(path, force = true)
  path = path.to_s if Symbol === path
  if not (Open.exists?(path) || Path === path || Path.located?(path))
    path = Scout.chats.find[path]
  end
  return if Open.exists?(path) && ! force
  Open.write path, self.print
end

#write_answer(path, force = true) ⇒ Object 



240
241
242
243
244
245
246
247
 
# File 'lib/scout/llm/chat/annotation.rb', line 240

def write_answer(path, force = true)
  path = path.to_s if Symbol === path
  if not (Open.exists?(path) || Path === path || Path.located?(path))
    path = Scout.chats.find[path]
  end
  return if Open.exists?(path) && ! force
  Open.write path, self.answer
end