Class: MiGA::Dataset

Inherits:
MiGA
  • Object
show all
Defined in:
lib/miga/dataset.rb

Overview

Dataset representation in MiGA.

Constant Summary collapse

@@RESULT_DIRS = 
{
  # Preprocessing
  raw_reads: "01.raw_reads", trimmed_reads: "02.trimmed_reads",
  read_quality: "03.read_quality", trimmed_fasta: "04.trimmed_fasta",
  assembly: "05.assembly", cds: "06.cds",
  # Annotation
  essential_genes: "07.annotation/01.function/01.essential",
  ssu: "07.annotation/01.function/02.ssu",
  mytaxa: "07.annotation/02.taxonomy/01.mytaxa",
  mytaxa_scan: "07.annotation/03.qa/02.mytaxa_scan",
  # Mapping
  mapping_on_contigs: "08.mapping/01.read-ctg",
  mapping_on_genes: "08.mapping/02.read-gene",
  # Distances (for single-species datasets)
  distances: "09.distances"
}
@@KNOWN_TYPES = 
{
  genome: {description: "The genome from an isolate.", multi: false},
  metagenome: {description: "A metagenome (excluding viromes).",
    multi: true},
  virome: {description: "A viral metagenome.", multi: true},
  scgenome: {description: "A genome from a single cell.", multi: false},
  popgenome: {description: "The genome of a population (including " +
    "microdiversity).", :multi=>false}
}
@@PREPROCESSING_TASKS = 
[:raw_reads, :trimmed_reads, :read_quality,
:trimmed_fasta, :assembly, :cds, :essential_genes, :ssu, :mytaxa,
:mytaxa_scan, :distances]
@@EXCLUDE_NOREF_TASKS =

Tasks to be excluded from query datasets.

[:essential_genes, :mytaxa_scan]
@@ONLY_NONMULTI_TASKS =

Tasks to be executed only in datasets that are not multi-organism. These tasks are ignored for multi-organism datasets or for unknown types.

[:mytaxa_scan, :distances]
@@ONLY_MULTI_TASKS =

Tasks to be executed only in datasets that are multi-organism. These tasks are ignored for single-organism datasets or for unknwon types.

[:mytaxa]

Constants included from MiGA

CITATION, VERSION, VERSION_DATE, VERSION_NAME

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from MiGA

CITATION, DEBUG, DEBUG_OFF, DEBUG_ON, DEBUG_TRACE_OFF, DEBUG_TRACE_ON, FULL_VERSION, LONG_VERSION, VERSION, VERSION_DATE, initialized?, #result_files_exist?, root_path, tabulate

Constructor Details

#initialize(project, name, is_ref = true, metadata = {}) ⇒ Dataset

Create a MiGA::Dataset object in a project MiGA::Project with a uniquely identifying name. is_ref indicates if the dataset is to be treated as reference (true, default) or query (false). Pass any additional metadata as a Hash.



98
99
100
101
102
103
104
105
106
 
# File 'lib/miga/dataset.rb', line 98

def initialize(project, name, is_ref=true, metadata={})
  raise "Invalid name '#{name}', please use only alphanumerics and " +
    "underscores." unless name.miga_name?
  @project = project
  @name = name
  metadata[:ref] = is_ref
  @metadata = MiGA::Metadata.new(project.path + "/metadata/" + name + ".json",
    metadata)
end

Instance Attribute Details

#metadata ⇒ Object (readonly)

MiGA::Metadata with information about the dataset.



91
92
93
 
# File 'lib/miga/dataset.rb', line 91

def metadata
  @metadata
end

#name ⇒ Object (readonly)

Datasets are uniquely identified by name in a project.



87
88
89
 
# File 'lib/miga/dataset.rb', line 87

def name
  @name
end

#project ⇒ Object (readonly)

MiGA::Project that contains the dataset.



83
84
85
 
# File 'lib/miga/dataset.rb', line 83

def project
  @project
end

Class Method Details

.exist?(project, name) ⇒ Boolean

Does the project already have a dataset with that name?

Returns:

  • (Boolean) 


69
70
71
 
# File 'lib/miga/dataset.rb', line 69

def self.exist?(project, name)
  File.exist? project.path + "/metadata/" + name + ".json"
end

.INFO_FIELDS ⇒ Object

Standard fields of metadata for datasets.



75
76
77
 
# File 'lib/miga/dataset.rb', line 75

def self.INFO_FIELDS
  %w(name created updated type ref user description comments)
end

.KNOWN_TYPES ⇒ Object

Supported dataset types.



35
 
# File 'lib/miga/dataset.rb', line 35

def self.KNOWN_TYPES ; @@KNOWN_TYPES end

.PREPROCESSING_TASKS ⇒ Object

Returns an Array of tasks to be executed before project-wide tasks.



48
 
# File 'lib/miga/dataset.rb', line 48

def self.PREPROCESSING_TASKS ; @@PREPROCESSING_TASKS ; end

.RESULT_DIRS ⇒ Object

Directories containing the results from dataset-specific tasks.



15
 
# File 'lib/miga/dataset.rb', line 15

def self.RESULT_DIRS ; @@RESULT_DIRS end

Instance Method Details

#add_result(result_type, save = true) ⇒ Object

Look for the result with symbol key result_type and register it in the dataset. If save is false, it doesn’t register the result, but it still returns a result if the expected files are complete. Returns MiGA::Result or nil.



175
176
177
178
179
180
181
182
183
184
 
# File 'lib/miga/dataset.rb', line 175

def add_result(result_type, save=true)
  return nil if @@RESULT_DIRS[result_type].nil?
  base = project.path + "/data/" + @@RESULT_DIRS[result_type] +
    "/" + name
  return MiGA::Result.load(base + ".json") unless save
  return nil unless result_files_exist?(base, ".done")
  r = self.send("add_result_#{result_type}", base)
  r.save unless r.nil?
  r
end

#done_preprocessing?(save = false) ⇒ Boolean

Are all the dataset-specific tasks done? Passes save to #add_result.

Returns:

  • (Boolean) 


219
220
221
 
# File 'lib/miga/dataset.rb', line 219

def done_preprocessing?(save=false)
  !first_preprocessing(save).nil? and next_preprocessing(save).nil?
end

#each_result(&blk) ⇒ Object

For each result executes the 2-ary blk block: key symbol and MiGA::Result.



164
165
166
167
168
 
# File 'lib/miga/dataset.rb', line 164

def each_result(&blk)
  @@RESULT_DIRS.keys.each do |k|
    blk.call(k, result(k)) unless result(k).nil?
  end
end

#first_preprocessing(save = false) ⇒ Object

Returns the key symbol of the first registered result (sorted by the execution order). This typically corresponds to the result used as the initial input. Passes save to #add_result.



190
191
192
 
# File 'lib/miga/dataset.rb', line 190

def first_preprocessing(save=false)
  @@PREPROCESSING_TASKS.find{ |t| not add_result(t, save).nil? }
end

#ignore_task?(task) ⇒ Boolean

Should I ignore task for this dataset?

Returns:

  • (Boolean) 


211
212
213
214
215
 
# File 'lib/miga/dataset.rb', line 211

def ignore_task?(task)
  ( (@@EXCLUDE_NOREF_TASKS.include?(task) and not is_ref?) or
    (@@ONLY_MULTI_TASKS.include?(task) and not is_multi?) or
    (@@ONLY_NONMULTI_TASKS.include?(task) and not is_nonmulti?))
end

#info ⇒ Object

Get standard metadata values for the dataset as Array.



126
127
128
129
130
 
# File 'lib/miga/dataset.rb', line 126

def info
  MiGA::Dataset.INFO_FIELDS.map do |k|
    (k=="name") ? self.name : self.metadata[k.to_sym]
  end
end

#is_multi? ⇒ Boolean

Is this dataset known to be multi-organism?

Returns:

  • (Boolean) 


138
139
140
141
 
# File 'lib/miga/dataset.rb', line 138

def is_multi?
  return false if self.metadata[:type].nil?
  return @@KNOWN_TYPES[self.metadata[:type]][:multi]
end

#is_nonmulti? ⇒ Boolean

Is this dataset known to be single-organism?

Returns:

  • (Boolean) 


145
146
147
148
 
# File 'lib/miga/dataset.rb', line 145

def is_nonmulti?
  return false if self.metadata[:type].nil?
  return !@@KNOWN_TYPES[self.metadata[:type]][:multi]
end

#is_ref? ⇒ Boolean

Is this dataset a reference?

Returns:

  • (Boolean) 


134
 
# File 'lib/miga/dataset.rb', line 134

def is_ref? ; !!self.metadata[:ref] ; end

#next_preprocessing(save = false) ⇒ Object

Returns the key symbol of the next task that needs to be executed. Passes save to #add_result.



197
198
199
200
201
202
203
204
205
206
207
 
# File 'lib/miga/dataset.rb', line 197

def next_preprocessing(save=false)
  after_first = false
  first = first_preprocessing(save)
  return nil if first.nil?
  @@PREPROCESSING_TASKS.each do |t|
    next if ignore_task? t
    return t if after_first and add_result(t, save).nil?
    after_first = (after_first or (t==first))
  end
  nil
end

#profile_advance(save = false) ⇒ Object

Returns an array indicating the stage of each task (sorted by execution order). The values are integers:

  • 0 for an undefined result (a task before the initial input).

  • 1 for a registered result (a completed task).

  • 2 for a queued result (a task yet to be executed).

It passes save to #add_result



230
231
232
233
234
235
236
237
238
239
240
241
242
 
# File 'lib/miga/dataset.rb', line 230

def profile_advance(save=false)
  first_task = first_preprocessing(save)
  return Array.new(@@PREPROCESSING_TASKS.size, 0) if first_task.nil?
  adv = []
  state = 0
  next_task = next_preprocessing(save)
  @@PREPROCESSING_TASKS.each do |task|
    state = 1 if first_task==task
    state = 2 if !next_task.nil? and next_task==task
    adv << state
  end
  adv
end

#remove! ⇒ Object

Delete the dataset with all it’s contents (including results) and returns nil.



119
120
121
122
 
# File 'lib/miga/dataset.rb', line 119

def remove!
  self.results.each{ |r| r.remove! }
  self.metadata.remove!
end

#result(k) ⇒ Object

Get the result MiGA::Result in this dataset identified by the symbol k.



152
153
154
155
156
 
# File 'lib/miga/dataset.rb', line 152

def result(k)
  return nil if @@RESULT_DIRS[k.to_sym].nil?
  MiGA::Result.load(project.path + "/data/" + @@RESULT_DIRS[k.to_sym] +
    "/" + name + ".json")
end

#results ⇒ Object

Get all the results (Array of MiGA::Result) in this dataset.



160
 
# File 'lib/miga/dataset.rb', line 160

def results ; @@RESULT_DIRS.keys.map{ |k| result k }.compact ; end

#save ⇒ Object

Save any changes you’ve made in the dataset.



110
111
112
113
114
 
# File 'lib/miga/dataset.rb', line 110

def save
  self.metadata[:type] = :metagenome if !metadata[:tax].nil? and
    !metadata[:tax][:ns].nil? and metadata[:tax][:ns]=="COMMUNITY"
  self.metadata.save
end