Class: COS::Slice
- Inherits:
-
Checkpoint
- Object
- COS::Struct::Base
- Checkpoint
- COS::Slice
- Includes:
- Logging
- Defined in:
- lib/cos/slice.rb
Overview
分片大文件上传, 支持断点续传, 支持多线程
Constant Summary collapse
- DEFAULT_SLICE_SIZE =
默认分片大小 3M
3 * 1024 * 1024
Constants included from Logging
Logging::DEFAULT_LOG_FILE, Logging::MAX_NUM_LOG, Logging::ROTATE_SIZE
Constants inherited from Checkpoint
Instance Attribute Summary collapse
-
#cpt_file ⇒ Object
Returns the value of attribute cpt_file.
-
#offset ⇒ Object
Returns the value of attribute offset.
-
#result ⇒ Object
Returns the value of attribute result.
-
#session ⇒ Object
Returns the value of attribute session.
-
#slice_size ⇒ Object
Returns the value of attribute slice_size.
Instance Method Summary collapse
-
#checkpoint ⇒ Object
断点续传状态记录.
-
#initialize(opts = {}) ⇒ Slice
constructor
A new instance of Slice.
-
#upload ⇒ Object
开始上传.
Methods included from Logging
Methods included from COS::Struct::Base::AttrHelper
#optional_attrs, #required_attrs
Constructor Details
#initialize(opts = {}) ⇒ Slice
Returns a new instance of Slice.
20 21 22 23 24 |
# File 'lib/cos/slice.rb', line 20 def initialize(opts = {}) super(opts) @cpt_file = [:cpt_file] || "#{File.(file_src)}.cpt" end |
Instance Attribute Details
#cpt_file ⇒ Object
Returns the value of attribute cpt_file.
18 19 20 |
# File 'lib/cos/slice.rb', line 18 def cpt_file @cpt_file end |
#offset ⇒ Object
Returns the value of attribute offset.
18 19 20 |
# File 'lib/cos/slice.rb', line 18 def offset @offset end |
#result ⇒ Object
Returns the value of attribute result.
18 19 20 |
# File 'lib/cos/slice.rb', line 18 def result @result end |
#session ⇒ Object
Returns the value of attribute session.
18 19 20 |
# File 'lib/cos/slice.rb', line 18 def session @session end |
#slice_size ⇒ Object
Returns the value of attribute slice_size.
18 19 20 |
# File 'lib/cos/slice.rb', line 18 def slice_size @slice_size end |
Instance Method Details
#checkpoint ⇒ Object
断点续传状态记录
103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 |
# File 'lib/cos/slice.rb', line 103 def checkpoint logger.debug("Make checkpoint, options[:disable_cpt]: #{[:disable_cpt] == true}") ensure_file_not_changed parts = sync_get_all_parts states = { :session => session, :slice_size => slice_size, :offset => offset, :file => file_src, :file_meta => @file_meta, :parts => parts } done = parts.count { |p| p[:done] } # 上传进度回调 if progress percent = (offset + done*slice_size).to_f / states[:file_meta][:size] progress.call(percent > 1 ? 1.to_f : percent) end write_checkpoint(states, cpt_file) unless [:disable_cpt] logger.debug("Upload Parts #{done}/#{states[:parts].count}") end |
#upload ⇒ Object
开始上传
27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 |
# File 'lib/cos/slice.rb', line 27 def upload logger.info("Begin upload, file: #{file_src}, threads: #{@num_threads}") # 重建断点续传或重新从服务器初始化分片上传 # 有可能sha命中直接完成 data = rebuild return data if data # 文件分片 divide_parts if @parts.empty? # 未完成的片段 @todo_parts = @parts.reject { |p| p[:done] } # 多线程上传 Thread.abort_on_exception = true threads = [] @num_threads.times do threads << Thread.new do loop do # 获取下一个未上传的片段 p = sync_get_todo_part break unless p # 上传片段 upload_part(p) end end end threads.each do |thread| begin thread.join rescue => error unless finish? # 部分服务端异常需要重新初始化, 可能上传已经完成了 if error.is_a?(ServerError) and error.error_code == -288 File.delete(cpt_file) unless [:disable_cpt] end threads.each {|t| t.exit} raise error end end end # 返回100%的进度 progress.call(1.to_f) if progress # 上传完成, 删除checkpoint文件 File.delete(cpt_file) unless [:disable_cpt] logger.info("Done upload, file: #{@file_src}") # 返回文件信息 result end |