Class: Rspider::MysqlUrlRelationStorage

Inherits:
Object
  • Object
show all
Defined in:
lib/rspider/MysqlUrlRelationStorage.rb

Overview

The class MysqlUrlRelationStorage store url relations in Mysql database For better performance, we create an UrlStorage object to cache urls in memory

Instance Method Summary collapse

Constructor Details

#initialize(hash, source = "default") ⇒ MysqlUrlRelationStorage

Returns a new instance of MysqlUrlRelationStorage.

Raises:



8
9
10
11
12
13
 
# File 'lib/rspider/MysqlUrlRelationStorage.rb', line 8

def initialize(hash,source="default")
	@seed=1024
	@source=source
	@my=Mysql::new(hash["host"],hash["user"],hash["pass"],hash["db"])
	raise MysqlException if @my.nil?
end

Instance Method Details

#md5(string) ⇒ Object

get the MD5 hash of string param “string”



15
16
17
18
19
 
# File 'lib/rspider/MysqlUrlRelationStorage.rb', line 15

def md5(string)
	t=Digest::MD5.new 
	t << string
	t.to_s
end

#save(referer, url) ⇒ Object 



20
21
22
23
24
25
26
27
28
29
 
# File 'lib/rspider/MysqlUrlRelationStorage.rb', line 20

def save(referer,url)
	url_crc=Zlib::crc32(url,@seed)
	referer_crc=Zlib::crc32(referer,@seed)
	sql="INSERT INTO url_relations (url,referer,url_crc32,referer_crc32) values('#{@my.quote(url)}','#{@my.quote(referer)}','#{url_crc}','#{referer_crc}')"
	begin
	@my.query(sql)
	rescue Mysql::Error
	rescue
	end
end