2. Data Preparation

The Data Fetchers is a crawler module of pricing data, fundamental data and other market data like market value. Fetchers allow us to crawl all of the data we will need from Tushare or other sources.

2.1. Proxy Pool

To get fundamental Data, we need set up a proxy pool for our Fundcrawler.

Some recommended proxy pool project:

dungproxy	https://github.com/virjar/dungproxy
proxyspider	https://github.com/zhangchenchen/proxyspider
ProxyPool	https://github.com/henson/ProxyPool
ProxyPool	https://github.com/WiseDoge/ProxyPool
IPProxyTool	https://github.com/awolfly9/IPProxyTool
IPProxyPool	https://github.com/qiyeboy/IPProxyPool
proxy_list	https://github.com/gavin66/proxy_list
proxy_pool	https://github.com/lujqme/proxy_pool
haipproxy	https://github.com/SpiderClub/haipproxy
proxy_pool	https://github.com/jhao104/proxy_pool

Note

factorset uses proxy_pool, see in Proxy_start. You may also install redis for your proxy pool.

2.2. Configuration file

[TARGET]
;all = all
;all = hs300
all = 000001.SZ, 000002.SZ

[OPTIONS]
MONGO = False
CSV = True
;succ and fail list of symbol
SFL = True

[STORE]
hqdir = ./hq
;Only fund crawler will use proxy_pool
funddir = ./fund
otherdir =  ./other

[ARCTIC]
;host = 127.0.0.1:27017
host = localhost

[READ]
proxypool = 127.0.0.1:5010
proxymin = 5
encode = gbk

2.3. Run Fetcher

from factorset.Run import data_fetch
data_fetch.data_fetch()

Start Fetching Stock & Index Data!
000001.SZ写入完成
000002.SZ写入完成
Finish Fetching Stock & Index Data!

Start Fetching Other Data!
2017-06-15写入完成
2017-06-16写入完成
2017-06-19写入完成
2017-06-20写入完成
2017-06-21写入完成
2017-06-22写入完成
....
[Getting data:]##################
Finish Fetching Other Data!

Start Fetching Fundamental Data!
proxy: http://xxxx
proxy: http://xxxx
Put 000001.SZ in queue!
proxy: http://xxxx
Put 000001.SZ in queue!
proxy: http://xxxx
000002.SZ写入84条数据,
000001.SZ写入88条数据,
BS表数据导入成功！

proxy: http://xxxx
proxy: http://xxxx
Cannot connect to host xxxx ssl:False [Connect call failed ('xxxx', xxxx)]
Put 000001.SZ in queue!
proxy: http://xxxx
Put 000001.SZ in queue!
proxy: http://xxxx
000002.SZ写入85条数据,
000001.SZ写入91条数据,
IS表数据导入成功！

proxy: http://xxxx
proxy: http://xxxx
Put 000001.SZ in queue!
Put 000002.SZ in queue!
proxy: http://xxxx
proxy: http://xxxx
000001.SZ写入72条数据,
000002.SZ写入71条数据,
CF表数据导入成功！

Finish Fetching Fundamental Data!

Note

If the number of proxies in your proxy pool is less than proxy min you set in the CONFIG factorset uses proxy_pool, see in Proxy_start. You may also install redis for your proxy pool.

import os
from factorset.data import CSVParser as cp
cp.concat_fund(os.path.abspath('.'), cp.all_fund_symbol(os.path.abspath('.'), 'BS'), 'BS').loc[:,['ticker', 121, 101, 68, 109, 0]]

date	ticker	121	101	68	109	0
2018/3/31	000002.SZ	1.22E+12	8.81E+11	3.42E+09	7.80E+09	4.11E+10
2017/12/31	000002.SZ	1.17E+12	8.47E+11	3.33E+09	1.61E+10	4.62E+10
2017/9/30	000002.SZ	1.02E+12	7.43E+11	2.79E+09	1.73E+10	3.69E+10
2017/6/30	000002.SZ	9.29E+11	6.76E+11	2.22E+09	1.34E+10	3.67E+10
2017/3/31	000002.SZ	8.87E+11	6.28E+11	2.63E+09	1.59E+10	3.25E+10
2016/12/31	000002.SZ	8.31E+11	5.80E+11	3.60E+09	1.66E+10	2.68E+10
2016/9/30	000002.SZ	7.56E+11	5.53E+11	5.35E+09	6.84E+09	2.41E+10
2016/6/30	000002.SZ	7.12E+11	5.10E+11	8.34E+09	2.85E+09	2.64E+10
2016/3/31	000002.SZ	6.59E+11	4.57E+11	1.15E+10	3.30E+09	2.37E+10
2015/12/31	000002.SZ	6.11E+11	4.20E+11	1.67E+10	1.90E+09	2.47E+10
2015/9/30	000002.SZ	5.71E+11	4.02E+11	1.83E+10	7.63E+08	2.34E+10
2015/6/30	000002.SZ	5.37E+11	3.79E+11	2.16E+10	4.77E+08	2.33E+10
2015/3/31	000002.SZ	5.26E+11	3.69E+11	2.47E+10	1.02E+09	2.26E+10
2014/12/31	000002.SZ	5.08E+11	3.46E+11	2.13E+10	2.38E+09	2.04E+10
2014/9/30	000002.SZ	5.20E+11	3.56E+11	2.13E+10	6.03E+09	1.38E+10
2014/6/30	000002.SZ	5.02E+11	3.36E+11	1.66E+10	6.58E+09	1.57E+10
2014/3/31	000002.SZ	4.95E+11	3.41E+11	1.40E+10	4.38E+09	2.35E+10
2013/12/31	000002.SZ	4.79E+11	3.29E+11	1.48E+10	5.10E+09	2.75E+10
2013/9/30	000002.SZ	4.61E+11	3.30E+11	1.32E+10	1.25E+10	2.64E+10
2013/6/30	000002.SZ	4.32E+11	3.14E+11	1.09E+10	1.20E+10	3.29E+10
2013/3/31	000002.SZ	4.18E+11	2.99E+11	7.87E+09	1.31E+10	3.10E+10
2012/12/31	000002.SZ	3.79E+11	2.60E+11	4.98E+09	9.93E+09	2.56E+10
2012/9/30	000002.SZ	3.48E+11	2.34E+11	1.69E+09	4.48E+09	1.41E+10
2012/6/30	000002.SZ	3.30E+11	2.17E+11	2.72E+08	4.80E+09	1.55E+10
2012/3/31	000002.SZ	3.10E+11	2.03E+11	0.00E+00	2.04E+09	1.58E+10
2011/12/31	000002.SZ	2.96E+11	2.01E+11	3.13E+07	1.72E+09	2.18E+10
2011/9/30	000002.SZ	2.83E+11	1.95E+11	3.13E+07	9.21E+08	2.28E+10
2011/6/30	000002.SZ	2.61E+11	1.73E+11	0.00E+00	1.61E+09	2.14E+10
2011/3/31	000002.SZ	2.35E+11	1.52E+11	0.00E+00	1.72E+09	1.86E+10
2010/12/31	000002.SZ	2.16E+11	1.30E+11	0.00E+00	1.48E+09	1.53E+10
	...	...	...	...	...	...
2004/9/30	000002.SZ	1.58E+10	7.06E+09	0.00E+00	2.81E+09	6.00E+07
2004/6/30	000002.SZ	1.32E+10	6.43E+09	4.00E+06	3.24E+09	1.60E+08
2004/3/31	000002.SZ	1.18E+10	5.76E+09	3.15E+09	0.00E+00	1.60E+08
2003/12/31	000002.SZ	1.06E+10	4.80E+09	3.00E+06	1.68E+09	1.60E+08
2003/9/30	000002.SZ	1.02E+10	4.88E+09	9.72E+06	1.56E+09	0.00E+00
2003/6/30	000002.SZ	9.47E+09	4.17E+09	9.69E+06	1.04E+09	0.00E+00
2003/3/31	000002.SZ	8.32E+09	3.14E+09	9.69E+06	5.20E+08	0.00E+00
2002/12/31	000002.SZ	8.22E+09	3.08E+09	9.69E+06	4.60E+08	0.00E+00
2002/9/30	000002.SZ	8.21E+09	3.18E+09	5.75E+08	7.60E+08	0.00E+00
2002/6/30	000002.SZ	8.22E+09	3.27E+09	0.00E+00	1.12E+09	0.00E+00
2002/3/31	000002.SZ	6.60E+09	3.23E+09	0.00E+00	1.50E+09	0.00E+00
2001/12/31	000002.SZ	6.48E+09	3.05E+09	0.00E+00	1.35E+09	0.00E+00
2001/6/30	000002.SZ	6.07E+09	2.73E+09	0.00E+00	9.86E+08	0.00E+00
2000/12/31	000002.SZ	5.58E+09	2.53E+09	0.00E+00	5.66E+08	0.00E+00
2000/6/30	000002.SZ	5.05E+09	2.06E+09	0.00E+00	6.96E+08	2.00E+07
1999/12/31	000002.SZ	4.49E+09	2.29E+09	0.00E+00	8.95E+08	0.00E+00
1999/6/30	000002.SZ	4.34E+09	2.11E+09	0.00E+00	8.15E+08	0.00E+00
1998/12/31	000002.SZ	4.04E+09	1.92E+09	0.00E+00	6.19E+08	0.00E+00
1998/6/30	000002.SZ	3.95E+09	1.93E+09	0.00E+00	5.74E+08	0.00E+00
1997/12/31	000002.SZ	3.96E+09	1.97E+09	0.00E+00	5.48E+08	0.00E+00
1997/6/30	000002.SZ	3.62E+09	2.04E+09	0.00E+00	0.00E+00	0.00E+00
1996/12/31	000002.SZ	3.47E+09	1.94E+09	0.00E+00	8.28E+08	4.05E+07
1996/6/30	000002.SZ	3.22E+09	1.66E+09	0.00E+00	0.00E+00	0.00E+00
1995/12/31	000002.SZ	3.23E+09	1.75E+09	0.00E+00	6.47E+08	3.01E+07
1995/6/30	000002.SZ	2.83E+09	1.57E+09	0.00E+00	0.00E+00	0.00E+00
1994/12/31	000002.SZ	2.68E+09	1.44E+09	0.00E+00	4.08E+08	2.49E+05
1994/6/30	000002.SZ	2.24E+09	1.19E+09	0.00E+00	0.00E+00	0.00E+00
1993/12/31	000002.SZ	2.14E+09	1.14E+09	0.00E+00	3.37E+08	0.00E+00
1992/12/31	000002.SZ	9.63E+08	7.05E+08	0.00E+00	1.97E+08	0.00E+00
1970/1/1	000002.SZ	0.00E+00	0.00E+00	0.00E+00	0.00E+00	0.00E+00

2.4. Fund Dict

Balance sheet

Income Statement

Cash Flow Statement