2008-01-10 15:05:39 | 评论 (5) | 被访问(723)次
|
最近股票很火,打算研究用代码下载股票数据,于是我做了一个Java代码的模拟IE网页登录的程序,
登录新浪的股票页面并下载一些数据来分析, 基本原理是利用Java自带的HttpURLConnection 类抓取网页,
不过有时有的网页返回不正常,需要用一些http协议的sniffer工具来辅助验证,比如
urlConn.setRequestProperty("Host","biz.finance.sina.com.cn");
urlConn.setRequestProperty("Accept","text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2");
这两句就是发现返回的页面不正常,后来抓取http头发现后加上的。
下面上程序:
import java.net.*;
import java.io.*;
public class test{
public static void main(String[] args){
try{
String url = "http://biz.finance.sina.com.cn/suggest/lookup_n.php?q=600018&country=cn";
String cookie = "";
String data = "test!!!";
HttpURLConnection urlConn=(HttpURLConnection)(new URL(url).openConnection());
urlConn.addRequestProperty("Cookie",cookie);
urlConn.setRequestMethod("GET");
urlConn.setRequestProperty("User-Agent","Mozilla/4.0 (compatible; MSIE 6.0; Windows 2000)");
urlConn.setFollowRedirects(true);
urlConn.setDoOutput(true); // 需要向服务器写数据
urlConn.setDoInput(true); //
urlConn.setUseCaches(false); // 获得服务器最新的信息
urlConn.setAllowUserInteraction(false);
urlConn.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
urlConn.setRequestProperty("Accept-Language","zh-cn" );
// urlConn.setRequestProperty("Content-Length", ""+data.length());
urlConn.setRequestProperty("Cache-Control","no-cache");
urlConn.setRequestProperty("Pragma","no-cache");
urlConn.setRequestProperty("Host","biz.finance.sina.com.cn");
urlConn.setRequestProperty("Accept","text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2");
urlConn.setRequestProperty("Connection","keep-alive");
cookie=urlConn.getHeaderField("Set-Cookie");
//BufferedReader br=new BufferedReader(new InputStreamReader(urlConn.getInputStream(),"utf-8"));
BufferedInputStream br = new BufferedInputStream(urlConn.getInputStream());
FileOutputStream out = new FileOutputStream(new File("out.html"));
int chByte = br.read();
while (chByte != -1){
out.write(chByte);
chByte = br.read();
}
}catch(Exception e){
e.printStackTrace();
}
}
}
抓下来后,可以用一些html分析工具对返回的out.html文件进行分析,得到需要的股票数据,后述。
@author mikemelon(maomao)
2008-1-10
联系客服