前几天拿到了一个400多M的.csv文件,在电脑上打开要好长时间,打开后里面的数据都是乱码。因此,做了一个先转码再导入数据库的程序。100多万条的数据转码+导入花了4分钟,感觉效率还可以。各位网友有更好的方法,请在留言中指点下,一起学习学习,多谢了。
static void Main(string[] args) { int count = 0; string readerPath=@"C:\Users\Administrator\Desktop\readerDemo.csv"; string writerPath=@"C:\Users\Administrator\Desktop\writeDemo.csv"; if (File.Exists(writerPath)) { File.Delete(writerPath); } using (StreamReader reader = new StreamReader(readerPath,Encoding.UTF8)) { while (!reader.EndOfStream) { string line = reader.ReadLine(); using (StreamWriter writer = new StreamWriter(writerPath, true,Encoding.Default)) { writer.WriteLine(line); } count++; Console.WriteLine("正在转码第{0}行,请稍等",count); } } Console.WriteLine("转码完成,共转码{0}条数据",count); Console.WriteLine("开始导入数据,请稍等"); string sql = "BULK INSERT Test.dbo.BagDataTable FROM ‘C:\\Users\\Administrator\\Desktop\\writeDemo.csv‘ WITH(FIELDTERMINATOR=‘,‘,BATCHSIZE=100000,FIRSTROW=2)"; try { DbHelper.ExecuteSql(sql); } catch(Exception ex) { using (StreamWriter writerLog = new StreamWriter(@"C:\Users\Administrator\Desktop\Log.txt")) { writerLog.WriteLine(ex.ToString()); } } Console.WriteLine("数据导入完毕"); Console.ReadKey(); } }
Bulk Insert命令详细:http://blog.csdn.net/jackmacro/article/details/5959321/
FIELDTERMINATOR表示csv文件中的结束符标志(csv默认列结束标志是,行结束标志是\r\n),具体使用参看:http://www.cnblogs.com/sunice/p/6367332.html
时间: 2024-10-27 08:19:59