pandas dataframe 做机器学习训练数据=》直接使用iloc或者as_matrix即可

样本示意，为kdd99数据源：

0,udp,private,SF,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00,normal.
0,udp,private,SF,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00,normal.
0,udp,private,SF,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00,normal.
0,udp,private,SF,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00,snmpgetattack.
0,udp,private,SF,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.01,0.00,0.00,0.00,0.00,0.00,snmpgetattack.
0,udp,private,SF,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,255,1.00,0.00,0.01,0.00,0.00,0.00,0.00,0.00,snmpgetattack.
0,udp,domain_u,SF,29,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,1,0.00,0.00,0.00,0.00,0.50,1.00,0.00,10,3,0.30,0.30,0.30,0.00,0.00,0.00,0.00,0.00,normal.
0,udp,private,SF,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,253,0.99,0.01,0.00,0.00,0.00,0.00,0.00,0.00,normal.
0,udp,private,SF,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00,snmpgetattack.
0,tcp,http,SF,223,185,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,4,4,0.00,0.00,0.00,0.00,1.00,0.00,0.00,71,255,1.00,0.00,0.01,0.01,0.00,0.00,0.00,0.00,normal.
0,udp,private,SF,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00,snmpgetattack.
0,tcp,http,SF,230,260,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,1,19,0.00,0.00,0.00,0.00,1.00,0.00,0.11,3,255,1.00,0.00,0.33,0.07,0.33,0.00,0.00,0.00,normal.
0,udp,private,SF,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.01,0.00,0.00,0.00,0.00,0.00,normal.
0,udp,private,SF,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,252,0.99,0.01,0.00,0.00,0.00,0.00,0.00,0.00,snmpgetattack.
1,tcp,smtp,SF,3170,329,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,1,2,0.00,0.00,0.00,0.00,1.00,0.00,1.00,54,39,0.72,0.11,0.02,0.00,0.02,0.00,0.09,0.13,normal.
0,tcp,http,SF,297,13787,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,177,255,1.00,0.00,0.01,0.01,0.00,0.00,0.00,0.00,normal.
0,tcp,http,SF,291,3542,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,12,12,0.00,0.00,0.00,0.00,1.00,0.00,0.00,187,255,1.00,0.00,0.01,0.01,0.00,0.00,0.00,0.00,normal.
0,tcp,http,SF,295,753,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,21,22,0.00,0.00,0.00,0.00,1.00,0.00,0.09,196,255,1.00,0.00,0.01,0.01,0.00,0.00,0.00,0.00,normal.
0,udp,private,SF,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.01,0.00,0.00,0.00,0.00,0.00,snmpgetattack.
0,udp,private,SF,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00,snmpgetattack.
0,tcp,http,SF,268,9235,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,5,5,0.00,0.00,0.00,0.00,1.00,0.00,0.00,58,255,1.00,0.00,0.02,0.05,0.00,0.00,0.00,0.00,normal.
0,udp,private,SF,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,253,0.99,0.01,0.00,0.00,0.00,0.00,0.00,0.00,snmpgetattack.
0,tcp,http,SF,223,185,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,3,3,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,255,1.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,normal.
0,tcp,http,SF,227,8841,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,13,13,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,255,1.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,normal.
0,tcp,http,SF,222,19564,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,22,23,0.00,0.00,0.00,0.00,1.00,0.00,0.09,255,255,1.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,normal.
0,tcp,ftp_data,SF,740,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,77,33,0.34,0.08,0.34,0.06,0.00,0.00,0.00,0.00,normal.
0,udp,private,SF,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00,normal.
0,tcp,ftp_data,SF,35195,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,10,10,0.00,0.00,0.00,0.00,1.00,0.00,0.00,92,44,0.43,0.07,0.43,0.05,0.00,0.00,0.00,0.00,normal.
0,tcp,ftp_data,SF,8325,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,20,20,0.00,0.00,0.00,0.00,1.00,0.00,0.00,103,54,0.49,0.06,0.49,0.04,0.00,0.00,0.00,0.00,normal.

代码：

# -*- coding:utf-8 -*-

import re
import matplotlib.pyplot as plt
import os
from sklearn.feature_extraction.text import CountVectorizer
from sklearn import preprocessing
from sklearn import cross_validation
import os
from sklearn.datasets import load_iris
from sklearn import tree
import pydotplus
from sklearn.preprocessing import LabelEncoder
import numpy as np
import pandas as pd
from sklearn_pandas import DataFrameMapper

def label(x):
    if x == "normal.":
        return 0
    else:
        return 1

if __name__ == ‘__main__‘:
    data = pd.read_csv(‘../data/kddcup99/corrected‘, sep=",", header=None)
    print data.columns
    print data.iloc[0,0], data.iloc[0,1]
    print len(data)
    col_cnt = len(data.columns)

    normal = data.loc[data.loc[:, col_cnt-1] == "normal.", :]
    print "normal len:", len(normal)
    guess = data.loc[data.loc[:, col_cnt-1] == "guess_passwd.", :]
    print "normal len:", len(guess)

    data = pd.concat([normal, guess])
    print len(data)

    le = preprocessing.LabelEncoder()
    for i in range(col_cnt-1):
        if isinstance(data.iloc[0,i], str):
            print "tranform string column only:", i
            data.loc[:,i] = le.fit_transform(data.loc[:,i])
    data.loc[:,col_cnt-1] = data.loc[:,col_cnt-1].apply(label)
    print data.iloc[0,0], data.iloc[0,1]
    x = data.iloc[:, range(col_cnt-1)]
    #x = data.iloc[:, [0,4,5,6,7,8,22,23,24,25,26,27,28,29,30]]
    y = data.iloc[:, col_cnt-1]
　　    ‘‘‘ also OK    data = data.as_matrix()    x = data[:, range(col_cnt-1)]    y = data[:, col_cnt-1]    ‘‘‘
    print "x=>"
    print x.iloc[0:3, :]
    print "y=>"
    print y[-3:]
    #v=load_kdd99("../data/kddcup99/corrected")
    #x,y=get_guess_passwdandNormal(v)
    clf = tree.DecisionTreeClassifier()
    clf = clf.fit(x, y)
    print clf

    print  cross_validation.cross_val_score(clf, x, y, n_jobs=-1, cv=10)

    clf = clf.fit(x, y)
    dot_data = tree.export_graphviz(clf, out_file=None)
    graph = pydotplus.graph_from_dot_data(dot_data)
    graph.write_pdf("../photo/6/iris-dt.pdf")

结果：

Int64Index([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
            17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
            34, 35, 36, 37, 38, 39, 40, 41],
           dtype=‘int64‘)
0 udp
311029
normal len: 60593
normal len: 4367
64960
tranform string column only: 1
tranform string column only: 2
tranform string column only: 3
0 2
x=>
   0   1   2   3    4    5   6   7   8   9  ...    31   32   33    34   35  0   0   2  15   7  105  146   0   0   0   0 ...   255  254  1.0  0.01  0.0
1   0   2  15   7  105  146   0   0   0   0 ...   255  254  1.0  0.01  0.0
2   0   2  15   7  105  146   0   0   0   0 ...   255  254  1.0  0.01  0.0   

    36   37   38   39   40
0  0.0  0.0  0.0  0.0  0.0
1  0.0  0.0  0.0  0.0  0.0
2  0.0  0.0  0.0  0.0  0.0  

[3 rows x 41 columns]
y=>
142098    1
142099    1
142101    1
Name: 41, dtype: int64
DecisionTreeClassifier(class_weight=None, criterion=‘gini‘, max_depth=None,
            max_features=None, max_leaf_nodes=None,
            min_impurity_decrease=0.0, min_impurity_split=None,
            min_samples_leaf=1, min_samples_split=2,
            min_weight_fraction_leaf=0.0, presort=False, random_state=None,
            splitter=‘best‘)
fg[ 0.9561336   0.99892258  0.99938433  0.99984606  0.99984606  0.99969212
  1.          0.99984604  0.99969207  1.        ]

时间： 2024-08-30 02:02:09

pandas dataframe 做机器学习训练数据=》直接使用iloc或者as_matrix即可的相关文章

python pandas.DataFrame选取、修改数据最好用.loc，.iloc，.ix

先手工生出一个数据框吧 [python] view plain copy import numpy as np import pandas as pd df = pd.DataFrame(np.arange(0,60,2).reshape(10,3),columns=list('abc')) df 是这样子滴那么这三种选取数据的方式该怎么选择呢? 一.当每列已有column name时,用 df [ 'a' ] 就能选取出一整列数据.如果你知道column names 和index,且两者都很

机器学习之数据预处理，Pandas读取excel数据

Python读写excel的工具库很多,比如最耳熟能详的xlrd.xlwt,xlutils,openpyxl等.其中xlrd和xlwt库通常配合使用,一个用于读,一个用于写excel.xlutils结合xlrd可以达到修改excel文件目的.openpyxl可以对excel文件同时进行读写操作. 而说到数据预处理,pandas就体现除了它的强大之处,并且它还支持可读写多种文档格式,其中就包括对excel的读写.本文重点就是介绍pandas对excel数据集的预处理. 机器学习常用的模型对数据输入

Pandas DataFrame 数据选取和过滤

This would allow chaining operations like: pd.read_csv('imdb.txt') .sort(columns='year') .filter(lambda x: x['year']>1990) # <---this is missing in Pandas .to_csv('filtered.csv') For current alternatives see: http://stackoverflow.com/questions/11869

用PyQt5来即时显示pandas Dataframe的数据，附qdarkstyle黑夜主题样式（美美哒的黑夜主题）

import sys from qdarkstyle import load_stylesheet_pyqt5 from PyQt5.QtWidgets import QApplication, QTableView from PyQt5.QtCore import QAbstractTableModel, Qt class QtTable(QAbstractTableModel): def __init__(self, data): QAbstractTableModel.__init__(s

pandas.DataFrame学习系列2——函数方法(1)

DataFrame类具有很多方法,下面做用法的介绍和举例. pandas.DataFrame学习系列2--函数方法(1) 1.abs(),返回DataFrame每个数值的绝对值,前提是所有元素均为数值型 1 import pandas as pd 2 import numpy as np 3 4 df=pd.read_excel('南京银行.xlsx',index_col='Date') 5 df1=df[:5] 6 df1.iat[0,1]=-df1.iat[0,1] 7 df1 8 Open

数据分析--pandas DataFrame

pandas DataFrame是一个表格类型的数据,含有一组有序的列,每列可以是不同的值类型(数值,字符串,布尔值).DataFrame即有行索引,也有列索引,可以看作由Series组成的字典(公用同一个索引). DataFrame是以一个或者多个二维块存放的(而不是列表,字典或别的一维数据结构) 构建DataFrame 传入一个等长列表或Numpy数组组成的字典 DataFrame会自动加上索引,且全部列会被有序排列可以指定序列的排序传入的列在数据中找不到,会产生Na值从DataFra

面向机器学习：数据平台设计与搭建实践

机器学习作为近几年的一项热门技术,不仅凭借众多"人工智能"产品而为人所熟知,更是从根本上增能了传统的互联网产品.在近期举办的2018 ArchSummit全球架构师峰会上,个推首席数据架构师袁凯,基于他在数据平台的建设以及数据产品研发的多年经验,分享了<面向机器学习数据平台的设计与搭建>. 一.背景:机器学习在个推业务中的应用场景作为独立的智能大数据服务商,个推主要业务包括开发者服务.精准营销服务和各垂直领域的大数据服务.而机器学习技术在多项业务及产品中均有涉及:基于用户

如何解决python训练数据内存泄露问题？

Python语言越来越广泛的应用于机器学习/ 深度学习领域,是目前最火的该领域编程语言,各大深度学习框架基本都支持 python 接口. 在TensorFlow 训练模型的过程中,一般数据加载分情景各有不同. 1. 当数据量能直接全部载入内存时,当然最是方便,直接全部载入内存,然后训练即可. 2. 数据无法全部载入内存时,有多种方法.介绍其中 2 种用的多的,其一是边训练边读取(可以用多线程进行优化),其二是随机打乱所有训练数据的索引然后随机选择部分数据进行训练测试. 第2 类情景中,有可

如何解决机器学习中数据不平衡问题

作者:无影随想时间:2016年1月. 出处:http://www.zhaokv.com/2016/01/learning-from-imbalanced-data.html 声明:版权所有,转载请联系作者并注明出处这几年来,机器学习和数据挖掘非常火热,它们逐渐为世界带来实际价值.与此同时,越来越多的机器学习算法从学术界走向工业界,而在这个过程中会有很多困难.数据不平衡问题虽然不是最难的,但绝对是最重要的问题之一. 一.数据不平衡在学术研究与教学中,很多算法都有一个基本假设,那就是数据分布是