【Flume】flume中sink到source的性能测试,以Avro为例来测试

从官方下载的flume-ng的源码里有单元测试的代码,本文就通过单元测试来体验下flume中avro的性能如何

本文中的一个event的body内容大小是1KB,读者可自行组织文本,达到1KB即可,方便测试时,性能的计算

 for (int i = 0; i < client.getBatchSize(); i++) {
        Event event = channel.take();

        if (event == null) {
          break;
        }

        batch.add(event);
      }

flume中avroClient也有一个batchSize的概念,这里获取的仍然是sink端的batchsize,默认值是100,

batch-size 100 number of event to batch together for send.

也就是当event的数量达到了100的话,就会触发方法

public void testPerformance() {
		setUp();
		Event event = EventBuilder
				.withBody("XXXXX",Charsets.UTF_8);
		sink.start();
		long allBegin = System.currentTimeMillis();
		int a;
		for (a = 0; a < 1000; a++) {
			long begin = System.currentTimeMillis();
			Transaction transaction = channel.getTransaction();

			transaction.begin();
			int i;
			for (i = 0; i < 100; i++) {
				channel.put(event);
			}
			transaction.commit();
			transaction.close();
			try {
				sink.process();
			} catch (EventDeliveryException e) {
				e.printStackTrace();
			}
			System.out.println("a= "+a+"i= "+i+" ====time="
					+ (System.currentTimeMillis() - begin));
		}
		System.out.println("all time = "
				+ (System.currentTimeMillis() - allBegin));
		System.out.println("====================== " + a);
	}

这是我的单元测试方法,XXX是内容,请自行组织1KB文本内容,方便计算

a= 0i= 100 ====time=385
a= 1i= 100 ====time=352
a= 2i= 100 ====time=476
a= 3i= 100 ====time=119
a= 4i= 100 ====time=111
a= 5i= 100 ====time=127
a= 6i= 100 ====time=111
a= 7i= 100 ====time=162
a= 8i= 100 ====time=129
a= 9i= 100 ====time=126
a= 10i= 100 ====time=132
a= 11i= 100 ====time=218
a= 12i= 100 ====time=226
a= 13i= 100 ====time=117
a= 14i= 100 ====time=141
a= 15i= 100 ====time=113
a= 16i= 100 ====time=112
a= 17i= 100 ====time=116
a= 18i= 100 ====time=173
a= 19i= 100 ====time=258
a= 20i= 100 ====time=184
a= 21i= 100 ====time=220
a= 22i= 100 ====time=104
a= 23i= 100 ====time=111
a= 24i= 100 ====time=202
a= 25i= 100 ====time=88
a= 26i= 100 ====time=163
a= 27i= 100 ====time=183
a= 28i= 100 ====time=207
a= 29i= 100 ====time=131
a= 30i= 100 ====time=162
a= 31i= 100 ====time=191
a= 32i= 100 ====time=279
a= 33i= 100 ====time=95
a= 34i= 100 ====time=136
a= 35i= 100 ====time=168
a= 36i= 100 ====time=168
a= 37i= 100 ====time=146
a= 38i= 100 ====time=195
a= 39i= 100 ====time=119
a= 40i= 100 ====time=106
a= 41i= 100 ====time=179
a= 42i= 100 ====time=200
a= 43i= 100 ====time=87
a= 44i= 100 ====time=131
a= 45i= 100 ====time=186
a= 46i= 100 ====time=482
a= 47i= 100 ====time=166
a= 48i= 100 ====time=180
a= 49i= 100 ====time=95
a= 50i= 100 ====time=127
a= 51i= 100 ====time=256
a= 52i= 100 ====time=89
a= 53i= 100 ====time=117
a= 54i= 100 ====time=176
a= 55i= 100 ====time=291
a= 56i= 100 ====time=77
a= 57i= 100 ====time=103
a= 58i= 100 ====time=124
a= 59i= 100 ====time=174
a= 60i= 100 ====time=87
a= 61i= 100 ====time=163
a= 62i= 100 ====time=78
a= 63i= 100 ====time=159
a= 64i= 100 ====time=86
a= 65i= 100 ====time=589
a= 66i= 100 ====time=114
a= 67i= 100 ====time=133
a= 68i= 100 ====time=169
a= 69i= 100 ====time=102
a= 70i= 100 ====time=93
a= 71i= 100 ====time=115
a= 72i= 100 ====time=108
a= 73i= 100 ====time=556
a= 74i= 100 ====time=107
a= 75i= 100 ====time=86
a= 76i= 100 ====time=193
a= 77i= 100 ====time=145
a= 78i= 100 ====time=221
a= 79i= 100 ====time=103
a= 80i= 100 ====time=125
a= 81i= 100 ====time=172
a= 82i= 100 ====time=136
a= 83i= 100 ====time=67
a= 84i= 100 ====time=178
a= 85i= 100 ====time=71
a= 86i= 100 ====time=118
a= 87i= 100 ====time=77
a= 88i= 100 ====time=101
a= 89i= 100 ====time=117
a= 90i= 100 ====time=325
a= 91i= 100 ====time=517
a= 92i= 100 ====time=511
a= 93i= 100 ====time=371
a= 94i= 100 ====time=397
a= 95i= 100 ====time=524
a= 96i= 100 ====time=998
a= 97i= 100 ====time=153
a= 98i= 100 ====time=110
a= 99i= 100 ====time=179
a= 100i= 100 ====time=75
a= 101i= 100 ====time=173
a= 102i= 100 ====time=85
a= 103i= 100 ====time=101
a= 104i= 100 ====time=84
a= 105i= 100 ====time=111
a= 106i= 100 ====time=165
a= 107i= 100 ====time=99
a= 108i= 100 ====time=118
a= 109i= 100 ====time=490
a= 110i= 100 ====time=96
a= 111i= 100 ====time=123
a= 112i= 100 ====time=180
a= 113i= 100 ====time=172
a= 114i= 100 ====time=117
a= 115i= 100 ====time=115
a= 116i= 100 ====time=137
a= 117i= 100 ====time=72
a= 118i= 100 ====time=176
a= 119i= 100 ====time=64
a= 120i= 100 ====time=61
a= 121i= 100 ====time=128
a= 122i= 100 ====time=168
a= 123i= 100 ====time=93
a= 124i= 100 ====time=127
a= 125i= 100 ====time=174
a= 126i= 100 ====time=419
a= 127i= 100 ====time=126
a= 128i= 100 ====time=148
a= 129i= 100 ====time=128
a= 130i= 100 ====time=111
a= 131i= 100 ====time=364
a= 132i= 100 ====time=338
a= 133i= 100 ====time=285
a= 134i= 100 ====time=81
a= 135i= 100 ====time=64
a= 136i= 100 ====time=74
a= 137i= 100 ====time=72
a= 138i= 100 ====time=154
a= 139i= 100 ====time=61
a= 140i= 100 ====time=57
a= 141i= 100 ====time=50
a= 142i= 100 ====time=63
a= 143i= 100 ====time=62
a= 144i= 100 ====time=60
a= 145i= 100 ====time=59
a= 146i= 100 ====time=59
a= 147i= 100 ====time=56
a= 148i= 100 ====time=116
a= 149i= 100 ====time=170
a= 150i= 100 ====time=54
a= 151i= 100 ====time=59
a= 152i= 100 ====time=117
a= 153i= 100 ====time=60
a= 154i= 100 ====time=59
a= 155i= 100 ====time=61
a= 156i= 100 ====time=416
a= 157i= 100 ====time=59
a= 158i= 100 ====time=54
a= 159i= 100 ====time=57
a= 160i= 100 ====time=51
a= 161i= 100 ====time=61
a= 162i= 100 ====time=59
a= 163i= 100 ====time=149
a= 164i= 100 ====time=52
a= 165i= 100 ====time=56
a= 166i= 100 ====time=63
a= 167i= 100 ====time=57
a= 168i= 100 ====time=62
a= 169i= 100 ====time=60
a= 170i= 100 ====time=55
a= 171i= 100 ====time=54
a= 172i= 100 ====time=108
a= 173i= 100 ====time=55
a= 174i= 100 ====time=62
a= 175i= 100 ====time=56
a= 176i= 100 ====time=59
a= 177i= 100 ====time=83
a= 178i= 100 ====time=56
a= 179i= 100 ====time=56
a= 180i= 100 ====time=159
a= 181i= 100 ====time=60
a= 182i= 100 ====time=52
a= 183i= 100 ====time=54
a= 184i= 100 ====time=60
a= 185i= 100 ====time=55
a= 186i= 100 ====time=59
a= 187i= 100 ====time=59
a= 188i= 100 ====time=59
a= 189i= 100 ====time=151
a= 190i= 100 ====time=54
a= 191i= 100 ====time=64
a= 192i= 100 ====time=114
a= 193i= 100 ====time=66
a= 194i= 100 ====time=67
a= 195i= 100 ====time=59
a= 196i= 100 ====time=370
a= 197i= 100 ====time=69
a= 198i= 100 ====time=170
a= 199i= 100 ====time=52
a= 200i= 100 ====time=63
a= 201i= 100 ====time=53
a= 202i= 100 ====time=55
a= 203i= 100 ====time=60
a= 204i= 100 ====time=57
a= 205i= 100 ====time=59
a= 206i= 100 ====time=124
a= 207i= 100 ====time=69
a= 208i= 100 ====time=59
a= 209i= 100 ====time=53
a= 210i= 100 ====time=103
a= 211i= 100 ====time=55
a= 212i= 100 ====time=60
a= 213i= 100 ====time=58
a= 214i= 100 ====time=51
a= 215i= 100 ====time=61
a= 216i= 100 ====time=122
a= 217i= 100 ====time=100
a= 218i= 100 ====time=57
a= 219i= 100 ====time=365
a= 220i= 100 ====time=129
a= 221i= 100 ====time=61
a= 222i= 100 ====time=56
a= 223i= 100 ====time=58
a= 224i= 100 ====time=66
a= 225i= 100 ====time=54
a= 226i= 100 ====time=130
a= 227i= 100 ====time=51
a= 228i= 100 ====time=60
a= 229i= 100 ====time=61
a= 230i= 100 ====time=55
a= 231i= 100 ====time=59
a= 232i= 100 ====time=59
a= 233i= 100 ====time=102
a= 234i= 100 ====time=60
a= 235i= 100 ====time=55
a= 236i= 100 ====time=63
a= 237i= 100 ====time=55
a= 238i= 100 ====time=59
a= 239i= 100 ====time=142
a= 240i= 100 ====time=160
a= 241i= 100 ====time=57
a= 242i= 100 ====time=60
a= 243i= 100 ====time=61
a= 244i= 100 ====time=67
a= 245i= 100 ====time=68
a= 246i= 100 ====time=132
a= 247i= 100 ====time=52
a= 248i= 100 ====time=56
a= 249i= 100 ====time=63
a= 250i= 100 ====time=60
a= 251i= 100 ====time=131
a= 252i= 100 ====time=64
a= 253i= 100 ====time=56
a= 254i= 100 ====time=65
a= 255i= 100 ====time=59
a= 256i= 100 ====time=53
a= 257i= 100 ====time=57
a= 258i= 100 ====time=53
a= 259i= 100 ====time=58
a= 260i= 100 ====time=56
a= 261i= 100 ====time=58
a= 262i= 100 ====time=58
a= 263i= 100 ====time=111
a= 264i= 100 ====time=57
a= 265i= 100 ====time=68
a= 266i= 100 ====time=54
a= 267i= 100 ====time=55
a= 268i= 100 ====time=116
a= 269i= 100 ====time=73
a= 270i= 100 ====time=65
a= 271i= 100 ====time=59
a= 272i= 100 ====time=58
a= 273i= 100 ====time=82
a= 274i= 100 ====time=53
a= 275i= 100 ====time=62
a= 276i= 100 ====time=61
a= 277i= 100 ====time=59
a= 278i= 100 ====time=62
a= 279i= 100 ====time=55
a= 280i= 100 ====time=56
a= 281i= 100 ====time=150
a= 282i= 100 ====time=61
a= 283i= 100 ====time=58
a= 284i= 100 ====time=97
a= 285i= 100 ====time=377
a= 286i= 100 ====time=67
a= 287i= 100 ====time=52
a= 288i= 100 ====time=75
a= 289i= 100 ====time=60
a= 290i= 100 ====time=56
a= 291i= 100 ====time=56
a= 292i= 100 ====time=55
a= 293i= 100 ====time=60
a= 294i= 100 ====time=51
a= 295i= 100 ====time=57
a= 296i= 100 ====time=57
a= 297i= 100 ====time=61
a= 298i= 100 ====time=57
a= 299i= 100 ====time=128
a= 300i= 100 ====time=57
a= 301i= 100 ====time=65
a= 302i= 100 ====time=63
a= 303i= 100 ====time=52
a= 304i= 100 ====time=59
a= 305i= 100 ====time=59
a= 306i= 100 ====time=52
a= 307i= 100 ====time=56
a= 308i= 100 ====time=57
a= 309i= 100 ====time=117
a= 310i= 100 ====time=74
a= 311i= 100 ====time=58
a= 312i= 100 ====time=56
a= 313i= 100 ====time=65
a= 314i= 100 ====time=63
a= 315i= 100 ====time=256
a= 316i= 100 ====time=51
a= 317i= 100 ====time=61
a= 318i= 100 ====time=73
a= 319i= 100 ====time=53
a= 320i= 100 ====time=59
a= 321i= 100 ====time=64
a= 322i= 100 ====time=225
a= 323i= 100 ====time=53
a= 324i= 100 ====time=60
a= 325i= 100 ====time=57
a= 326i= 100 ====time=53
a= 327i= 100 ====time=59
a= 328i= 100 ====time=51
a= 329i= 100 ====time=75
a= 330i= 100 ====time=57
a= 331i= 100 ====time=113
a= 332i= 100 ====time=69
a= 333i= 100 ====time=52
a= 334i= 100 ====time=90
a= 335i= 100 ====time=175
a= 336i= 100 ====time=133
a= 337i= 100 ====time=53
a= 338i= 100 ====time=54
a= 339i= 100 ====time=58
a= 340i= 100 ====time=67
a= 341i= 100 ====time=57
a= 342i= 100 ====time=56
a= 343i= 100 ====time=127
a= 344i= 100 ====time=62
a= 345i= 100 ====time=167
a= 346i= 100 ====time=193
a= 347i= 100 ====time=60
a= 348i= 100 ====time=58
a= 349i= 100 ====time=55
a= 350i= 100 ====time=56
a= 351i= 100 ====time=151
a= 352i= 100 ====time=132
a= 353i= 100 ====time=57
a= 354i= 100 ====time=60
a= 355i= 100 ====time=55
a= 356i= 100 ====time=49
a= 357i= 100 ====time=55
a= 358i= 100 ====time=59
a= 359i= 100 ====time=56
a= 360i= 100 ====time=54
a= 361i= 100 ====time=76
a= 362i= 100 ====time=147
a= 363i= 100 ====time=56
a= 364i= 100 ====time=122
a= 365i= 100 ====time=61
a= 366i= 100 ====time=58
a= 367i= 100 ====time=59
a= 368i= 100 ====time=54
a= 369i= 100 ====time=60
a= 370i= 100 ====time=55
a= 371i= 100 ====time=52
a= 372i= 100 ====time=91
a= 373i= 100 ====time=109
a= 374i= 100 ====time=56
a= 375i= 100 ====time=54
a= 376i= 100 ====time=53
a= 377i= 100 ====time=87
a= 378i= 100 ====time=121
a= 379i= 100 ====time=54
a= 380i= 100 ====time=51
a= 381i= 100 ====time=70
a= 382i= 100 ====time=110
a= 383i= 100 ====time=75
a= 384i= 100 ====time=53
a= 385i= 100 ====time=59
a= 386i= 100 ====time=62
a= 387i= 100 ====time=59
a= 388i= 100 ====time=56
a= 389i= 100 ====time=56
a= 390i= 100 ====time=55
a= 391i= 100 ====time=55
a= 392i= 100 ====time=57
a= 393i= 100 ====time=52
a= 394i= 100 ====time=78
a= 395i= 100 ====time=57
a= 396i= 100 ====time=127
a= 397i= 100 ====time=58
a= 398i= 100 ====time=58
a= 399i= 100 ====time=85
a= 400i= 100 ====time=122
a= 401i= 100 ====time=63
a= 402i= 100 ====time=130
a= 403i= 100 ====time=55
a= 404i= 100 ====time=62
a= 405i= 100 ====time=70
a= 406i= 100 ====time=57
a= 407i= 100 ====time=69
a= 408i= 100 ====time=51
a= 409i= 100 ====time=53
a= 410i= 100 ====time=85
a= 411i= 100 ====time=156
a= 412i= 100 ====time=178
a= 413i= 100 ====time=154
a= 414i= 100 ====time=56
a= 415i= 100 ====time=63
a= 416i= 100 ====time=384
a= 417i= 100 ====time=73
a= 418i= 100 ====time=59
a= 419i= 100 ====time=57
a= 420i= 100 ====time=140
a= 421i= 100 ====time=52
a= 422i= 100 ====time=56
a= 423i= 100 ====time=56
a= 424i= 100 ====time=54
a= 425i= 100 ====time=52
a= 426i= 100 ====time=55
a= 427i= 100 ====time=113
a= 428i= 100 ====time=59
a= 429i= 100 ====time=55
a= 430i= 100 ====time=53
a= 431i= 100 ====time=66
a= 432i= 100 ====time=56
a= 433i= 100 ====time=65
a= 434i= 100 ====time=60
a= 435i= 100 ====time=57
a= 436i= 100 ====time=53
a= 437i= 100 ====time=62
a= 438i= 100 ====time=68
a= 439i= 100 ====time=51
a= 440i= 100 ====time=58
a= 441i= 100 ====time=132
a= 442i= 100 ====time=91
a= 443i= 100 ====time=89
a= 444i= 100 ====time=62
a= 445i= 100 ====time=54
a= 446i= 100 ====time=54
a= 447i= 100 ====time=56
a= 448i= 100 ====time=52
a= 449i= 100 ====time=101
a= 450i= 100 ====time=54
a= 451i= 100 ====time=69
a= 452i= 100 ====time=56
a= 453i= 100 ====time=59
a= 454i= 100 ====time=54
a= 455i= 100 ====time=59
a= 456i= 100 ====time=55
a= 457i= 100 ====time=62
a= 458i= 100 ====time=58
a= 459i= 100 ====time=150
a= 460i= 100 ====time=117
a= 461i= 100 ====time=90
a= 462i= 100 ====time=84
a= 463i= 100 ====time=83
a= 464i= 100 ====time=56
a= 465i= 100 ====time=65
a= 466i= 100 ====time=58
a= 467i= 100 ====time=57
a= 468i= 100 ====time=60
a= 469i= 100 ====time=50
a= 470i= 100 ====time=55
a= 471i= 100 ====time=61
a= 472i= 100 ====time=143
a= 473i= 100 ====time=70
a= 474i= 100 ====time=55
a= 475i= 100 ====time=91
a= 476i= 100 ====time=191
a= 477i= 100 ====time=84
a= 478i= 100 ====time=73
a= 479i= 100 ====time=51
a= 480i= 100 ====time=151
a= 481i= 100 ====time=51
a= 482i= 100 ====time=70
a= 483i= 100 ====time=60
a= 484i= 100 ====time=62
a= 485i= 100 ====time=57
a= 486i= 100 ====time=54
a= 487i= 100 ====time=58
a= 488i= 100 ====time=50
a= 489i= 100 ====time=57
a= 490i= 100 ====time=58
a= 491i= 100 ====time=58
a= 492i= 100 ====time=56
a= 493i= 100 ====time=103
a= 494i= 100 ====time=55
a= 495i= 100 ====time=60
a= 496i= 100 ====time=66
a= 497i= 100 ====time=441
a= 498i= 100 ====time=208
a= 499i= 100 ====time=58
a= 500i= 100 ====time=56
a= 501i= 100 ====time=63
a= 502i= 100 ====time=78
a= 503i= 100 ====time=62
a= 504i= 100 ====time=59
a= 505i= 100 ====time=59
a= 506i= 100 ====time=58
a= 507i= 100 ====time=64
a= 508i= 100 ====time=54
a= 509i= 100 ====time=54
a= 510i= 100 ====time=125
a= 511i= 100 ====time=51
a= 512i= 100 ====time=65
a= 513i= 100 ====time=56
a= 514i= 100 ====time=53
a= 515i= 100 ====time=104
a= 516i= 100 ====time=58
a= 517i= 100 ====time=59
a= 518i= 100 ====time=54
a= 519i= 100 ====time=58
a= 520i= 100 ====time=61
a= 521i= 100 ====time=55
a= 522i= 100 ====time=55
a= 523i= 100 ====time=58
a= 524i= 100 ====time=52
a= 525i= 100 ====time=58
a= 526i= 100 ====time=57
a= 527i= 100 ====time=58
a= 528i= 100 ====time=157
a= 529i= 100 ====time=56
a= 530i= 100 ====time=69
a= 531i= 100 ====time=94
a= 532i= 100 ====time=52
a= 533i= 100 ====time=53
a= 534i= 100 ====time=62
a= 535i= 100 ====time=55
a= 536i= 100 ====time=59
a= 537i= 100 ====time=78
a= 538i= 100 ====time=58
a= 539i= 100 ====time=49
a= 540i= 100 ====time=56
a= 541i= 100 ====time=60
a= 542i= 100 ====time=77
a= 543i= 100 ====time=120
a= 544i= 100 ====time=53
a= 545i= 100 ====time=55
a= 546i= 100 ====time=148
a= 547i= 100 ====time=57
a= 548i= 100 ====time=61
a= 549i= 100 ====time=101
a= 550i= 100 ====time=60
a= 551i= 100 ====time=52
a= 552i= 100 ====time=55
a= 553i= 100 ====time=53
a= 554i= 100 ====time=61
a= 555i= 100 ====time=55
a= 556i= 100 ====time=70
a= 557i= 100 ====time=64
a= 558i= 100 ====time=56
a= 559i= 100 ====time=61
a= 560i= 100 ====time=65
a= 561i= 100 ====time=54
a= 562i= 100 ====time=62
a= 563i= 100 ====time=148
a= 564i= 100 ====time=51
a= 565i= 100 ====time=56
a= 566i= 100 ====time=62
a= 567i= 100 ====time=58
a= 568i= 100 ====time=88
a= 569i= 100 ====time=63
a= 570i= 100 ====time=55
a= 571i= 100 ====time=79
a= 572i= 100 ====time=53
a= 573i= 100 ====time=51
a= 574i= 100 ====time=388
a= 575i= 100 ====time=61
a= 576i= 100 ====time=58
a= 577i= 100 ====time=57
a= 578i= 100 ====time=59
a= 579i= 100 ====time=61
a= 580i= 100 ====time=123
a= 581i= 100 ====time=155
a= 582i= 100 ====time=65
a= 583i= 100 ====time=50
a= 584i= 100 ====time=115
a= 585i= 100 ====time=54
a= 586i= 100 ====time=54
a= 587i= 100 ====time=57
a= 588i= 100 ====time=51
a= 589i= 100 ====time=53
a= 590i= 100 ====time=60
a= 591i= 100 ====time=62
a= 592i= 100 ====time=65
a= 593i= 100 ====time=64
a= 594i= 100 ====time=56
a= 595i= 100 ====time=54
a= 596i= 100 ====time=59
a= 597i= 100 ====time=58
a= 598i= 100 ====time=57
a= 599i= 100 ====time=55
a= 600i= 100 ====time=114
a= 601i= 100 ====time=93
a= 602i= 100 ====time=56
a= 603i= 100 ====time=58
a= 604i= 100 ====time=58
a= 605i= 100 ====time=59
a= 606i= 100 ====time=54
a= 607i= 100 ====time=57
a= 608i= 100 ====time=55
a= 609i= 100 ====time=53
a= 610i= 100 ====time=55
a= 611i= 100 ====time=51
a= 612i= 100 ====time=50
a= 613i= 100 ====time=87
a= 614i= 100 ====time=53
a= 615i= 100 ====time=58
a= 616i= 100 ====time=60
a= 617i= 100 ====time=55
a= 618i= 100 ====time=155
a= 619i= 100 ====time=87
a= 620i= 100 ====time=64
a= 621i= 100 ====time=68
a= 622i= 100 ====time=82
a= 623i= 100 ====time=53
a= 624i= 100 ====time=55
a= 625i= 100 ====time=60
a= 626i= 100 ====time=53
a= 627i= 100 ====time=55
a= 628i= 100 ====time=57
a= 629i= 100 ====time=54
a= 630i= 100 ====time=49
a= 631i= 100 ====time=64
a= 632i= 100 ====time=56
a= 633i= 100 ====time=56
a= 634i= 100 ====time=63
a= 635i= 100 ====time=166
a= 636i= 100 ====time=56
a= 637i= 100 ====time=53
a= 638i= 100 ====time=110
a= 639i= 100 ====time=51
a= 640i= 100 ====time=76
a= 641i= 100 ====time=57
a= 642i= 100 ====time=55
a= 643i= 100 ====time=56
a= 644i= 100 ====time=54
a= 645i= 100 ====time=63
a= 646i= 100 ====time=62
a= 647i= 100 ====time=57
a= 648i= 100 ====time=70
a= 649i= 100 ====time=96
a= 650i= 100 ====time=85
a= 651i= 100 ====time=58
a= 652i= 100 ====time=54
a= 653i= 100 ====time=84
a= 654i= 100 ====time=151
a= 655i= 100 ====time=58
a= 656i= 100 ====time=55
a= 657i= 100 ====time=131
a= 658i= 100 ====time=61
a= 659i= 100 ====time=57
a= 660i= 100 ====time=56
a= 661i= 100 ====time=57
a= 662i= 100 ====time=59
a= 663i= 100 ====time=57
a= 664i= 100 ====time=53
a= 665i= 100 ====time=59
a= 666i= 100 ====time=52
a= 667i= 100 ====time=56
a= 668i= 100 ====time=105
a= 669i= 100 ====time=53
a= 670i= 100 ====time=365
a= 671i= 100 ====time=72
a= 672i= 100 ====time=56
a= 673i= 100 ====time=55
a= 674i= 100 ====time=141
a= 675i= 100 ====time=62
a= 676i= 100 ====time=53
a= 677i= 100 ====time=53
a= 678i= 100 ====time=58
a= 679i= 100 ====time=52
a= 680i= 100 ====time=56
a= 681i= 100 ====time=58
a= 682i= 100 ====time=74
a= 683i= 100 ====time=56
a= 684i= 100 ====time=63
a= 685i= 100 ====time=59
a= 686i= 100 ====time=56
a= 687i= 100 ====time=160
a= 688i= 100 ====time=58
a= 689i= 100 ====time=54
a= 690i= 100 ====time=95
a= 691i= 100 ====time=56
a= 692i= 100 ====time=62
a= 693i= 100 ====time=66
a= 694i= 100 ====time=53
a= 695i= 100 ====time=53
a= 696i= 100 ====time=54
a= 697i= 100 ====time=55
a= 698i= 100 ====time=86
a= 699i= 100 ====time=425
a= 700i= 100 ====time=68
a= 701i= 100 ====time=58
a= 702i= 100 ====time=52
a= 703i= 100 ====time=55
a= 704i= 100 ====time=62
a= 705i= 100 ====time=80
a= 706i= 100 ====time=131
a= 707i= 100 ====time=103
a= 708i= 100 ====time=58
a= 709i= 100 ====time=59
a= 710i= 100 ====time=63
a= 711i= 100 ====time=53
a= 712i= 100 ====time=56
a= 713i= 100 ====time=62
a= 714i= 100 ====time=53
a= 715i= 100 ====time=66
a= 716i= 100 ====time=58
a= 717i= 100 ====time=214
a= 718i= 100 ====time=55
a= 719i= 100 ====time=59
a= 720i= 100 ====time=51
a= 721i= 100 ====time=56
a= 722i= 100 ====time=96
a= 723i= 100 ====time=57
a= 724i= 100 ====time=369
a= 725i= 100 ====time=121
a= 726i= 100 ====time=123
a= 727i= 100 ====time=58
a= 728i= 100 ====time=54
a= 729i= 100 ====time=55
a= 730i= 100 ====time=54
a= 731i= 100 ====time=52
a= 732i= 100 ====time=61
a= 733i= 100 ====time=56
a= 734i= 100 ====time=55
a= 735i= 100 ====time=60
a= 736i= 100 ====time=55
a= 737i= 100 ====time=54
a= 738i= 100 ====time=68
a= 739i= 100 ====time=58
a= 740i= 100 ====time=55
a= 741i= 100 ====time=114
a= 742i= 100 ====time=57
a= 743i= 100 ====time=58
a= 744i= 100 ====time=81
a= 745i= 100 ====time=55
a= 746i= 100 ====time=108
a= 747i= 100 ====time=57
a= 748i= 100 ====time=56
a= 749i= 100 ====time=52
a= 750i= 100 ====time=58
a= 751i= 100 ====time=56
a= 752i= 100 ====time=57
a= 753i= 100 ====time=56
a= 754i= 100 ====time=55
a= 755i= 100 ====time=59
a= 756i= 100 ====time=64
a= 757i= 100 ====time=51
a= 758i= 100 ====time=56
a= 759i= 100 ====time=69
a= 760i= 100 ====time=100
a= 761i= 100 ====time=72
a= 762i= 100 ====time=63
a= 763i= 100 ====time=56
a= 764i= 100 ====time=65
a= 765i= 100 ====time=54
a= 766i= 100 ====time=51
a= 767i= 100 ====time=59
a= 768i= 100 ====time=59
a= 769i= 100 ====time=53
a= 770i= 100 ====time=57
a= 771i= 100 ====time=54
a= 772i= 100 ====time=55
a= 773i= 100 ====time=57
a= 774i= 100 ====time=56
a= 775i= 100 ====time=147
a= 776i= 100 ====time=377
a= 777i= 100 ====time=65
a= 778i= 100 ====time=50
a= 779i= 100 ====time=54
a= 780i= 100 ====time=82
a= 781i= 100 ====time=59
a= 782i= 100 ====time=58
a= 783i= 100 ====time=128
a= 784i= 100 ====time=56
a= 785i= 100 ====time=55
a= 786i= 100 ====time=57
a= 787i= 100 ====time=53
a= 788i= 100 ====time=112
a= 789i= 100 ====time=54
a= 790i= 100 ====time=100
a= 791i= 100 ====time=58
a= 792i= 100 ====time=63
a= 793i= 100 ====time=64
a= 794i= 100 ====time=56
a= 795i= 100 ====time=61
a= 796i= 100 ====time=50
a= 797i= 100 ====time=53
a= 798i= 100 ====time=83
a= 799i= 100 ====time=142
a= 800i= 100 ====time=72
a= 801i= 100 ====time=58
a= 802i= 100 ====time=72
a= 803i= 100 ====time=138
a= 804i= 100 ====time=125
a= 805i= 100 ====time=179
a= 806i= 100 ====time=275
a= 807i= 100 ====time=171
a= 808i= 100 ====time=205
a= 809i= 100 ====time=104
a= 810i= 100 ====time=102
a= 811i= 100 ====time=469
a= 812i= 100 ====time=160
a= 813i= 100 ====time=115
a= 814i= 100 ====time=56
a= 815i= 100 ====time=66
a= 816i= 100 ====time=60
a= 817i= 100 ====time=55
a= 818i= 100 ====time=58
a= 819i= 100 ====time=58
a= 820i= 100 ====time=58
a= 821i= 100 ====time=53
a= 822i= 100 ====time=61
a= 823i= 100 ====time=57
a= 824i= 100 ====time=63
a= 825i= 100 ====time=64
a= 826i= 100 ====time=51
a= 827i= 100 ====time=67
a= 828i= 100 ====time=154
a= 829i= 100 ====time=57
a= 830i= 100 ====time=124
a= 831i= 100 ====time=53
a= 832i= 100 ====time=56
a= 833i= 100 ====time=106
a= 834i= 100 ====time=133
a= 835i= 100 ====time=96
a= 836i= 100 ====time=57
a= 837i= 100 ====time=61
a= 838i= 100 ====time=63
a= 839i= 100 ====time=60
a= 840i= 100 ====time=55
a= 841i= 100 ====time=52
a= 842i= 100 ====time=60
a= 843i= 100 ====time=57
a= 844i= 100 ====time=56
a= 845i= 100 ====time=145
a= 846i= 100 ====time=52
a= 847i= 100 ====time=57
a= 848i= 100 ====time=55
a= 849i= 100 ====time=58
a= 850i= 100 ====time=57
a= 851i= 100 ====time=55
a= 852i= 100 ====time=128
a= 853i= 100 ====time=54
a= 854i= 100 ====time=71
a= 855i= 100 ====time=58
a= 856i= 100 ====time=135
a= 857i= 100 ====time=58
a= 858i= 100 ====time=57
a= 859i= 100 ====time=60
a= 860i= 100 ====time=55
a= 861i= 100 ====time=127
a= 862i= 100 ====time=51
a= 863i= 100 ====time=56
a= 864i= 100 ====time=55
a= 865i= 100 ====time=56
a= 866i= 100 ====time=55
a= 867i= 100 ====time=57
a= 868i= 100 ====time=57
a= 869i= 100 ====time=193
a= 870i= 100 ====time=181
a= 871i= 100 ====time=62
a= 872i= 100 ====time=82
a= 873i= 100 ====time=51
a= 874i= 100 ====time=61
a= 875i= 100 ====time=59
a= 876i= 100 ====time=56
a= 877i= 100 ====time=50
a= 878i= 100 ====time=391
a= 879i= 100 ====time=159
a= 880i= 100 ====time=54
a= 881i= 100 ====time=59
a= 882i= 100 ====time=66
a= 883i= 100 ====time=60
a= 884i= 100 ====time=53
a= 885i= 100 ====time=92
a= 886i= 100 ====time=54
a= 887i= 100 ====time=57
a= 888i= 100 ====time=55
a= 889i= 100 ====time=53
a= 890i= 100 ====time=60
a= 891i= 100 ====time=153
a= 892i= 100 ====time=55
a= 893i= 100 ====time=56
a= 894i= 100 ====time=89
a= 895i= 100 ====time=60
a= 896i= 100 ====time=51
a= 897i= 100 ====time=60
a= 898i= 100 ====time=58
a= 899i= 100 ====time=75
a= 900i= 100 ====time=155
a= 901i= 100 ====time=60
a= 902i= 100 ====time=87
a= 903i= 100 ====time=61
a= 904i= 100 ====time=55
a= 905i= 100 ====time=57
a= 906i= 100 ====time=59
a= 907i= 100 ====time=66
a= 908i= 100 ====time=56
a= 909i= 100 ====time=58
a= 910i= 100 ====time=152
a= 911i= 100 ====time=117
a= 912i= 100 ====time=54
a= 913i= 100 ====time=57
a= 914i= 100 ====time=59
a= 915i= 100 ====time=51
a= 916i= 100 ====time=65
a= 917i= 100 ====time=53
a= 918i= 100 ====time=55
a= 919i= 100 ====time=65
a= 920i= 100 ====time=50
a= 921i= 100 ====time=59
a= 922i= 100 ====time=61
a= 923i= 100 ====time=458
a= 924i= 100 ====time=86
a= 925i= 100 ====time=52
a= 926i= 100 ====time=55
a= 927i= 100 ====time=63
a= 928i= 100 ====time=167
a= 929i= 100 ====time=57
a= 930i= 100 ====time=58
a= 931i= 100 ====time=61
a= 932i= 100 ====time=123
a= 933i= 100 ====time=58
a= 934i= 100 ====time=51
a= 935i= 100 ====time=56
a= 936i= 100 ====time=55
a= 937i= 100 ====time=57
a= 938i= 100 ====time=55
a= 939i= 100 ====time=54
a= 940i= 100 ====time=52
a= 941i= 100 ====time=57
a= 942i= 100 ====time=66
a= 943i= 100 ====time=112
a= 944i= 100 ====time=62
a= 945i= 100 ====time=85
a= 946i= 100 ====time=62
a= 947i= 100 ====time=56
a= 948i= 100 ====time=61
a= 949i= 100 ====time=62
a= 950i= 100 ====time=598
a= 951i= 100 ====time=56
a= 952i= 100 ====time=387
a= 953i= 100 ====time=61
a= 954i= 100 ====time=58
a= 955i= 100 ====time=54
a= 956i= 100 ====time=76
a= 957i= 100 ====time=128
a= 958i= 100 ====time=64
a= 959i= 100 ====time=57
a= 960i= 100 ====time=58
a= 961i= 100 ====time=71
a= 962i= 100 ====time=81
a= 963i= 100 ====time=83
a= 964i= 100 ====time=70
a= 965i= 100 ====time=61
a= 966i= 100 ====time=66
a= 967i= 100 ====time=132
a= 968i= 100 ====time=60
a= 969i= 100 ====time=53
a= 970i= 100 ====time=63
a= 971i= 100 ====time=59
a= 972i= 100 ====time=114
a= 973i= 100 ====time=75
a= 974i= 100 ====time=53
a= 975i= 100 ====time=117
a= 976i= 100 ====time=51
a= 977i= 100 ====time=65
a= 978i= 100 ====time=53
a= 979i= 100 ====time=56
a= 980i= 100 ====time=60
a= 981i= 100 ====time=153
a= 982i= 100 ====time=53
a= 983i= 100 ====time=56
a= 984i= 100 ====time=61
a= 985i= 100 ====time=55
a= 986i= 100 ====time=145
a= 987i= 100 ====time=50
a= 988i= 100 ====time=56
a= 989i= 100 ====time=63
a= 990i= 100 ====time=57
a= 991i= 100 ====time=56
a= 992i= 100 ====time=56
a= 993i= 100 ====time=108
a= 994i= 100 ====time=62
a= 995i= 100 ====time=53
a= 996i= 100 ====time=55
a= 997i= 100 ====time=149
a= 998i= 100 ====time=58
a= 999i= 100 ====time=51
all time = 91795
====================== 1000

1000批,一批100个event,一个事务

从测试结果来看,100MB数据,共计耗时92秒

这92秒做了如下几件事情:

1、AvroSink-->AvroSource

2、AvroSource-->RollingFileSink

3、RollingFileSink-->本地磁盘文件

那么如果我们要计算AvroSink到AvroSource的时间的话,应该要取出RollingFileSink的时间,我们通过JMX远程连接JVM来抽样看下这个时间是多少,减去就可以了

从抽样结果可以看出RollingFileSink花费时间26秒

去除改时间,那么AvroSink到AvroSource所花费的时间就是92-26=66秒

性能也就是:100MB/66S = 1.5MB/S

望各位不吝指教!!

时间: 2024-10-05 14:25:55

【Flume】flume中sink到source的性能测试,以Avro为例来测试的相关文章

Flume NG 学习笔记(十) Transaction、Sink、Source和Channel开发

版权声明:本文为博主原创文章,未经博主允许不得转载. 目录(?)[+] 一.Transaction interface Transaction接口是基于flume的稳定性考虑的.所有主要的组件(sources.sinks.channels)都必须使用Flume Transaction.我们也可以理解Transaction接口就是flume的事务,sources和sinks的发送数据与接受数据都是在一个Transaction里完成的. 从上图中可以看出,一个Transaction在Channel实

flume 自定义 hbase sink 类

参考(向原作者致敬) http://ydt619.blog.51cto.com/316163/1230586 https://blogs.apache.org/flume/entry/streaming_data_into_apache_hbase flume 1.5 的配置文件示例 #Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the

[Flume] - flume安装

Apache Flume是一个分布式的.可靠的.高效的系统,可以将不同来源的数据收集.聚合并移动到集中的数据存储中心上.Apache Flume不仅仅只是用到日志收集中.由于数据来源是可以定制的,flume可以使用传输大量的自定义event数据,包括但不限于网站流量信息.社会媒体信息.email信息以及其它可能的数据.Flume是Apache软件基金组织的顶级项目.官网http://flume.apache.org/. 一.安装 flume提供了二进制安装版本,所有我们可以选择直接下载二进制安装

Xcode中怎么引用Source Trees定义路径里的头文件

做的项目中需要引用Source Trees定义路径里的头文件,刚开始没有头绪,稍加思索发现太简单了. 此Source Trees非彼Source Tree,是Xcode"偏好设置"中的一个选项,见下图: 以上面的"QUICK_COCOS2DX_ROOT"一项为例,我遇到的问题是想在工程中引用这个路径下include文件夹里的头文件. 一般情况下如果想把某个文件夹中的头文件加入到工程中,可以在Build Setting中的Hearder Search Path里添加形

tomcat启动提示server.xml的context节点中不支持source属性警告的解决方法

Tomcat启动时提示: 2011-4-17 0:33:27 org.apache.tomcat.util.digester.SetPropertiesRule begin 警告: [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'source' to 'org.eclipse.jst.j2ee.server:Epai-0.1' did not find a matching property. [

shell中sh, exec, source, fork, ./的区别

https://www.cnblogs.com/cord/p/9226498.html shell中sh, exec, source, fork, ./的区别 1,sh sh test.sh ? sh是通过创建子进程(subshell)去执行脚本,父进程无法使用子进程中的变量,而子进程对环境变量的修改也不会影响到父进程.父进程中的局部变量子进程也无法使用,子进程会继承父进程的环境变量; ? 脚本不需要执行权限 2,exec exec + command ./test.sh exec ? 使用ex

浅谈服务器性能测试的全生命周期——从测试、结果分析到优化策略

本文由腾讯WeTest授权发布 作者:Micheal,腾讯资深后台开发工程师. 链接:http://wetest.qq.com/lab/view/?id=102 著作权归作者所有.商业转载请联系WeTest获得授权,非商业转载请注明出处. WeTest导读 服务器性能测试是一项非常重要而且必要的工作,本文是作者Micheal在对服务器进行性能测试的过程中不断摸索出来的一些实用策略,通过定位问题,分析原因以及解决问题,实现对服务器进行更有针对性的优化,提升服务器的性能. 1.服务器性能测试小结 讲

转:LR性能测试结果样例分析 测试结果分析

LoadRunner性能测试结果分析是个复杂的过程,通常可以从结果摘要.并发数.平均事务响应时间.每秒点击数.业务成功率.系统资源.网页细分图.Web服务器资源.数据库服务器资源等几个方面分析,如图1- 1所示.性能测试结果分析的一个重要的原则是以性能测试的需求指标为导向.我们回顾一下本次性能测试的目的,正如 所列的指标,本次测试的要求是验证在30分钟内完成2000次用户登录系统,然后进行考勤业务,最后退出,在业务操作过程中页面的响应时间不超过3秒,并且服务器的CPU使用率.内存使用率分别不超过

如何用VS2010在SharePoint中创建自定义字段类型(以eWebEditor为例)

如何用VS2010在SharePoint中创建自定义字段类型(以eWebEditor为例) 前提 项目中用到eWebEditor作为在线编辑器替换sharepoint2010自动的多行编辑器,下面以eWebEditor作为自定义字段类型为例来讲述如何用VS2010在sharepoint中创建自定义字段类型. 开发 1. 首先用VS2010创建一个空的sharepoint2010项目,如下图: 指向sharepoint站点,部署为场解决方案,如下图: 2. 在解决方案上添加“映射文件”,指向TEM