You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-16 - 10:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Tue Apr 16, 2024 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
378699280,1cyclictest0-21swapper/322:16:433
3786992620,5cyclictest0-21swapper/322:11:483
3786992620,5cyclictest0-21swapper/321:41:393
378699260,25cyclictest0-21swapper/319:41:323
378699260,25cyclictest0-21swapper/300:06:403
378699260,21cyclictest1249-21apt-get22:56:383
3777992624,2cyclictest0-21swapper/023:56:410
3786992523,1cyclictest0-21swapper/321:21:393
3786992522,3cyclictest0-21swapper/321:19:543
378699250,24cyclictest0-21swapper/322:34:223
378699250,24cyclictest0-21swapper/300:02:073
378699250,1cyclictest0-21swapper/300:33:193
378699250,0cyclictest0-21swapper/322:29:193
378299250,24cyclictest29814-21diskmemload21:43:152
378099250,24cyclictest0-21swapper/121:26:481
3786992420,4cyclictest0-21swapper/319:21:333
3786992419,4cyclictest29814-21diskmemload23:01:363
378699240,23cyclictest29814-21diskmemload00:27:073
378699240,23cyclictest17974-21ssh21:31:363
378699240,23cyclictest0-21swapper/323:41:073
378699240,23cyclictest0-21swapper/322:05:483
378699240,23cyclictest0-21swapper/322:02:073
378699240,23cyclictest0-21swapper/321:36:323
378699240,21cyclictest0-21swapper/322:52:083
378699240,0cyclictest29814-21diskmemload22:42:023
3782992422,1cyclictest29814-21diskmemload21:51:532
378299240,24cyclictest0-21swapper/221:34:222
378299240,23cyclictest0-21swapper/223:37:442
378299240,23cyclictest0-21swapper/223:30:002
378299240,23cyclictest0-21swapper/222:05:542
3780992422,2cyclictest0-21swapper/123:05:471
3780992421,2cyclictest22-21ksoftirqd/120:25:011
378099240,24cyclictest0-21swapper/100:39:481
378099240,24cyclictest0-21swapper/100:34:361
378099240,23cyclictest30162-21ssh21:10:101
378099240,23cyclictest29814-21diskmemload00:24:381
378099240,1cyclictest29814-21diskmemload22:23:411
378099240,1cyclictest29814-21diskmemload00:00:391
3777992424,0cyclictest62150irq/126-eno122:10:090
3777992423,1cyclictest29814-21diskmemload22:22:520
3777992423,0cyclictest8076-21apt-get21:21:380
3777992421,2cyclictest24405-21basename00:30:000
377799240,2cyclictest32523-21munin-plugin-st22:55:000
377799240,24cyclictest0-21swapper/021:18:080
377799240,23cyclictest2846-21ssh21:48:390
377799240,0cyclictest29814-21diskmemload21:55:160
377799240,0cyclictest0-21swapper/023:18:310
3786992323,0cyclictest0-21swapper/321:10:093
3786992321,2cyclictest0-21swapper/321:50:253
3786992321,1cyclictest0-21swapper/300:14:373
3786992320,2cyclictest28240-21sh23:59:093
3786992319,4cyclictest0-21swapper/300:16:363
378699230,23cyclictest0-21swapper/300:24:413
378699230,22cyclictest4590-21munin-plugin-st22:25:003
378699230,22cyclictest28162-21turbostat23:25:013
378699230,20cyclictest0-21swapper/323:36:503
378699230,0cyclictest0-21swapper/321:45:303
378699230,0cyclictest0-21swapper/321:27:133
3782992323,0cyclictest0-21swapper/223:53:472
3782992323,0cyclictest0-21swapper/223:30:032
3782992323,0cyclictest0-21swapper/221:21:212
3782992322,1cyclictest28-21ksoftirqd/223:05:102
3782992321,1cyclictest27168-21apt22:14:592
3782992320,3cyclictest0-21swapper/221:11:362
378299230,23cyclictest0-21swapper/223:47:172
378299230,23cyclictest0-21swapper/223:02:052
378299230,22cyclictest597-21dbus-daemon22:59:592
378299230,22cyclictest25726-21fschecks_time23:56:442
378299230,22cyclictest16668-21df_inode00:21:402
378299230,22cyclictest10133-21sh00:14:312
378299230,22cyclictest0-21swapper/222:52:382
378299230,22cyclictest0-21swapper/200:30:082
378299230,1cyclictest0-21swapper/221:46:512
378299230,0cyclictest0-21swapper/223:40:172
378299230,0cyclictest0-21swapper/221:35:052
378299230,0cyclictest0-21swapper/200:18:402
3780992323,0cyclictest29814-21diskmemload22:53:371
3780992323,0cyclictest29814-21diskmemload22:40:141
3780992323,0cyclictest29814-21diskmemload22:16:331
3780992322,1cyclictest29814-21diskmemload00:06:151
3780992321,1cyclictest597-21dbus-daemon21:40:001
3780992321,1cyclictest32104-21ssh21:46:121
378099230,2cyclictest24542-21ssh23:55:281
378099230,23cyclictest0-21swapper/123:45:321
378099230,23cyclictest0-21swapper/123:14:391
378099230,22cyclictest8180-21/usr/sbin/munin21:21:391
378099230,22cyclictest6845-21sh21:52:111
378099230,22cyclictest0-21swapper/123:41:451
378099230,22cyclictest0-21swapper/122:28:351
378099230,22cyclictest0-21swapper/122:06:371
378099230,1cyclictest18640-21systemd-journal20:45:001
378099230,0cyclictest29814-21diskmemload23:52:401
378099230,0cyclictest29814-21diskmemload22:13:481
378099230,0cyclictest0-21swapper/123:33:191
378099230,0cyclictest0-21swapper/121:40:591
3777992323,0cyclictest0-21swapper/023:47:230
3777992323,0cyclictest0-21swapper/023:02:520
3777992322,1cyclictest1313-21ssh21:13:260
3777992321,1cyclictest7420-21dump-pmu-power20:25:000
3777992321,1cyclictest13068-21ssh21:26:250
377799230,23cyclictest29814-21diskmemload21:52:310
377799230,23cyclictest29814-21diskmemload00:14:140
377799230,23cyclictest0-21swapper/023:10:380
377799230,23cyclictest0-21swapper/022:09:420
377799230,23cyclictest0-21swapper/021:40:580
377799230,23cyclictest0-21swapper/021:31:340
377799230,22cyclictest6110-21idleruntime-cro19:15:000
377799230,22cyclictest20109-21rm23:50:520
377799230,22cyclictest17830-21dump-pmu-power20:45:000
377799230,22cyclictest17454-21rm22:38:050
377799230,0cyclictest29814-21diskmemload00:24:480
377799230,0cyclictest0-21swapper/023:42:570
377799230,0cyclictest0-21swapper/023:27:480
3786992220,2cyclictest0-21swapper/323:54:233
3786992220,2cyclictest0-21swapper/323:10:073
3786992220,1cyclictest605-21rs:main3
3786992220,1cyclictest0-21swapper/321:56:423
3786992219,2cyclictest0-21swapper/323:27:003
3786992218,3cyclictest0-21swapper/300:36:413
378699220,22cyclictest0-21swapper/323:05:473
378699220,1cyclictest15890-21apt-get23:46:393
378699220,1cyclictest12625-21munin-run20:35:003
378699220,1cyclictest0-21swapper/322:35:093
378699220,0cyclictest29814-21diskmemload23:34:153
3782992222,0cyclictest29814-21diskmemload23:16:232
3782992222,0cyclictest0-21swapper/223:20:332
3782992222,0cyclictest0-21swapper/222:19:572
3782992222,0cyclictest0-21swapper/222:04:082
3782992222,0cyclictest0-21swapper/200:29:112
3782992221,1cyclictest0-21swapper/221:25:342
3782992221,1cyclictest0-21swapper/220:25:022
3782992220,2cyclictest0-21swapper/222:45:572
3782992220,1cyclictest27813-21apt-get21:06:372
3782992220,1cyclictest27813-21apt-get21:06:372
3782992220,1cyclictest15755-21munin-run19:35:002
3782992220,0cyclictest0-21swapper/200:00:202
378299220,22cyclictest0-21swapper/200:38:212
378299220,22cyclictest0-21swapper/200:06:242
378299220,21cyclictest29814-21diskmemload21:15:462
378299220,21cyclictest0-21swapper/222:30:282
378299220,1cyclictest10492-21kernelversion21:56:462
378299220,1cyclictest0-21swapper/222:28:562
3780992222,0cyclictest0-21swapper/100:11:421
3780992220,1cyclictest18540-21munin-run22:40:001
3780992220,1cyclictest11790-21sh21:57:301
378099220,22cyclictest29814-21diskmemload22:58:151
378099220,22cyclictest28480-1kworker/u9:221:30:091
378099220,22cyclictest0-21swapper/100:15:571
378099220,21cyclictest8790-21ssh23:38:221
378099220,21cyclictest13238-21apt-get20:35:021
378099220,21cyclictest10598-21apt-get20:31:341
378099220,1cyclictest2894-21ssh21:15:521
378099220,1cyclictest27174-21rm23:23:151
378099220,1cyclictest26250-21sh22:47:031
378099220,0cyclictest0-21swapper/123:01:381
378099220,0cyclictest0-21swapper/122:34:431
378099220,0cyclictest0-21swapper/122:01:281
3777992222,0cyclictest0-21swapper/023:20:190
3777992222,0cyclictest0-21swapper/022:25:250
3777992221,1cyclictest18533-21apt-get19:41:320
377799220,2cyclictest19396-21rm22:41:130
377799220,22cyclictest29814-21diskmemload00:15:030
377799220,22cyclictest0-21swapper/023:36:040
377799220,22cyclictest0-21swapper/000:30:490
377799220,21cyclictest0-21swapper/022:01:040
377799220,1cyclictest25028-21dump-pmu-power21:00:000
377799220,1cyclictest23446-21ssh22:45:040
377799220,1cyclictest15395-21ls20:39:590
377799220,1cyclictest0-21swapper/023:09:150
377799220,0cyclictest29814-21diskmemload00:38:120
377799220,0cyclictest0-21swapper/022:56:110
377799220,0cyclictest0-21swapper/022:17:490
3786992121,0cyclictest0-21swapper/323:17:403
3786992119,1cyclictest8559-21perf19:20:003
3782992121,0cyclictest29814-21diskmemload22:24:362
378299210,21cyclictest0-21swapper/222:36:122
378299210,21cyclictest0-21swapper/219:56:332
378299210,20cyclictest20598-21apt-get20:51:322
378299210,1cyclictest13389-21apt-get20:36:332
378299210,0cyclictest0-21swapper/223:10:072
378299210,0cyclictest0-21swapper/222:41:482
378299210,0cyclictest0-21swapper/220:21:342
3780992121,0cyclictest0-21swapper/123:15:411
3780992121,0cyclictest0-21swapper/119:51:331
3780992120,0cyclictest20898-21apt-get00:26:411
378099210,21cyclictest0-21swapper/123:25:361
378099210,20cyclictest0-21swapper/121:06:381
378099210,20cyclictest0-21swapper/121:06:381
378099210,20cyclictest0-21swapper/119:21:331
3777992121,0cyclictest23324-21apt-get19:51:330
3777992121,0cyclictest0-21swapper/023:31:310
377799210,21cyclictest0-21swapper/022:30:270
377799210,21cyclictest0-21swapper/019:16:330
377799210,20cyclictest0-21swapper/021:06:390
377799210,20cyclictest0-21swapper/021:06:380
377799210,1cyclictest141rcuc/020:20:010
377799210,0cyclictest0-21swapper/000:05:420
377799210,0cyclictest0-21swapper/000:00:070
3786992020,0cyclictest0-21swapper/320:46:353
3786992020,0cyclictest0-21swapper/320:16:323
3786992020,0cyclictest0-21swapper/319:51:333
3786992018,1cyclictest30145-21cron20:05:003
3786992017,3cyclictest0-21swapper/320:51:323
378699200,0cyclictest0-21swapper/321:01:393
3782992020,0cyclictest0-21swapper/220:56:402
3782992020,0cyclictest0-21swapper/220:41:322
3780992018,2cyclictest22-21ksoftirqd/120:05:001
378099200,20cyclictest0-21swapper/119:46:331
378099200,0cyclictest0-21swapper/121:01:391
3777992020,0cyclictest0-21swapper/020:46:340
3777992020,0cyclictest0-21swapper/020:26:330
3777992020,0cyclictest0-21swapper/020:11:330
3777992020,0cyclictest0-21swapper/020:06:340
3777992020,0cyclictest0-21swapper/019:46:330
3786991919,0cyclictest0-21swapper/320:06:343
378699190,1cyclictest17804-21cron20:45:003
3782991919,0cyclictest20930-21apt-get19:46:342
3782991919,0cyclictest0-21swapper/220:16:332
3780991919,0cyclictest0-21swapper/120:46:331
3780991919,0cyclictest0-21swapper/120:16:331
3780991919,0cyclictest0-21swapper/119:56:331
3780991919,0cyclictest0-21swapper/119:31:331
3780991919,0cyclictest0-21swapper/119:26:331
3777991919,0cyclictest0-21swapper/020:31:340
3777991919,0cyclictest0-21swapper/019:21:340
3782991818,0cyclictest8927-21apt-get19:21:322
3782991818,0cyclictest0-21swapper/219:51:332
3782991818,0cyclictest0-21swapper/219:11:322
3782991816,1cyclictest16114-21apt-get19:36:322
3782991816,1cyclictest13350-21grep19:30:002
3780991814,3cyclictest10209-21cron20:30:011
378099180,0cyclictest30543-21apt-get20:06:341
3777991818,0cyclictest0-21swapper/020:51:320
3786991717,0cyclictest0-21swapper/321:06:383
3786991717,0cyclictest0-21swapper/321:06:383
3786991717,0cyclictest0-21swapper/320:36:323
3782991717,0cyclictest0-21swapper/220:06:342
378699160,16cyclictest0-21swapper/319:29:413
3782991616,0cyclictest0-21swapper/221:01:392
3782991616,0cyclictest0-21swapper/219:16:342
3780991616,0cyclictest23004-21apt-get20:56:411
3780991616,0cyclictest0-21swapper/120:11:341
3780991612,3cyclictest22-21ksoftirqd/119:36:441
378099160,1cyclictest20753-21sed19:45:011
3777991616,0cyclictest0-21swapper/019:36:320
3786991515,0cyclictest0-21swapper/320:26:333
3786991515,0cyclictest0-21swapper/320:21:343
378699150,15cyclictest0-21swapper/320:58:043
378699150,15cyclictest0-21swapper/320:14:023
3782991515,0cyclictest0-21swapper/220:31:342
3780991515,0cyclictest6506-21apt-get19:16:331
3777991513,1cyclictest27427-21ls21:05:000
3786991414,0cyclictest0-21swapper/319:45:233
378699140,14cyclictest0-21swapper/319:55:243
378699140,14cyclictest0-21swapper/319:35:013
378699140,0cyclictest0-21swapper/319:12:453
3782991414,0cyclictest0-21swapper/220:10:222
378299140,14cyclictest0-21swapper/220:02:222
378299140,0cyclictest0-21swapper/220:45:022
378299140,0cyclictest0-21swapper/219:40:392
3780991414,0cyclictest0-21swapper/119:11:061
378099140,14cyclictest0-21swapper/120:51:041
3777991414,0cyclictest0-21swapper/019:55:330
3777991414,0cyclictest0-21swapper/019:30:400
3777991414,0cyclictest0-21swapper/019:25:390
377799140,0cyclictest0-21swapper/020:00:490
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional