You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-05-04 - 04:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Fri May 03, 2024 12:44:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1632399280,27cyclictest9709-21diskmemload09:42:592
1632399270,26cyclictest0-21swapper/210:33:282
16323992622,4cyclictest9709-21diskmemload10:05:162
16323992622,4cyclictest9709-21diskmemload09:51:572
1632399260,0cyclictest0-21swapper/212:35:132
1632399260,0cyclictest0-21swapper/209:25:152
16326992523,1cyclictest0-21swapper/312:06:533
1632699250,24cyclictest0-21swapper/311:21:113
1632699250,24cyclictest0-21swapper/309:08:313
16323992522,3cyclictest0-21swapper/210:18:132
16323992521,4cyclictest9709-21diskmemload11:11:452
1632399250,2cyclictest20509-21awk08:25:002
1632399250,21cyclictest0-21swapper/212:30:082
1632399250,20cyclictest0-21swapper/210:53:302
1632199250,0cyclictest0-21swapper/109:36:281
1631599250,1cyclictest9880-21dump-pmu-power10:50:000
1632699240,2cyclictest21774-21ssh09:53:413
1632699240,24cyclictest0-21swapper/311:30:243
1632699240,23cyclictest9709-21diskmemload10:52:553
1632699240,23cyclictest26477-21apt-get07:33:303
1632699240,23cyclictest0-21swapper/310:46:003
16323992423,1cyclictest25339-21sh09:57:562
16323992421,2cyclictest21694-21sh09:23:332
16323992420,3cyclictest9709-21diskmemload11:29:052
16323992420,0cyclictest9709-21diskmemload10:13:302
16323992420,0cyclictest0-21swapper/211:42:032
16323992419,0cyclictest0-21swapper/210:43:282
1632399240,20cyclictest0-21swapper/212:17:322
16321992424,0cyclictest0-21swapper/112:18:141
16321992420,3cyclictest14508-21dump-pmu-power10:55:001
1632199240,2cyclictest14998-21ssh10:55:301
1632199240,24cyclictest0-21swapper/109:56:001
1632199240,23cyclictest9710-21cron09:10:001
1632199240,23cyclictest0-21swapper/109:21:211
1632199240,22cyclictest18640-21systemd-journal08:20:001
16315992424,0cyclictest0-21swapper/009:57:580
16315992423,1cyclictest9709-21diskmemload09:53:580
16315992421,2cyclictest24820-21rm12:16:340
16315992420,3cyclictest18640-21systemd-journal07:24:590
1631599240,24cyclictest0-21swapper/010:21:000
1631599240,24cyclictest0-21swapper/009:28:200
1631599240,23cyclictest9709-21diskmemload11:25:110
1631599240,23cyclictest3440-21rm09:36:450
1631599240,23cyclictest13040-21ssh09:45:160
1631599240,1cyclictest9709-21diskmemload09:17:400
1631599240,1cyclictest62150irq/126-eno112:08:160
1631599240,0cyclictest9709-21diskmemload12:26:550
16326992323,0cyclictest9709-21diskmemload11:56:153
16326992323,0cyclictest9709-21diskmemload09:41:023
16326992323,0cyclictest9709-21diskmemload09:30:373
16326992323,0cyclictest9709-21diskmemload09:16:133
16326992323,0cyclictest0-21swapper/312:32:393
16326992323,0cyclictest0-21swapper/311:16:553
16326992323,0cyclictest0-21swapper/311:00:413
16326992323,0cyclictest0-21swapper/310:11:533
16326992323,0cyclictest0-21swapper/309:12:423
16326992321,1cyclictest24603-21rm09:25:363
1632699230,23cyclictest9709-21diskmemload10:39:423
1632699230,23cyclictest0-21swapper/311:36:083
1632699230,23cyclictest0-21swapper/310:05:103
1632699230,22cyclictest30869-21turbostat.cron07:40:003
1632699230,21cyclictest151rcu_preempt09:48:383
1632699230,1cyclictest9709-21diskmemload09:35:533
1632699230,1cyclictest0-21swapper/312:11:083
1632699230,0cyclictest9709-21diskmemload10:57:043
1632699230,0cyclictest0-21swapper/310:41:163
1632699230,0cyclictest0-21swapper/309:58:093
1632699230,0cyclictest0-21swapper/309:22:123
16323992323,0cyclictest9709-21diskmemload09:30:312
16323992323,0cyclictest0-21swapper/210:57:372
16323992319,0cyclictest0-21swapper/211:54:562
1632399230,2cyclictest30160-21munin-plugin-st08:45:002
1632399230,23cyclictest0-21swapper/211:02:512
1632399230,23cyclictest0-21swapper/210:26:482
1632399230,23cyclictest0-21swapper/209:38:322
1632399230,22cyclictest14179-21apt-get12:05:012
1632399230,22cyclictest0-21swapper/209:45:262
1632399230,20cyclictest9709-21diskmemload10:04:122
1632399230,20cyclictest0-21swapper/211:06:042
1632399230,0cyclictest0-21swapper/211:20:142
1632399230,0cyclictest0-21swapper/210:22:542
16321992323,0cyclictest9709-21diskmemload10:26:311
16321992323,0cyclictest0-21swapper/112:11:211
16321992323,0cyclictest0-21swapper/111:30:541
16321992323,0cyclictest0-21swapper/110:14:191
16321992323,0cyclictest0-21swapper/110:08:261
16321992323,0cyclictest0-21swapper/109:27:341
16321992323,0cyclictest0-21swapper/109:16:491
16321992321,1cyclictest0-21swapper/112:40:001
1632199230,2cyclictest29856-21sh09:30:361
1632199230,23cyclictest9709-21diskmemload10:19:001
1632199230,23cyclictest0-21swapper/111:36:471
1632199230,23cyclictest0-21swapper/111:06:371
1632199230,23cyclictest0-21swapper/109:51:261
1632199230,22cyclictest2676-21sh10:43:231
1632199230,1cyclictest9386-21ssh09:42:491
1632199230,1cyclictest0-21swapper/109:10:191
1632199230,0cyclictest9709-21diskmemload11:25:141
1632199230,0cyclictest0-21swapper/112:26:151
1632199230,0cyclictest0-21swapper/112:08:191
1632199230,0cyclictest0-21swapper/111:15:281
1632199230,0cyclictest0-21swapper/110:02:321
1632199230,0cyclictest0-21swapper/109:45:251
16315992323,0cyclictest9709-21diskmemload12:21:500
16315992323,0cyclictest0-21swapper/010:44:210
16315992322,1cyclictest9709-21diskmemload12:35:350
16315992322,1cyclictest9709-21diskmemload11:06:140
16315992322,1cyclictest25651-21sh10:32:530
16315992322,1cyclictest0-21swapper/009:11:210
16315992321,1cyclictest30345-21ssh09:31:210
16315992321,1cyclictest30163-21ssh10:37:440
1631599231,1cyclictest62150irq/126-eno111:46:020
1631599230,23cyclictest62150irq/126-eno110:02:240
1631599230,22cyclictest9709-21diskmemload11:33:390
1631599230,22cyclictest22687-21ssh12:13:400
1631599230,22cyclictest11003-21ssh10:16:260
1631599230,22cyclictest0-21swapper/011:50:490
1631599230,0cyclictest0-21swapper/012:01:320
16326992222,0cyclictest9709-21diskmemload11:28:003
16326992222,0cyclictest0-21swapper/312:26:363
16326992222,0cyclictest0-21swapper/311:41:483
16326992222,0cyclictest0-21swapper/311:07:523
16326992221,1cyclictest9709-21diskmemload10:02:153
16326992220,1cyclictest5662-21perf07:55:003
16326992220,1cyclictest0-21swapper/312:15:403
1632699220,2cyclictest10844-21sh08:05:003
1632699220,22cyclictest0-21swapper/312:36:293
1632699220,22cyclictest0-21swapper/311:45:153
1632699220,21cyclictest28959-21sh12:20:383
1632699220,21cyclictest0-21swapper/312:00:353
1632699220,0cyclictest0-21swapper/310:31:453
16323992222,0cyclictest0-21swapper/212:24:052
16323992221,1cyclictest14805-21ssh09:16:552
16323992221,1cyclictest0-21swapper/211:49:132
16323992221,0cyclictest0-21swapper/212:25:042
16323992220,1cyclictest7866-21apt-get09:08:322
16323992220,1cyclictest0-21swapper/211:56:162
1632399220,22cyclictest0-21swapper/211:36:002
1632399220,22cyclictest0-21swapper/211:34:212
1632399220,21cyclictest3048-21apt-get08:58:282
1632399220,21cyclictest21242-21apt-get12:13:292
1632399220,21cyclictest1582-21cron07:50:002
1632399220,21cyclictest0-21swapper/207:55:022
1632399220,1cyclictest27743-21perf08:40:002
1632399220,1cyclictest0-21swapper/211:17:102
1632399220,1cyclictest0-21swapper/207:20:002
1632399220,0cyclictest9709-21diskmemload09:12:082
16321992222,0cyclictest9709-21diskmemload11:58:441
16321992222,0cyclictest0-21swapper/112:01:411
16321992222,0cyclictest0-21swapper/111:20:311
16321992222,0cyclictest0-21swapper/111:02:061
16321992222,0cyclictest0-21swapper/110:34:421
16321992222,0cyclictest0-21swapper/110:23:221
16321992220,1cyclictest0-21swapper/111:10:301
1632199220,22cyclictest0-21swapper/111:51:051
1632199220,22cyclictest0-21swapper/111:40:011
1632199220,21cyclictest65742chrt07:58:271
1632199220,21cyclictest15697-21turbostat.cron08:14:591
1632199220,21cyclictest0-21swapper/112:20:091
1632199220,1cyclictest0-21swapper/112:31:041
1632199220,1cyclictest0-21swapper/107:20:001
16315992222,0cyclictest9709-21diskmemload10:50:410
16315992222,0cyclictest0-21swapper/010:12:300
16315992220,1cyclictest790-21ssh11:15:130
1631599220,22cyclictest0-21swapper/011:10:220
1631599220,22cyclictest0-21swapper/010:25:010
1631599220,22cyclictest0-21swapper/010:07:500
1631599220,1cyclictest7630-21ssh09:40:070
1631599220,1cyclictest22876-21ssh11:38:460
1631599220,0cyclictest9709-21diskmemload11:00:060
16326992121,0cyclictest1010-21kworker/u8:011:52:503
16326992121,0cyclictest0-21swapper/311:10:083
16326992121,0cyclictest0-21swapper/310:26:293
16326992120,1cyclictest18654-21apt-get08:23:273
16326992120,0cyclictest31400-21apt-get07:43:283
16326992119,1cyclictest25953-21munin-run07:30:003
16326992119,1cyclictest0-21swapper/309:05:003
1632699210,20cyclictest0-21swapper/307:18:263
1632699210,1cyclictest9709-21diskmemload10:15:433
1632699210,0cyclictest0-21swapper/308:13:273
16323992121,0cyclictest9709-21diskmemload10:38:062
16323992121,0cyclictest0-21swapper/212:02:382
16323992120,0cyclictest0-21swapper/207:28:272
16321992119,1cyclictest0-21swapper/111:45:041
1632199210,21cyclictest0-21swapper/110:47:571
1632199210,21cyclictest0-21swapper/110:36:151
1632199210,0cyclictest0-21swapper/108:43:261
1632199210,0cyclictest0-21swapper/107:13:291
16315992120,1cyclictest14823-21ssh10:55:100
16315992120,0cyclictest11380-21apt-get08:08:270
1631599210,21cyclictest0-21swapper/011:20:440
1631599210,20cyclictest0-21swapper/008:53:270
1631599210,0cyclictest0-21swapper/012:30:090
1631599210,0cyclictest0-21swapper/011:55:410
1631599210,0cyclictest0-21swapper/008:33:310
1631599210,0cyclictest0-21swapper/007:48:300
1631599210,0cyclictest0-21swapper/007:33:300
16326992020,0cyclictest0-21swapper/310:20:023
16326992020,0cyclictest0-21swapper/308:43:263
16326992020,0cyclictest0-21swapper/307:48:303
16326992019,0cyclictest0-21swapper/308:58:283
16323992020,0cyclictest0-21swapper/210:46:172
16321992020,0cyclictest0-21swapper/109:03:291
16321992020,0cyclictest0-21swapper/108:48:281
16321992020,0cyclictest0-21swapper/108:28:271
16321992020,0cyclictest0-21swapper/108:08:261
16321992020,0cyclictest0-21swapper/107:53:271
16321992020,0cyclictest0-21swapper/107:43:281
16315992020,0cyclictest0-21swapper/008:23:270
16326991919,0cyclictest0-21swapper/308:08:263
16326991918,0cyclictest0-21swapper/307:13:293
1632699190,0cyclictest0-21swapper/308:48:273
16323991919,0cyclictest0-21swapper/208:03:292
16323991918,0cyclictest0-21swapper/207:53:282
16321991919,0cyclictest0-21swapper/108:58:271
16321991919,0cyclictest0-21swapper/108:33:311
16321991918,0cyclictest0-21swapper/107:33:301
16321991917,1cyclictest10836-21cron08:05:001
16315991917,1cyclictest22914-21apt08:30:000
16326991818,0cyclictest0-21swapper/308:33:313
16326991818,0cyclictest0-21swapper/308:28:293
16323991818,0cyclictest0-21swapper/208:48:272
16323991817,1cyclictest619-21apt-get08:53:282
16321991818,0cyclictest0-21swapper/107:48:301
16315991818,0cyclictest0-21swapper/007:43:280
16315991817,1cyclictest26822-21avahi-daemon08:45:010
16315991817,0cyclictest0-21swapper/007:38:240
16326991717,0cyclictest0-21swapper/308:18:283
16323991717,0cyclictest0-21swapper/207:38:242
16323991715,1cyclictest16803-21apt-get07:13:292
16321991716,0cyclictest0-21swapper/108:53:271
16315991717,0cyclictest0-21swapper/008:58:280
16326991615,1cyclictest21644-21apt-get07:23:283
16326991614,1cyclictest8471-21munin-run08:00:013
1632399160,16cyclictest0-21swapper/207:43:282
1632399160,15cyclictest13797-21/usr/sbin/munin08:13:352
16321991616,0cyclictest0-21swapper/108:23:271
16321991615,1cyclictest27757-21basename08:40:001
16321991614,1cyclictest0-21swapper/107:23:311
1632199160,15cyclictest25124-21ntp_states07:28:351
16315991614,1cyclictest0-21swapper/008:40:010
16323991515,0cyclictest0-21swapper/207:23:262
16323991514,1cyclictest605-21rs:main2
1632399150,1cyclictest0-21swapper/207:30:482
1632399150,14cyclictest0-21swapper/208:15:262
16315991515,0cyclictest0-21swapper/008:18:280
16315991515,0cyclictest0-21swapper/008:03:290
16315991515,0cyclictest0-21swapper/007:58:270
16315991513,1cyclictest9732-21perf09:10:010
1631599150,8cyclictest587-21irqbalance08:46:200
1631599150,1cyclictest19196-21/usr/sbin/munin07:18:350
1631599150,14cyclictest597-21dbus-daemon08:14:590
1632699140,14cyclictest0-21swapper/308:50:173
1632699140,14cyclictest0-21swapper/308:35:163
1632399140,14cyclictest0-21swapper/209:01:132
1632399140,0cyclictest0-21swapper/208:30:172
1632399140,0cyclictest0-21swapper/208:05:222
1632199140,14cyclictest0-21swapper/107:35:011
16315991414,0cyclictest0-21swapper/007:25:460
16315991414,0cyclictest0-21swapper/007:10:300
1631599140,14cyclictest0-21swapper/007:50:370
1631599140,0cyclictest0-21swapper/009:00:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional