You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-08 - 04:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot5s.osadl.org (updated Sat Aug 08, 2026 00:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1488999280,27cyclictest8254-21latency_hist00:00:011
1488999280,27cyclictest0-21swapper/122:32:091
1488999280,27cyclictest0-21swapper/100:22:581
14889992720,6cyclictest3923-21cstates23:29:101
14889992720,6cyclictest20471-21sendmail21:20:001
1488999270,26cyclictest0-21swapper/123:09:111
1488999270,26cyclictest0-21swapper/120:34:111
1488999270,26cyclictest0-21swapper/100:29:171
1488999270,26cyclictest0-21swapper/100:09:091
14889992622,4cyclictest0-21swapper/122:17:071
14889992620,5cyclictest0-21swapper/121:34:091
1488999260,5cyclictest0-21swapper/123:49:141
1488999260,26cyclictest29277-21/usr/sbin/munin21:29:141
1488999260,26cyclictest14125-21ssh00:34:061
1488999260,26cyclictest0-21swapper/121:44:431
1488999260,25cyclictest32288-21sh22:57:071
1488999260,25cyclictest25938-21ssh00:15:371
1488999260,25cyclictest22-21ksoftirqd/122:38:271
1488999260,25cyclictest0-21swapper/123:38:211
1488999260,25cyclictest0-21swapper/100:12:211
1489599250,24cyclictest8935-21diskmemload22:28:243
14893992520,4cyclictest0-21swapper/223:29:002
1489399250,22cyclictest22311-21rm21:22:092
14889992523,2cyclictest0-21swapper/123:15:081
14889992522,3cyclictest0-21swapper/121:21:371
14889992520,5cyclictest8935-21diskmemload22:41:151
1488999250,5cyclictest8935-21diskmemload21:47:281
1488999250,4cyclictest0-21swapper/100:35:481
1488999250,3cyclictest0-21swapper/121:52:011
1488999250,25cyclictest0-21swapper/122:53:541
1488999250,25cyclictest0-21swapper/122:12:471
1488999250,24cyclictest8935-21diskmemload22:02:461
1488999250,24cyclictest0-21swapper/119:19:111
1488599250,24cyclictest0-21swapper/021:52:540
14895992421,2cyclictest32025-21sh21:59:153
1489599240,24cyclictest8935-21diskmemload23:47:083
1489599240,24cyclictest8935-21diskmemload23:38:513
1489599240,24cyclictest0-21swapper/322:22:583
1489599240,24cyclictest0-21swapper/322:10:503
1489599240,23cyclictest9395-21ssh21:10:163
1489599240,23cyclictest8935-21diskmemload23:23:133
1489599240,23cyclictest8935-21diskmemload22:41:193
1489599240,23cyclictest8935-21diskmemload21:18:113
1489599240,23cyclictest11599-21apt-get00:04:123
1489599240,23cyclictest0-21swapper/323:07:503
14893992423,1cyclictest0-21swapper/222:41:162
14893992421,0cyclictest0-21swapper/200:30:242
1489399240,23cyclictest6088-21ssh21:35:512
1489399240,23cyclictest14535-21ssh21:14:252
1489399240,23cyclictest12700-21ssh23:36:042
1489399240,0cyclictest8935-21diskmemload00:14:192
14889992421,3cyclictest0-21swapper/122:45:521
14889992420,3cyclictest0-21swapper/122:06:201
14889992419,4cyclictest0-21swapper/121:14:021
1488999240,3cyclictest0-21swapper/121:55:411
1488999240,24cyclictest0-21swapper/123:40:161
1488999240,23cyclictest0-21swapper/123:21:441
1488999240,23cyclictest0-21swapper/121:39:561
14885992424,0cyclictest0-21swapper/023:11:440
14885992423,1cyclictest62550irq/127-eno123:30:170
14885992423,1cyclictest62550irq/127-eno121:48:370
14885992423,1cyclictest31883-21ssh00:20:500
14885992423,1cyclictest0-21swapper/021:36:430
1488599240,24cyclictest0-21swapper/023:53:550
1488599240,24cyclictest0-21swapper/022:51:400
1488599240,23cyclictest8935-21diskmemload22:26:070
1488599240,23cyclictest8935-21diskmemload21:55:450
1488599240,23cyclictest22152-21perf20:30:000
1488599240,23cyclictest0-21swapper/023:37:210
1488599240,23cyclictest0-21swapper/023:25:130
1488599240,23cyclictest0-21swapper/022:58:000
1488599240,0cyclictest0-21swapper/023:09:230
1488599240,0cyclictest0-21swapper/021:17:090
14895992323,0cyclictest0-21swapper/323:55:013
14895992323,0cyclictest0-21swapper/322:51:593
14895992323,0cyclictest0-21swapper/300:25:363
14895992323,0cyclictest0-21swapper/300:09:013
14895992320,2cyclictest18256-21ssh22:15:103
1489599230,23cyclictest0-21swapper/323:15:483
1489599230,23cyclictest0-21swapper/321:45:153
1489599230,23cyclictest0-21swapper/300:12:463
1489599230,22cyclictest3550-21sh22:31:163
1489599230,22cyclictest30971-21sh00:20:003
1489599230,22cyclictest1336-21ssh00:23:273
1489599230,22cyclictest11475-21ssh00:30:333
1489599230,22cyclictest0-21swapper/321:27:553
1489599230,0cyclictest8935-21diskmemload21:50:143
1489599230,0cyclictest0-21swapper/322:46:013
14893992323,0cyclictest0-21swapper/223:55:022
14893992323,0cyclictest0-21swapper/223:30:182
14893992323,0cyclictest0-21swapper/219:39:102
14893992323,0cyclictest0-21swapper/200:21:032
14893992320,2cyclictest0-21swapper/222:22:102
14893992320,0cyclictest0-21swapper/222:52:042
1489399230,2cyclictest28674-21chrt21:28:122
1489399230,23cyclictest8935-21diskmemload23:04:062
1489399230,23cyclictest0-21swapper/223:51:122
1489399230,23cyclictest0-21swapper/221:50:102
1489399230,22cyclictest30695-21apt-get19:44:072
1489399230,22cyclictest17232-21sh21:45:472
1489399230,22cyclictest16472-21ssh23:11:192
1489399230,22cyclictest16388-21ssh21:16:472
1489399230,22cyclictest14453-21ssh22:12:402
1489399230,22cyclictest13654-21ssh00:04:492
1489399230,22cyclictest0-21swapper/222:45:252
1489399230,22cyclictest0-21swapper/219:15:012
1489399230,0cyclictest0-21swapper/223:20:162
1489399230,0cyclictest0-21swapper/222:38:062
1489399230,0cyclictest0-21swapper/200:36:472
14889992321,1cyclictest5864-21apt-get19:59:101
14889992320,2cyclictest12487-21idleruntime-cro20:10:011
14889992319,3cyclictest0-21swapper/122:26:251
1488999230,23cyclictest0-21swapper/123:12:131
1488999230,23cyclictest0-21swapper/122:20:271
1488999230,22cyclictest30123-21perf19:40:001
1488999230,22cyclictest19867-21grep20:25:001
1488999230,22cyclictest0-21swapper/123:01:221
1488999230,22cyclictest0-21swapper/100:03:001
1488999230,0cyclictest0-21swapper/123:32:221
14885992323,0cyclictest8935-21diskmemload22:07:140
14885992323,0cyclictest0-21swapper/022:37:310
14885992323,0cyclictest0-21swapper/022:24:010
14885992323,0cyclictest0-21swapper/022:15:200
14885992323,0cyclictest0-21swapper/000:14:280
14885992321,2cyclictest8935-21diskmemload00:08:020
14885992321,1cyclictest0-21swapper/023:16:560
1488599230,2cyclictest32638-21sh23:24:540
1488599230,23cyclictest0-21swapper/022:10:460
1488599230,23cyclictest0-21swapper/022:02:480
1488599230,23cyclictest0-21swapper/000:18:390
1488599230,22cyclictest10363-21sh21:11:260
1488599230,22cyclictest0-21swapper/023:00:540
1488599230,22cyclictest0-21swapper/000:30:160
1488599230,1cyclictest0-21swapper/000:03:170
1488599230,0cyclictest0-21swapper/022:47:330
1488599230,0cyclictest0-21swapper/021:41:520
14895992222,0cyclictest8935-21diskmemload22:00:413
14895992222,0cyclictest0-21swapper/323:01:453
14895992221,1cyclictest16833-21ssh23:11:523
14895992220,1cyclictest11491-21ssh21:40:363
1489599221,21cyclictest25835-21kworker/3:123:55:473
1489599220,2cyclictest21145-21sh21:20:313
1489599220,22cyclictest0-21swapper/323:40:593
1489599220,22cyclictest0-21swapper/322:36:583
1489599220,21cyclictest9006-21ssh23:33:383
1489599220,21cyclictest8935-21diskmemload21:30:013
1489599220,21cyclictest17546-21ssh00:35:493
1489599220,21cyclictest0-21swapper/319:29:103
1489599220,21cyclictest0-21swapper/319:19:113
1489599220,1cyclictest7342-21ssh22:05:423
1489599220,1cyclictest31009-21ssh22:55:233
14893992222,0cyclictest8935-21diskmemload23:08:062
14893992222,0cyclictest8935-21diskmemload00:29:192
14893992222,0cyclictest0-21swapper/223:15:272
14893992222,0cyclictest0-21swapper/221:30:022
14893992222,0cyclictest0-21swapper/220:24:102
14893992222,0cyclictest0-21swapper/200:07:502
14893992220,2cyclictest29703-21ssh21:57:282
14893992220,1cyclictest0-21swapper/219:30:012
14893992220,0cyclictest0-21swapper/223:40:582
1489399220,2cyclictest2500-21apt-get19:50:022
1489399220,22cyclictest0-21swapper/221:42:582
1489399220,21cyclictest0-21swapper/220:05:012
1489399220,1cyclictest3028-21ssh22:02:322
1489399220,1cyclictest25914-21rm00:15:362
1489399220,0cyclictest0-21swapper/223:48:232
14889992220,1cyclictest0-21swapper/119:55:001
1488999220,21cyclictest9123-21grep20:04:121
1488999220,21cyclictest28154-21sed20:44:121
1488999220,21cyclictest13070-21apt-get20:14:101
1488999220,21cyclictest0-21swapper/119:49:121
1488999220,21cyclictest0-21swapper/119:29:101
1488999220,21cyclictest0-21swapper/119:14:091
14885992221,0cyclictest0-21swapper/023:55:250
14885992220,1cyclictest20110-21perf19:20:010
14885992220,1cyclictest0-21swapper/021:30:330
1488599220,21cyclictest4358-21apt-get00:25:030
1488599220,21cyclictest0-21swapper/023:45:180
1488599220,21cyclictest0-21swapper/023:40:310
1488599220,21cyclictest0-21swapper/022:40:430
1488599220,21cyclictest0-21swapper/021:25:570
1488599220,21cyclictest0-21swapper/021:21:260
1488599220,1cyclictest8945-21turbostat.cron21:10:000
1488599220,1cyclictest0-21swapper/000:35:300
14895992120,0cyclictest3472-21apt-get19:54:103
14895992120,0cyclictest0-21swapper/320:24:103
14895992119,1cyclictest0-21swapper/323:25:283
1489599210,20cyclictest30128-21sendmail-msp19:40:003
1489599210,20cyclictest2326-21apt-get20:59:103
1489599210,20cyclictest0-21swapper/319:49:113
1489599210,0cyclictest15831-21apt-get19:14:093
14893992121,0cyclictest8935-21diskmemload22:32:032
14893992120,1cyclictest27553-21apt-get20:44:102
14893992119,1cyclictest22168-21munin-run20:30:002
1489399210,21cyclictest0-21swapper/222:26:212
1489399210,0cyclictest0-21swapper/222:56:082
1488999210,20cyclictest0-21swapper/120:59:201
1488999210,20cyclictest0-21swapper/120:39:111
1488999210,0cyclictest0-21swapper/121:09:101
14885992120,1cyclictest17895-21apt-get20:24:100
14885992120,1cyclictest17316-21sendmail-msp20:20:010
1488599210,0cyclictest0-21swapper/020:44:100
1488599210,0cyclictest0-21swapper/019:34:090
14895992020,0cyclictest0-21swapper/319:44:073
14895992019,1cyclictest34-21ksoftirqd/320:00:013
1489599200,0cyclictest0-21swapper/321:04:113
14893992020,0cyclictest654-21apt-get19:49:112
14893992020,0cyclictest32342-21apt-get20:54:102
14893992020,0cyclictest18232-21apt-get19:19:112
1489399200,0cyclictest20699-21apt-get19:24:102
14889992020,0cyclictest0-21swapper/120:49:101
14889992020,0cyclictest0-21swapper/119:24:091
14889992019,1cyclictest0-21swapper/119:30:021
1488999200,20cyclictest0-21swapper/120:19:141
1488999200,19cyclictest0-21swapper/121:04:171
1488999200,19cyclictest0-21swapper/119:44:081
14885992020,0cyclictest29948-21apt-get20:49:100
14885992020,0cyclictest0-21swapper/020:34:110
1488599200,0cyclictest0-21swapper/020:40:000
1488599200,0cyclictest0-21swapper/019:39:100
14895991918,0cyclictest8280-21apt-get20:04:083
14895991918,0cyclictest0-21swapper/320:54:093
1489599190,18cyclictest0-21swapper/320:09:103
14893991919,0cyclictest29948-21apt-get20:49:102
14893991919,0cyclictest0-21swapper/220:14:102
14893991919,0cyclictest0-21swapper/220:09:112
14893991919,0cyclictest0-21swapper/219:30:022
1489399190,1cyclictest15478-21apt-get20:19:112
14889991919,0cyclictest0-21swapper/120:29:101
14885991919,0cyclictest0-21swapper/021:04:110
14895991817,0cyclictest0-21swapper/320:44:103
14893991818,0cyclictest0-21swapper/221:04:112
14893991817,1cyclictest22742-21apt-get20:34:102
1489399180,17cyclictest0-21swapper/220:59:112
14889991818,0cyclictest0-21swapper/120:54:091
14885991818,0cyclictest0-21swapper/020:59:100
14885991818,0cyclictest0-21swapper/020:54:090
14885991818,0cyclictest0-21swapper/020:04:080
14885991818,0cyclictest0-21swapper/019:49:110
14893991715,1cyclictest8924-21cron21:10:002
14885991717,0cyclictest0-21swapper/019:29:100
14885991717,0cyclictest0-21swapper/019:24:090
14885991716,1cyclictest14810-21apt-get19:10:030
14895991616,0cyclictest0-21swapper/320:49:103
14895991616,0cyclictest0-21swapper/320:29:113
14893991616,0cyclictest0-21swapper/220:40:002
14885991616,0cyclictest0-21swapper/019:59:100
14885991614,2cyclictest5448-21sed19:55:010
14895991515,0cyclictest25137-21apt-get20:39:113
14895991515,0cyclictest0-21swapper/319:24:093
14895991513,1cyclictest0-21swapper/320:19:113
14893991515,0cyclictest0-21swapper/219:58:502
14885991515,0cyclictest0-21swapper/020:09:110
1489599143,10cyclictest10060-21munin-node20:30:283
1489599140,14cyclictest0-21swapper/321:05:593
1489599140,0cyclictest0-21swapper/320:10:463
1488599140,14cyclictest0-21swapper/019:40:310
1488599140,0cyclictest0-21swapper/020:10:230
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional