You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-18 - 10:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Sat Apr 18, 2026 00:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2551499270,23cyclictest12163-21ssh23:09:340
2552199260,1cyclictest6690-21ssh00:38:023
25516992623,0cyclictest151rcu_preempt23:47:111
25514992624,1cyclictest0-21swapper/021:30:160
25514992620,5cyclictest2739-1kworker/u9:000:39:540
2551499260,25cyclictest0-21swapper/022:21:160
2552199250,24cyclictest19767-21diskmemload00:08:573
2552199250,24cyclictest0-21swapper/323:24:533
2552199250,1cyclictest14864-21sh23:10:453
2551699252,22cyclictest22-21ksoftirqd/100:36:501
2551699251,23cyclictest10455-21idleruntime-cro00:10:011
2551699250,24cyclictest23390-21ssh22:48:171
2551699250,24cyclictest0-21swapper/122:34:331
2551699250,24cyclictest0-21swapper/122:07:101
2551699250,24cyclictest0-21swapper/121:12:241
25514992523,1cyclictest0-21swapper/023:50:020
25514992522,3cyclictest0-21swapper/022:19:530
25514992519,5cyclictest0-21swapper/021:20:070
25514992519,5cyclictest0-21swapper/000:11:490
2551499250,24cyclictest19767-21diskmemload22:48:590
2551499250,24cyclictest10692-21apt-get19:45:030
2551499250,24cyclictest0-21swapper/023:30:390
2551499250,24cyclictest0-21swapper/022:32:110
2551499250,24cyclictest0-21swapper/022:28:580
2551499250,24cyclictest0-21swapper/021:12:550
2551499250,1cyclictest29478-21apt-get22:54:530
2551499250,0cyclictest0-21swapper/022:07:340
2551499250,0cyclictest0-21swapper/000:30:210
2552199240,23cyclictest3911-21apt-get19:34:533
2552199240,23cyclictest0-21swapper/300:25:363
25520992424,0cyclictest0-21swapper/200:24:392
25520992420,4cyclictest0-21swapper/223:18:022
25520992419,4cyclictest0-21swapper/222:08:252
2552099240,23cyclictest14229-21grep19:50:022
2552099240,23cyclictest0-21swapper/221:52:012
2552099240,1cyclictest1834-21ssh00:33:272
25516992424,0cyclictest0-21swapper/100:12:431
25516992423,1cyclictest19767-21diskmemload22:57:381
2551699242,21cyclictest22-21ksoftirqd/123:19:211
2551699240,2cyclictest12065-21grep22:05:011
2551699240,24cyclictest0-21swapper/123:33:161
2551699240,0cyclictest0-21swapper/123:42:031
25514992423,1cyclictest17863-21diskmemload23:45:410
25514992421,3cyclictest62550irq/127-eno121:51:150
25514992421,2cyclictest0-21swapper/023:59:330
25514992420,3cyclictest19767-21diskmemload21:41:220
25514992419,4cyclictest19767-21diskmemload22:41:290
2551499240,23cyclictest17801-21taskset00:16:330
2551499240,0cyclictest19767-21diskmemload00:09:180
2551499240,0cyclictest0-21swapper/023:24:230
2551499240,0cyclictest0-21swapper/023:16:020
2551499240,0cyclictest0-21swapper/021:28:450
2551499240,0cyclictest0-21swapper/021:16:550
2551499240,0cyclictest0-21swapper/000:26:170
25521992323,0cyclictest0-21swapper/323:56:183
25521992323,0cyclictest0-21swapper/321:55:303
25521992323,0cyclictest0-21swapper/300:15:573
25521992323,0cyclictest0-21swapper/300:13:203
25521992322,1cyclictest30752-21kworker/u8:221:20:353
25521992322,1cyclictest3031-21ssh00:03:363
25521992322,1cyclictest19767-21diskmemload22:18:013
25521992322,1cyclictest19767-21diskmemload00:21:183
25521992320,2cyclictest24410-21ssh21:46:243
25521992320,2cyclictest12774-21ssh22:05:073
2552199230,23cyclictest0-21swapper/323:16:413
2552199230,23cyclictest0-21swapper/323:03:223
2552199230,23cyclictest0-21swapper/322:00:263
2552199230,22cyclictest3436-21ssh21:25:183
2552199230,22cyclictest27335-21sh22:51:313
2552199230,22cyclictest13563-21sh22:38:513
2552199230,22cyclictest11083-21ssh21:34:233
2552199230,0cyclictest19767-21diskmemload23:48:143
2552199230,0cyclictest0-21swapper/321:40:403
2552199230,0cyclictest0-21swapper/300:32:023
25520992323,0cyclictest0-21swapper/223:29:322
25520992323,0cyclictest0-21swapper/222:45:162
25520992323,0cyclictest0-21swapper/222:35:282
25520992323,0cyclictest0-21swapper/221:19:072
25520992321,1cyclictest2732-21awk21:25:002
25520992321,1cyclictest22907-21sh22:15:052
25520992320,2cyclictest2913-21ssh23:31:102
25520992320,2cyclictest18656-21ssh22:11:252
25520992320,2cyclictest16133-21latency_hist00:15:012
2552099230,2cyclictest15486-21ssh23:44:532
2552099230,23cyclictest0-21swapper/222:43:232
2552099230,22cyclictest3696-21sh00:04:462
2552099230,22cyclictest30588-21munin-run20:25:012
2552099230,22cyclictest25029-21ssh23:54:012
2552099230,22cyclictest19767-21diskmemload23:05:272
2552099230,22cyclictest16925-21sh23:14:132
2552099230,22cyclictest0-21swapper/221:36:122
2552099230,0cyclictest0-21swapper/223:37:212
2552099230,0cyclictest0-21swapper/222:55:232
2552099230,0cyclictest0-21swapper/221:47:032
2552099230,0cyclictest0-21swapper/200:25:032
25516992323,0cyclictest0-21swapper/123:10:101
25516992323,0cyclictest0-21swapper/123:08:121
25516992323,0cyclictest0-21swapper/122:13:301
25516992323,0cyclictest0-21swapper/119:54:521
25516992323,0cyclictest0-21swapper/100:01:311
25516992322,1cyclictest19767-21diskmemload21:22:001
25516992321,1cyclictest25775-21/usr/sbin/munin22:20:001
25516992321,1cyclictest0-21swapper/123:38:041
25516992321,1cyclictest0-21swapper/123:29:481
25516992321,1cyclictest0-21swapper/100:20:291
2551699230,2cyclictest28131-21rm23:55:131
2551699230,23cyclictest0-21swapper/123:20:571
2551699230,22cyclictest3471-21sh21:25:211
2551699230,22cyclictest29074-21sh21:50:411
2551699230,22cyclictest28014-21ssh22:52:321
2551699230,1cyclictest19767-21diskmemload23:03:071
2551699230,0cyclictest19767-21diskmemload22:25:181
2551699230,0cyclictest0-21swapper/123:50:251
2551699230,0cyclictest0-21swapper/100:30:161
25514992321,2cyclictest0-21swapper/023:02:050
25514992321,1cyclictest15877-21munin-run22:40:010
25514992321,1cyclictest0-21swapper/021:55:160
2551499230,23cyclictest0-21swapper/021:37:540
2551499230,23cyclictest0-21swapper/000:20:450
2551499230,23cyclictest0-21swapper/000:01:200
2551499230,22cyclictest18491-21ssh22:11:090
2551499230,22cyclictest13002-21sh23:40:520
2551499230,22cyclictest0-21swapper/023:27:180
2551499230,22cyclictest0-21swapper/021:45:020
2551499230,0cyclictest0-21swapper/023:10:350
2551499230,0cyclictest0-21swapper/022:00:580
25521992222,0cyclictest19767-21diskmemload23:06:013
25521992222,0cyclictest19767-21diskmemload21:11:143
25521992222,0cyclictest0-21swapper/322:46:053
25521992222,0cyclictest0-21swapper/322:25:503
25521992220,1cyclictest28528-21ssh22:20:573
25521992220,1cyclictest0-21swapper/323:35:043
2552199220,2cyclictest23083-21sh23:50:493
2552199220,22cyclictest0-21swapper/323:27:113
2552199220,21cyclictest7984-21ssh22:33:033
2552199220,21cyclictest16250-21munin-run19:55:003
2552199220,21cyclictest0-21swapper/322:57:173
2552199220,21cyclictest0-21swapper/321:15:073
2552199220,1cyclictest17138-21apt-get19:59:523
2552199220,1cyclictest0-21swapper/323:31:493
2552199220,1cyclictest0-21swapper/322:12:373
25520992222,0cyclictest19767-21diskmemload22:31:452
25520992222,0cyclictest0-21swapper/221:55:172
25520992222,0cyclictest0-21swapper/221:26:422
25520992220,1cyclictest7447-21ssh00:06:582
25520992220,1cyclictest0-21swapper/223:55:032
25520992220,1cyclictest0-21swapper/223:48:162
25520992220,0cyclictest0-21swapper/222:00:102
25520992219,0cyclictest0-21swapper/221:43:402
25520992218,3cyclictest606-21dbus-daemon20:05:002
2552099220,21cyclictest8807-21apt-get20:49:522
2552099220,21cyclictest30516-21munin-plugin-st19:20:002
2552099220,21cyclictest20755-21ssh21:10:312
2552099220,21cyclictest19767-21diskmemload23:00:452
2552099220,21cyclictest17045-21ssh00:15:182
2552099220,1cyclictest3423-21ssh22:29:382
2552099220,1cyclictest19801-21munin-run21:10:012
25516992222,0cyclictest0-21swapper/122:38:541
25516992222,0cyclictest0-21swapper/121:30:201
25516992221,1cyclictest0-21swapper/122:21:091
25516992220,1cyclictest783-21snmpd21:35:541
25516992220,1cyclictest29561-21ssh21:19:581
25516992219,2cyclictest16756-21/usr/sbin/munin00:15:021
2551699220,22cyclictest19767-21diskmemload22:43:591
2551699220,21cyclictest5345-21dpkg19:35:011
2551699220,21cyclictest522-21munin-run20:30:011
2551699220,21cyclictest0-21swapper/121:45:011
2551699220,21cyclictest0-21swapper/121:40:411
2551699220,21cyclictest0-21swapper/121:04:521
2551699220,21cyclictest0-21swapper/119:19:531
2551699220,1cyclictest0-21swapper/121:55:411
25514992222,0cyclictest0-21swapper/019:25:010
25514992220,1cyclictest0-21swapper/020:30:010
25514992220,1cyclictest0-21swapper/019:14:510
2551499220,21cyclictest0-21swapper/019:44:530
25521992121,0cyclictest0-21swapper/323:40:523
25521992120,1cyclictest21949-21apt-get20:09:523
25521992120,1cyclictest1497-21apt-get19:29:523
25521992120,1cyclictest1497-21apt-get19:29:523
2552199210,21cyclictest0-21swapper/321:09:533
2552199210,20cyclictest19767-21diskmemload22:40:163
2552199210,20cyclictest0-21swapper/320:29:533
25520992121,0cyclictest0-21swapper/221:30:252
2552099210,1cyclictest5368-21perf20:40:002
25516992121,0cyclictest0-21swapper/100:25:461
25516992119,1cyclictest22-21ksoftirqd/119:45:011
25516992119,1cyclictest18616-21sendmail-msp20:00:001
25514992121,0cyclictest16013-21apt-get21:04:520
25514992120,1cyclictest29109-21apt19:19:530
2551499210,21cyclictest0-21swapper/019:39:520
2551499210,20cyclictest0-21swapper/020:49:520
2551499210,0cyclictest14744-21apt-get19:54:520
25521992020,0cyclictest6413-21apt-get20:44:503
25521992020,0cyclictest29165-21apt-get20:24:533
25521992020,0cyclictest11204-21apt-get20:54:533
25521992020,0cyclictest0-21swapper/320:59:523
25521992020,0cyclictest0-21swapper/320:04:533
25521992020,0cyclictest0-21swapper/319:39:523
2552199200,20cyclictest0-21swapper/319:14:513
2552199200,1cyclictest13320-21apt-get19:50:003
2552199200,0cyclictest29110-21apt-get19:19:533
25520992020,0cyclictest26746-21apt-get20:19:522
25520992020,0cyclictest0-21swapper/220:09:512
25520992020,0cyclictest0-21swapper/219:34:522
25516992020,0cyclictest0-21swapper/120:49:521
25516992020,0cyclictest0-21swapper/120:19:521
25516992019,0cyclictest24345-21apt-get20:14:531
25516992019,0cyclictest0-21swapper/119:29:521
25516992019,0cyclictest0-21swapper/119:29:521
2551699200,0cyclictest0-21swapper/119:14:511
25514992020,0cyclictest0-21swapper/020:54:530
25514992019,1cyclictest0-21swapper/020:09:510
2551499200,0cyclictest30587-21munin-plugin-st20:25:010
25521991919,0cyclictest0-21swapper/320:14:533
25520991919,0cyclictest0-21swapper/220:54:532
25520991919,0cyclictest0-21swapper/220:44:502
25520991919,0cyclictest0-21swapper/220:14:532
25520991919,0cyclictest0-21swapper/219:59:522
25520991918,1cyclictest26712-21apt-get19:14:512
25520991917,1cyclictest1538-21apt-get20:34:532
25520991917,1cyclictest15098-21turbostat.cron21:00:012
25516991917,1cyclictest211rcuc/121:00:011
25516991917,1cyclictest0-21swapper/120:40:001
25514991919,0cyclictest0-21swapper/021:09:530
25514991919,0cyclictest0-21swapper/020:04:530
25514991919,0cyclictest0-21swapper/019:29:520
25514991919,0cyclictest0-21swapper/019:29:520
25514991918,0cyclictest0-21swapper/020:34:530
25521991818,0cyclictest0-21swapper/319:24:523
25520991818,0cyclictest31559-21apt-get20:29:532
25520991818,0cyclictest0-21swapper/219:24:522
25520991817,1cyclictest9170-21apt-get19:44:532
25520991817,1cyclictest5314-21apt-get19:35:022
25520991816,1cyclictest17505-21awk21:05:012
25516991818,0cyclictest0-21swapper/120:04:521
25516991815,2cyclictest22-21ksoftirqd/120:54:531
25521991717,0cyclictest0-21swapper/321:04:523
25521991717,0cyclictest0-21swapper/320:19:513
25516991717,0cyclictest0-21swapper/119:45:031
2551699170,2cyclictest8221-21latency_hist20:45:001
25514991717,0cyclictest0-21swapper/020:39:520
25514991717,0cyclictest0-21swapper/020:14:530
25514991715,1cyclictest5352-21latency_hist19:35:010
25521991615,0cyclictest0-21swapper/319:44:523
25516991616,0cyclictest0-21swapper/121:09:531
25516991614,1cyclictest21949-21apt-get20:09:521
25514991616,0cyclictest0-21swapper/020:44:500
2552199150,7cyclictest0-21swapper/320:39:523
2552099151,7cyclictest313-21systemd-journal19:45:022
25516991514,0cyclictest0-21swapper/120:34:531
25514991515,0cyclictest0-21swapper/019:59:520
25514991514,1cyclictest0-21swapper/020:56:250
2552199140,14cyclictest0-21swapper/320:45:433
2552199140,14cyclictest0-21swapper/320:31:533
25516991414,0cyclictest0-21swapper/119:20:371
2551499140,0cyclictest0-21swapper/020:15:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional