You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-23 - 06:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot5s.osadl.org (updated Mon Mar 23, 2026 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6914992726,1cyclictest331rcuc/320:30:003
691499260,26cyclictest0-21swapper/322:11:203
691199260,2cyclictest15975-21ls22:27:452
690999261,4cyclictest16494-21awk19:30:001
6914992523,1cyclictest0-21swapper/300:29:383
691499250,24cyclictest24785-21ssh23:38:293
6911992522,2cyclictest9717-21ssh22:22:162
6911992522,2cyclictest16805-21ssh21:25:462
6911992520,2cyclictest5454-21rm22:48:282
6909992524,1cyclictest0-21swapper/123:32:501
6909992524,1cyclictest0-21swapper/100:33:031
6909992522,2cyclictest313-21systemd-journal23:25:011
690999250,24cyclictest14088-21apt19:25:011
690999250,24cyclictest0-21swapper/121:48:281
6904992524,1cyclictest62550irq/127-eno121:35:340
690499250,24cyclictest30471-21ssh22:42:300
690499250,24cyclictest14084-21latency_hist19:25:010
690499250,24cyclictest0-21swapper/021:48:590
690499250,1cyclictest0-21swapper/023:58:340
691499240,3cyclictest509-21diskmemload23:45:013
691499240,23cyclictest0-21swapper/323:02:413
6911992424,0cyclictest509-21diskmemload21:49:472
691199242,21cyclictest2781-21ssh00:20:352
6911992422,1cyclictest0-21swapper/200:02:152
691199240,23cyclictest509-21diskmemload23:25:572
691199240,23cyclictest0-21swapper/223:55:562
691199240,23cyclictest0-21swapper/223:00:572
6909992424,0cyclictest0-21swapper/100:00:231
6909992422,1cyclictest13124-21idleruntime-cro22:25:011
6909992422,1cyclictest0-21swapper/122:49:181
690999240,24cyclictest0-21swapper/122:42:301
690999240,24cyclictest0-21swapper/100:27:251
690999240,23cyclictest0-21swapper/122:33:561
690999240,23cyclictest0-21swapper/100:08:341
6904992424,0cyclictest0-21swapper/000:33:030
6904992422,1cyclictest2356-21ssh21:12:340
690499241,22cyclictest0-21swapper/000:00:200
690499241,1cyclictest0-21swapper/022:57:270
690499240,23cyclictest0-21swapper/023:43:300
690499240,23cyclictest0-21swapper/022:55:010
690499240,23cyclictest0-21swapper/000:09:320
690499240,0cyclictest509-21diskmemload23:45:120
6914992323,0cyclictest509-21diskmemload00:21:383
6914992323,0cyclictest509-21diskmemload00:15:263
691499232,20cyclictest34-21ksoftirqd/322:18:333
6914992321,1cyclictest9943-21apt-get22:22:413
6914992321,1cyclictest0-21swapper/322:00:223
6914992321,1cyclictest0-21swapper/321:40:103
6914992321,1cyclictest0-21swapper/321:14:463
6914992320,2cyclictest7044-21sh21:47:483
6914992320,2cyclictest32564-21ssh23:46:353
6914992320,2cyclictest20412-21ntp_states00:37:493
691499230,2cyclictest521-21sh22:43:383
691499230,22cyclictest8383-21sh22:52:423
691499230,22cyclictest537-21perf21:10:003
691499230,22cyclictest509-21diskmemload22:45:533
691499230,22cyclictest509-21diskmemload00:30:483
691499230,22cyclictest4088-21ssh23:18:533
691499230,22cyclictest26832-21ssh22:37:493
691499230,22cyclictest25953-21sh23:08:563
691499230,22cyclictest25060-21ssh21:33:023
691499230,22cyclictest14094-21wc19:25:013
691499230,0cyclictest0-21swapper/321:52:093
6911992323,0cyclictest0-21swapper/223:22:242
6911992323,0cyclictest0-21swapper/222:43:432
6911992323,0cyclictest0-21swapper/221:42:072
6911992323,0cyclictest0-21swapper/221:38:312
6911992323,0cyclictest0-21swapper/200:16:322
6911992321,1cyclictest0-21swapper/223:30:522
6911992320,2cyclictest26689-21ssh23:41:042
6911992318,0cyclictest0-21swapper/222:35:222
691199231,21cyclictest0-21swapper/200:37:442
691199230,2cyclictest30478-21switchtime23:12:502
691199230,2cyclictest11336-21ssh21:20:052
691199230,23cyclictest0-21swapper/223:45:192
691199230,23cyclictest0-21swapper/221:52:022
691199230,22cyclictest7101-21ssh21:17:022
691199230,22cyclictest5646-21idleruntime-cro21:15:002
691199230,22cyclictest509-21diskmemload23:08:252
691199230,22cyclictest31280-21ssh22:11:582
691199230,22cyclictest21617-21sh21:30:132
691199230,22cyclictest18960-21ssh21:59:322
691199230,22cyclictest18535-21sendmail-msp20:40:012
691199230,22cyclictest11374-21ssh00:28:312
691199230,22cyclictest0-21swapper/222:56:452
691199230,0cyclictest509-21diskmemload00:08:392
691199230,0cyclictest0-21swapper/222:53:452
6909992323,0cyclictest0-21swapper/123:53:301
6909992323,0cyclictest0-21swapper/123:47:371
6909992323,0cyclictest0-21swapper/122:28:311
6909992323,0cyclictest0-21swapper/121:36:401
6909992322,1cyclictest0-21swapper/123:29:271
6909992321,1cyclictest9450-21ssh23:55:181
6909992321,1cyclictest0-21swapper/122:00:021
690999230,23cyclictest0-21swapper/100:15:261
690999230,22cyclictest509-21diskmemload00:12:251
690999230,22cyclictest4045-21sh23:18:501
690999230,22cyclictest26100-21ssh23:40:021
690999230,22cyclictest15834-21ssh21:57:201
690999230,22cyclictest14202-21apt20:32:421
690999230,22cyclictest0-21swapper/123:01:431
690999230,22cyclictest0-21swapper/121:51:061
690999230,1cyclictest0-21swapper/122:12:471
690999230,0cyclictest0-21swapper/122:07:541
690999230,0cyclictest0-21swapper/100:38:551
6904992323,0cyclictest509-21diskmemload23:53:300
6904992323,0cyclictest0-21swapper/000:26:280
6904992322,1cyclictest0-21swapper/023:18:050
6904992321,2cyclictest0-21swapper/021:57:410
6904992321,1cyclictest12945-21ssh21:22:390
6904992321,1cyclictest0-21swapper/022:27:240
6904992321,1cyclictest0-21swapper/022:20:510
6904992321,1cyclictest0-21swapper/022:15:240
6904992320,2cyclictest28137-21perf21:00:010
6904992320,2cyclictest26199-21ssh22:07:010
6904992320,2cyclictest24097-21latency_hist19:45:000
6904992320,2cyclictest22462-21sendmail-msp00:40:010
690499230,23cyclictest509-21diskmemload21:54:180
690499230,22cyclictest5629-21ssh23:20:210
690499230,22cyclictest509-21diskmemload23:12:410
690499230,22cyclictest4025-21diskmemload00:22:350
690499230,22cyclictest32537-21ssh21:41:250
690499230,22cyclictest23553-21df_inode21:32:440
690499230,22cyclictest0-21swapper/022:10:040
690499230,22cyclictest0-21swapper/020:25:020
690499230,21cyclictest0-21swapper/019:57:430
690499230,1cyclictest22454-21ssh23:05:570
690499230,0cyclictest0-21swapper/023:38:090
690499230,0cyclictest0-21swapper/022:35:250
690499230,0cyclictest0-21swapper/021:26:520
690499230,0cyclictest0-21swapper/000:15:260
6914992222,0cyclictest0-21swapper/323:51:293
6914992222,0cyclictest0-21swapper/321:58:423
6914992222,0cyclictest0-21swapper/321:23:373
6914992221,0cyclictest6588-21apt-get20:17:433
6914992220,2cyclictest5907-21ssh21:15:033
6914992220,1cyclictest509-21diskmemload22:28:013
6914992220,1cyclictest24981-21ssh22:05:013
6914992220,1cyclictest18820-21apt-get22:30:023
6914992220,1cyclictest11376-21diskstats23:57:433
6914992220,1cyclictest0-21swapper/300:05:413
691499220,22cyclictest0-21swapper/323:34:553
691499220,22cyclictest0-21swapper/300:14:033
691499220,21cyclictest509-21diskmemload22:55:513
691499220,21cyclictest30582-21awk21:05:003
691499220,21cyclictest27191-21ssh21:36:113
691499220,21cyclictest0-21swapper/319:47:423
691499220,1cyclictest18893-21perf19:35:003
691499220,0cyclictest0-21swapper/323:11:133
6911992221,1cyclictest0-21swapper/222:00:052
6911992220,1cyclictest2979-21gpgv22:15:012
6911992220,1cyclictest28158-21munin-run21:00:012
6911992220,1cyclictest0-21swapper/223:36:092
6911992219,2cyclictest20938-21awk20:45:012
6911992219,0cyclictest0-21swapper/222:07:352
6911992214,0cyclictest28-21ksoftirqd/200:34:262
691199220,21cyclictest509-21diskmemload23:15:192
691199220,21cyclictest509-21diskmemload00:10:142
691199220,21cyclictest31350-21perf20:00:012
691199220,21cyclictest19365-21ssh22:30:512
691199220,21cyclictest0-21swapper/223:50:132
691199220,1cyclictest1323-21latency_hist20:05:012
6909992222,0cyclictest0-21swapper/123:07:431
6909992222,0cyclictest0-21swapper/122:55:161
6909992222,0cyclictest0-21swapper/121:30:571
6909992222,0cyclictest0-21swapper/121:29:291
6909992220,1cyclictest24093-21perf19:45:001
6909992220,1cyclictest0-21swapper/121:10:141
6909992220,1cyclictest0-21swapper/121:05:001
690999220,21cyclictest509-21diskmemload23:35:491
690999220,21cyclictest29396-21apt-get19:57:431
690999220,21cyclictest0-21swapper/123:10:441
690999220,21cyclictest0-21swapper/122:35:561
690999220,21cyclictest0-21swapper/121:40:161
690999220,21cyclictest0-21swapper/119:37:421
690999220,1cyclictest2444-21rm00:20:041
690999220,1cyclictest10948-21apt-get20:25:021
690999220,0cyclictest0-21swapper/122:18:151
6904992221,1cyclictest62550irq/127-eno122:30:060
6904992220,1cyclictest0-21swapper/023:30:120
690499220,22cyclictest0-21swapper/023:01:430
690499220,1cyclictest2262-21ssh22:45:580
690499220,1cyclictest0-21swapper/023:25:120
690499220,1cyclictest0-21swapper/022:00:100
690499220,1cyclictest0-21swapper/021:19:530
690499220,1cyclictest0-21swapper/000:14:540
690499220,0cyclictest0-21swapper/019:52:420
6914992120,1cyclictest1792-21apt-get20:07:423
6914992119,1cyclictest0-21swapper/321:26:313
6914992119,1cyclictest0-21swapper/321:00:013
691499210,0cyclictest0-21swapper/319:57:433
6911992120,1cyclictest9738-21apt-get19:17:422
6911992120,1cyclictest0-21swapper/219:15:012
6911992119,1cyclictest18901-21munin-run19:35:002
691199210,20cyclictest23801-21apt-get20:52:422
691199210,0cyclictest0-21swapper/219:47:422
690999210,20cyclictest0-21swapper/120:07:421
6904992119,1cyclictest7345-21apt-get19:12:430
690499210,20cyclictest23801-21apt-get20:52:420
6914992020,0cyclictest26946-21apt-get19:52:423
6914992020,0cyclictest22158-21apt-get19:42:423
6914992017,2cyclictest16149-21munin-run20:35:003
691499200,20cyclictest1332-21perf20:05:013
691499200,0cyclictest0-21swapper/319:12:433
6911992020,0cyclictest0-21swapper/220:47:422
6911992020,0cyclictest0-21swapper/220:25:022
6911992020,0cyclictest0-21swapper/219:52:422
691199200,0cyclictest0-21swapper/220:07:422
691199200,0cyclictest0-21swapper/219:35:022
6909992020,0cyclictest24552-21apt-get19:47:421
6909992020,0cyclictest0-21swapper/121:00:011
6909992018,1cyclictest19016-21apt-get20:42:431
690999200,20cyclictest0-21swapper/120:52:421
6904992020,0cyclictest0-21swapper/019:47:420
6904992018,1cyclictest0-21swapper/019:40:000
690499200,20cyclictest0-21swapper/020:07:420
6914991919,0cyclictest19016-21apt-get20:42:423
6914991917,1cyclictest16491-21perf19:30:003
6914991916,2cyclictest313-21systemd-journal20:40:013
6911991919,0cyclictest0-21swapper/220:17:442
6911991915,3cyclictest16146-21perf20:35:012
6909991919,0cyclictest0-21swapper/120:47:421
6904991919,0cyclictest21406-21apt-get20:47:420
6904991919,0cyclictest0-21swapper/020:42:430
6904991917,1cyclictest4188-21apt-get20:12:410
6914991818,0cyclictest0-21swapper/320:47:423
6914991818,0cyclictest0-21swapper/319:37:423
6911991818,0cyclictest0-21swapper/219:22:422
6911991817,1cyclictest14553-21apt-get19:27:422
6911991816,1cyclictest29397-21modprobe21:02:442
6911991816,1cyclictest0-21swapper/220:12:422
6909991818,0cyclictest0-21swapper/119:12:431
690999180,1cyclictest16597-21apt-get20:37:421
6914991717,0cyclictest0-21swapper/320:52:423
6911991715,2cyclictest0-21swapper/221:07:502
6911991714,2cyclictest0-21swapper/219:42:442
6904991716,1cyclictest16495-21munin-run19:30:000
6911991616,0cyclictest0-21swapper/220:22:422
6909991616,0cyclictest0-21swapper/121:07:411
6909991616,0cyclictest0-21swapper/120:12:411
6909991614,1cyclictest10958-21munin-run20:25:001
6904991616,0cyclictest0-21swapper/021:07:410
6904991616,0cyclictest0-21swapper/020:17:440
6904991614,1cyclictest10948-21apt-get20:25:000
6904991613,2cyclictest14840-21tail20:32:440
690499160,15cyclictest0-21swapper/021:02:430
691499155,9cyclictest611-21irqbalance19:16:313
691499150,6cyclictest0-21swapper/320:12:413
691499140,14cyclictest0-21swapper/320:20:033
6909991414,0cyclictest0-21swapper/119:30:151
690999140,14cyclictest0-21swapper/120:00:301
690999140,14cyclictest0-21swapper/119:15:461
6904991414,0cyclictest0-21swapper/019:30:150
690499140,14cyclictest0-21swapper/020:35:370
690499140,14cyclictest0-21swapper/019:15:460
690499140,0cyclictest0-21swapper/020:00:050
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional