You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-17 - 08:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sun May 17, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23336996639,21cyclictest35966irq/58-eth022:45:113
23318995719,33cyclictest5915-21ssh22:20:121
23318995719,33cyclictest5915-21ssh22:20:121
2331899551,45cyclictest13197-21ssh21:30:131
23328995429,20cyclictest27-21ksoftirqd/220:20:122
23309995241,5cyclictest22594-21apt-get20:50:110
23309995145,3cyclictest19897-21apt-get19:40:130
23318995041,4cyclictest32217-21apt-get21:00:131
2332899483,27cyclictest10294-21ssh23:35:132
2332899481,30cyclictest3879-21ssh22:55:142
2332899461,25cyclictest24944-21ssh23:25:122
2333699456,25cyclictest35966irq/58-eth000:05:133
23336994537,5cyclictest35966irq/58-eth023:25:123
2332899452,32cyclictest29324-21ssh22:15:132
23318994535,7cyclictest9343-21ssh22:40:141
23309994436,4cyclictest11485-21apt-get20:05:130
2333699437,25cyclictest35966irq/58-eth022:25:163
2333699432,25cyclictest35966irq/58-eth023:20:123
2332899432,21cyclictest930-21snmpd20:50:112
23318994337,3cyclictest17679-21apt-get20:45:121
23318994329,10cyclictest2414-21apt-get00:25:151
23309994317,10cyclictest16859-21/usr/sbin/munin23:00:300
23309994315,12cyclictest7491-21latency_hist21:45:010
23309994236,2cyclictest3876-21apt-get22:55:140
23309994235,3cyclictest4583-21apt-get21:05:140
23309994218,20cyclictest0-21swapper/000:25:150
23318994135,3cyclictest25895-21apt-get00:20:131
2330999413,27cyclictest13167-21ssh21:30:120
23309994114,10cyclictest20751-21ssh23:58:180
23309994114,10cyclictest20136-21ssh22:26:050
23309994113,10cyclictest1741-21ssh22:52:550
23309994112,11cyclictest24382-21ssh00:18:440
23318994033,4cyclictest28226-21apt-get23:45:141
23318994018,18cyclictest0-21swapper/121:45:131
23309994031,4cyclictest647-21apt-get19:20:130
23309994014,9cyclictest11675-21swap20:35:260
23309994014,8cyclictest13773-21ssh00:11:400
23318993913,10cyclictest4617-21ssh23:12:211
23309993932,3cyclictest29312-21apt-get22:15:130
23309993927,10cyclictest9335-21apt-get22:40:140
23309993922,15cyclictest0-21swapper/020:30:130
23309993917,18cyclictest0-21swapper/000:20:130
23309993915,7cyclictest9185-21diskmemload22:45:110
23309993914,9cyclictest2224-21ssh23:47:260
23309993914,9cyclictest12777-21sh21:30:000
23309993816,16cyclictest0-21swapper/021:15:140
23309993814,8cyclictest24312-21ssh23:24:450
23309993813,9cyclictest12066-21ssh23:15:350
23309993813,8cyclictest0-21swapper/019:50:180
23309993812,9cyclictest30801-21sh23:26:470
23309993812,10cyclictest30873-21fschecks_count23:10:160
23318993731,3cyclictest10284-21apt-get23:35:141
23318993711,9cyclictest10018-21ssh21:26:411
23318993711,10cyclictest7852-21sh22:01:221
23318993711,10cyclictest7852-21sh22:01:221
23318993711,10cyclictest32124-21ssh22:34:451
23309993714,9cyclictest31490-21cpu00:05:130
23309993714,8cyclictest0-21swapper/000:32:400
23309993713,8cyclictest26671-21ssh23:05:310
23309993712,7cyclictest9185-21diskmemload21:56:270
23309993712,13cyclictest0-21swapper/021:22:320
2332899361,16cyclictest1618066ssh00:30:342
2331899369,9cyclictest22306-21ssh21:51:211
2331899369,9cyclictest22306-21ssh21:51:211
23318993629,3cyclictest27385-21apt-get20:55:141
23318993626,7cyclictest25975-21apt-get20:20:121
23318993611,9cyclictest9185-21diskmemload23:29:201
23309993623,9cyclictest0-21swapper/021:10:130
23309993617,15cyclictest0-21swapper/022:00:130
23309993617,15cyclictest0-21swapper/022:00:130
23309993612,9cyclictest0-21swapper/023:42:240
23309993611,11cyclictest29285-21ssh21:38:130
23309993611,10cyclictest27545-21ssh00:00:440
2331899359,11cyclictest14676-21ssh00:12:461
23318993528,4cyclictest32721-21apt-get22:35:121
23318993513,7cyclictest29900-21sh23:09:311
23318993513,7cyclictest28133-21ntpq19:45:241
23318993512,7cyclictest24755-21ssh21:17:081
23318993511,8cyclictest1262-21ssh21:59:161
23318993510,7cyclictest6616-21ssh23:30:591
23318993510,15cyclictest30516-21sh21:39:431
2330999359,10cyclictest29017-21ssh22:30:580
23309993528,3cyclictest12899-21apt-get20:40:130
23309993518,13cyclictest32747-21ssh22:35:140
23309993513,7cyclictest9185-21diskmemload23:53:190
23309993513,7cyclictest28330-21ssh22:14:230
23309993513,10cyclictest11185-21sh22:21:000
23309993513,10cyclictest11185-21sh22:21:000
23309993512,7cyclictest13776-21ssh21:46:560
2331899348,10cyclictest21342-21ssh23:20:581
2331899347,10cyclictest21143-21cat00:15:311
23318993424,6cyclictest10261-21apt-get19:30:191
23318993412,7cyclictest23374-21ssh22:30:001
23318993411,10cyclictest2265-21rm22:53:321
23318993410,7cyclictest0-21swapper/121:44:581
23309993429,2cyclictest16725-21apt-get21:50:120
23309993429,2cyclictest16725-21apt-get21:50:120
23309993413,9cyclictest11031-21kthreadcore19:30:260
23309993413,8cyclictest16730-21ssh23:37:240
23309993413,7cyclictest28493-21ssh00:39:380
23309993412,8cyclictest18959-21sh22:08:540
2331899339,9cyclictest5145-21rm00:07:041
2331899335,13cyclictest13012-21ssh23:16:451
23318993326,4cyclictest15068-21apt-get19:35:121
23318993326,4cyclictest15068-21apt-get19:35:121
23318993325,4cyclictest23477-21apt-get19:10:131
23318993311,9cyclictest9185-21diskmemload00:01:361
23318993311,10cyclictest26242-21ssh23:43:041
2330999336,9cyclictest0-21swapper/023:30:240
23309993312,13cyclictest18385-21memory19:35:220
23309993312,13cyclictest18385-21memory19:35:220
23309993311,9cyclictest1993-21cut19:55:160
23309993311,8cyclictest6931-21cut19:25:210
2331899329,7cyclictest0-21swapper/122:12:361
23318993222,6cyclictest3229-21apt-get20:30:131
23318993217,12cyclictest18301-21apt-get22:45:111
23318993213,15cyclictest0-21swapper/123:00:131
23318993212,8cyclictest21522-21sh23:59:161
23318993211,12cyclictest9185-21diskmemload21:24:041
23309993217,12cyclictest20883-21kthreadcore20:45:210
23309993210,7cyclictest0-21swapper/019:15:230
2333699313,18cyclictest35966irq/58-eth022:50:133
23336993115,9cyclictest35966irq/58-eth022:35:133
23328993111,14cyclictest11318-21ssh22:05:132
2331899317,10cyclictest21266-21/usr/sbin/munin20:15:151
2331899317,10cyclictest21266-21/usr/sbin/munin20:15:151
23318993110,7cyclictest9940-21ssh22:57:011
23318993110,7cyclictest9185-21diskmemload00:35:041
2330999319,8cyclictest23317-21kthreadcore20:15:190
2330999319,8cyclictest23317-21kthreadcore20:15:190
2330999318,9cyclictest0-21swapper/019:50:020
23309993123,6cyclictest27469-21/usr/sbin/munin20:55:160
23309993112,7cyclictest2479-21cat20:25:290
2333699308,15cyclictest35966irq/58-eth022:05:123
2333699304,16cyclictest35966irq/58-eth023:35:133
23336993018,6cyclictest0-21swapper/319:40:133
2331899308,9cyclictest15067-21ssh22:05:271
2331899308,8cyclictest0-21swapper/121:12:071
2331899308,10cyclictest4637-21ssh23:50:081
2331899307,10cyclictest0-21swapper/122:17:131
2331899306,14cyclictest11287-21apt-get00:30:141
2331899304,11cyclictest23253-21/usr/sbin/munin19:40:231
2332899292,21cyclictest0-21swapper/221:05:152
2332899292,15cyclictest20214-21sh00:35:142
2331899295,11cyclictest2125-21cut19:20:191
23309992921,5cyclictest16066-21apt-get20:10:050
23309992918,8cyclictest32307-21df_inode21:00:150
23309992918,8cyclictest11241-21sshd20:02:590
23309992915,11cyclictest27001-21grep19:10:270
2332899285,18cyclictest27575-21kworker/2:223:10:122
2332899284,17cyclictest27346irq/40-dwc2_hso20:15:132
2332899284,17cyclictest27346irq/40-dwc2_hso20:15:122
2332899283,17cyclictest27346irq/40-dwc2_hso19:30:202
23318992815,7cyclictest536-21usb-storage20:10:111
23318992810,4cyclictest5954-21ps19:25:191
23309992817,8cyclictest0-21swapper/020:20:120
2332899273,8cyclictest0-21swapper/223:30:142
2332899272,19cyclictest0-21swapper/220:05:152
2332899272,19cyclictest0-21swapper/219:45:132
2331899278,6cyclictest504-21ntpq19:50:261
23318992713,11cyclictest0-21swapper/120:25:111
23309992713,10cyclictest23311-21latency_hist19:10:010
2333699262,15cyclictest35966irq/58-eth021:10:143
2332899266,16cyclictest19331-21diskmemload23:40:142
2332899265,12cyclictest27346irq/40-dwc2_hso20:25:112
2332899261,19cyclictest28251-21ssh23:45:142
2332899261,15cyclictest0-21swapper/219:40:142
2332899261,13cyclictest23789-21ssh22:30:132
23318992615,8cyclictest14119-21grep20:40:191
2333699254,17cyclictest0-21swapper/300:20:123
23336992512,9cyclictest35966irq/58-eth023:40:143
2332899253,4cyclictest14914-21ssh22:25:162
2332899253,17cyclictest27346irq/40-dwc2_hso00:05:122
2332899252,19cyclictest27-21ksoftirqd/223:15:142
2332899252,14cyclictest27346irq/40-dwc2_hso21:15:142
23328992512,6cyclictest31184-21ssh21:40:142
2332899251,17cyclictest8346-21tr00:10:182
2332899251,15cyclictest32748-21ssh22:35:132
2332899251,11cyclictest25751-21ssh21:55:142
2331899259,14cyclictest12577-21kthreadcore20:05:201
23318992516,6cyclictest8077-21ntp_states21:05:271
23318992515,8cyclictest6426-21sshd19:55:591
23318992515,8cyclictest28240-21munin-node19:15:121
23318992514,8cyclictest26339-21sed20:50:281
23318992513,9cyclictest10006-21cat20:00:221
2333699245,9cyclictest35966irq/58-eth021:55:133
2333699243,12cyclictest0-21swapper/322:30:133
2333699242,14cyclictest35966irq/58-eth021:50:113
2333699242,14cyclictest35966irq/58-eth021:50:113
2332899248,7cyclictest27-21ksoftirqd/221:35:172
2332899244,10cyclictest131rcu_preempt00:25:202
2332899243,15cyclictest27346irq/40-dwc2_hso22:50:142
2332899241,5cyclictest22512-21ssh00:00:112
23328992410,7cyclictest4346-21cron21:05:012
23318992414,8cyclictest23295-21cyclictest19:10:011
23318992413,8cyclictest8025-21/usr/sbin/munin20:35:241
2333699233,12cyclictest0-21swapper/323:45:133
2333699233,12cyclictest0-21swapper/323:30:143
2332899233,15cyclictest27346irq/40-dwc2_hso22:45:122
2332899232,13cyclictest0-21swapper/220:55:152
2332899231,7cyclictest3162666ssh21:20:292
2332899231,19cyclictest25523-21apt-config00:20:012
2332899231,18cyclictest5401-21cut23:50:192
2332899231,13cyclictest381-21munin-run19:20:012
2333699222,15cyclictest0-21swapper/323:10:123
2333699222,15cyclictest0-21swapper/321:40:143
23336992215,4cyclictest35966irq/58-eth021:25:143
23336992212,6cyclictest0-21swapper/300:30:223
23336992210,8cyclictest35966irq/58-eth022:20:273
23336992210,8cyclictest35966irq/58-eth022:20:273
2332899225,9cyclictest10551-21grep20:00:282
2332899222,9cyclictest5611-21fschecks_time19:25:182
2332899222,17cyclictest24946-21kthreadcore23:05:212
2332899222,17cyclictest20885-21grep21:50:272
2332899222,17cyclictest20885-21grep21:50:272
2332899222,16cyclictest23624-21fschecks_count19:10:162
23328992214,5cyclictest8977-21ssh22:20:222
23328992214,5cyclictest8977-21ssh22:20:222
23328992214,5cyclictest0-21swapper/219:35:122
23328992214,5cyclictest0-21swapper/219:35:122
2332899221,11cyclictest18170-21latency_hist21:15:012
2332899221,10cyclictest14358-21kthreadcore23:00:212
2332899221,10cyclictest12676-21tr22:40:232
2332899220,15cyclictest4987-21kthreadcore21:25:202
2333699213,10cyclictest35966irq/58-eth019:10:133
23336992114,5cyclictest0-21swapper/322:10:273
23336992112,6cyclictest35966irq/58-eth021:20:143
2332899219,4cyclictest26647-21kthreadcore00:20:212
2332899219,4cyclictest22261-21kthreadcore22:10:222
2332899218,10cyclictest17396-21cat21:30:312
2332899213,2cyclictest0-21swapper/223:55:132
2332899212,11cyclictest17386-21kthreadcore20:10:182
2332899211,8cyclictest18990-21cut23:20:222
23328992114,4cyclictest16396-21ssh21:50:012
23328992111,8cyclictest0-21swapper/219:20:272
2332899211,16cyclictest2466-21fschecks_count22:00:172
2332899211,16cyclictest2466-21fschecks_count22:00:162
2332899211,10cyclictest17447-21cron20:45:012
2333699208,8cyclictest35966irq/58-eth000:00:113
23336992011,5cyclictest0-21swapper/322:15:323
2332899207,10cyclictest18181-21ps20:45:182
2332899202,9cyclictest12860-21latency_hist20:40:022
2332899202,13cyclictest0-21swapper/220:30:052
2332899201,10cyclictest30238-21kthreadcore19:50:192
2333699198,8cyclictest0-21swapper/300:12:383
2333699196,10cyclictest0-21swapper/322:55:303
2333699193,10cyclictest0-21swapper/300:35:143
2333699192,11cyclictest0-21swapper/320:50:043
23336991911,5cyclictest35966irq/58-eth022:40:133
2333699191,12cyclictest0-21swapper/320:35:213
2333699191,12cyclictest0-21swapper/319:20:273
23336991910,7cyclictest0-21swapper/300:25:343
2332899191,9cyclictest23347-21latency_hist19:10:012
2332899191,13cyclictest1884-21apt-get19:55:142
2333699189,6cyclictest0-21swapper/323:57:403
2333699189,6cyclictest0-21swapper/321:35:093
2333699188,8cyclictest0-21swapper/323:17:213
2333699188,6cyclictest35966irq/58-eth023:05:273
2333699182,11cyclictest0-21swapper/321:00:153
2333699182,10cyclictest0-21swapper/320:30:053
2333699182,10cyclictest0-21swapper/320:25:143
2333699182,10cyclictest0-21swapper/319:50:143
23336991812,4cyclictest0-21swapper/321:30:123
23336991812,3cyclictest0-21swapper/323:00:273
2333699181,12cyclictest0-21swapper/322:00:163
2333699181,12cyclictest0-21swapper/322:00:163
2333699181,12cyclictest0-21swapper/319:25:293
2333699181,11cyclictest0-21swapper/319:45:213
2333699179,5cyclictest0-21swapper/321:45:273
2333699179,3cyclictest0-21swapper/321:15:533
2333699178,6cyclictest35966irq/58-eth023:50:173
2333699172,9cyclictest0-21swapper/320:55:153
2333699172,12cyclictest0-21swapper/300:15:153
2333699172,10cyclictest0-21swapper/320:40:143
2333699172,10cyclictest0-21swapper/320:20:143
2333699172,10cyclictest0-21swapper/320:05:143
2333699172,10cyclictest0-21swapper/319:30:213
2333699157,5cyclictest35966irq/58-eth020:15:123
2333699157,5cyclictest35966irq/58-eth020:15:113
2333699152,8cyclictest0-21swapper/319:35:143
2333699152,8cyclictest0-21swapper/319:35:133
2333699147,5cyclictest35966irq/58-eth020:50:013
2333699142,8cyclictest0-21swapper/321:05:153
2333699141,8cyclictest0-21swapper/320:05:013
2333699136,4cyclictest0-21swapper/320:10:153
2333699132,9cyclictest0-21swapper/319:15:203
2333699131,9cyclictest0-21swapper/319:55:123
2333699121,7cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional