You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-28 - 18:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Thu May 28, 2026 12:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2480299752,55cyclictest5860-21ssh09:25:111
2479399641,48cyclictest16869-21ssh11:00:130
24802995912,8cyclictest131rcu_preempt08:40:131
2479399583,41cyclictest7752-21ssh10:55:130
2479399583,41cyclictest7752-21ssh10:55:120
2479399562,46cyclictest23980-21ssh12:15:120
24802995131,8cyclictest21-21ksoftirqd/110:35:111
24793995138,9cyclictest14141-21apt-get10:05:110
24824994712,25cyclictest35966irq/58-eth012:00:113
24812994726,16cyclictest0-21swapper/209:55:122
2481299462,18cyclictest23702-21usage_EthcTask12:30:282
24802994628,10cyclictest6403-21ssh11:30:131
24793994637,4cyclictest14364-21apt-get08:40:120
24793994539,2cyclictest21372-21apt-get07:40:120
24793994432,8cyclictest3932-21apt-get10:35:120
2479399443,27cyclictest11304-21ssh11:15:140
24793994337,3cyclictest3414-21apt-get07:55:130
2480299423,34cyclictest20502-21ssh11:20:121
2479399423,30cyclictest10757-21ssh11:50:140
2479399423,30cyclictest10757-21ssh11:50:130
24793994228,10cyclictest1979-21apt-get11:10:120
24793994227,9cyclictest14956-21ssh09:30:110
24793994213,11cyclictest8836-21rm12:24:150
24793994213,11cyclictest8836-21rm12:24:140
24793994114,10cyclictest16931-21ssh12:28:010
24793994113,11cyclictest14053-21ssh11:33:430
2479399411,30cyclictest24089-21ssh09:35:120
24802994027,8cyclictest536-21usb-storage08:10:121
24802994027,8cyclictest536-21usb-storage08:10:111
2480299401,32cyclictest28984-21ssh12:00:131
24793994011,10cyclictest27043-21date10:30:020
2482499391,21cyclictest35966irq/58-eth011:25:123
2481299396,15cyclictest6401-21sh11:30:132
24793993934,2cyclictest31136-21apt-get10:50:130
24793993934,2cyclictest31136-21apt-get10:50:120
24793993932,4cyclictest1487-21apt-get11:45:110
24793993927,9cyclictest9906-21apt-get09:45:140
24793993911,12cyclictest5463-21cat09:40:310
24793993832,3cyclictest13023-21apt-get08:05:120
24793993828,5cyclictest17904-21apt-get08:10:130
24793993828,5cyclictest17904-21apt-get08:10:120
24793993823,11cyclictest9032-21apt-get10:20:140
24793993819,14cyclictest5001-21ssh10:00:110
24793993817,11cyclictest12-21ksoftirqd/007:30:120
2480299376,19cyclictest29693-21ssh11:25:131
24793993731,4cyclictest29666-21apt-get11:25:130
24793993714,11cyclictest11415-21ssh12:06:330
24793993629,3cyclictest24949-21apt-get07:10:130
24793993629,3cyclictest24949-21apt-get07:10:120
24793993629,3cyclictest15616-21apt-get11:35:130
24793993628,5cyclictest28145-21apt-get09:55:130
24793993613,8cyclictest25850-21ssh09:52:440
24793993613,8cyclictest21481-21rm10:44:430
24793993613,14cyclictest0-21swapper/010:30:110
24793993612,10cyclictest413-21ssh11:44:140
24793993610,9cyclictest22683-21ssh10:45:180
24793993610,11cyclictest7125-21ssh10:18:170
24793993526,4cyclictest22685-21apt-get08:15:110
24793993521,10cyclictest10238-21memory07:25:210
24793993514,6cyclictest14284-21sh09:29:430
24793993513,8cyclictest2259-21ssh12:37:200
24793993512,8cyclictest20500-21ssh12:11:320
24793993511,7cyclictest0-21swapper/009:17:150
2480299340,28cyclictest10567-21diskmemload10:45:121
24793993416,13cyclictest0-21swapper/011:55:100
24793993413,5cyclictest32287-21cut12:00:220
24793993412,8cyclictest24033-21missed_timers11:20:230
2479399341,25cyclictest23292-21sh10:10:120
2480299332,22cyclictest21-21ksoftirqd/107:55:121
24802993319,7cyclictest0-21swapper/108:05:111
24793993313,8cyclictest27438-21ssh12:34:330
24793993312,7cyclictest32223-21awk08:55:240
24793993312,10cyclictest21930-21kthreadcore08:45:230
2482499322,19cyclictest35966irq/58-eth011:10:123
2479399329,7cyclictest10567-21diskmemload11:06:570
24793993222,7cyclictest9504-21apt-get08:35:140
24793993222,7cyclictest29167-21ssh09:20:120
24793993213,7cyclictest12511-21cut09:10:190
2482499316,17cyclictest35966irq/58-eth012:10:103
24824993118,8cyclictest0-21swapper/311:45:123
24824993118,7cyclictest321rcuc/308:40:123
2480299314,16cyclictest21-21ksoftirqd/110:50:121
2480299314,16cyclictest21-21ksoftirqd/110:50:111
2479399319,9cyclictest0-21swapper/007:35:170
2479399317,13cyclictest7735-21kthreadcore09:05:180
24793993119,8cyclictest29848-21diskstats07:15:140
24793993118,10cyclictest5359-21irqstats08:30:170
2481299302,21cyclictest0-21swapper/208:10:132
2481299302,21cyclictest0-21swapper/208:10:122
2480299301,17cyclictest700-21ssh12:20:131
2480299301,17cyclictest700-21ssh12:20:121
2479399306,8cyclictest0-21swapper/008:55:010
24793993021,7cyclictest26216-21apt-get07:45:140
24793993016,10cyclictest32391-21df08:25:150
24793993016,10cyclictest27617-21df08:20:140
2482499293,16cyclictest35966irq/58-eth010:45:113
24793992917,8cyclictest24804-21sed07:10:020
24793992915,11cyclictest2013-21/usr/sbin/munin07:50:220
24793992910,10cyclictest10066-21kthreadcore08:00:200
2482499283,19cyclictest33-21ksoftirqd/310:50:123
2482499283,19cyclictest33-21ksoftirqd/310:50:113
2481299282,19cyclictest0-21swapper/207:40:142
2480299286,13cyclictest0-21swapper/108:35:131
2481299271,15cyclictest9059-21ssh10:20:142
24812992710,13cyclictest27-21ksoftirqd/210:35:122
24793992716,8cyclictest4614-21ntp_192.168.11409:00:240
24793992716,8cyclictest4614-21ntp_192.168.11409:00:230
24793992715,9cyclictest2333-21fschecks_time07:20:170
24824992612,9cyclictest36066irq/59-eth010:55:133
24824992612,9cyclictest36066irq/59-eth010:55:123
2481299263,7cyclictest0-21swapper/212:00:142
2481299263,16cyclictest27346irq/40-dwc2_hso07:25:112
2481299263,16cyclictest27-21ksoftirqd/208:40:122
2481299263,15cyclictest27346irq/40-dwc2_hso07:10:142
2481299263,15cyclictest27346irq/40-dwc2_hso07:10:132
2481299262,16cyclictest27346irq/40-dwc2_hso11:45:122
24812992610,9cyclictest27-21ksoftirqd/212:05:112
2480299262,18cyclictest1511-21sh11:45:111
2482499255,15cyclictest35966irq/58-eth012:30:113
2482499252,19cyclictest35966irq/58-eth008:50:123
24824992514,8cyclictest35966irq/58-eth012:35:143
24824992513,9cyclictest35966irq/58-eth009:15:143
2481299251,7cyclictest2977166ssh09:36:292
24812992517,5cyclictest973-21ssh11:25:252
2481299251,20cyclictest14878-21ls11:15:242
2481299251,18cyclictest9913-21rm09:45:132
2480299258,4cyclictest131rcu_preempt09:20:291
2480299258,10cyclictest131rcu_preempt12:05:111
2480299255,3cyclictest21-21ksoftirqd/109:40:131
2480299253,17cyclictest9931-21ssh09:45:131
2480299251,3cyclictest131rcu_preempt12:15:111
2480299251,3cyclictest131rcu_preempt09:00:131
2480299251,3cyclictest131rcu_preempt09:00:121
24802992512,3cyclictest10567-21diskmemload10:15:381
24824992412,5cyclictest35966irq/58-eth011:50:133
24824992412,5cyclictest35966irq/58-eth011:50:133
2481299246,9cyclictest11172-21kthreadcore08:00:222
2481299242,17cyclictest27346irq/40-dwc2_hso07:45:122
2481299241,8cyclictest2835766ssh11:59:462
2481299241,6cyclictest3057866ssh11:41:352
24812992412,9cyclictest0-21swapper/211:05:132
2480299248,4cyclictest32123-21cat12:35:281
24802992411,10cyclictest28804-21cat10:30:201
24802992410,4cyclictest201rcuc/112:25:131
2482499239,8cyclictest35966irq/58-eth010:10:133
2481299231,7cyclictest885466ssh12:24:162
2481299231,7cyclictest885466ssh12:24:152
2481299231,7cyclictest2462066ssh11:04:292
2481299231,7cyclictest1683966ssh11:51:562
2481299231,7cyclictest1683966ssh11:51:552
2481299231,6cyclictest2172166ssh09:32:412
2481299231,6cyclictest2080466ssh10:07:342
2481299231,6cyclictest1929566rm10:42:102
2481299231,6cyclictest1429866ssh09:29:432
2481299231,6cyclictest1171266/usr/sbin/munin07:30:302
24812992311,9cyclictest27-21ksoftirqd/210:30:132
2480299238,7cyclictest131rcu_preempt11:15:231
2482499229,8cyclictest0-21swapper/310:20:153
2482499224,13cyclictest0-21swapper/310:15:123
2482499223,11cyclictest35966irq/58-eth007:25:123
2481299229,5cyclictest30855-21kthreadcore09:20:222
2481299222,6cyclictest0-21swapper/212:35:032
2481299222,6cyclictest0-21swapper/211:35:142
2481299222,5cyclictest1250566ssh12:25:232
2481299221,7cyclictest779666ssh10:19:052
2481299221,7cyclictest1706366ssh09:12:002
2481299221,6cyclictest3059966ssh12:17:332
24812992212,3cyclictest21072-21irqrtprio11:20:192
24812992211,9cyclictest3617-21ssh10:50:342
24812992211,9cyclictest3617-21ssh10:50:332
24812992211,9cyclictest16173-21ssh10:59:422
24812992211,9cyclictest16173-21ssh10:59:422
24812992210,9cyclictest9410-21wc08:35:022
24812992210,5cyclictest0-21swapper/208:25:232
2480299222,5cyclictest0-21swapper/112:30:371
2480299222,15cyclictest23274-21ssh10:10:111
24802992214,5cyclictest18728-21ssh09:30:281
2480299221,17cyclictest19860-21ssh11:55:131
2482499219,9cyclictest0-21swapper/311:55:283
2482499214,10cyclictest35966irq/58-eth012:25:133
24824992113,6cyclictest35966irq/58-eth011:15:153
2481299219,9cyclictest23716-21cron08:50:022
2481299217,11cyclictest6123-21unixbench_multi11:10:302
2481299212,11cyclictest20106-21kthreadcore10:25:202
2481299211,6cyclictest2876966ssh10:11:172
2481299211,6cyclictest2132666ssh12:12:302
2481299211,15cyclictest13178-21cut08:05:162
2481299211,11cyclictest1929-21grep07:50:222
2481299211,10cyclictest19983-21ssh09:50:192
24812992110,9cyclictest6557-21ssh09:41:322
24812992110,9cyclictest29801-21ssh10:48:532
24812992110,9cyclictest25821-21ssh09:16:342
24812992110,9cyclictest12147-21ssh10:03:062
2480299212,6cyclictest16986-21diskstats11:00:141
2480299212,14cyclictest844-21munin-node11:35:121
2480299212,11cyclictest1990-21ssh11:10:121
2480299211,9cyclictest10734-21apt-get11:50:131
2480299211,9cyclictest10734-21apt-get11:50:121
24802992113,6cyclictest12589-21ssh10:55:311
24802992113,6cyclictest12589-21ssh10:55:301
2480299211,14cyclictest10210-21ls07:25:221
2480299211,12cyclictest14881-21df12:10:141
2482499203,13cyclictest0-21swapper/310:35:113
2482499203,13cyclictest0-21swapper/310:25:113
2482499202,9cyclictest0-21swapper/309:45:143
2482499201,2cyclictest0-21swapper/309:40:123
24824992011,6cyclictest35966irq/58-eth008:10:133
24824992011,6cyclictest35966irq/58-eth008:10:123
24824992011,4cyclictest0-21swapper/311:05:233
2481299208,9cyclictest0-21swapper/208:15:212
2481299202,10cyclictest9597-21diskstats08:35:162
2481299202,10cyclictest16524-21/usr/sbin/munin07:35:162
2481299201,9cyclictest9768-21timerwakeupswit09:05:272
2481299201,10cyclictest31371-21sed08:20:292
2481299201,10cyclictest24109-21cut08:50:152
2480299208,7cyclictest28170-21sh09:55:121
2480299204,11cyclictest21-21ksoftirqd/107:45:121
2480299201,16cyclictest14168-21ssh10:05:121
2480299200,16cyclictest16309-21latency_hist07:35:011
2482499193,9cyclictest35966irq/58-eth010:05:103
2482499193,12cyclictest0-21swapper/309:55:123
24824991912,5cyclictest0-21swapper/311:40:403
24824991911,6cyclictest0-21swapper/312:05:093
2482499191,13cyclictest0-21swapper/309:50:163
2482499191,13cyclictest0-21swapper/308:25:233
24824991910,6cyclictest0-21swapper/309:10:583
2481299193,9cyclictest3539-21tune2fs07:55:162
2481299192,9cyclictest31099-21kthreadcore08:55:202
2481299192,8cyclictest29736-21/usr/sbin/munin07:15:152
2481299191,8cyclictest1332-21fschecks_time09:00:162
2481299191,8cyclictest1332-21fschecks_time09:00:152
2481299190,14cyclictest2364-21ls07:20:182
2480299197,10cyclictest11368-21ssh10:20:231
2480299195,5cyclictest201rcuc/109:50:291
2480299195,3cyclictest21-21ksoftirqd/108:45:131
2480299191,4cyclictest8645-21ntp_192.168.11410:00:251
24802991911,6cyclictest12771-21sshd08:02:041
24802991911,6cyclictest10961-21ssh09:10:111
2480299191,10cyclictest0-21swapper/109:18:371
24802991910,3cyclictest10567-21diskmemload10:40:061
2482499189,6cyclictest35966irq/58-eth011:00:123
2482499183,11cyclictest0-21swapper/312:20:143
2482499183,11cyclictest0-21swapper/312:20:133
2482499182,13cyclictest0-21swapper/309:35:123
2482499182,10cyclictest0-21swapper/309:00:133
2482499182,10cyclictest0-21swapper/309:00:133
2482499182,10cyclictest0-21swapper/308:20:133
2482499182,10cyclictest0-21swapper/308:05:133
2482499182,10cyclictest0-21swapper/307:45:143
2482499182,10cyclictest0-21swapper/307:40:133
2482499182,10cyclictest0-21swapper/307:15:143
24824991812,4cyclictest0-21swapper/310:30:133
2482499181,13cyclictest0-21swapper/311:30:123
2482499181,12cyclictest0-21swapper/309:05:163
24824991810,6cyclictest0-21swapper/310:41:273
2480299189,6cyclictest770-21sshd11:09:041
2480299189,6cyclictest29801-21cut08:20:211
2480299181,8cyclictest29516-21cron07:15:011
2480299181,8cyclictest29516-21cron07:15:001
2480299181,8cyclictest18739-21kthreadcore07:35:191
2480299181,12cyclictest25199-21cut11:40:171
2482499179,6cyclictest0-21swapper/309:25:443
2482499178,6cyclictest0-21swapper/309:31:323
2482499172,12cyclictest0-21swapper/310:00:143
2482499172,10cyclictest0-21swapper/312:15:053
2482499172,10cyclictest0-21swapper/307:10:143
2482499172,10cyclictest0-21swapper/307:10:133
24824991711,4cyclictest0-21swapper/311:20:013
2482499171,13cyclictest0-21swapper/308:55:163
2482499171,13cyclictest0-21swapper/307:55:143
2480299172,4cyclictest21-21ksoftirqd/110:25:201
2480299172,2cyclictest131rcu_preempt08:30:121
24802991712,3cyclictest9650irq/52-mmc007:15:211
24802991710,3cyclictest21-21ksoftirqd/109:35:111
2482499162,9cyclictest0-21swapper/308:15:133
2482499162,10cyclictest0-21swapper/311:35:143
2482499161,9cyclictest35966irq/58-eth008:35:153
24824991611,3cyclictest0-21swapper/308:04:533
2482499161,11cyclictest0-21swapper/308:45:183
2482499161,10cyclictest0-21swapper/309:20:143
2482499161,10cyclictest0-21swapper/307:50:273
2480299168,6cyclictest3159-21sshd07:51:051
2480299167,6cyclictest6738-21cut09:05:181
2480299165,8cyclictest0-21swapper/107:40:131
2480299162,10cyclictest28893-21fschecks_count08:55:141
2480299162,10cyclictest25756-21kthreadcore08:15:211
2480299161,10cyclictest2260-21fschecks_count07:20:161
2481299150,11cyclictest24854-21cat07:10:022
2480299151,2cyclictest131rcu_preempt08:25:111
2480299151,11cyclictest24332-21sed08:50:181
2482499149,3cyclictest0-21swapper/307:35:263
2482499131,9cyclictest0-21swapper/308:30:113
2482499131,10cyclictest0-21swapper/307:20:303
2480299131,9cyclictest24843-21latency_hist07:10:011
2482499121,8cyclictest0-21swapper/307:35:013
2482499102,5cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional