You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-24 - 18:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun May 24, 2026 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31469997034,22cyclictest35966irq/58-eth012:35:113
31438995916,6cyclictest12-21ksoftirqd/012:25:140
3144999553,40cyclictest5604-21ssh10:15:121
3143899535,38cyclictest30542-21ssh09:35:130
3143899511,31cyclictest7813-21ssh11:10:140
31469994621,15cyclictest35966irq/58-eth009:40:123
3146099463,31cyclictest21509-21ssh09:30:132
3144999444,3cyclictest131rcu_preempt12:30:141
3146099435,26cyclictest1889-21ssh09:55:142
31449994315,4cyclictest131rcu_preempt09:50:121
31449994313,3cyclictest131rcu_preempt07:45:111
31460994115,10cyclictest22976-21irqstats11:00:212
3143899404,22cyclictest131rcu_preempt08:50:130
3146099382,18cyclictest11378-21needreboot11:10:232
31438993826,6cyclictest12-21ksoftirqd/010:25:130
3146099370,23cyclictest11736-21head09:00:312
3144999373,26cyclictest24263-21ssh11:55:121
3144999371,30cyclictest0-21swapper/107:15:131
31438993717,11cyclictest131rcu_preempt11:35:130
3146999363,19cyclictest35966irq/58-eth010:40:133
31469993623,10cyclictest35966irq/58-eth012:15:113
3146999362,22cyclictest35966irq/58-eth007:15:113
3146099362,17cyclictest27822-21ssh10:45:182
3146099362,17cyclictest27762-21ssh11:38:172
3144999361,28cyclictest9457-21ssh10:35:131
31469993518,11cyclictest0-21swapper/312:05:133
31469993518,11cyclictest0-21swapper/312:05:123
31460993518,13cyclictest14656-21apt-get10:20:142
31460993510,17cyclictest31218-21ssh11:05:132
3144999353,26cyclictest28968-21ssh10:10:111
3143899353,10cyclictest131rcu_preempt11:40:110
3143899351,26cyclictest22271-21ssh11:00:160
3143899351,25cyclictest18832-21ssh12:10:110
3146099342,16cyclictest24827-21ssh10:42:172
3146099341,16cyclictest16266-21ssh09:45:132
3143899342,24cyclictest27851-21ssh12:15:140
3146999335,13cyclictest35966irq/58-eth009:55:143
3146999333,18cyclictest35966irq/58-eth012:25:133
31460993318,9cyclictest30530-21apt-get09:35:132
3144999330,24cyclictest0-21swapper/111:15:131
3146099321,17cyclictest0-21swapper/209:40:132
3144999324,24cyclictest20582-21ssh11:35:121
3143899321,3cyclictest131rcu_preempt12:05:130
3143899321,3cyclictest131rcu_preempt12:05:130
3146099316,15cyclictest27346irq/40-dwc2_hso09:05:122
3146099316,13cyclictest0-21swapper/211:16:262
3146099314,16cyclictest27346irq/40-dwc2_hso07:35:132
3146099308,5cyclictest27-21ksoftirqd/209:20:272
3146099304,15cyclictest27346irq/40-dwc2_hso08:40:122
3146099303,6cyclictest27-21ksoftirqd/211:42:292
3146099303,21cyclictest27346irq/40-dwc2_hso10:30:132
3146099303,19cyclictest0-21swapper/209:15:132
3146099302,21cyclictest11021-21cat11:45:312
3146099294,17cyclictest0-21swapper/209:25:132
3146099294,17cyclictest0-21swapper/209:25:122
3146099293,9cyclictest0-21swapper/212:00:172
3146099293,19cyclictest0-21swapper/212:05:132
3146099293,19cyclictest0-21swapper/212:05:122
3146099293,11cyclictest0-21swapper/211:53:172
3146099292,4cyclictest27-21ksoftirqd/210:25:022
3146099292,20cyclictest0-21swapper/211:20:142
3146099292,20cyclictest0-21swapper/210:55:132
31460992917,9cyclictest22495-21apt-get12:30:142
31460992917,3cyclictest5591-21apt-get10:15:112
3144999294,6cyclictest131rcu_preempt12:25:141
3144999294,3cyclictest131rcu_preempt10:25:131
3144999292,21cyclictest2009-21usage_EthcTask11:40:301
3143899292,19cyclictest16859-21ssh11:15:130
3146999284,9cyclictest35966irq/58-eth010:35:123
3146999282,16cyclictest35966irq/58-eth010:45:133
31469992816,8cyclictest0-21swapper/311:35:133
3146099287,8cyclictest27-21ksoftirqd/211:30:012
3146099284,6cyclictest27-21ksoftirqd/210:05:252
3146099283,17cyclictest27346irq/40-dwc2_hso07:55:142
3146099282,19cyclictest0-21swapper/209:50:142
3144999282,20cyclictest8042-21ssh12:03:171
3143899282,3cyclictest131rcu_preempt07:20:120
3146999273,15cyclictest0-21swapper/312:00:183
3146999272,16cyclictest35966irq/58-eth009:50:133
31469992715,5cyclictest0-21swapper/311:55:143
3146099277,3cyclictest0-21swapper/207:47:112
3146099273,17cyclictest27346irq/40-dwc2_hso08:00:132
3146099273,14cyclictest27346irq/40-dwc2_hso07:50:132
3146099273,13cyclictest27-21ksoftirqd/212:35:272
3146099273,11cyclictest27-21ksoftirqd/212:26:452
3146099272,20cyclictest0-21swapper/210:51:172
3146099272,18cyclictest4721-21ssh12:20:162
31460992718,8cyclictest261rcuc/212:19:182
31460992715,8cyclictest9423-21apt-get10:35:142
31460992710,10cyclictest27-21ksoftirqd/211:58:222
3144999276,14cyclictest131rcu_preempt08:20:121
3144999272,20cyclictest15654-21grep11:30:281
3144999272,19cyclictest32296-21ssh11:22:171
3144999272,18cyclictest3437-21ssh09:20:121
3144999272,18cyclictest24534-21apt-get08:10:121
3143899278,3cyclictest12-21ksoftirqd/011:45:130
31438992710,9cyclictest12-21ksoftirqd/010:20:130
3146099263,8cyclictest27-21ksoftirqd/212:13:362
3146099263,5cyclictest27-21ksoftirqd/210:10:222
3146099263,14cyclictest27346irq/40-dwc2_hso08:30:122
31460992617,5cyclictest17619-21apt-get09:10:122
3146099261,20cyclictest3124-21ls08:55:152
3143899263,17cyclictest22515-21ssh12:30:130
3143899262,18cyclictest29678-21/usr/sbin/munin11:20:260
3143899262,18cyclictest15657-21sed11:30:280
3143899261,3cyclictest131rcu_preempt10:55:120
3146999252,14cyclictest35966irq/58-eth009:25:143
3146999252,14cyclictest35966irq/58-eth009:25:133
3146999251,16cyclictest35966irq/58-eth009:45:113
3146099257,14cyclictest0-21swapper/208:05:122
3146099255,4cyclictest27-21ksoftirqd/207:15:212
3146099252,4cyclictest271ksoftirqd/208:35:252
3146099252,12cyclictest790-21grep08:15:302
3146099250,20cyclictest0-21swapper/208:20:122
3143899252,3cyclictest131rcu_preempt08:20:140
31438992515,6cyclictest0-21swapper/009:05:120
3146999247,11cyclictest0-21swapper/311:42:183
3146999243,15cyclictest0-21swapper/311:45:313
3146999243,14cyclictest0-21swapper/311:24:173
3146999243,14cyclictest0-21swapper/311:05:183
3146099245,5cyclictest27-21ksoftirqd/207:20:282
3146099245,5cyclictest27-21ksoftirqd/207:10:232
3146099243,9cyclictest27-21ksoftirqd/208:50:222
3146099243,5cyclictest27-21ksoftirqd/207:30:192
3146099243,4cyclictest27-21ksoftirqd/210:00:392
3146099241,12cyclictest13861-21tr11:30:212
3144999249,7cyclictest0-21swapper/109:10:141
3144999242,6cyclictest131rcu_preempt11:06:181
3144999241,18cyclictest18660-21ssh10:56:171
3143899247,6cyclictest18678-21cut09:45:200
3146999237,10cyclictest0-21swapper/311:11:173
3146999233,13cyclictest0-21swapper/311:25:183
3146099234,12cyclictest27-21ksoftirqd/207:40:292
3146099234,12cyclictest27-21ksoftirqd/207:40:292
3146099232,5cyclictest27-21ksoftirqd/207:10:022
3146099231,9cyclictest27-21ksoftirqd/208:25:302
3146099231,14cyclictest27-21ksoftirqd/207:25:162
3144999239,3cyclictest0-21swapper/110:50:241
3144999236,7cyclictest20304-21timerwakeupswit09:45:281
3143899238,7cyclictest12-21ksoftirqd/008:30:110
3143899237,12cyclictest131rcu_preempt10:30:140
3143899236,4cyclictest131rcu_preempt10:50:290
3143899233,4cyclictest131rcu_preempt09:50:230
3143899233,11cyclictest0-21swapper/010:14:100
3143899232,16cyclictest15221-21ssh11:50:130
3143899231,4cyclictest17231-21diskmemload10:45:180
3143899231,17cyclictest31522-21ssh12:35:120
3146999229,8cyclictest35966irq/58-eth009:35:133
3146999227,10cyclictest0-21swapper/311:16:173
3146999222,13cyclictest35966irq/58-eth010:30:143
3146999221,11cyclictest35966irq/58-eth011:50:143
31449992214,3cyclictest131rcu_preempt09:30:121
3143899223,5cyclictest131rcu_preempt09:25:130
3143899223,5cyclictest131rcu_preempt09:25:130
3143899222,16cyclictest16233-21apt-get08:35:120
3146999214,9cyclictest0-21swapper/312:10:133
3146999212,13cyclictest0-21swapper/310:05:123
31469992111,6cyclictest35966irq/58-eth012:20:143
31469992110,8cyclictest0-21swapper/310:12:283
3146099211,12cyclictest24715-21fschecks_time08:10:162
3144999211,4cyclictest0-21swapper/109:07:121
3144999211,2cyclictest131rcu_preempt12:20:511
3144999211,2cyclictest131rcu_preempt11:10:191
31449992112,6cyclictest1-21systemd10:40:151
3143899217,3cyclictest131rcu_preempt09:10:120
3143899216,3cyclictest131rcu_preempt08:15:210
3143899215,3cyclictest131rcu_preempt12:00:150
3143899212,3cyclictest131rcu_preempt08:00:130
3143899212,16cyclictest22175-21ssh10:40:260
3143899211,5cyclictest5840-21uniq10:15:160
3143899211,5cyclictest0-21swapper/012:21:080
3143899211,4cyclictest23277-21sendmail_mailqu08:05:260
3143899211,3cyclictest131rcu_preempt10:35:290
3143899211,3cyclictest131rcu_preempt09:40:140
3143899211,2cyclictest131rcu_preempt09:55:130
3143899211,10cyclictest9923-21missed_timers08:25:230
31438992110,8cyclictest3130-21df08:55:160
3143899210,8cyclictest5237-21kthreadcore09:20:210
3143899210,8cyclictest23332-21fschecks_time07:35:160
3143899210,4cyclictest131rcu_preempt11:05:140
3146999209,8cyclictest0-21swapper/309:11:493
3146999209,6cyclictest0-21swapper/310:20:063
3146999203,12cyclictest0-21swapper/311:00:173
3146999202,11cyclictest0-21swapper/307:40:053
3146999202,11cyclictest0-21swapper/307:40:053
3146099202,4cyclictest0-21swapper/208:45:122
3144999205,5cyclictest24441-21latency_hist08:10:021
3144999201,3cyclictest131rcu_preempt08:40:281
31449992013,5cyclictest4249-21mailstats10:30:271
3144999201,2cyclictest131rcu_preempt09:15:311
31449992012,5cyclictest9813-21/usr/sbin/munin12:05:331
31449992012,5cyclictest9813-21/usr/sbin/munin12:05:331
31449992010,7cyclictest14915-21sed10:00:281
3144999200,2cyclictest131rcu_preempt08:00:151
3143899208,4cyclictest0-21swapper/010:00:290
3143899205,3cyclictest131rcu_preempt07:20:010
3143899202,4cyclictest131rcu_preempt09:15:200
3143899201,9cyclictest0-21swapper/009:32:100
3143899201,4cyclictest0-21swapper/011:25:190
3143899201,2cyclictest0-21swapper/007:30:190
3143899201,15cyclictest24254-21ssh11:55:130
3143899200,2cyclictest131rcu_preempt07:10:100
3146999199,8cyclictest0-21swapper/310:25:113
3146999199,8cyclictest0-21swapper/309:33:203
3146999194,11cyclictest35966irq/58-eth008:25:133
3146999192,10cyclictest0-21swapper/308:15:143
3146999192,10cyclictest0-21swapper/308:05:043
3146999191,12cyclictest0-21swapper/308:50:283
3146999191,12cyclictest0-21swapper/307:10:253
31469991910,6cyclictest0-21swapper/310:01:003
3144999199,7cyclictest15703-21sed11:50:181
3144999192,13cyclictest29088-21apt-get08:15:031
31449991914,3cyclictest12491-21ssh09:25:141
31449991914,3cyclictest12491-21ssh09:25:131
31449991914,3cyclictest0-21swapper/110:45:151
3144999191,3cyclictest131rcu_preempt11:25:211
31449991913,4cyclictest28570-21sed07:40:181
31449991913,4cyclictest28570-21sed07:40:181
3144999191,2cyclictest131rcu_preempt12:35:151
3144999191,2cyclictest131rcu_preempt12:18:251
31449991912,5cyclictest21-21ksoftirqd/109:35:161
31449991912,5cyclictest0-21swapper/107:50:301
31449991912,4cyclictest0-21swapper/108:50:131
31449991910,6cyclictest1-21systemd11:00:201
3144999190,2cyclictest131rcu_preempt09:00:161
3143899191,14cyclictest19950-21ssh10:05:110
3143899191,10cyclictest27979-21ntp_states08:10:250
3146999189,6cyclictest0-21swapper/312:32:003
3146999189,6cyclictest0-21swapper/310:51:163
3146999187,4cyclictest0-21swapper/310:15:203
3146999183,9cyclictest0-21swapper/311:30:273
3146999183,10cyclictest0-21swapper/310:56:173
3146999182,10cyclictest0-21swapper/309:00:153
3146999182,10cyclictest0-21swapper/308:45:143
3146999182,10cyclictest0-21swapper/308:20:143
31469991810,6cyclictest0-21swapper/309:15:323
31469991810,6cyclictest0-21swapper/308:00:293
3144999189,7cyclictest20229-21cat12:10:241
3144999188,7cyclictest614-21dbus-daemon10:05:071
3144999185,4cyclictest0-21swapper/109:42:051
3144999185,10cyclictest2209-21ssh09:55:191
3144999182,7cyclictest131rcu_preempt08:25:201
3144999181,3cyclictest131rcu_preempt07:30:121
3144999181,2cyclictest131rcu_preempt11:47:571
3144999181,2cyclictest131rcu_preempt10:20:281
3143899184,3cyclictest10502-21irqcore07:55:190
3143899184,2cyclictest131rcu_preempt09:00:210
3143899181,3cyclictest0-21swapper/007:45:180
3143899181,3cyclictest0-21swapper/007:25:150
3143899180,7cyclictest21113-21awk08:40:140
3143899180,3cyclictest131rcu_preempt08:45:180
3146999179,6cyclictest0-21swapper/309:22:053
3146999172,9cyclictest0-21swapper/307:35:133
3146999172,10cyclictest0-21swapper/307:20:143
3146999171,11cyclictest0-21swapper/307:35:023
3144999178,6cyclictest7796-21latency_hist09:00:021
3144999173,10cyclictest0-21swapper/107:10:111
3144999172,3cyclictest131rcu_preempt07:35:161
3144999171,3cyclictest131rcu_preempt07:25:181
3144999171,2cyclictest131rcu_preempt08:35:171
3144999170,3cyclictest131rcu_preempt07:20:261
3143899171,7cyclictest5881-21irqstats07:50:200
3143899171,3cyclictest131rcu_preempt07:10:010
3143899170,7cyclictest28077-21apt07:40:120
3143899170,7cyclictest28077-21apt07:40:110
3146999162,10cyclictest0-21swapper/308:10:143
3146999161,11cyclictest35966irq/58-eth007:55:123
3144999168,6cyclictest13472-21ntpq07:55:251
3144999164,10cyclictest13982-21kthreadcore08:30:191
3144999161,8cyclictest25925-21df08:45:141
3146999151,9cyclictest0-21swapper/307:25:273
3144999152,10cyclictest31489-21latency_hist07:10:021
3146999149,3cyclictest0-21swapper/307:50:243
3146999141,9cyclictest0-21swapper/308:30:173
3146999141,6cyclictest0-21swapper/308:40:003
3146999141,10cyclictest0-21swapper/309:10:023
3146999138,3cyclictest0-21swapper/308:56:443
3146999121,7cyclictest0-21swapper/308:40:143
3146999111,8cyclictest0-21swapper/307:45:113
314699991,5cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional