You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-22 - 09:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed Jul 22, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1152929566,9sleep20-21swapper/219:08:072
1135328775,8sleep00-21swapper/019:05:490
1149828064,12sleep10-21swapper/119:07:421
1139327866,8sleep30-21swapper/319:06:193
1177099330,32cyclictest5673-21kworker/3:121:56:133
1177099290,28cyclictest5673-21kworker/3:121:46:323
1177099281,25cyclictest4571-21diskmemload21:31:313
11770992724,2cyclictest31267-21kworker/3:223:57:023
11770992624,2cyclictest31267-21kworker/3:222:53:433
1177099250,25cyclictest15380-21kworker/3:000:32:533
1177099250,24cyclictest15380-21kworker/3:000:11:123
1177099240,23cyclictest31267-21kworker/3:223:18:133
1176299240,23cyclictest0-21swapper/023:26:530
11770992321,2cyclictest31267-21kworker/3:200:01:533
11770992320,2cyclictest31267-21kworker/3:223:13:323
1177099230,22cyclictest15380-21kworker/3:000:15:033
1176499230,21cyclictest0-21swapper/122:09:031
1177099221,20cyclictest5673-21kworker/3:122:02:423
1177099220,22cyclictest15380-21kworker/3:000:09:323
1177099220,21cyclictest31267-21kworker/3:223:21:423
11767992220,1cyclictest477-21rs:main2
1176499223,19cyclictest0-21swapper/122:27:321
1176499222,2cyclictest23059-21kworker/1:100:36:021
11770992120,1cyclictest0-21swapper/322:36:223
11770992120,1cyclictest0-21swapper/322:27:323
1177099210,21cyclictest31267-21kworker/3:223:28:333
1176499210,2cyclictest0-21swapper/100:17:401
298142200,0sleep20-21swapper/223:20:112
1177099200,20cyclictest31267-21kworker/3:223:07:233
1177099200,20cyclictest31267-21kworker/3:223:01:233
11764992020,0cyclictest0-21swapper/122:44:351
11764992019,1cyclictest0-21swapper/123:25:131
1177099190,16cyclictest0-21swapper/300:35:133
11767991917,1cyclictest477-21rs:main2
1176799190,0cyclictest0-21swapper/222:23:112
11764991916,1cyclictest0-21swapper/122:04:151
1176799180,18cyclictest0-21swapper/200:05:442
1176499181,17cyclictest4571-21diskmemload00:00:541
145652170,0sleep30-21swapper/323:49:253
11767991717,0cyclictest0-21swapper/222:42:442
11764991716,1cyclictest0-21swapper/121:13:271
1176499170,1cyclictest0-21swapper/122:21:031
1176499170,17cyclictest9966-21ssh00:25:011
1176299176,6cyclictest5781-21munin-run20:05:000
11770991615,1cyclictest0-21swapper/322:49:523
11767991614,1cyclictest55750irq/128-enp2s0-00:04:412
1176799160,2cyclictest0-21swapper/223:17:262
1176499162,2cyclictest0-21swapper/123:50:501
11764991615,1cyclictest0-21swapper/123:37:521
11764991615,1cyclictest0-21swapper/100:24:521
1176499160,15cyclictest0-21swapper/122:52:251
1176499160,15cyclictest0-21swapper/122:45:011
1176499160,14cyclictest0-21swapper/121:25:411
1176299168,4cyclictest12792-21apache223:20:230
11770991513,1cyclictest0-21swapper/321:17:133
1176799151,14cyclictest0-21swapper/221:47:352
11767991510,4cyclictest3968-21ssh22:45:152
1176799150,0cyclictest0-21swapper/221:11:492
1176499152,5cyclictest0-21swapper/121:15:021
1176499152,2cyclictest0-21swapper/100:05:191
1176499150,14cyclictest0-21swapper/121:46:291
1176299157,7cyclictest14280-21/usr/sbin/munin23:30:220
1176299152,1cyclictest141rcu_preempt23:10:250
1176299151,2cyclictest0-21swapper/020:35:110
1176299150,9cyclictest27064-21/usr/sbin/munin00:35:200
1176299150,6cyclictest0-21swapper/000:09:060
1177099143,4cyclictest225-21systemd-journal21:00:003
1177099143,4cyclictest0-21swapper/323:40:123
11770991413,1cyclictest0-21swapper/322:06:463
11770991412,1cyclictest0-21swapper/300:20:133
1177099140,14cyclictest0-21swapper/323:54:173
1177099140,13cyclictest0-21swapper/321:40:153
1176799149,3cyclictest12838-21apache222:10:412
11767991413,1cyclictest0-21swapper/223:13:552
11767991410,3cyclictest28690-21ls20:50:202
11767991410,3cyclictest26137-21grep00:15:162
1176799140,14cyclictest0-21swapper/222:18:312
1176799140,14cyclictest0-21swapper/222:18:302
1176799140,0cyclictest0-21swapper/200:39:122
11764991413,1cyclictest0-21swapper/123:45:321
11764991412,1cyclictest21498-21apache_processe23:15:151
1176499141,12cyclictest4571-21diskmemload23:20:051
1176499140,7cyclictest0-21swapper/123:40:191
1176499140,14cyclictest0-21swapper/122:59:031
1176299141,7cyclictest23706-21perf19:34:590
11762991414,0cyclictest0-21swapper/020:55:150
11762991414,0cyclictest0-21swapper/020:46:290
11762991411,2cyclictest19543-21wc22:15:250
11762991411,2cyclictest19543-21wc22:15:250
11762991410,3cyclictest30939-21cat19:50:010
1176299140,13cyclictest15503-21wc21:15:250
1177099139,3cyclictest23717-21grep20:40:193
1177099139,3cyclictest21650-21head22:55:233
1177099137,5cyclictest12838-21apache222:10:163
1177099134,4cyclictest0-21swapper/322:30:153
11770991310,2cyclictest32076-21head19:50:163
11770991310,2cyclictest27902-21cat22:20:253
11770991310,2cyclictest21593-21ssh22:16:523
11770991310,2cyclictest21593-21ssh22:16:513
11770991310,2cyclictest14704-21grep19:15:163
1177099130,2cyclictest0-21swapper/322:43:043
1176799139,3cyclictest24407-21tr21:40:202
1176799133,2cyclictest0-21swapper/221:30:042
1176799131,4cyclictest225-21systemd-journal22:36:422
11767991310,2cyclictest6068-21grep20:05:142
11767991310,2cyclictest17958-21ssh22:34:362
11767991310,2cyclictest1344-21ssh22:05:192
11767991310,2cyclictest11507-21grep20:15:192
11767991310,2cyclictest0-21swapper/221:55:172
1176799130,12cyclictest13891-21basename21:15:132
1176499137,5cyclictest195822sleep100:10:351
1176499136,3cyclictest4571-21diskmemload21:24:021
1176499133,5cyclictest4571-21diskmemload21:37:161
1176499132,10cyclictest38150irq/131-snd_hda23:33:191
1176499131,7cyclictest0-21swapper/123:58:271
11764991312,1cyclictest0-21swapper/121:42:121
11764991310,2cyclictest26590-21cron00:35:001
1176299139,3cyclictest26106-21sed23:55:270
1176299139,3cyclictest12068-21wc00:25:270
1176299139,3cyclictest11229-21sh22:49:590
1176299139,3cyclictest11039-21tr20:15:160
1176299131,6cyclictest21325-21munin-run19:29:590
11762991310,2cyclictest470-21grep19:50:230
11762991310,2cyclictest3901-21wc21:05:280
11762991310,2cyclictest28817-21grep23:00:150
11762991310,2cyclictest27257-21wc00:15:220
11762991310,2cyclictest18980-21grep00:30:140
11762991310,2cyclictest13-21ksoftirqd/021:25:080
11762991310,2cyclictest12003-21grep22:50:130
1176299130,5cyclictest0-21swapper/023:50:360
1176299130,12cyclictest0-21swapper/022:34:080
235632126,1sleep125-21ksoftirqd/122:18:251
235632126,1sleep125-21ksoftirqd/122:18:241
1177099129,2cyclictest7416-21ssh21:50:013
1177099129,2cyclictest5466-21cat21:10:173
1177099129,2cyclictest22285-21grep19:30:173
1177099129,2cyclictest16384-21grep20:25:183
1177099129,2cyclictest16246-21grep23:30:253
1177099129,2cyclictest12234-21cat20:15:283
1177099129,2cyclictest12155-21grep00:25:283
1177099127,1cyclictest11998-21apt-get19:10:143
1176799129,2cyclictest3646-21cat00:20:252
1176799129,2cyclictest31645-21cut23:40:142
1176799129,2cyclictest29476-21head19:45:192
1176799129,2cyclictest21513-21uname20:35:172
1176799129,2cyclictest19316-21ssh00:30:152
1176799129,2cyclictest16594-21cut20:25:202
1176799129,2cyclictest10564-21tr00:25:162
1176799128,3cyclictest6478-21ssh23:25:152
1176799128,3cyclictest5998-21cat21:29:592
1176799128,3cyclictest25566-21sed19:35:262
1176799121,7cyclictest15836-21find23:30:212
11767991210,2cyclictest18650-21ssh21:36:462
1176799120,2cyclictest0-21swapper/222:54:122
1176799120,1cyclictest0-21swapper/219:25:142
1176799120,11cyclictest0-21swapper/223:08:352
1176799120,11cyclictest0-21swapper/222:25:152
1176499129,3cyclictest0-21swapper/123:08:101
1176499129,2cyclictest9083-21head21:50:211
1176499129,2cyclictest8138-21sh20:10:011
1176499129,2cyclictest4157-21grep21:05:291
1176499129,2cyclictest2898-21sed19:55:321
1176499129,2cyclictest27104-21cut19:40:161
1176499129,2cyclictest20812-21sed19:25:261
1176499129,2cyclictest14791-21wc23:10:261
1176499129,2cyclictest12067-21ssh22:30:251
1176499128,3cyclictest22892-21ssh21:58:581
1176499128,3cyclictest10747-21ssh21:32:131
1176499123,4cyclictest225-21systemd-journal22:11:171
1176499120,10cyclictest0-21swapper/120:55:141
1176299129,2cyclictest8287-21cat21:30:260
1176299129,2cyclictest6480-21sed23:05:260
1176299129,2cyclictest29333-21sh23:19:590
1176299129,2cyclictest27501-21grep22:40:140
1176299129,2cyclictest20907-21ssh22:55:190
1176299129,2cyclictest18969-21sed20:30:180
1176299129,2cyclictest18499-21grep19:20:260
1176299129,2cyclictest17933-21cron20:30:010
1176299128,3cyclictest8148-21cron00:05:000
1176299128,3cyclictest24756-21grep21:40:220
1176299128,3cyclictest14693-21grep20:20:220
1176299128,3cyclictest14693-21grep20:20:210
1176299125,4cyclictest32057-21cpuspeed_turbos21:45:160
1176299124,4cyclictest225-21systemd-journal00:24:560
1176299123,8cyclictest28480-21perf19:44:590
1176299121,6cyclictest15345-21munin-run23:50:000
11762991211,1cyclictest0-21swapper/021:00:150
11762991210,1cyclictest26468-21grep22:20:150
11762991210,1cyclictest19813-21tail22:35:180
1177099119,2cyclictest23494-21ntp_states21:20:243
1177099119,2cyclictest15968-21sh21:35:193
1177099119,1cyclictest8435-21hostname20:10:143
1177099119,1cyclictest3448-21sort21:05:223
1177099119,1cyclictest31446-21cut21:25:183
1177099119,1cyclictest31003-21cat20:55:203
1177099119,1cyclictest23257-21tr23:35:153
1177099119,1cyclictest19679-21expr19:25:153
1177099118,2cyclictest8134-21cron20:10:013
1177099118,2cyclictest4370-21grep20:00:163
1177099118,2cyclictest28373-21grep20:50:163
1177099118,2cyclictest27455-21cut19:40:203
1177099118,2cyclictest25315-21taskset20:45:073
1177099118,2cyclictest19718-21cat20:30:263
1177099113,5cyclictest20583-21apt-get20:35:123
1176799119,1cyclictest30370-21grep20:55:162
1176799118,2cyclictest9823-21ssh21:50:322
1176799118,2cyclictest9437-21cat20:10:202
1176799118,2cyclictest31569-21grep19:50:122
1176799118,2cyclictest30105-21wc23:00:232
1176799118,2cyclictest29750-21sh23:58:102
1176799118,2cyclictest28030-21ssh21:23:282
1176799118,2cyclictest27578-21grep19:40:212
1176799118,2cyclictest2704-21cut21:05:172
1176799118,2cyclictest26902-21cat20:45:272
1176799118,2cyclictest24851-21sh22:00:142
1176799118,2cyclictest23024-21cat19:30:262
1176799118,2cyclictest16455-21grep23:50:172
1176799118,2cyclictest13279-21grep19:10:232
1176799118,2cyclictest10331-21rm23:46:102
1176799112,8cyclictest18211-21apt-get20:30:132
1176799110,10cyclictest3176-21seq19:57:382
1176799110,10cyclictest24118-21ntpq20:40:232
1176499119,1cyclictest20720-21grep22:35:261
1176499118,2cyclictest9533-21ntpq20:10:221
1176499118,2cyclictest4098-21expr20:00:141
1176499118,2cyclictest32464-21basename21:00:131
1176499118,2cyclictest29081-21ls23:00:161
1176499118,2cyclictest28724-21cat20:50:201
1176499118,2cyclictest27494-21sort20:50:011
1176499118,2cyclictest24618-21grep19:35:171
1176499118,2cyclictest21706-21cut19:30:141
1176499118,2cyclictest17318-21grep19:20:151
1176499118,2cyclictest15213-21cut19:15:191
1176499118,2cyclictest10740-21chrt20:15:051
1176299119,1cyclictest13-21ksoftirqd/023:37:290
1176299119,1cyclictest11304-21sed22:10:280
1176299118,2cyclictest8597-21cron22:10:000
1176299118,2cyclictest6104-21tr20:05:130
1176299118,2cyclictest5089-21ssh22:26:110
1176299118,2cyclictest4552-21chrt23:43:140
1176299118,2cyclictest2988-21grep19:55:320
1176299118,2cyclictest25647-21cut22:00:180
1176299118,2cyclictest22994-21grep21:20:200
1176299118,2cyclictest19498-21ssh00:10:310
1176299118,2cyclictest18437-21ssh21:36:370
1176299118,2cyclictest18420-21ssh21:55:290
1176299118,2cyclictest13771-21grep19:10:260
1176299118,2cyclictest13222-21ssh21:53:080
1177099109,1cyclictest15463-21chrt20:23:493
1177099109,1cyclictest15463-21chrt20:23:493
1177099108,1cyclictest28943-21sort19:45:173
1177099108,1cyclictest17215-21expr19:20:143
1177099107,2cyclictest24229-21tr19:35:143
1176799108,1cyclictest921-21if_enp2s021:00:192
1176799108,1cyclictest17014-21cpu19:20:142
1176799107,2cyclictest20107-21ssh22:55:092
1176799104,6cyclictest537-21snmpd20:23:052
1176799104,6cyclictest537-21snmpd20:23:042
1176799101,8cyclictest14401-21/usr/sbin/munin19:15:172
1176499108,2cyclictest7282sleep119:51:491
1176499108,1cyclictest248662sleep120:42:181
1176499108,1cyclictest19864-21grep20:30:271
1176499107,2cyclictest29013-21tr19:45:171
1176499107,2cyclictest22391-21chrt20:36:181
1176499107,2cyclictest15083-21grep20:20:271
1176499107,2cyclictest15083-21grep20:20:261
1176499107,2cyclictest119512sleep119:10:031
1176499100,10cyclictest17165-21sendmail_mailst20:25:251
1176299108,1cyclictest29579-21taskset20:51:520
1176299108,1cyclictest14922-21df19:15:170
1176299107,2cyclictest8886-21chrt20:10:160
1176299107,2cyclictest5156-21ssh21:10:160
1176299107,2cyclictest25843-21chrt19:37:190
117709997,1cyclictest1270-21aten2_r7power_c19:55:153
117679997,1cyclictest4957-21taskset20:00:222
117629997,1cyclictest23007-21cut20:40:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional