You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-15 - 03:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Tue Sep 15, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1016729174,13sleep00-21swapper/019:05:400
836428459,9sleep30-21swapper/319:05:033
1023428370,9sleep20-21swapper/219:06:352
930927564,6sleep10-21swapper/119:05:211
261382560,0sleep20-21swapper/200:37:012
10595993026,1cyclictest39-21ksoftirqd/322:21:263
10593992926,3cyclictest0-21swapper/223:13:442
10593992926,3cyclictest0-21swapper/223:08:432
1059399270,27cyclictest0-21swapper/220:55:232
10593992622,3cyclictest0-21swapper/221:25:332
10593992510,15cyclictest0-21swapper/221:43:452
1059399250,25cyclictest0-21swapper/222:40:232
1059099250,1cyclictest0-21swapper/119:48:441
10595992422,1cyclictest39-21ksoftirqd/323:57:103
10595992414,9cyclictest39-21ksoftirqd/322:16:043
1059399240,23cyclictest0-21swapper/200:06:132
1059399240,23cyclictest0-21swapper/200:06:132
10595992317,3cyclictest39-21ksoftirqd/323:11:193
10593992320,3cyclictest0-21swapper/222:22:442
1059399230,23cyclictest0-21swapper/200:17:442
10588992323,0cyclictest0-21swapper/021:40:420
1058899230,1cyclictest0-21swapper/000:06:520
1058899230,1cyclictest0-21swapper/000:06:510
1059599220,3cyclictest141rcu_preempt22:30:153
10593992220,2cyclictest0-21swapper/223:49:342
10593992219,3cyclictest0-21swapper/223:23:132
10593992219,3cyclictest0-21swapper/223:02:232
10593992219,3cyclictest0-21swapper/221:20:032
10593992219,3cyclictest0-21swapper/221:15:532
1059399220,22cyclictest0-21swapper/221:45:242
1059399220,22cyclictest0-21swapper/221:32:132
1059399220,22cyclictest0-21swapper/221:10:542
1059099223,4cyclictest141rcu_preempt20:35:001
10588992218,3cyclictest0-21swapper/021:51:350
1059599217,11cyclictest39-21ksoftirqd/323:07:563
10593992119,2cyclictest0-21swapper/221:50:232
10593992118,3cyclictest0-21swapper/223:19:232
10593992118,3cyclictest0-21swapper/200:31:532
1059399210,21cyclictest0-21swapper/222:50:232
1059399210,21cyclictest0-21swapper/222:37:132
1059099210,21cyclictest0-21swapper/100:11:541
10595992019,1cyclictest0-21swapper/322:26:463
10595992017,2cyclictest39-21ksoftirqd/321:39:593
10595992012,7cyclictest39-21ksoftirqd/300:30:163
1059399200,20cyclictest0-21swapper/222:01:112
10590992012,5cyclictest10938-21ssh22:50:511
1059599195,2cyclictest141rcu_preempt21:19:123
1059599190,18cyclictest3378-21diskmemload21:11:533
1059099190,18cyclictest0-21swapper/123:30:381
1059599183,1cyclictest141rcu_preempt20:45:123
1058899186,6cyclictest26869-21munin-run23:40:010
1058899180,1cyclictest0-21swapper/000:21:420
35252170,0sleep10-21swapper/122:07:531
1059599176,3cyclictest39-21ksoftirqd/323:25:093
1059599171,7cyclictest39-21ksoftirqd/322:50:153
1059599171,4cyclictest39-21ksoftirqd/323:30:113
10595991712,3cyclictest39-21ksoftirqd/322:35:073
1059599171,15cyclictest0-21swapper/323:44:333
1059599170,17cyclictest580-21kworker/3:020:12:343
1059399172,2cyclictest141rcu_preempt23:40:232
1058899170,1cyclictest0-21swapper/022:20:000
1059599166,2cyclictest141rcu_preempt22:46:063
1059599161,3cyclictest141rcu_preempt21:30:193
10595991612,4cyclictest2650-21kworker/u8:000:23:593
10595991611,4cyclictest39-21ksoftirqd/321:54:503
10590991614,1cyclictest241rcuc/120:45:261
10590991613,2cyclictest0-21swapper/122:21:501
1059099160,3cyclictest0-21swapper/100:28:071
1058899161,15cyclictest0-21swapper/023:31:260
10588991610,4cyclictest17873-21ssh23:15:110
1059599157,2cyclictest141rcu_preempt22:04:353
1059599152,8cyclictest39-21ksoftirqd/321:29:583
1059599152,1cyclictest0-21swapper/319:40:223
10595991513,2cyclictest39-21ksoftirqd/322:57:573
10595991513,1cyclictest381rcuc/321:41:303
10595991511,3cyclictest19131-21head19:25:193
1059599150,15cyclictest0-21swapper/322:43:043
1059399159,2cyclictest13370-21basename00:30:002
10593991513,1cyclictest0-21swapper/221:57:232
10593991512,2cyclictest23906-21turbostat20:45:012
10593991512,2cyclictest23906-21turbostat20:45:002
1059099159,6cyclictest79242sleep123:09:091
10590991514,1cyclictest26237-21kworker/u8:223:47:051
1059099150,14cyclictest0-21swapper/122:14:261
1058899157,7cyclictest4382-21apt-get20:05:010
1058899153,1cyclictest0-21swapper/019:45:130
1058899151,1cyclictest0-21swapper/021:20:450
1058899150,2cyclictest0-21swapper/023:22:170
1058899150,2cyclictest0-21swapper/021:59:240
282732140,3sleep30-21swapper/300:38:433
1059599148,3cyclictest141rcu_preempt22:11:543
1059599143,1cyclictest141rcu_preempt00:11:343
1059599142,2cyclictest39-21ksoftirqd/319:55:283
10595991411,3cyclictest39-21ksoftirqd/323:38:443
10595991411,2cyclictest39-21ksoftirqd/300:30:013
10595991410,3cyclictest39-21ksoftirqd/323:51:483
1059599140,3cyclictest141rcu_preempt19:15:273
1059399140,2cyclictest0-21swapper/223:30:132
1059099142,9cyclictest0-21swapper/119:35:131
10590991411,3cyclictest38150irq/131-snd_hda21:41:031
10590991411,2cyclictest25-21ksoftirqd/121:40:001
10588991410,3cyclictest9674-21grep23:10:120
10588991410,3cyclictest707-21cron23:05:000
1058899140,14cyclictest0-21swapper/022:23:560
1059599139,3cyclictest32314-21uname22:05:223
1059599139,3cyclictest21620-21grep19:30:203
1059599134,1cyclictest39-21ksoftirqd/300:05:083
1059599134,1cyclictest39-21ksoftirqd/300:05:073
10595991312,1cyclictest381rcuc/300:15:343
10595991310,2cyclictest381rcuc/320:50:173
10595991310,2cyclictest29133-21grep00:00:173
10595991310,2cyclictest14731-21grep20:25:133
10595991310,1cyclictest39-21ksoftirqd/323:02:033
1059599130,4cyclictest1747-21kworker/u8:021:23:453
1059599130,1cyclictest0-21swapper/323:45:253
1059599130,12cyclictest28643-21grep19:45:203
1059399139,2cyclictest13824-21sed00:10:182
10593991313,0cyclictest0-21swapper/222:56:272
10593991311,2cyclictest94-21kswapd021:37:192
10593991310,2cyclictest3667-21sh00:05:002
10593991310,2cyclictest16328-21grep19:20:162
1059099132,6cyclictest6668-21ssh23:27:151
10590991312,1cyclictest21325-21ssh00:34:461
10590991311,1cyclictest0-21swapper/120:05:001
10590991310,2cyclictest866-21cut22:45:161
10590991310,2cyclictest28360-21cat21:24:591
1058899139,3cyclictest28603-21tr00:00:160
1058899136,1cyclictest2005-21readlink20:00:000
1058899134,3cyclictest9361-21awk20:15:000
1058899132,10cyclictest12939-21tail20:20:150
10588991310,2cyclictest32155-21ntpq22:05:210
10588991310,2cyclictest12922-21cut21:15:160
1058899130,13cyclictest0-21swapper/022:34:300
1059599129,2cyclictest27928-21head23:20:253
1059599129,2cyclictest22992-21tr19:35:133
1059599129,2cyclictest18710-21head20:30:273
1059599129,2cyclictest13839-21grep20:20:253
1059599129,2cyclictest11077-21tr19:10:153
1059599127,3cyclictest39-21ksoftirqd/320:05:003
1059599123,5cyclictest21983-21idleruntime-cro22:00:003
1059599121,7cyclictest29810-21aten2_r7power_e21:45:143
1059599120,2cyclictest0-21swapper/323:16:573
1059399129,2cyclictest32480-21cpuspeed_turbos19:55:152
1059399129,2cyclictest32325-21cut22:25:152
1059399129,2cyclictest29951-21cut19:50:132
1059399129,2cyclictest28166-21sort19:45:172
1059399129,2cyclictest16775-21grep20:30:002
1059399129,2cyclictest1485-21grep21:05:132
1059399129,2cyclictest12435-21grep19:10:262
1059399128,3cyclictest20744-21grep23:55:152
1059399128,3cyclictest10559-21ssh22:11:532
1059399121,6cyclictest225-21systemd-journal23:50:002
1059399121,1cyclictest26536-21apache_accesses20:50:112
1059399121,11cyclictest2401-21ssh23:25:132
1059099129,2cyclictest9520-21grep22:30:271
1059099129,2cyclictest6694-21grep21:50:171
1059099129,2cyclictest5508-21sed00:05:201
1059099129,2cyclictest5508-21sed00:05:191
1059099129,2cyclictest30255-21grep00:00:271
1059099129,2cyclictest25241-21tr19:40:151
1059099129,2cyclictest19986-21ssh21:58:251
1059099129,2cyclictest1968-21cron20:00:001
1059099129,2cyclictest11743-21tr23:50:141
1059099124,2cyclictest225-21systemd-journal19:20:001
1059099122,6cyclictest25-21ksoftirqd/121:15:141
1059099121,1cyclictest6603-21ssh21:11:001
10590991211,1cyclictest0-21swapper/122:35:271
10590991210,1cyclictest12526-21munin-plugin-st21:34:591
1059099120,11cyclictest23560-21wc00:35:221
1058899129,2cyclictest6829-21grep00:25:200
1058899129,2cyclictest5584-21grep21:30:170
1058899129,2cyclictest31623-21head19:50:260
1058899129,2cyclictest30395-21cut21:45:160
1058899129,2cyclictest2767-21ssh23:05:220
1058899129,2cyclictest27245-21grep20:50:160
1058899129,2cyclictest25526-21mailstats20:45:220
1058899129,2cyclictest24909-21tr19:40:000
1058899129,2cyclictest24365-21cut22:40:140
1058899129,2cyclictest23354-21head22:00:180
1058899129,2cyclictest21980-21sort00:15:150
1058899129,2cyclictest21898-21grep20:40:160
1058899129,2cyclictest21898-21grep20:40:160
1058899129,2cyclictest16654-21ssh22:55:010
1058899129,2cyclictest16252-21grep20:25:250
1058899129,2cyclictest14850-21wc21:35:250
1058899129,2cyclictest13778-21grep00:10:180
1058899129,2cyclictest11706-21tr23:50:130
1058899129,2cyclictest11257-21cat23:50:000
1058899121,7cyclictest11683-21cron21:15:000
10588991210,1cyclictest5346-21df20:05:150
1058899120,6cyclictest23708-21perf22:40:010
1059599118,2cyclictest31713-21chrt19:51:113
1059599118,2cyclictest2337-21cut20:00:153
1059599118,2cyclictest20322-21taskset20:35:203
1059599118,2cyclictest17035-21mailstats19:20:233
1059599117,3cyclictest11366-21wc20:15:253
1059399119,1cyclictest25142-21cat19:40:132
1059399118,2cyclictest6309-21cat20:05:252
1059399118,2cyclictest32351-21ssh00:21:022
1059399118,2cyclictest2740-21cat20:00:162
1059399118,2cyclictest24179-21cut20:45:132
1059399118,2cyclictest23499-21grep19:35:162
1059399118,2cyclictest19384-21cat23:35:152
1059399118,2cyclictest17081-21ssh22:15:272
1059399118,2cyclictest14328-21tr19:15:202
1059399118,2cyclictest10980-21ssh22:31:342
1059399118,2cyclictest1062-21ssh22:05:562
1059399115,2cyclictest225-21systemd-journal21:05:002
10593991110,1cyclictest0-21swapper/222:45:142
1059099119,1cyclictest30065-21cat21:45:141
1059099118,2cyclictest4379-21turbostat20:05:011
1059099118,2cyclictest31773-21rm21:26:101
1059099118,2cyclictest30463-21grep00:20:181
1059099118,2cyclictest28449-21ssh23:40:181
1059099118,2cyclictest25053-21ssh23:00:021
1059099118,2cyclictest24328-21cut22:40:141
1059099118,2cyclictest22541-21ssh22:19:461
1059099118,2cyclictest20497-21cat20:35:221
1059099118,2cyclictest19068-21ssh23:35:141
1059099118,2cyclictest16767-21munin-plugin-st20:30:001
1059099117,4cyclictest32116-21expr22:25:131
1059099117,4cyclictest12350-21timerwakeupswit19:10:251
1059099112,6cyclictest23140-21unixbench-2d00:15:261
1059099112,5cyclictest3378-21diskmemload23:10:131
1058899118,2cyclictest9623-21ssh22:11:100
1058899118,2cyclictest5656-21ssh23:26:270
1058899118,2cyclictest4886-21ssh22:47:380
1058899118,2cyclictest32352-21cut22:25:150
1058899118,2cyclictest29121-21grep21:25:150
1058899118,2cyclictest28907-21ssh00:39:130
1058899118,2cyclictest25148-21ssh23:57:570
1058899118,2cyclictest2160-21ssh23:44:330
1058899118,2cyclictest20202-21wc19:30:000
1058899118,2cyclictest17994-21ssh00:32:100
1058899118,2cyclictest17781-21grep20:30:170
1058899118,2cyclictest10082-21cat22:50:250
1058899117,3cyclictest32525-21sed21:00:230
1058899115,2cyclictest225-21systemd-journal19:34:590
1058899114,4cyclictest225-21systemd-journal21:10:010
1059599108,1cyclictest21804-21aten2_r7power_c20:40:173
1059599108,1cyclictest21804-21aten2_r7power_c20:40:163
1059599107,2cyclictest31447-21cut21:00:163
1059599107,2cyclictest29671-21expr20:55:153
1059599107,2cyclictest2913-21tail21:05:253
1059399108,1cyclictest7269-21grep20:10:162
1059399108,1cyclictest214202sleep220:39:032
1059399108,1cyclictest20707-21cstates19:30:142
1059399108,1cyclictest19388-21run-parts19:25:222
1059399108,1cyclictest11718-21taskset20:17:212
1059399107,2cyclictest17839-21wc20:30:182
1059099109,1cyclictest32692-21mailstats21:00:241
1059099108,1cyclictest30256-21expr19:50:141
1059099108,1cyclictest28173-21grep20:50:251
1059099108,1cyclictest2785-21cat21:05:241
1059099108,1cyclictest19790-21rm23:55:061
1059099107,2cyclictest29967-21cat20:55:181
1059099107,2cyclictest22902-21cut20:40:221
1059099107,2cyclictest22902-21cut20:40:221
1059099107,2cyclictest20186-21cat19:29:591
1059099107,2cyclictest18055-21ssh22:55:191
1059099107,2cyclictest17912-21cat19:24:591
1059099107,2cyclictest11185-21run-parts20:15:221
1059099106,1cyclictest141rcu_preempt23:15:071
1059099105,4cyclictest22545-21apt-get22:00:141
1059099101,1cyclictest225-21systemd-journal23:25:001
1058899107,2cyclictest30849-21chrt20:57:570
1058899107,2cyclictest19658-21cut20:35:160
1058899107,2cyclictest17326-21sed19:20:260
1058899107,2cyclictest14375-21open_inodes19:15:210
1058899107,2cyclictest11009-21ls19:10:140
1058899107,2cyclictest10360-21taskset20:15:150
105939997,1cyclictest12538-21expr20:20:132
105909996,2cyclictest20434-21cut19:30:131
105909996,2cyclictest13831-21grep20:20:241
105889997,1cyclictest259592sleep019:40:180
105909986,1cyclictest7501-21grep20:10:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional