You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-22 - 08:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sat Aug 22, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3099321160,0sleep30-21swapper/321:51:413
3192429685,6sleep00-21swapper/019:08:110
3189929669,8sleep20-21swapper/219:07:532
3201529267,9sleep30-21swapper/319:09:223
2997629171,14sleep10-21swapper/119:05:151
59262720,0sleep10-21swapper/123:52:251
255122660,0sleep20-21swapper/222:07:542
3215399250,2cyclictest0-21swapper/121:26:511
3215399250,25cyclictest0-21swapper/100:04:471
3215699221,3cyclictest141rcu_preempt22:15:262
3216099213,18cyclictest24955-21diskmemload23:22:373
3215399200,20cyclictest24955-21diskmemload23:39:021
3215399200,1cyclictest0-21swapper/121:53:291
3216099194,5cyclictest0-21swapper/321:39:453
32156991916,1cyclictest32-21ksoftirqd/222:55:162
32156991916,1cyclictest32-21ksoftirqd/222:55:152
32156991914,1cyclictest32-21ksoftirqd/222:41:422
3215399192,11cyclictest11751-21apt-get22:00:131
3216099185,3cyclictest0-21swapper/322:30:123
3216099181,4cyclictest0-21swapper/300:35:143
3215699181,0cyclictest141rcu_preempt23:14:102
32153991817,1cyclictest0-21swapper/100:30:171
32153991815,2cyclictest0-21swapper/120:15:181
3215399180,5cyclictest0-21swapper/121:37:091
3215399180,3cyclictest141rcu_preempt19:25:141
3215399180,0cyclictest141rcu_preempt23:08:261
3215399180,0cyclictest141rcu_preempt00:27:151
3215099188,5cyclictest19633-21perf22:05:000
32160991715,1cyclictest39-21ksoftirqd/300:06:233
32156991712,2cyclictest32-21ksoftirqd/222:36:102
3215699171,12cyclictest24955-21diskmemload00:35:152
3215399174,1cyclictest25-21ksoftirqd/122:46:471
3215399170,0cyclictest0-21swapper/122:39:141
3216099162,11cyclictest0-21swapper/323:00:193
32160991613,3cyclictest55750irq/128-enp2s0-22:27:043
3216099161,2cyclictest141rcu_preempt21:22:063
3216099160,0cyclictest0-21swapper/319:35:123
3215699165,9cyclictest32-21ksoftirqd/220:40:172
3215699163,12cyclictest20895-21unixbench_multi21:45:252
3215699161,5cyclictest141rcu_preempt00:15:142
32156991614,2cyclictest32-21ksoftirqd/221:52:132
32156991614,1cyclictest32-21ksoftirqd/219:50:252
3215699161,3cyclictest141rcu_preempt21:20:182
3215699161,3cyclictest0-21swapper/200:10:142
3215699160,8cyclictest0-21swapper/223:35:272
3215399162,12cyclictest24955-21diskmemload21:59:161
32153991614,2cyclictest25-21ksoftirqd/122:13:531
32153991614,1cyclictest25-21ksoftirqd/123:03:361
32153991612,1cyclictest25-21ksoftirqd/121:46:341
3215399160,1cyclictest24955-21diskmemload00:37:401
3215399160,1cyclictest141rcu_preempt21:44:031
3215099163,7cyclictest6857-21munin-run19:25:010
3215099162,7cyclictest12019-21munin-run00:35:000
3215099161,8cyclictest0-21swapper/023:20:010
32160991512,2cyclictest39-21ksoftirqd/323:40:003
32160991512,2cyclictest3260-21idleruntime-cro20:25:003
3216099150,1cyclictest0-21swapper/322:53:513
3215699150,13cyclictest10397-21wc21:40:012
3215399157,4cyclictest25-21ksoftirqd/100:05:241
3215399153,7cyclictest516-21gdbus00:10:231
3215399150,15cyclictest0-21swapper/123:46:561
3215399150,15cyclictest0-21swapper/122:22:231
3215399150,0cyclictest141rcu_preempt22:40:461
3215099156,4cyclictest225-21systemd-journal20:02:550
3215099153,5cyclictest225-21systemd-journal21:45:010
32150991511,3cyclictest26132-21ssh23:45:150
32160991412,2cyclictest28547-21kworker/u8:023:27:543
3216099140,12cyclictest0-21swapper/321:15:313
3215699147,3cyclictest32-21ksoftirqd/223:16:112
3215699145,3cyclictest141rcu_preempt23:40:112
3215699144,5cyclictest12288-21df_abs22:00:152
3215699143,4cyclictest16439-21grep23:00:212
32156991413,1cyclictest0-21swapper/223:50:052
32156991412,1cyclictest0-21swapper/220:25:112
32156991410,3cyclictest32-21ksoftirqd/220:10:102
32156991410,2cyclictest27382-21sed21:30:202
3215699140,6cyclictest225-21systemd-journal21:06:562
3215399148,5cyclictest25-21ksoftirqd/100:20:111
32153991413,1cyclictest25-21ksoftirqd/120:45:181
32153991412,1cyclictest7080-21diskmemload23:34:071
32153991411,2cyclictest25-21ksoftirqd/123:15:211
3215399140,1cyclictest0-21swapper/121:15:321
3215399140,13cyclictest0-21swapper/122:55:491
3215399140,13cyclictest0-21swapper/122:55:481
3215099147,6cyclictest28183-21munin-run22:09:590
3215099141,7cyclictest25974-21munin-run21:30:010
3216099139,1cyclictest39-21ksoftirqd/321:34:513
3216099137,4cyclictest39-21ksoftirqd/322:22:463
3216099136,4cyclictest487-21avahi-daemon00:10:233
3216099136,2cyclictest225-21systemd-journal00:00:003
3216099131,9cyclictest9244-21idleruntime-cro19:30:013
32160991312,1cyclictest0-21swapper/323:46:563
32160991310,2cyclictest24955-21diskmemload21:58:583
3216099130,1cyclictest0-21swapper/322:38:503
3215699139,3cyclictest10609-21cat20:40:012
3215699131,7cyclictest32-21ksoftirqd/220:55:212
3215699131,5cyclictest32714-21users22:50:272
32156991311,1cyclictest32-21ksoftirqd/223:32:122
32156991311,1cyclictest32-21ksoftirqd/200:09:542
3215699131,10cyclictest18516-21cron21:45:002
3215699131,0cyclictest141rcu_preempt22:48:242
32156991310,2cyclictest4109-21munin-plugin-st22:15:002
32156991310,2cyclictest29346-21grep00:25:192
32156991310,2cyclictest26964-21sed20:05:182
32156991310,2cyclictest18947-21grep21:25:162
32156991310,2cyclictest18064-21grep23:20:262
32156991310,2cyclictest14555-21ssh22:20:232
32156991310,2cyclictest1430-21tr20:20:142
3215699130,6cyclictest24955-21diskmemload23:47:442
3215699130,2cyclictest0-21swapper/221:19:042
3215699130,13cyclictest0-21swapper/222:26:102
3215399139,3cyclictest20073-21ssh22:05:011
3215399139,1cyclictest25-21ksoftirqd/122:16:041
3215399138,5cyclictest4525-21cut19:20:011
3215399133,7cyclictest225-21systemd-journal19:40:001
3215399132,6cyclictest0-21swapper/122:30:131
3215399131,2cyclictest17070-21apache_volume23:40:101
32153991311,2cyclictest25-21ksoftirqd/120:25:111
3215399130,0cyclictest141rcu_preempt19:30:171
3215099139,3cyclictest29305-21sed21:50:270
3215099131,6cyclictest29150-21cron22:29:590
3215099131,1cyclictest141rcu_preempt23:21:560
32150991311,2cyclictest0-21swapper/022:53:560
32150991311,1cyclictest10030-21apache221:12:140
32150991311,1cyclictest10030-21apache221:12:130
32150991310,2cyclictest6502-21sort20:30:160
32150991310,2cyclictest24506-21ssh23:05:180
32150991310,2cyclictest21719-21grep19:55:140
32150991310,2cyclictest18315-21grep19:45:250
3215099130,1cyclictest0-21swapper/000:19:040
3215099130,0cyclictest0-21swapper/000:01:000
3216099129,2cyclictest9054-21ssh23:54:523
3216099129,2cyclictest688-21grep23:30:153
3216099129,2cyclictest27195-21cat20:05:203
3216099129,2cyclictest26289-21tail21:10:163
3216099129,2cyclictest26289-21tail21:10:153
3216099129,2cyclictest26107-21cat21:30:013
3216099129,2cyclictest19707-21tail20:55:263
3216099129,2cyclictest19622-21ssh23:40:503
3216099129,2cyclictest19229-21latency_hist00:19:593
3216099129,2cyclictest17933-21sh21:44:333
3216099129,2cyclictest10148-21ssh00:33:343
3216099123,9cyclictest0-21swapper/300:22:063
3216099123,8cyclictest21086-21/usr/sbin/munin22:05:183
3216099122,1cyclictest141rcu_preempt23:11:263
32160991210,2cyclictest39-21ksoftirqd/319:20:013
3216099120,11cyclictest3624-21cron00:30:013
3216099120,11cyclictest0-21swapper/322:46:523
3216099120,0cyclictest0-21swapper/322:10:463
3215699129,2cyclictest3113-21grep19:15:182
3215699129,2cyclictest30507-21head22:30:162
3215699128,3cyclictest7038-21cat19:25:002
3215699128,3cyclictest5890-21sed00:30:252
3215699126,4cyclictest311rcuc/221:15:012
3215699126,4cyclictest311rcuc/221:15:002
3215699121,2cyclictest141rcu_preempt20:00:172
32156991210,1cyclictest0-21swapper/223:55:232
32156991210,1cyclictest0-21swapper/200:20:072
3215399129,2cyclictest5770-21wc19:20:171
3215399129,2cyclictest15275-21wc19:40:181
3215399129,2cyclictest1443-21tr20:20:141
3215399128,3cyclictest339-21ssh22:50:311
3215399128,3cyclictest17223-21ssh21:24:411
3215399128,3cyclictest15715-21sh00:17:161
3215399126,1cyclictest24955-21diskmemload23:14:251
3215399124,6cyclictest25-21ksoftirqd/121:05:001
32153991210,2cyclictest28486-21grep21:30:301
3215399120,10cyclictest31010-21ssh21:13:141
3215399120,10cyclictest31010-21ssh21:13:131
3215099129,2cyclictest8872-21grep20:35:140
3215099129,2cyclictest32382-21grep23:10:160
3215099129,2cyclictest28315-21sh20:10:000
3215099129,2cyclictest27374-21grep21:30:200
3215099129,2cyclictest26945-21wc21:49:590
3215099129,2cyclictest20610-21head00:20:160
3215099129,2cyclictest15309-21ssh23:00:140
3215099129,2cyclictest13991-21sort20:45:140
3215099129,2cyclictest13909-21grep00:35:210
3215099129,2cyclictest13267-21grep19:35:200
3215099129,2cyclictest10316-21head19:30:170
3215099128,3cyclictest4136-21sed21:55:160
3215099128,3cyclictest2243-21grep23:50:170
3215099128,3cyclictest2142-21wc20:20:190
3215099127,1cyclictest1142-21runrttasks23:35:130
3215099122,8cyclictest2934-21/usr/sbin/munin00:10:200
32150991210,1cyclictest4458-21sh19:20:010
3215099120,11cyclictest4275-21sed20:25:170
3216099119,2cyclictest14292-21ssh22:00:513
3216099119,1cyclictest8502-21grep22:55:243
3216099119,1cyclictest8502-21grep22:55:233
3216099119,1cyclictest6597-21grep22:15:273
3216099119,1cyclictest5806-21cat19:20:173
3216099119,1cyclictest31463-21cat20:15:173
3216099119,1cyclictest28861-21wget20:10:133
3216099119,1cyclictest23975-21grep20:00:143
3216099119,1cyclictest21435-21apache_processe19:55:113
3216099119,1cyclictest20512-21basename21:00:113
3216099119,1cyclictest15170-21rm22:40:183
3216099119,1cyclictest11271-21ssh23:16:383
3216099118,2cyclictest5998-21cut20:30:143
3216099118,2cyclictest20933-21tail21:45:253
3216099118,2cyclictest19069-21tail00:00:183
3216099116,4cyclictest4317-21kworker/3:120:40:223
3216099115,2cyclictest225-21systemd-journal19:35:003
3216099110,10cyclictest0-21swapper/323:06:343
3215699118,2cyclictest4217-21sort21:55:172
3215699118,2cyclictest32267-21tr20:15:222
3215699118,2cyclictest28122-21chrt23:26:462
3215699118,2cyclictest26538-21ssh23:06:172
3215699118,2cyclictest22153-21grep21:00:242
3215699118,2cyclictest2136-21cat19:15:002
3215699118,2cyclictest21172-21ssh00:01:132
3215699118,2cyclictest16939-21cut19:45:132
3215699118,1cyclictest141rcu_preempt19:55:122
3215699110,2cyclictest0-21swapper/219:40:192
3215399118,2cyclictest306102sleep120:11:211
3215399118,2cyclictest26406-21tr20:05:161
3215399118,2cyclictest25704-21chrt20:02:041
3215399118,2cyclictest22382-21cut19:55:181
3215399118,2cyclictest19464-21ssh23:21:311
3215399118,2cyclictest13048-21rm23:56:241
3215399117,3cyclictest6514-21cut20:30:161
3215399117,3cyclictest21382-21ssh22:25:121
3215399116,5cyclictest25803-21grep23:25:201
3215399110,5cyclictest21163-21cron19:55:001
3215099118,2cyclictest9035-21taskset19:27:410
3215099118,2cyclictest8529-21sh22:36:080
3215099118,2cyclictest6884-21tr22:55:120
3215099118,2cyclictest6884-21tr22:55:110
3215099118,2cyclictest4434-21ssh22:15:040
3215099118,2cyclictest3624-21strings21:35:210
3215099118,2cyclictest29769-21ssh22:30:130
3215099118,2cyclictest29113-21cat22:10:150
3215099118,2cyclictest28897-21ssh00:05:340
3215099118,2cyclictest26708-21ssh22:47:160
3215099118,2cyclictest25294-21sed23:25:160
3215099118,2cyclictest22476-21chrt21:03:570
3215099118,2cyclictest22415-21chrt23:43:010
3215099118,2cyclictest19764-21sed20:55:260
3215099118,2cyclictest18766-21ssh22:42:340
3215099118,2cyclictest1621-21ssh23:30:190
3215099118,2cyclictest15570-21wc19:40:210
3215099118,2cyclictest15010-21grep22:20:270
3215099118,2cyclictest13282-21ssh21:21:360
3215099118,2cyclictest11715-21sed23:55:240
3215099118,2cyclictest10737-21chrt20:40:140
3215099117,3cyclictest29475-21sed00:25:210
3216099109,1cyclictest32734-21ls19:10:143
3216099109,1cyclictest24808-21chrt21:08:263
3216099109,1cyclictest16543-21sed19:45:003
3216099109,1cyclictest13687-21expr20:45:133
3216099108,2cyclictest17592-21chrt20:52:303
3216099108,1cyclictest5362-21taskset20:26:383
3215699108,1cyclictest7901-21chrt20:33:462
3215699108,1cyclictest177752sleep220:54:532
3215699108,1cyclictest13675-21taskset19:35:352
3215699107,2cyclictest9041-21taskset19:27:472
3215699107,2cyclictest13595-21tr20:45:132
3215699103,3cyclictest225-21systemd-journal19:31:552
3215399107,2cyclictest898-21cut19:10:181
3215399107,2cyclictest8354-21cut20:35:121
3215399107,2cyclictest17367-21sort19:45:151
3215399107,2cyclictest16052-21cat20:50:161
3215399107,2cyclictest12219-21cat20:40:251
3215399104,5cyclictest25-21ksoftirqd/121:05:151
3215399101,2cyclictest225-21systemd-journal20:55:551
3215099108,1cyclictest22795-21taskset21:05:060
3215099108,1cyclictest15903-21wc20:50:150
3215099108,1cyclictest1113-21ssh21:15:020
3215099107,2cyclictest905-21latency19:10:190
3215099107,2cyclictest20394-21awk19:50:220
321609996,2cyclictest20506-21cat19:50:253
321609990,6cyclictest16828-21grep19:45:143
321609990,5cyclictest0-21swapper/320:35:113
321509997,1cyclictest31164-21apache_accesses20:15:140
321509997,1cyclictest29017-21cut20:10:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional