You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-25 - 18:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Mon May 25, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2235929463,9sleep20-21swapper/207:08:562
2239929267,9sleep00-21swapper/007:09:270
2223829277,11sleep30-21swapper/307:07:203
2234828871,13sleep10-21swapper/107:08:471
286272720,0sleep10-21swapper/109:36:471
85452520,0sleep10-21swapper/110:04:121
190072460,0sleep10-21swapper/109:30:461
22533992716,5cyclictest25-21ksoftirqd/110:23:451
2253899260,26cyclictest0-21swapper/312:30:123
2253399260,25cyclictest0-21swapper/111:14:321
2253699240,24cyclictest0-21swapper/211:03:212
2253399241,2cyclictest141rcu_preempt11:02:371
2252999240,24cyclictest0-21swapper/010:54:010
2252999240,24cyclictest0-21swapper/010:54:010
2253699230,21cyclictest0-21swapper/211:21:502
22533992322,1cyclictest0-21swapper/111:34:161
2253399227,1cyclictest141rcu_preempt12:03:511
22533992214,7cyclictest25-21ksoftirqd/109:12:431
22533992214,7cyclictest25-21ksoftirqd/109:12:421
2253399220,21cyclictest20679-21latency10:30:191
2253399220,1cyclictest0-21swapper/111:22:121
2253899214,2cyclictest141rcu_preempt12:18:533
22536992120,1cyclictest0-21swapper/210:37:422
2253699210,1cyclictest141rcu_preempt12:14:052
2253399214,1cyclictest141rcu_preempt11:58:431
2253399211,19cyclictest25-21ksoftirqd/110:25:201
2253399210,21cyclictest0-21swapper/112:35:221
2253399210,21cyclictest0-21swapper/112:35:211
2253399210,19cyclictest0-21swapper/107:40:271
2252999210,3cyclictest0-21swapper/012:02:460
22538992019,1cyclictest0-21swapper/311:54:073
22538992017,1cyclictest0-21swapper/307:10:123
22536992018,1cyclictest32-21ksoftirqd/211:11:252
2253699201,2cyclictest0-21swapper/211:45:012
2253699200,1cyclictest141rcu_preempt12:39:382
2253699200,1cyclictest141rcu_preempt12:39:372
2253699200,19cyclictest0-21swapper/210:51:022
2253699200,19cyclictest0-21swapper/210:51:012
2253699200,18cyclictest0-21swapper/210:59:542
2253399204,2cyclictest141rcu_preempt10:17:531
2253399200,1cyclictest0-21swapper/109:57:231
2252999207,7cyclictest9129-21munin-run07:50:000
22538991915,1cyclictest39-21ksoftirqd/309:22:323
2253899190,19cyclictest0-21swapper/310:03:163
2253699191,18cyclictest0-21swapper/210:00:352
22536991910,6cyclictest32-21ksoftirqd/210:13:262
2253699190,19cyclictest0-21swapper/211:28:292
2253699190,19cyclictest0-21swapper/209:43:122
2253399195,13cyclictest795-21apt-get09:40:121
2253399191,6cyclictest0-21swapper/110:43:371
22533991915,2cyclictest25-21ksoftirqd/111:45:191
22533991911,7cyclictest0-21swapper/111:20:011
2253399190,2cyclictest0-21swapper/110:46:101
2252999190,1cyclictest0-21swapper/010:56:160
22538991816,1cyclictest39-21ksoftirqd/309:54:393
2253699180,18cyclictest0-21swapper/211:51:042
2253699180,18cyclictest0-21swapper/210:18:502
2253699180,17cyclictest0-21swapper/209:13:472
2253699180,17cyclictest0-21swapper/209:13:462
22533991813,4cyclictest24549-21kworker/u8:309:46:321
22533991811,5cyclictest25-21ksoftirqd/110:37:321
22529991815,1cyclictest0-21swapper/011:50:420
2252999180,3cyclictest0-21swapper/010:42:140
2252999180,16cyclictest0-21swapper/010:03:090
2253899173,11cyclictest39-21ksoftirqd/312:25:143
22538991713,3cyclictest39-21ksoftirqd/312:38:053
22538991713,3cyclictest39-21ksoftirqd/312:38:043
22538991713,3cyclictest39-21ksoftirqd/309:38:033
2253899171,1cyclictest141rcu_preempt11:17:053
22536991715,2cyclictest0-21swapper/211:34:102
2253699170,13cyclictest0-21swapper/209:17:332
2253399173,14cyclictest6795-21kworker/u8:210:06:431
2253399172,2cyclictest25-21ksoftirqd/111:39:241
22533991714,2cyclictest25-21ksoftirqd/112:15:211
2253399170,3cyclictest0-21swapper/110:58:331
2252999170,1cyclictest0-21swapper/009:58:460
2253899164,1cyclictest39-21ksoftirqd/312:11:533
22538991616,0cyclictest0-21swapper/309:44:263
22538991613,3cyclictest39-21ksoftirqd/309:28:493
2253899161,1cyclictest141rcu_preempt12:24:543
22538991611,3cyclictest20034-21sh10:50:013
2253899161,0cyclictest141rcu_preempt10:16:003
22536991616,0cyclictest0-21swapper/210:45:502
2253399167,8cyclictest26060-21apt-get12:30:141
2253399162,4cyclictest25-21ksoftirqd/110:50:351
2253399162,4cyclictest25-21ksoftirqd/110:50:351
2252999160,15cyclictest0-21swapper/011:09:590
22538991514,1cyclictest0-21swapper/310:24:363
2253899151,1cyclictest141rcu_preempt09:18:043
22538991511,1cyclictest39-21ksoftirqd/312:09:203
22536991515,0cyclictest0-21swapper/212:31:442
2253699150,15cyclictest0-21swapper/212:17:372
2253699150,15cyclictest0-21swapper/212:05:412
2253399158,2cyclictest25-21ksoftirqd/111:09:331
2253399152,4cyclictest25-21ksoftirqd/108:55:131
2253399150,1cyclictest0-21swapper/109:28:491
2253399150,14cyclictest15059-21diskmemload09:53:201
2253399150,13cyclictest15059-21diskmemload09:23:591
22529991515,0cyclictest0-21swapper/009:21:410
22529991514,1cyclictest0-21swapper/009:26:140
22538991412,2cyclictest39-21ksoftirqd/310:30:003
2253899141,1cyclictest141rcu_preempt09:30:133
2253899141,0cyclictest141rcu_preempt11:36:523
22538991410,3cyclictest11099-21chrt10:05:193
2253899140,13cyclictest5588-21fw_packets11:00:183
22536991411,2cyclictest30806-21cron08:34:592
2253699140,14cyclictest0-21swapper/212:00:392
22533991411,2cyclictest241rcuc/111:41:191
2253399140,3cyclictest20258-21ssh12:25:511
2252999143,10cyclictest10493-21awk10:25:000
22529991410,3cyclictest17979-21ssh09:30:190
22529991410,3cyclictest1348-21cat07:30:250
2252999140,1cyclictest0-21swapper/010:13:590
264212130,1sleep10-21swapper/112:10:231
2253899138,4cyclictest537-21snmpd11:23:273
2253899136,3cyclictest225-21systemd-journal11:45:003
22538991312,1cyclictest39-21ksoftirqd/311:10:143
22538991311,1cyclictest39-21ksoftirqd/310:10:143
22538991310,2cyclictest31533-21grep07:25:283
22538991310,2cyclictest30350-21grep08:30:253
22538991310,1cyclictest0-21swapper/311:46:543
2253899130,1cyclictest0-21swapper/309:10:473
2253899130,1cyclictest0-21swapper/309:10:463
2253699139,3cyclictest9189-21wc07:50:002
22536991312,1cyclictest291irq_work/210:25:572
22536991311,2cyclictest3441-21kworker/u8:311:08:322
22536991310,2cyclictest9743-21wc08:55:262
22536991310,2cyclictest8770-21wc11:40:262
22536991310,2cyclictest5940-21cut08:50:152
22536991310,2cyclictest5715-21grep10:40:252
22536991310,2cyclictest30625-21grep07:25:192
22536991310,2cyclictest27258-21sh07:20:012
22536991310,2cyclictest12743-21wc07:55:162
2253699130,2cyclictest0-21swapper/209:25:012
2253699130,1cyclictest0-21swapper/210:07:582
2253699130,13cyclictest0-21swapper/211:57:112
2253399139,3cyclictest25-21ksoftirqd/111:50:221
2253399139,3cyclictest25-21ksoftirqd/110:10:201
2253399135,2cyclictest32029-21latency_hist07:30:011
22533991310,2cyclictest21479-21cut08:15:151
22533991310,2cyclictest14422-21tr08:00:141
2252999139,2cyclictest29967-21ssh11:15:120
22529991310,2cyclictest28612-21grep10:15:240
22529991310,2cyclictest28218-21grep07:20:160
22529991310,2cyclictest26256-21cut09:35:210
22529991310,2cyclictest12485-21grep10:45:150
2252999130,12cyclictest594-21grep11:55:160
2253899129,2cyclictest7104-21ps08:50:233
2253899129,2cyclictest32578-21wc11:55:133
2253899129,2cyclictest30584-21sort10:55:243
2253899129,2cyclictest3019-21grep07:35:173
2253899129,2cyclictest24742-21grep08:20:143
2253899129,2cyclictest23850-21ntpq11:30:193
2253899129,2cyclictest19733-21egrep10:30:153
2253899129,2cyclictest1972-21cron07:35:003
2253899129,2cyclictest18262-21sed08:05:263
2253899129,2cyclictest14767-21sh11:25:143
2253899129,2cyclictest11512-21basename07:54:593
2253899129,2cyclictest11378-21sh09:45:383
2253899127,1cyclictest601-21ssh10:37:533
2253899121,7cyclictest0-21swapper/310:50:173
2253899121,7cyclictest0-21swapper/310:50:163
22538991210,1cyclictest26234-21grep07:15:203
2253899120,11cyclictest2752-21grep08:40:253
2253899120,11cyclictest26607-21grep08:25:163
2253699129,2cyclictest4985-21ssh11:38:502
2253699129,2cyclictest31371-21tr08:35:162
2253699129,2cyclictest29153-21sed07:20:262
2253699129,2cyclictest25455-21cat09:54:592
2253699129,2cyclictest21994-21taskset08:15:182
2253699129,2cyclictest17311-21cut09:30:152
2253699129,2cyclictest11239-21tail12:20:262
2253699127,5cyclictest15454-21apache210:31:482
2253699121,3cyclictest225-21systemd-journal10:23:182
22536991210,1cyclictest0-21swapper/209:55:572
2253399129,3cyclictest0-21swapper/107:45:121
2253399129,2cyclictest2969-21grep07:35:161
2253399129,2cyclictest27084-21sed08:25:191
2253399129,2cyclictest20616-21tail08:10:251
2253399129,2cyclictest17343-21grep12:05:171
2253399129,2cyclictest13665-21head09:05:171
2253399129,2cyclictest13369-21sed07:55:231
2253399128,1cyclictest141rcu_preempt08:35:351
2253399123,6cyclictest25-21ksoftirqd/109:20:001
2253399121,2cyclictest25-21ksoftirqd/112:21:011
2253399120,7cyclictest25-21ksoftirqd/108:50:151
2253399120,11cyclictest10062-21cut07:50:181
2252999129,2cyclictest6775-21cat07:45:010
2252999129,2cyclictest32258-21ssh11:35:180
2252999129,2cyclictest3210-21grep12:35:200
2252999129,2cyclictest3210-21grep12:35:200
2252999129,2cyclictest26281-21ssh11:12:360
2252999129,2cyclictest25340-21cut08:20:190
2252999129,2cyclictest22042-21cron11:29:590
2252999129,2cyclictest19516-21grep09:50:280
2252999129,2cyclictest17310-21head08:05:170
2252999129,2cyclictest14220-21cut08:00:120
2252999129,2cyclictest10882-21tail10:05:170
2252999129,2cyclictest10293-21munin-run09:00:000
2252999124,2cyclictest225-21systemd-journal09:45:000
2252999121,6cyclictest16995-21awk09:50:000
2252999120,12cyclictest0-21swapper/012:07:080
2252999120,11cyclictest26374-21grep07:15:210
2253899119,1cyclictest9498-21expr08:55:223
2253899119,1cyclictest4150-21grep08:45:173
2253899119,1cyclictest27614-21tr07:20:133
2253899119,1cyclictest21967-21sort08:15:173
2253899118,2cyclictest7788-21cut07:45:173
2253899118,2cyclictest32404-21ssh09:59:033
2253899118,2cyclictest17226-21runrttasks11:07:003
2253899118,2cyclictest10451-21rm12:00:393
22538991110,1cyclictest0-21swapper/309:05:133
2253899110,10cyclictest749-21sh08:40:013
2253699118,2cyclictest5767-21ntpq07:40:202
2253699118,2cyclictest2673-21ssh09:21:052
2253699118,2cyclictest24878-21ls07:15:002
2253699118,2cyclictest21071-21dump-pmu-power08:14:592
2253699118,2cyclictest1966-21taskset07:34:582
2253699118,2cyclictest1406-21sh11:16:552
2253699118,2cyclictest13650-21grep09:05:172
2253699118,2cyclictest12653-21taskset09:04:522
2253699117,3cyclictest26288-21grep09:35:202
2253699117,3cyclictest24672-21grep08:20:142
2253699110,7cyclictest225-21systemd-journal08:40:082
2253699110,0cyclictest141rcu_preempt12:25:172
2253399118,2cyclictest27612-21grep07:20:131
2253399118,2cyclictest19146-21taskset11:27:421
2253399118,2cyclictest12584-21chrt09:03:581
2253399115,5cyclictest25150-21pmu-power08:20:181
2253399111,9cyclictest0-21swapper/107:10:121
2253399111,4cyclictest225-21systemd-journal07:18:161
2253399110,7cyclictest18240-21tail08:05:251
2252999118,2cyclictest8096-21cat11:40:200
2252999118,2cyclictest5771-21rm11:00:180
2252999118,2cyclictest32700-21grep08:35:260
2252999118,2cyclictest30181-21grep07:25:150
2252999118,2cyclictest29189-21ssh12:31:200
2252999118,2cyclictest28172-21ssh12:11:290
2252999118,2cyclictest26525-21egrep08:25:150
2252999118,2cyclictest2644-21cat08:40:240
2252999118,2cyclictest24697-21uniq09:15:180
2252999118,2cyclictest23615-21cut11:30:180
2252999118,2cyclictest23429-21seq08:19:500
2252999118,2cyclictest21937-21ssh10:30:430
2252999118,2cyclictest19744-21head08:10:160
2252999118,2cyclictest17847-21wget12:25:140
2252999118,2cyclictest17487-21sed09:10:270
2252999118,2cyclictest17487-21sed09:10:260
2252999118,2cyclictest1398-21df12:15:150
2252999118,2cyclictest12276-21head09:00:260
2252999118,2cyclictest10488-21sort12:20:200
2252999117,3cyclictest12855-21grep07:55:170
2252999110,7cyclictest0-21swapper/011:46:170
2252999110,2cyclictest0-21swapper/010:35:040
2253899108,1cyclictest3747-21sshd10:40:093
2253899108,1cyclictest161342sleep308:02:463
2253899108,1cyclictest11784-21hostname07:55:103
2253899100,5cyclictest6059-21sort07:40:243
2253899100,5cyclictest15994-21kworker/3:209:00:233
2253699108,1cyclictest8940-21ssh09:45:052
2253699108,1cyclictest173812sleep208:05:162
2253699108,1cyclictest16059-21taskset08:01:472
2253699107,2cyclictest3711-21tr08:45:142
2253699107,2cyclictest26573-21tr08:25:162
2253699103,3cyclictest141rcu_preempt07:35:182
2253699100,1cyclictest141rcu_preempt07:50:182
2253399108,1cyclictest53232sleep108:47:271
2253399108,1cyclictest1953-21chrt07:34:501
2253399107,2cyclictest28718-21cut08:30:121
2253399100,9cyclictest3130-21taskset08:44:481
2252999108,1cyclictest9364-21basename07:50:130
2252999108,1cyclictest3431-21dpkg08:45:120
2252999108,1cyclictest133162sleep010:25:310
2252999107,2cyclictest6029-21grep08:50:160
2252999107,2cyclictest5966-21basename11:20:140
2252999107,2cyclictest29464-21head08:30:160
2252999107,2cyclictest2937-21tune2fs07:35:160
2252999107,2cyclictest13247-21cat09:05:160
225299997,2cyclictest24398-21unixbench_multi07:10:250
225389981,2cyclictest0-21swapper/308:10:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional