You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-01 - 02:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Fri May 01, 2026 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3816210062,33sleep00-21swapper/019:09:400
365329267,9sleep20-21swapper/219:07:322
369229164,9sleep30-21swapper/319:08:013
170628565,16sleep10-21swapper/119:05:031
114372770,3sleep125-21ksoftirqd/121:37:331
210012530,0sleep00-21swapper/021:24:250
394399230,0cyclictest0-21swapper/322:35:243
3936992320,2cyclictest16099-21sed23:55:261
3936992316,3cyclictest25-21ksoftirqd/122:14:591
393699231,1cyclictest21952-21cat20:55:011
393699231,1cyclictest21952-21cat20:55:001
393399239,7cyclictest13-21ksoftirqd/000:15:270
3933992210,5cyclictest13-21ksoftirqd/023:01:580
3933992116,2cyclictest13-21ksoftirqd/000:07:210
393399210,20cyclictest0-21swapper/000:32:090
394199200,1cyclictest0-21swapper/200:36:432
393699202,17cyclictest0-21swapper/121:14:441
394399190,2cyclictest0-21swapper/323:08:423
394199195,12cyclictest12082-21runrttasks22:16:232
394199192,1cyclictest0-21swapper/222:41:282
394199190,18cyclictest0-21swapper/221:31:502
3933991917,1cyclictest13-21ksoftirqd/023:34:210
3933991911,1cyclictest13-21ksoftirqd/022:17:070
393399190,19cyclictest0-21swapper/023:51:090
394199181,2cyclictest0-21swapper/221:11:412
3936991816,2cyclictest25-21ksoftirqd/121:33:491
3933991811,2cyclictest13-21ksoftirqd/000:26:220
394399170,1cyclictest0-21swapper/322:06:313
393399170,0cyclictest141rcu_preempt22:33:220
394399162,14cyclictest0-21swapper/321:48:263
3943991613,1cyclictest0-21swapper/300:01:163
394199162,3cyclictest32-21ksoftirqd/220:50:012
394199160,1cyclictest0-21swapper/222:49:052
394199160,1cyclictest0-21swapper/220:40:112
394199160,16cyclictest0-21swapper/200:27:432
394199160,14cyclictest0-21swapper/223:37:552
393699162,8cyclictest29448-21idleruntime-cro23:44:591
393699160,3cyclictest0-21swapper/122:26:331
393699160,1cyclictest0-21swapper/123:21:151
393699160,16cyclictest0-21swapper/100:24:151
393399167,1cyclictest13-21ksoftirqd/000:01:410
393399166,4cyclictest21732-21apt-get20:55:030
393399163,0cyclictest141rcu_preempt22:39:070
99362150,0sleep10-21swapper/122:15:211
394399156,8cyclictest1370-21apache223:45:503
394399152,11cyclictest14366-21runrttasks23:15:343
394399150,2cyclictest0-21swapper/323:52:343
394199155,8cyclictest29146-21diskmemload22:35:162
394199155,4cyclictest7569-21apt-config21:55:002
3936991515,0cyclictest0-21swapper/122:44:571
3936991515,0cyclictest0-21swapper/121:52:471
393699150,1cyclictest0-21swapper/123:36:231
393399156,3cyclictest141rcu_preempt21:40:450
393399151,7cyclictest18177-21munin-run19:39:590
3933991513,1cyclictest13-21ksoftirqd/021:13:040
393399151,1cyclictest141rcu_preempt21:29:000
3933991511,3cyclictest13-21ksoftirqd/021:30:170
3933991510,4cyclictest487-21avahi-daemon00:38:190
3933991510,2cyclictest13-21ksoftirqd/023:36:360
393399150,1cyclictest141rcu_preempt20:40:160
393399150,0cyclictest141rcu_preempt00:21:550
394399141,6cyclictest225-21systemd-journal22:25:503
3943991413,1cyclictest0-21swapper/322:22:203
394399141,1cyclictest0-21swapper/322:32:183
3943991411,2cyclictest28654-21cut20:00:163
3943991410,3cyclictest803-21tr20:10:173
394399140,13cyclictest0-21swapper/323:25:203
394199143,9cyclictest18164-21cron19:39:592
394199142,2cyclictest26828-21/usr/sbin/munin22:25:202
3941991411,2cyclictest24706-21ls00:00:292
394199140,0cyclictest0-21swapper/220:50:142
394199140,0cyclictest0-21swapper/220:50:132
393699146,5cyclictest27264-21apache222:06:591
3936991411,2cyclictest12203-21cron20:35:001
3936991410,3cyclictest0-21swapper/121:24:261
393699140,1cyclictest29146-21diskmemload22:58:401
393399143,1cyclictest141rcu_preempt21:54:220
3933991413,1cyclictest13-21ksoftirqd/023:24:390
3933991413,1cyclictest0-21swapper/022:50:170
3933991411,3cyclictest0-21swapper/020:50:140
3933991411,3cyclictest0-21swapper/020:50:130
393399141,0cyclictest141rcu_preempt23:07:080
3933991410,3cyclictest10612-21grep19:20:260
393399140,14cyclictest0-21swapper/022:13:540
394399139,3cyclictest29146-21diskmemload21:34:173
394399139,3cyclictest17079-21wc00:35:253
394399139,2cyclictest22288-21wc00:20:013
3943991310,2cyclictest9834-21sed21:55:263
3943991310,2cyclictest3672-21grep20:15:163
3943991310,2cyclictest19277-21taskset23:00:053
3943991310,2cyclictest16291-21grep19:35:123
394199133,9cyclictest13716-21cron23:54:592
394199133,7cyclictest22941-21ls19:49:592
3941991312,1cyclictest0-21swapper/200:12:332
3941991310,2cyclictest7162-21grep19:15:172
3941991310,2cyclictest31055-21wc23:45:192
3941991310,2cyclictest29537-21cat20:00:252
3941991310,2cyclictest28785-21grep23:05:202
3941991310,2cyclictest24887-21grep22:05:132
3941991310,2cyclictest23848-21grep21:25:242
3941991310,2cyclictest21276-21cut23:20:172
3941991310,2cyclictest15517-21tr23:55:202
394199130,12cyclictest20139-21cat23:19:592
394199130,12cyclictest18843-21sh22:20:242
393699139,3cyclictest30351-21cron21:30:011
393699138,4cyclictest6482-21ntpq20:20:201
393699138,4cyclictest6183-21taskset19:13:591
393699138,3cyclictest11238-21grep19:25:001
393699132,11cyclictest38150irq/131-snd_hda00:10:191
3936991310,2cyclictest24086-21cron22:04:591
3936991310,2cyclictest23670-21grep00:00:191
3936991310,2cyclictest14841-21df00:15:161
393699130,11cyclictest0-21swapper/100:05:081
393399139,3cyclictest5360-21wc23:10:240
393399139,2cyclictest28630-21wc22:45:280
393399134,5cyclictest141rcu_preempt22:28:390
393399133,2cyclictest141rcu_preempt23:16:360
393399132,1cyclictest141rcu_preempt20:25:110
393399131,9cyclictest25582-21apt-get19:55:140
3933991311,1cyclictest13-21ksoftirqd/021:45:180
3933991310,2cyclictest9042-21wc21:55:190
3933991310,2cyclictest7322-21cut19:15:170
3933991310,2cyclictest29666-21grep20:00:260
3933991310,2cyclictest14688-21wc21:40:000
3933991310,2cyclictest13193-21tail20:35:150
393399130,5cyclictest13-21ksoftirqd/000:10:110
393399130,1cyclictest0-21swapper/022:41:420
394399129,2cyclictest6136-21grep20:20:163
394399129,2cyclictest5641-21grep23:30:183
394399129,2cyclictest32343-21ssh00:06:063
394399129,2cyclictest31368-21grep20:05:173
394399129,2cyclictest27076-21cut22:45:153
394399129,2cyclictest24532-21grep00:20:253
394399129,2cyclictest22770-21grep21:25:153
394399129,2cyclictest22088-21cat23:20:233
394399129,2cyclictest20263-21sed22:40:273
394399129,2cyclictest14339-21cut19:30:153
394399129,2cyclictest14119-21tail20:35:253
394399128,3cyclictest9283-21seq21:35:523
394399128,3cyclictest8649-21cron19:20:013
394399128,3cyclictest30882-21grep21:10:193
394399128,3cyclictest21493-21ssh23:59:323
394399123,3cyclictest21815-21apt-key20:55:003
394399123,3cyclictest21815-21apt-key20:55:003
394399122,5cyclictest9644-21cut22:15:173
3943991212,0cyclictest0-21swapper/321:40:463
394199129,2cyclictest8697-21grep21:35:272
394199129,2cyclictest27738-21cron20:00:002
394199129,2cyclictest13698-21grep20:35:202
394199122,7cyclictest0-21swapper/221:22:552
3941991211,1cyclictest0-21swapper/222:12:402
3941991210,1cyclictest18643-21ssh21:41:392
393699129,2cyclictest8205-21tr21:55:141
393699129,2cyclictest8016-21grep20:25:131
393699129,2cyclictest29740-21ssh23:05:281
393699129,2cyclictest23283-21perf00:40:001
393699129,2cyclictest23189-21wc20:55:191
393699129,2cyclictest17715-21grep19:35:241
393699128,3cyclictest13521-21ntpq23:15:211
393699127,2cyclictest7363-21ssh22:52:291
393699120,1cyclictest0-21swapper/121:49:191
393399129,2cyclictest27197-21tr21:05:150
393399129,2cyclictest25123-21df22:05:150
393399129,2cyclictest19163-21grep19:40:170
393399129,2cyclictest12354-21tr22:55:210
393399129,2cyclictest11386-21grep19:25:130
393399128,3cyclictest15005-21sort23:55:160
393399123,2cyclictest141rcu_preempt21:17:060
393399122,4cyclictest13-21ksoftirqd/023:49:080
3933991211,1cyclictest161rcuc/022:04:070
3933991210,1cyclictest3548-21sort20:15:150
394399118,2cyclictest9871-21ssh00:12:103
394399118,2cyclictest9009-21wget19:20:133
394399118,2cyclictest8607-21ssh21:16:153
394399118,2cyclictest8274-21ssh22:53:133
394399118,2cyclictest6719-21ssh23:11:073
394399118,2cyclictest4869-21grep19:10:173
394399118,2cyclictest3858-21ssh21:52:243
394399118,2cyclictest30819-21taskset00:25:073
394399118,2cyclictest28352-21wc21:05:243
394399118,2cyclictest2780-21chrt22:10:373
394399118,2cyclictest27507-21taskset19:57:083
394399118,2cyclictest26700-21cron21:05:003
394399118,2cyclictest22160-21mailstats19:45:223
394399118,2cyclictest21351-21sh23:40:053
394399118,2cyclictest19661-21ssh22:01:323
394399118,2cyclictest19090-21cut19:40:173
394399118,2cyclictest18753-21wc20:45:253
394399118,2cyclictest12044-21ssh00:32:473
394399118,2cyclictest11785-21grep20:30:283
394399118,2cyclictest11280-21taskset19:25:083
394399118,2cyclictest10823-21ssh22:55:023
394399113,4cyclictest225-21systemd-journal20:26:433
394399111,5cyclictest0-21swapper/321:20:143
394399110,10cyclictest12874-21ssh23:35:013
394199119,1cyclictest24545-21grep00:20:252
394199119,1cyclictest12070-21grep19:25:172
394199118,2cyclictest9760-21basename19:20:162
394199118,2cyclictest9139-21rm00:30:322
394199118,2cyclictest8702-21ssh21:16:192
394199118,2cyclictest6079-21cat23:30:202
394199118,2cyclictest4942-21ssh22:50:342
394199118,2cyclictest4568-21sh22:31:142
394199118,2cyclictest4437-21grep19:10:142
394199118,2cyclictest31531-21ssh21:50:022
394199118,2cyclictest26559-21ssh21:46:202
394199118,2cyclictest25822-21grep21:00:222
394199118,2cyclictest24372-21ssh23:41:042
394199118,2cyclictest20844-21ls23:00:212
394199118,2cyclictest15809-21awk19:35:002
394199118,2cyclictest15570-21cat00:15:222
394199116,2cyclictest452-21systemd-logind00:08:162
3941991110,1cyclictest0-21swapper/223:11:462
393699119,1cyclictest13329-21if_enp0s31f620:35:171
393699118,2cyclictest8675-21ssh22:34:271
393699118,2cyclictest8494-21sh23:31:441
393699118,2cyclictest7272-21head19:15:171
393699118,2cyclictest6768-21head23:50:191
393699118,2cyclictest6623-21rm21:15:201
393699118,2cyclictest32268-21cat00:25:221
393699118,2cyclictest31124-21taskset22:47:141
393699118,2cyclictest29136-21sh21:10:001
393699118,2cyclictest29009-21ssh23:25:141
393699118,2cyclictest24902-21head19:50:251
393699118,2cyclictest19837-21expr23:00:141
393699118,2cyclictest17286-21cut22:20:131
393699118,2cyclictest16259-21wc21:40:191
393699118,2cyclictest16133-21ssh22:38:391
393699118,2cyclictest15971-21head20:40:171
393699118,2cyclictest10900-21ssh00:31:541
393699117,3cyclictest30497-21ssh23:45:151
393399118,2cyclictest5780-21grep19:10:260
393399118,2cyclictest30604-21wc23:25:260
393399118,2cyclictest26975-21ssh23:43:070
393399118,2cyclictest25382-21cut21:00:190
393399118,2cyclictest20601-21ssh22:21:310
393399118,2cyclictest14678-21wc19:30:190
393399116,3cyclictest13-21ksoftirqd/020:10:290
393399110,3cyclictest19376-21idleruntime-cro20:50:000
394399108,1cyclictest24953-21taskset19:50:343
394399107,2cyclictest22908-21grep20:55:173
394399107,2cyclictest16632-21taskset20:40:413
394199108,2cyclictest24202-21sed19:50:182
394199108,1cyclictest7726-21apache_processe20:25:112
394199108,1cyclictest5353-21apache_processe20:20:122
394199108,1cyclictest31740-21ntpq20:05:202
394199108,1cyclictest22938-21grep20:55:172
394199108,1cyclictest17131-21grep22:00:172
394199108,1cyclictest10866-21ssh22:55:052
394199107,2cyclictest3414-21expr20:15:142
394199107,2cyclictest27680-21grep21:05:172
394199100,9cyclictest10494-21expr20:30:162
394199100,1cyclictest141rcu_preempt23:25:192
393699108,2cyclictest26482-21interrupts19:55:191
393699108,1cyclictest3612-21ssh23:10:041
393699108,1cyclictest25018-21expr21:00:151
393699108,1cyclictest203652sleep119:42:341
393699108,1cyclictest18059-21ls20:45:181
393699108,1cyclictest15650-21taskset19:33:151
393699108,1cyclictest11725-21expr19:25:151
393699107,2cyclictest847-21expr20:10:181
393699107,2cyclictest30922-21cat20:05:141
393699107,2cyclictest28470-21expr20:00:141
393699107,2cyclictest21067-21tr19:45:151
393399108,1cyclictest5379-21cut20:20:130
393399108,1cyclictest326072sleep020:08:360
393399107,2cyclictest22457-21sed19:45:260
393399103,1cyclictest141rcu_preempt20:35:000
39419997,1cyclictest18894-21cat19:40:152
39419996,2cyclictest1741-21taskset20:10:232
39369997,1cyclictest4115-21ntpq20:15:201
39339997,1cyclictest25114-21taskset19:52:350
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional