You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-17 - 08:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Fri Jul 17, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
663629566,9sleep00-21swapper/019:06:210
660828977,8sleep20-21swapper/219:05:592
674828769,14sleep10-21swapper/119:07:481
677028471,9sleep30-21swapper/319:08:053
175612310,0sleep30-21swapper/322:17:303
699699230,0cyclictest0-21swapper/100:06:441
699699210,18cyclictest0-21swapper/122:40:151
7000992015,3cyclictest291irq_work/200:05:152
699699200,1cyclictest0-21swapper/121:36:411
699699200,19cyclictest0-21swapper/122:25:141
7000991916,1cyclictest0-21swapper/221:19:302
7000991916,1cyclictest0-21swapper/221:19:302
699699191,1cyclictest12066-21sed23:50:261
699699190,1cyclictest0-21swapper/119:35:121
699699190,18cyclictest0-21swapper/122:05:271
699599196,7cyclictest20799-21idleruntime-cro22:20:010
700199180,18cyclictest27044-21kworker/3:000:35:323
7000991818,0cyclictest0-21swapper/223:40:332
700099180,2cyclictest0-21swapper/222:54:422
700099180,17cyclictest0-21swapper/223:34:142
6996991817,1cyclictest0-21swapper/123:26:111
699599186,6cyclictest25588-21munin-run21:25:000
7001991716,1cyclictest27044-21kworker/3:023:10:123
7001991716,1cyclictest27044-21kworker/3:023:09:223
7001991716,1cyclictest27044-21kworker/3:021:34:523
700099170,16cyclictest0-21swapper/200:12:092
699699178,6cyclictest12892-21apache222:48:581
699699173,14cyclictest404-21diskmemload00:24:071
6996991717,0cyclictest0-21swapper/100:18:071
6996991716,1cyclictest0-21swapper/122:57:361
6996991715,1cyclictest7261-21grep22:50:151
699699170,17cyclictest0-21swapper/123:41:201
6995991714,2cyclictest94-21kswapd021:39:270
700199164,10cyclictest18910-21cut23:55:143
700199162,13cyclictest0-21swapper/323:50:053
700099160,1cyclictest0-21swapper/223:14:242
6995991616,0cyclictest0-21swapper/021:57:300
699599161,12cyclictest18805-21/usr/sbin/munin23:35:210
69262160,0sleep10-21swapper/122:10:461
700199156,3cyclictest225-21systemd-journal23:22:113
700199153,1cyclictest15880-21ssh22:35:253
7001991514,0cyclictest0-21swapper/300:25:313
7001991514,0cyclictest0-21swapper/300:25:303
7001991512,1cyclictest55650irq/127-enp2s0-00:02:483
700199150,3cyclictest0-21swapper/321:43:243
700199150,1cyclictest0-21swapper/323:30:493
700199150,0cyclictest0-21swapper/323:47:283
700099157,8cyclictest12887-21apache223:05:522
7000991510,3cyclictest22920-21sh22:20:242
700099150,5cyclictest0-21swapper/222:19:252
700099150,3cyclictest0-21swapper/200:32:282
700099150,1cyclictest141rcu_preempt21:35:002
699699157,8cyclictest38150irq/131-snd_hda00:25:251
699699157,8cyclictest38150irq/131-snd_hda00:25:251
6996991513,2cyclictest0-21swapper/123:12:521
6996991513,1cyclictest0-21swapper/121:40:131
6996991511,2cyclictest25-21ksoftirqd/123:02:241
6995991512,2cyclictest94-21kswapd000:35:020
699599150,2cyclictest5533-21taskset22:49:180
699599150,1cyclictest0-21swapper/000:28:320
699599150,1cyclictest0-21swapper/000:28:320
7001991411,2cyclictest13817-21sh20:30:003
700199141,0cyclictest0-21swapper/321:54:133
7001991410,2cyclictest32192-21grep20:00:143
7000991413,1cyclictest0-21swapper/221:55:432
7000991413,1cyclictest0-21swapper/221:37:072
7000991410,3cyclictest26700-21grep20:55:172
700099140,3cyclictest0-21swapper/200:39:342
699699149,2cyclictest21110-21ls22:00:201
699699143,2cyclictest11944-21/usr/sbin/munin00:10:151
6996991410,3cyclictest8988-21cron20:20:001
699599149,4cyclictest404-21diskmemload21:28:090
699599149,3cyclictest10736-21grep21:15:240
699599149,3cyclictest10736-21grep21:15:230
699599147,6cyclictest7550-21munin-run23:10:000
699599144,6cyclictest3146-21/usr/sbin/munin21:30:310
700199134,4cyclictest404-21diskmemload23:19:493
700199132,6cyclictest7482-21wget19:10:133
7001991311,2cyclictest1736-21kworker/u8:100:24:503
7001991310,2cyclictest7610-21grep20:15:193
7001991310,2cyclictest6619-21grep20:15:003
7001991310,2cyclictest31603-21wc20:00:013
7001991310,2cyclictest21935-21grep20:45:173
7001991310,2cyclictest19167-21grep21:20:233
7001991310,2cyclictest14335-21ntpq00:30:213
700199130,1cyclictest0-21swapper/322:00:223
700199130,12cyclictest13401-21wc21:55:263
700199130,10cyclictest0-21swapper/322:49:433
700099139,3cyclictest31470-21run-parts20:00:002
700099139,2cyclictest15091-21ssh22:35:172
700099135,7cyclictest28097-21systemctl00:20:002
7000991310,2cyclictest4005-21sed21:50:172
7000991310,2cyclictest26738-21turbostat.cron00:00:002
7000991310,2cyclictest20752-21sed21:40:262
7000991310,2cyclictest18874-21cat21:20:212
700099130,12cyclictest0-21swapper/222:30:282
699699139,3cyclictest12702-21tr19:20:121
699699139,3cyclictest12702-21tr19:20:121
699699139,2cyclictest20940-21cat22:20:001
699699136,6cyclictest0-21swapper/121:20:141
699699135,6cyclictest38150irq/131-snd_hda21:31:181
6996991310,2cyclictest24830-21tr19:45:161
6996991310,2cyclictest17329-21sed23:15:171
699699130,13cyclictest0-21swapper/121:45:361
699699130,13cyclictest0-21swapper/121:27:441
699699130,12cyclictest27261-21tr19:50:141
699599139,3cyclictest28155-21cut00:00:190
699599139,3cyclictest2801-21sed23:45:170
699599138,3cyclictest11966-21dpkg19:20:000
699599131,6cyclictest8821-21idleruntime-cro21:15:000
6995991312,1cyclictest13-21ksoftirqd/023:44:130
6995991310,2cyclictest3967-21apt-config22:10:000
6995991310,2cyclictest23290-21cat23:00:000
699599130,12cyclictest5252-21sed20:10:170
700199129,2cyclictest6246-21cat22:10:273
700199129,2cyclictest31236-21df21:05:163
700199129,2cyclictest29855-21wc21:00:253
700199129,2cyclictest27125-21ssh21:25:193
700199129,2cyclictest20115-21wc20:40:213
700199129,2cyclictest18227-21head19:30:163
700199129,2cyclictest17778-21cut23:35:143
700199129,2cyclictest15739-21tail20:30:283
700199129,2cyclictest11938-21sh19:20:013
700199128,3cyclictest13613-21wc00:10:273
700199128,3cyclictest1114-21chrt21:48:493
700199128,2cyclictest5178-21ssh00:05:303
700199125,3cyclictest28536-21sshd22:43:293
700199120,0cyclictest0-21swapper/321:13:203
700099129,2cyclictest8282sleep223:25:132
700099129,2cyclictest5317-21grep00:25:172
700099129,2cyclictest5317-21grep00:25:162
700099129,2cyclictest29646-21rm22:05:212
700099129,2cyclictest28560-21ntpq21:45:242
700099129,2cyclictest24029-21sed22:40:232
700099129,2cyclictest21452-21grep20:45:152
700099129,2cyclictest14309-21tr20:30:152
700099129,2cyclictest11063-21grep23:50:172
700099129,2cyclictest10599-21sed19:15:172
700099128,3cyclictest487-21avahi-daemon22:02:312
700099124,3cyclictest21957-21munin-run19:40:002
700099120,1cyclictest0-21swapper/221:00:122
700099120,11cyclictest9906-21sort20:20:172
699699129,2cyclictest8125-21sort19:10:161
699699129,2cyclictest5628-21ntpq20:10:201
699699129,2cyclictest2843-21tail21:10:251
699699129,2cyclictest22141-21grep22:20:171
699699129,2cyclictest15375-21grep19:25:141
699699129,2cyclictest15345-21wc20:30:221
699699129,2cyclictest1476-21chrt00:04:041
699699129,2cyclictest13410-21rm00:30:141
699699128,2cyclictest26252-21grep20:55:141
699699127,2cyclictest10544-21ssh21:54:371
699699126,5cyclictest25404-21df_abs23:20:141
699699123,8cyclictest12892-21apache223:32:001
6996991210,1cyclictest90532sleep123:49:241
699599129,2cyclictest8767-21grep23:10:170
699599129,2cyclictest5301-21rm00:05:310
699599129,2cyclictest4777-21grep22:10:140
699599129,2cyclictest28876-21tr00:20:140
699599129,2cyclictest21057-21head00:15:190
699599129,2cyclictest19438-21ssh23:55:150
699599129,2cyclictest16592-21basename20:35:150
699599128,3cyclictest3781-21grep20:05:260
699599128,3cyclictest24334-21sleep23:20:000
699599128,2cyclictest32559-21sort20:00:160
699599128,2cyclictest18763-21ps19:30:210
699599128,2cyclictest14541-21rm00:30:230
6995991210,1cyclictest116362sleep000:10:110
699599120,12cyclictest0-21swapper/023:26:030
699599120,10cyclictest0-21swapper/022:02:080
700199118,2cyclictest9740-21expr20:20:153
700199118,2cyclictest9111-21sh21:15:043
700199118,2cyclictest9111-21sh21:15:043
700199118,2cyclictest7128-21tr22:50:163
700199118,2cyclictest3491-21ssh23:25:573
700199118,2cyclictest30537-21head22:25:193
700199118,2cyclictest28823-21cat22:05:163
700199118,2cyclictest28676-21grep19:50:273
700199118,2cyclictest27075-21ssh23:40:193
700199118,2cyclictest25543-21chrt20:52:593
700199118,2cyclictest24979-21ssh23:00:213
700199118,2cyclictest21960-21awk19:40:003
700199118,2cyclictest21234-21chrt00:15:193
700199118,2cyclictest21047-21sh22:20:013
700199118,2cyclictest18473-21ssh22:56:213
700199118,2cyclictest18308-21taskset20:36:473
700199118,2cyclictest12099-21sh22:33:563
700199110,1cyclictest0-21swapper/321:36:213
700099119,1cyclictest19075-21sed19:30:252
700099118,2cyclictest9505-21chrt19:14:292
700099118,2cyclictest8373-21cat20:15:282
700099118,2cyclictest6702-21cut20:15:002
700099118,2cyclictest5425-21ssh21:12:222
700099118,2cyclictest4859-21ssh23:46:092
700099118,2cyclictest32134-21ssh22:25:492
700099118,2cyclictest31697-21ssh00:21:172
700099118,2cyclictest28991-21taskset19:53:552
700099118,2cyclictest28260-21ssh23:21:312
700099118,2cyclictest28100-21ssh23:02:172
700099118,2cyclictest25875-21ssh21:25:042
700099118,2cyclictest24866-21tr19:45:162
700099118,2cyclictest21568-21ssh22:58:462
700099118,2cyclictest20333-21ssh23:16:542
700099118,2cyclictest12794-21tr20:25:212
700099115,5cyclictest17564-21/usr/sbin/munin23:35:202
700099115,4cyclictest24352-21sed20:50:172
700099114,3cyclictest28443-21missed_timers00:00:222
700099113,3cyclictest5031-21df_abs22:10:152
700099110,10cyclictest1048-21ssh22:45:482
699699118,2cyclictest26523-21ssh23:59:511
699699118,2cyclictest24032-21taskset23:38:521
699699118,2cyclictest23384-21ntpq19:40:211
699699118,2cyclictest1951-21ssh23:05:391
699699118,2cyclictest17532-21tr19:30:121
699699118,2cyclictest17203-21sleep21:20:001
699699118,2cyclictest17203-21sleep21:20:001
699699118,2cyclictest14265-21ssh22:35:131
699699118,2cyclictest11847-21expr21:55:141
699699118,2cyclictest10960-21tr19:15:201
699699118,2cyclictest0-21swapper/121:05:171
699699115,2cyclictest225-21systemd-journal22:34:551
699699113,2cyclictest225-21systemd-journal00:37:311
699699112,6cyclictest38150irq/131-snd_hda20:35:131
699599119,1cyclictest135542sleep020:26:430
699599118,2cyclictest4335-21sort21:50:200
699599118,2cyclictest29692-21ssh22:25:130
699599118,2cyclictest28450-21basename21:00:130
699599118,2cyclictest27222-21grep19:50:140
699599118,2cyclictest27203-21ssh21:45:120
699599118,2cyclictest26441-21ssh23:20:240
699599118,2cyclictest25574-21sh22:41:110
699599118,2cyclictest22376-21wc19:40:150
699599118,2cyclictest20185-21cut21:40:210
699599118,2cyclictest16301-21ssh22:35:280
699599118,2cyclictest14842-21grep20:30:180
699599118,2cyclictest12953-21ssh23:31:460
699599118,2cyclictest12156-21ssh22:52:590
699599117,3cyclictest7549-21sed22:30:270
699599117,3cyclictest25409-21ssh23:00:260
699599114,4cyclictest10345-21apache_volume23:50:120
699599114,3cyclictest225-21systemd-journal19:49:290
699599110,10cyclictest29142-21sh22:25:000
700199108,1cyclictest14997-21cat19:25:013
700199107,2cyclictest25091-21df_abs19:45:173
700199107,2cyclictest24144-21seq19:42:393
700199107,2cyclictest12471-21grep19:20:113
700199107,2cyclictest12471-21grep19:20:113
700199100,9cyclictest27570-21cat20:55:263
700199100,9cyclictest2093-21taskset20:05:073
700099108,1cyclictest3226-21sed20:05:212
700099108,1cyclictest32058-21gzip21:05:232
700099108,1cyclictest23974-21chrt19:40:292
700099108,1cyclictest20620-21chrt20:40:432
700099108,1cyclictest16935-21chrt19:26:482
700099107,2cyclictest16951-21df20:35:152
700099100,9cyclictest32382-21expr20:00:152
699699108,1cyclictest31192-21chrt19:56:431
699699108,1cyclictest301512sleep121:00:551
699699108,1cyclictest232312sleep120:48:481
699699108,1cyclictest19216-21expr20:40:141
699699108,1cyclictest11317-21seq20:24:161
699699107,2cyclictest638-21uname20:00:211
699699107,2cyclictest12203-21grep20:25:161
699699100,9cyclictest254252sleep120:51:281
699599108,1cyclictest9277-21apache_accesses20:20:130
699599108,1cyclictest31954-21cut21:05:220
699599108,1cyclictest29413-21hostname19:55:140
699599108,1cyclictest26110-21cut20:55:140
699599108,1cyclictest255802sleep020:53:280
699599108,1cyclictest19728-21cat20:40:170
699599108,1cyclictest12952-21cat19:20:140
699599108,1cyclictest12952-21cat19:20:130
699599107,2cyclictest7260-21rm19:10:000
699599100,9cyclictest19921-21cut19:35:130
70009997,1cyclictest13633-21ntpq19:20:192
70009997,1cyclictest13633-21ntpq19:20:192
69969997,1cyclictest0-21swapper/120:05:021
69959997,1cyclictest8456-21grep20:15:290
69959997,1cyclictest21336-21grep20:45:140
69959997,1cyclictest15149-21hostname19:25:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional