You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-08 - 04:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Aug 08, 2026 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6702210087,9sleep00-21swapper/019:07:290
686529380,9sleep20-21swapper/219:09:372
679128265,13sleep10-21swapper/119:08:401
680028169,8sleep30-21swapper/319:08:463
278662740,0sleep10-21swapper/123:41:431
6987992927,2cyclictest0-21swapper/223:07:332
698599250,4cyclictest15747-21unixbench_singl22:55:251
698799240,23cyclictest0-21swapper/221:14:362
698799240,22cyclictest0-21swapper/221:56:342
698799230,22cyclictest0-21swapper/221:42:462
232232230,0sleep20-21swapper/223:20:112
698799220,20cyclictest0-21swapper/221:48:002
698599220,20cyclictest0-21swapper/121:41:541
698799210,17cyclictest0-21swapper/222:22:332
698599210,21cyclictest0-21swapper/123:10:121
698799200,2cyclictest0-21swapper/221:37:202
698799200,20cyclictest0-21swapper/222:05:542
698599200,6cyclictest0-21swapper/123:04:401
698799193,14cyclictest16540-21awk19:30:002
6987991912,6cyclictest0-21swapper/222:27:332
698799190,1cyclictest0-21swapper/222:30:032
698799190,19cyclictest0-21swapper/221:50:212
698599190,3cyclictest0-21swapper/122:37:061
698599190,1cyclictest0-21swapper/121:53:031
698199190,2cyclictest0-21swapper/023:42:550
164392190,0sleep20-21swapper/222:55:552
13512190,0sleep30-21swapper/300:05:013
699199182,8cyclictest0-21swapper/322:15:133
6991991815,2cyclictest39-21ksoftirqd/321:40:133
699199181,3cyclictest141rcu_preempt21:10:123
699199180,18cyclictest0-21swapper/321:50:593
698799180,2cyclictest0-21swapper/223:03:542
698799180,1cyclictest0-21swapper/222:14:032
698799180,18cyclictest0-21swapper/222:16:282
6985991817,1cyclictest38150irq/131-snd_hda23:20:401
698599181,16cyclictest32034-21diskmemload21:18:521
6981991817,1cyclictest0-21swapper/023:19:560
6981991817,1cyclictest0-21swapper/021:44:310
698199180,18cyclictest0-21swapper/021:24:390
698199180,15cyclictest0-21swapper/022:40:090
6991991715,2cyclictest39-21ksoftirqd/322:06:523
6991991715,1cyclictest0-21swapper/323:58:173
6991991711,3cyclictest39-21ksoftirqd/300:04:493
699199170,16cyclictest0-21swapper/323:22:583
6987991716,1cyclictest0-21swapper/223:15:432
698799170,1cyclictest0-21swapper/222:38:202
698799170,1cyclictest0-21swapper/200:20:352
698799170,16cyclictest0-21swapper/223:39:262
6985991715,1cyclictest0-21swapper/122:40:371
698599171,15cyclictest32034-21diskmemload22:08:461
698599170,2cyclictest0-21swapper/122:15:001
698199170,17cyclictest0-21swapper/021:50:480
699199161,4cyclictest141rcu_preempt00:29:143
6991991614,2cyclictest39-21ksoftirqd/319:45:243
699199160,1cyclictest0-21swapper/300:35:193
698799162,13cyclictest1235-21ntpd22:40:192
6987991615,1cyclictest0-21swapper/222:50:532
698799160,1cyclictest0-21swapper/223:55:492
698799160,1cyclictest0-21swapper/221:30:202
698799160,1cyclictest0-21swapper/221:30:192
6985991615,0cyclictest0-21swapper/100:11:281
698199160,16cyclictest0-21swapper/023:27:350
699199156,3cyclictest32034-21diskmemload23:45:253
699199155,2cyclictest141rcu_preempt22:27:043
699199153,11cyclictest9642-21/usr/sbin/munin23:50:193
699199153,10cyclictest10028-21sensors21:15:233
6991991513,1cyclictest39-21ksoftirqd/320:55:143
699199151,14cyclictest0-21swapper/323:44:303
6987991515,0cyclictest0-21swapper/221:16:342
698599151,14cyclictest0-21swapper/122:53:391
698599150,14cyclictest0-21swapper/122:10:301
698199159,5cyclictest1606-21apt-get00:05:110
698199150,2cyclictest0-21swapper/021:19:040
698199150,15cyclictest0-21swapper/021:11:170
699199148,2cyclictest39-21ksoftirqd/323:04:063
699199143,9cyclictest4503-21kworker/3:220:40:143
6991991413,1cyclictest0-21swapper/321:20:113
6991991411,1cyclictest141rcu_preempt22:34:533
699199140,1cyclictest0-21swapper/323:37:583
699199140,1cyclictest0-21swapper/321:49:263
698799142,10cyclictest0-21swapper/200:15:132
6987991410,3cyclictest3607-21rm00:05:262
6987991410,3cyclictest17652-21tail21:20:182
698799140,4cyclictest0-21swapper/223:44:182
6985991410,3cyclictest12939-21cron20:30:001
698599140,14cyclictest0-21swapper/121:59:241
698599140,13cyclictest7962-21sed19:10:161
698199147,6cyclictest2897-21munin-run22:09:590
698199145,6cyclictest8447-21apache200:30:110
698199144,2cyclictest0-21swapper/022:19:030
6981991410,3cyclictest22956-21grep23:00:170
698199140,14cyclictest0-21swapper/021:30:500
698199140,14cyclictest0-21swapper/021:30:490
698199140,14cyclictest0-21swapper/000:15:130
699199139,3cyclictest19397-21seq23:17:253
699199139,3cyclictest15815-21tr20:35:163
699199139,3cyclictest10541-21grep00:10:143
699199135,4cyclictest225-21systemd-journal19:30:003
699199133,4cyclictest39-21ksoftirqd/300:20:193
699199131,1cyclictest141rcu_preempt22:45:253
6991991311,2cyclictest39-21ksoftirqd/300:15:363
6991991310,2cyclictest39-21ksoftirqd/323:06:183
6991991310,2cyclictest39-21ksoftirqd/321:05:003
6991991310,2cyclictest29367-21grep19:55:173
699199130,12cyclictest11055-21tr20:25:153
698799139,3cyclictest26065-21ls19:50:012
698799131,2cyclictest0-21swapper/223:13:022
6987991310,2cyclictest7248-21grep19:10:122
6987991310,2cyclictest26917-21cut19:50:162
6987991310,2cyclictest18902-21cron19:35:012
6987991310,2cyclictest14825-21grep20:30:262
698799130,1cyclictest0-21swapper/200:12:332
698799130,13cyclictest0-21swapper/221:28:342
698799130,12cyclictest0-21swapper/222:46:572
698599139,3cyclictest26292-21cat21:25:181
698599132,3cyclictest225-21systemd-journal20:00:001
698599131,8cyclictest225-21systemd-journal00:30:001
698599131,11cyclictest0-21swapper/122:25:161
6985991310,2cyclictest27944-21cron22:25:001
6985991310,1cyclictest29668-21ls21:05:001
698599130,13cyclictest0-21swapper/123:15:291
698599130,12cyclictest0-21swapper/100:17:571
698599130,11cyclictest0-21swapper/100:36:081
698199135,2cyclictest20626-21awk22:40:010
698199132,3cyclictest225-21systemd-journal21:50:000
698199131,1cyclictest141rcu_preempt22:28:410
6981991310,2cyclictest32647-21ssh21:29:440
6981991310,2cyclictest29290-21grep19:55:150
6981991310,2cyclictest24891-21latency_hist20:54:590
6981991310,2cyclictest24670-21grep19:45:170
6981991310,2cyclictest14098-21tr22:55:130
6981991310,2cyclictest11777-21cat21:55:190
698199130,11cyclictest0-21swapper/023:50:260
699199129,2cyclictest7208-21sed23:10:163
699199129,2cyclictest6649-21ssh22:50:183
699199129,2cyclictest21445-21ssh22:40:133
699199129,2cyclictest11799-21grep00:30:163
699199129,2cyclictest10535-21basename20:24:593
699199128,3cyclictest5322-21sed20:10:263
699199126,3cyclictest1078-21apt-get21:30:133
699199126,3cyclictest1078-21apt-get21:30:123
6991991211,1cyclictest39-21ksoftirqd/321:28:473
6991991211,1cyclictest381rcuc/321:35:043
6991991210,1cyclictest0-21swapper/322:58:033
698799129,2cyclictest8146-21basename20:20:002
698799129,2cyclictest7015-21cron23:30:002
698799129,2cyclictest4909-21sed00:25:242
698799129,2cyclictest31136-21sed21:05:212
698799129,2cyclictest27912-21sort00:40:012
698799129,2cyclictest26797-21grep20:55:262
698799129,2cyclictest26544-21cut00:00:202
698799129,2cyclictest24438-21sed20:50:262
698799129,2cyclictest21278-21sh19:40:012
698799129,2cyclictest21009-21sed22:00:262
698799129,2cyclictest1051-21cat20:05:012
698799126,2cyclictest225-21systemd-journal19:45:002
698599129,2cyclictest7685-21tail20:15:281
698599129,2cyclictest5320-21grep20:10:251
698599129,2cyclictest25898-21sed20:55:181
698599129,2cyclictest25053-21cut19:45:211
698599129,2cyclictest23686-21munin-plugin-st19:45:001
698599129,2cyclictest20653-21cat19:35:261
698599129,2cyclictest1977-21head20:05:171
698599129,2cyclictest19442-21wc20:40:251
698599129,2cyclictest16752-21head23:35:161
698599129,2cyclictest10334-21head19:15:171
698599128,3cyclictest38150irq/131-snd_hda00:06:301
698599123,7cyclictest9381-21/usr/sbin/munin21:35:161
698599122,1cyclictest141rcu_preempt21:22:341
698199129,2cyclictest6266-21tr20:15:170
698199129,2cyclictest2243-21cat20:05:190
698199129,2cyclictest21135-21grep20:45:160
698199129,2cyclictest21109-21grep22:00:260
698199129,2cyclictest19399-21tr19:35:150
698199129,2cyclictest12817-21sed19:20:170
698199129,2cyclictest10704-21uname19:15:200
698199129,2cyclictest10536-21tr00:10:150
698199128,3cyclictest28419-21cron19:55:010
698199128,3cyclictest22779-21ls19:40:180
698199128,2cyclictest7453-21ssh00:27:190
698199125,6cyclictest0-21swapper/022:10:230
698199124,4cyclictest225-21systemd-journal00:40:010
6981991211,1cyclictest0-21swapper/022:48:380
699199118,2cyclictest32031-21head20:00:163
699199118,2cyclictest31028-21uname21:05:203
699199118,2cyclictest22807-21tr20:50:133
699199118,2cyclictest22468-21ssh22:01:273
699199118,2cyclictest21761-21run-parts20:45:223
699199118,2cyclictest2005-21sed20:05:173
699199118,2cyclictest16047-21ssh22:36:263
699199118,2cyclictest14764-21ssh21:56:593
699199118,2cyclictest13752-21grep19:20:273
699199115,5cyclictest381rcuc/322:13:153
699199113,4cyclictest39-21ksoftirqd/323:31:593
699199113,4cyclictest39-21ksoftirqd/323:25:173
699199112,6cyclictest39-21ksoftirqd/322:21:133
698799118,2cyclictest8701-21cstates20:20:142
698799118,2cyclictest32624-21ssh23:45:142
698799118,2cyclictest28295-21head21:00:172
698799118,2cyclictest20683-21tr20:45:142
698799118,2cyclictest18305-21cut20:40:152
698799118,2cyclictest16850-21wc20:35:222
698799118,2cyclictest14050-21ssh00:30:512
698799118,2cyclictest11616-21ssh23:51:182
698799113,3cyclictest0-21swapper/219:20:142
698799110,4cyclictest3086-21sshd20:06:342
698799110,0cyclictest0-21swapper/223:30:132
698599119,1cyclictest1734-21grep21:50:001
698599118,2cyclictest7311-21ssh21:14:311
698599118,2cyclictest32654-21ssh23:45:151
698599118,2cyclictest32028-21sh21:10:011
698599118,2cyclictest31542-21expr20:00:131
698599118,2cyclictest31317-21sh00:03:261
698599118,2cyclictest29214-21ssh22:45:011
698599118,2cyclictest29025-21grep00:20:271
698599118,2cyclictest26763-21ssh22:04:511
698599118,2cyclictest2572-21ssh21:30:221
698599118,2cyclictest2572-21ssh21:30:221
698599118,2cyclictest24886-21turbostat.cron20:54:591
698599118,2cyclictest22841-21ssh23:58:221
698599118,2cyclictest17506-21grep19:30:161
698599118,2cyclictest15144-21ssh23:34:501
698599118,2cyclictest11531-21ssh22:34:281
698599117,3cyclictest14662-21wc20:30:251
698599117,3cyclictest10063-21grep20:20:261
698599110,8cyclictest32034-21diskmemload23:05:121
698199119,1cyclictest2681-21cat00:25:010
698199118,2cyclictest9621-21grep21:35:150
698199118,2cyclictest8967-21ssh22:51:390
698199118,2cyclictest8726-21grep23:30:190
698199118,2cyclictest561-21ssh23:05:510
698199118,2cyclictest4564-21uname20:10:180
698199118,2cyclictest27611-21ssh00:00:330
698199118,2cyclictest26564-21cat20:55:260
698199118,2cyclictest2635-21ssh23:45:480
698199118,2cyclictest23052-21ssh23:20:050
698199118,2cyclictest23035-21sh23:58:310
698199118,2cyclictest20175-21ssh22:20:140
698199118,2cyclictest15754-21ssh23:35:070
698199118,2cyclictest10127-21sed20:20:260
698199110,4cyclictest0-21swapper/022:30:460
699199108,1cyclictest22382-21basename19:40:143
699199107,2cyclictest13215-21apache_processe20:30:123
699199107,2cyclictest11074-21cat19:15:273
699199100,9cyclictest8350-21ntpq19:10:193
699199100,9cyclictest79092sleep320:17:243
698799108,1cyclictest54392sleep220:11:192
698799108,1cyclictest12031-21pluginstate20:25:222
698799108,1cyclictest114772sleep219:17:132
698599108,1cyclictest17179-21cat20:35:261
698599108,1cyclictest15133-21grep19:25:161
698599107,2cyclictest8639-21ssh23:50:131
698599107,2cyclictest27067-21sed19:50:171
698599107,2cyclictest12363-21sed00:30:181
698599105,3cyclictest1142-21runrttasks23:25:191
698599100,9cyclictest12498-21expr19:20:151
698199108,1cyclictest930-21taskset20:04:330
698199108,1cyclictest188482sleep020:40:180
698199108,1cyclictest17216-21cat20:35:260
698199108,1cyclictest15278-21chrt20:34:330
698199107,2cyclictest6360-21rm23:10:100
698199107,2cyclictest32049-21basename21:10:010
698199102,4cyclictest225-21systemd-journal19:35:000
69919998,1cyclictest18051-21proc_pri19:30:213
69919997,1cyclictest19416-21grep19:35:143
69919996,2cyclictest26456-21grep19:50:143
69879997,1cyclictest29325-21cut19:55:152
69859995,3cyclictest20677-21tr20:45:141
69819998,1cyclictest8063-21cat19:10:170
69819997,1cyclictest29089-21ls21:00:260
69819997,1cyclictest15119-21tune2fs19:25:160
69819986,1cyclictest11246-21expr20:25:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional