You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-29 - 07:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Wed Apr 29, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2190621300,0sleep00-21swapper/000:05:220
2436929850,37sleep20-21swapper/219:06:352
2466628874,9sleep00-21swapper/019:09:550
2434828568,13sleep10-21swapper/119:06:181
2444828471,9sleep30-21swapper/319:07:043
2476699271,5cyclictest17521-21diskmemload23:25:163
2476699261,2cyclictest0-21swapper/322:48:083
2476699253,2cyclictest17521-21diskmemload00:00:173
2476699251,23cyclictest0-21swapper/321:20:133
2476699250,2cyclictest0-21swapper/323:02:223
24756992421,2cyclictest23562-21python00:25:250
2476699231,2cyclictest0-21swapper/322:52:433
2476699230,2cyclictest0-21swapper/321:18:283
2476699223,1cyclictest27811-21apt-get23:50:133
2476699221,4cyclictest4651-21grep23:35:273
2476699221,4cyclictest17521-21diskmemload00:22:173
2476699220,0cyclictest0-21swapper/320:50:133
24763992219,2cyclictest27807-21memory21:15:212
2476399221,20cyclictest0-21swapper/223:49:552
24756992219,2cyclictest4447-21df_inode23:55:160
2476699215,14cyclictest16798-21apt-get23:05:023
24766992119,1cyclictest27176-21df23:30:153
2476699210,21cyclictest0-21swapper/322:05:143
2476699210,1cyclictest791-21fschecks_count22:55:173
2476699210,16cyclictest0-21swapper/319:40:143
24761992121,0cyclictest0-21swapper/123:52:031
24761992120,1cyclictest0-21swapper/122:10:231
2476199210,21cyclictest0-21swapper/123:36:151
2476199210,20cyclictest17552-21kworker/1:022:55:031
2476199210,19cyclictest0-21swapper/100:19:021
2476699207,7cyclictest16739-21sh21:47:323
2476699204,1cyclictest17521-21diskmemload21:35:133
2476699203,1cyclictest17521-21diskmemload21:54:233
2476699202,3cyclictest0-21swapper/322:00:153
2476699200,4cyclictest0-21swapper/323:23:193
2476699200,14cyclictest28997-21sh00:29:193
2476399200,20cyclictest0-21swapper/223:13:042
2476399200,1cyclictest0-21swapper/200:33:062
24761992020,0cyclictest0-21swapper/100:24:361
24761992018,1cyclictest2167-21munin-run23:35:001
24756992017,2cyclictest22799-21/usr/sbin/munin22:30:160
30442190,0chrt0-21swapper/319:30:163
2476699191,12cyclictest14421-21ssh22:25:123
2476699190,5cyclictest3326-21latency_hist20:40:013
24763991919,0cyclictest0-21swapper/223:56:152
2476399190,19cyclictest0-21swapper/200:02:022
2476399190,18cyclictest0-21swapper/223:26:072
24761991917,1cyclictest28347-21munin-run21:55:001
2476199191,18cyclictest16022-21kworker/1:000:06:131
2476199190,19cyclictest0-21swapper/122:09:221
24756991917,1cyclictest2646-21apache_accesses23:35:120
2475699191,14cyclictest0-21swapper/023:45:130
2476699183,1cyclictest28745-21apt-get20:25:143
2476699182,2cyclictest32018-21sh21:56:193
2476699181,1cyclictest22830-21apache_accesses22:30:123
2476699180,1cyclictest0-21swapper/322:24:163
2476699180,16cyclictest0-21swapper/321:40:143
2476399180,18cyclictest0-21swapper/222:29:442
2476199182,1cyclictest17521-21diskmemload21:23:541
24761991816,2cyclictest38150irq/131-snd_hda19:30:131
24761991815,1cyclictest0-21swapper/100:39:061
2476199180,1cyclictest0-21swapper/121:48:311
24756991816,1cyclictest11178-21ntp_states23:20:190
113832180,0sleep20-21swapper/223:01:082
2476699177,6cyclictest504-21taskset20:30:543
2476699177,6cyclictest15091-21taskset21:04:313
2476699173,1cyclictest17521-21diskmemload00:19:263
2476699171,4cyclictest32028-21/usr/sbin/munin00:30:253
2476699171,4cyclictest17521-21diskmemload21:13:383
2476699171,2cyclictest28602-21ssh22:14:173
2476699171,2cyclictest17521-21diskmemload23:56:073
2476699171,2cyclictest14284-21latency_hist19:55:003
2476699170,3cyclictest0-21swapper/300:05:183
24763991715,1cyclictest23871-21irqstats22:30:192
2476399170,17cyclictest0-21swapper/223:41:052
2476399170,17cyclictest0-21swapper/222:02:262
2476399170,17cyclictest0-21swapper/221:24:012
24761991717,0cyclictest0-21swapper/122:00:241
2476199170,16cyclictest0-21swapper/122:26:301
2475699175,6cyclictest14985-21apt-get22:45:000
2475699173,2cyclictest141rcu_preempt22:38:340
24766991613,2cyclictest20371-21irqstats21:30:193
2476699160,15cyclictest0-21swapper/322:36:253
24763991613,2cyclictest16999-21/usr/sbin/munin23:05:172
2476399160,1cyclictest0-21swapper/222:54:502
2476199162,2cyclictest26467-21kworker/1:021:35:531
2476199161,15cyclictest0-21swapper/123:13:051
24756991614,2cyclictest13-21ksoftirqd/000:12:420
24756991614,1cyclictest17525-21munin-run21:10:000
24756991611,3cyclictest31930-21sh19:25:000
2476699158,6cyclictest20780-21open_files23:45:213
2476699158,6cyclictest11734-21munin-plugin-st19:50:003
2476699157,7cyclictest1897-21seq22:17:103
2476699153,2cyclictest8274-21grep00:35:273
2476699153,2cyclictest12144-21memory21:25:203
2476699153,1cyclictest2627-21grep23:15:203
2476699152,3cyclictest4915-21/usr/sbin/munin20:40:263
2476699150,6cyclictest26667-21unixbench_multi19:10:273
2476699150,3cyclictest0-21swapper/323:13:133
24763991514,1cyclictest0-21swapper/200:06:472
2476399150,15cyclictest0-21swapper/221:46:132
2476399150,15cyclictest0-21swapper/200:36:352
2476199158,3cyclictest1436-21sendmail-mta23:05:281
24761991513,1cyclictest17521-21diskmemload21:28:081
24761991511,3cyclictest29257-21grep00:10:141
24761991510,4cyclictest1836-21head20:35:171
2476199150,12cyclictest0-21swapper/120:15:131
2476199150,12cyclictest0-21swapper/120:15:131
2476199150,0cyclictest0-21swapper/122:22:311
2475699155,4cyclictest225-21systemd-journal22:02:390
24756991513,1cyclictest11054-21apache_accesses23:40:130
24756991512,2cyclictest2611-21apache_processe21:20:120
24756991510,1cyclictest13522-21apache_volume00:20:120
2476699148,5cyclictest6318-21grep19:35:283
2476699147,6cyclictest31190-21run-parts19:20:223
2476699147,6cyclictest26366-21apache_volume20:20:133
2476699147,6cyclictest24363-21expr20:15:143
2476699147,6cyclictest24363-21expr20:15:133
2476699147,6cyclictest1514-21vmstat19:25:283
2476699147,6cyclictest14462-21rm23:41:323
2476699147,6cyclictest11526-21cat20:55:203
2476699146,7cyclictest16999-21grep20:00:143
2476399148,5cyclictest6017-21tail00:15:182
2476399145,5cyclictest225-21systemd-journal00:30:002
24763991410,3cyclictest7722-21ssh22:20:252
24763991410,3cyclictest25844-21kworker/u8:122:37:132
24763991410,2cyclictest28147-21grep19:15:172
2476399140,1cyclictest0-21swapper/223:21:542
2476399140,1cyclictest0-21swapper/222:14:302
2476199148,6cyclictest38150irq/131-snd_hda21:58:281
24761991412,1cyclictest31256-21apache_processe22:35:131
24761991412,1cyclictest11168-21ntp_states23:20:181
24761991411,2cyclictest18223-21/usr/sbin/munin23:25:211
24761991410,3cyclictest21095-21ls21:30:271
24761991410,3cyclictest11930-21grep23:40:191
2475699146,7cyclictest862-21munin-run20:35:000
2475699142,6cyclictest3196-21munin-run21:40:000
2475699142,6cyclictest18968-21munin-run20:05:000
24756991411,2cyclictest29624-21ls22:15:000
24756991411,2cyclictest16999-21/usr/sbin/munin23:05:180
2476699138,4cyclictest7223-21cat20:45:233
2476699138,4cyclictest293512sleep319:17:413
2476699136,6cyclictest16368-21chrt19:58:173
2476699132,2cyclictest225-21systemd-journal20:10:003
2476699131,2cyclictest225-21systemd-journal21:10:003
24766991310,2cyclictest29454-21tr00:10:153
2476699130,2cyclictest0-21swapper/322:42:343
2476699130,12cyclictest21859-21grep20:10:153
2476399138,3cyclictest27161-21cron21:35:002
2476399131,0cyclictest0-21swapper/221:11:302
24763991310,2cyclictest8738-21cat20:50:152
24763991310,2cyclictest30958-21head22:15:182
24763991310,2cyclictest28584-21wc20:25:002
24763991310,2cyclictest28567-21grep23:30:262
24763991310,2cyclictest14999-21awk22:45:002
2476399130,13cyclictest0-21swapper/222:47:552
2476399130,12cyclictest21359-21wc22:10:012
2476199139,3cyclictest9805-21wc20:50:261
2476199139,3cyclictest14616-21tr19:55:151
24761991313,0cyclictest0-21swapper/119:50:171
24761991310,2cyclictest7897-21grep22:40:181
24761991310,2cyclictest31933-21idleruntime-cro19:25:001
2476199130,11cyclictest0-21swapper/123:57:301
2475699139,4cyclictest13-21ksoftirqd/000:16:570
2475699139,3cyclictest26755-21tr23:10:220
2475699137,4cyclictest13-21ksoftirqd/019:35:150
2475699135,5cyclictest24290-21apache200:36:100
2475699135,4cyclictest225-21systemd-journal19:45:000
2475699131,1cyclictest141rcu_preempt00:33:060
24756991311,1cyclictest10979-21apache_volume21:25:130
24756991310,2cyclictest9867-21grep20:50:260
24756991310,2cyclictest27478-21head21:15:190
2475699130,8cyclictest5648-21idleruntime-cro20:45:010
2475699130,12cyclictest30518-21sed20:25:260
2476399129,2cyclictest4404-21taskset22:00:002
2476399129,2cyclictest3859-21cut21:40:152
2476399129,2cyclictest3324-21latency_hist23:55:002
2476399129,2cyclictest31950-21awk19:25:002
2476399129,2cyclictest29221-21grep21:35:212
2476399129,2cyclictest25064-21grep19:10:152
2476399129,2cyclictest12699-21head19:50:182
24763991211,1cyclictest0-21swapper/200:21:352
24763991210,1cyclictest0-21swapper/221:50:222
2476199129,2cyclictest9768-21tr23:00:201
2476199129,2cyclictest8836-21grep19:40:271
2476199129,2cyclictest3819-21grep21:40:141
2476199129,2cyclictest3370-21tail23:15:251
2476199129,2cyclictest30857-21cut22:15:181
24761991211,0cyclictest0-21swapper/122:46:131
2476199120,3cyclictest225-21systemd-journal20:50:011
2475699129,2cyclictest8793-21tr23:00:150
2475699129,2cyclictest5578-21run-parts22:20:000
2475699129,2cyclictest4412-21basename22:00:000
2475699129,2cyclictest18343-21tr23:25:150
2475699129,2cyclictest17670-21grep22:45:260
2475699129,2cyclictest12758-21sed00:00:170
2475699128,3cyclictest15142-21ls21:04:590
2475699125,6cyclictest17521-21diskmemload22:50:030
2475699125,5cyclictest17521-21diskmemload22:55:060
2475699121,6cyclictest28493-21awk20:25:000
2475699120,12cyclictest0-21swapper/022:23:300
2476399119,1cyclictest921-21tr22:55:172
2476399119,1cyclictest7878-21head19:40:162
2476399118,2cyclictest8172-21cat20:50:002
2476399118,2cyclictest4553-21sed23:35:272
2476399118,2cyclictest4008-21grep19:30:262
2476399118,2cyclictest21319-21cron20:10:002
2476399118,2cyclictest14294-21ssh21:26:292
2476399118,2cyclictest1125-21ssh00:12:162
2476399115,2cyclictest225-21systemd-journal21:00:002
2476399112,7cyclictest0-21swapper/220:14:342
2476399111,9cyclictest2492-21irqstats23:15:192
2476199118,2cyclictest4405-21kworker/1:119:35:171
2476199118,2cyclictest32273-21grep19:25:151
2476199118,2cyclictest27588-21head20:20:221
2476199118,2cyclictest26520-21ssh23:49:241
2476199118,2cyclictest24198-21cut22:50:141
2476199118,2cyclictest23861-21tail00:25:261
2476199118,2cyclictest20435-21grep20:05:211
2476199118,2cyclictest20335-21ssh21:10:401
2476199118,2cyclictest1331-21ssh00:31:301
2476199118,2cyclictest12750-21apt21:00:001
2476199118,2cyclictest11230-21head19:45:251
24761991110,0cyclictest0-21swapper/121:16:141
2476199110,10cyclictest17176-21ssh00:02:441
2475699118,2cyclictest6470-21ssh23:17:460
2475699118,2cyclictest5547-21grep21:40:270
2475699118,2cyclictest29540-21sh19:20:000
2475699118,2cyclictest29458-21ssh23:50:220
2475699118,2cyclictest28002-21open_files23:30:210
2475699118,2cyclictest2620-21cut19:30:140
2475699118,2cyclictest25132-21grep19:10:160
2475699118,2cyclictest22382-21sed21:50:270
2475699118,2cyclictest21544-21ssh22:29:240
2475699118,2cyclictest2119-21cp19:30:000
2475699118,2cyclictest19975-21head20:05:170
2475699118,2cyclictest19193-21ssh21:30:080
2475699118,2cyclictest18100-21ssh21:10:150
2475699118,2cyclictest15626-21uname19:55:220
2475699118,2cyclictest15385-21ssh22:05:320
2475699118,2cyclictest13387-21python19:50:250
2475699110,10cyclictest14459-21ssh21:45:440
2476399108,2cyclictest14858-21df_abs19:55:152
2476399108,1cyclictest324742sleep220:30:212
2476399108,1cyclictest25409-21wc20:15:242
2476399108,1cyclictest25409-21wc20:15:242
2476399108,1cyclictest14888-21taskset21:01:522
2476399107,2cyclictest566-21chrt19:25:182
2476399107,2cyclictest5207-21awk20:40:272
2476399107,2cyclictest3205-21taskset20:39:482
2476399107,2cyclictest17502-21grep20:00:162
2476399107,2cyclictest16521-21cut21:05:202
2476399106,3cyclictest11526-21systemd19:48:372
2476399100,9cyclictest30586-21taskset20:26:002
2476399100,10cyclictest0-21swapper/219:37:022
2476199108,1cyclictest3631-21awk20:40:151
2476199108,1cyclictest28456-21needreboot19:15:191
2476199108,1cyclictest25254-21ssh22:30:461
2476199108,1cyclictest22518-21cut20:10:191
2476199108,1cyclictest15723-21ls21:05:151
2476199107,2cyclictest31772-21sort20:30:161
2476199107,2cyclictest30434-21grep20:25:261
2476199107,2cyclictest17348-21grep20:00:151
2476199107,2cyclictest14911-21chrt21:02:101
2476199101,5cyclictest225-21systemd-journal19:15:001
2475699108,1cyclictest7220-21which20:45:230
2475699107,2cyclictest21550-21taskset20:10:050
2475699103,3cyclictest225-21systemd-journal19:48:370
2475699100,9cyclictest1380-21cut20:35:140
247569997,1cyclictest24503-21df_abs20:15:140
247569997,1cyclictest24503-21df_abs20:15:140
247569997,1cyclictest105952sleep020:55:040
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional