You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-06 - 22:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sat Jun 06, 2026 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
72142172104,40sleep00-21swapper/019:09:260
69612172103,26sleep30-21swapper/319:06:093
53562171102,27sleep20-21swapper/219:05:072
72052162104,28sleep10-21swapper/119:09:201
726027457,10sleep20-21swapper/219:10:012
193792710,3sleep20-21swapper/223:52:152
26402670,2sleep20-21swapper/200:28:062
284222620,2sleep10-21swapper/123:59:241
139032590,1sleep00-21swapper/000:35:540
173512560,2sleep10-21swapper/121:25:501
237932550,2sleep10-21swapper/123:07:151
44602530,3sleep00-21swapper/022:05:140
263422510,2sleep20-21swapper/221:56:522
210262500,3sleep30-21swapper/300:17:323
210262500,3sleep30-21swapper/300:17:323
34242490,1sleep10-21swapper/121:15:351
7531994123,3cyclictest13602-21ssh22:11:220
171932410,3chrt17192-21ssh23:50:223
7534993716,18cyclictest4705-21users21:40:253
7534993512,6cyclictest34-21ksoftirqd/322:40:123
753499342,13cyclictest28671-21sh21:35:113
753499341,31cyclictest7529-21cyclictest19:40:153
753199331,3cyclictest19336-21/usr/sbin/munin22:15:260
753499322,9cyclictest6795-21ssh22:55:113
753399325,25cyclictest28-21ksoftirqd/219:50:012
753299321,28cyclictest17473-21sh22:15:031
301472320,3chrt30144-21ssh00:24:550
753499318,6cyclictest52950irq/141-enp3s0-00:25:113
753499314,11cyclictest0-21swapper/322:30:273
7534993123,6cyclictest25091-21ssh21:31:513
7534993121,8cyclictest0-21swapper/322:11:313
7534993113,4cyclictest29457-21sh22:47:233
7534993113,4cyclictest29457-21sh22:47:233
7534993111,6cyclictest6309-21which20:20:223
7534993012,3cyclictest10497-21sh21:20:423
7534993011,6cyclictest28031-21swap19:55:233
7534993011,5cyclictest34-21ksoftirqd/320:30:133
7533993023,5cyclictest17595-21/usr/sbin/munin22:15:262
7532993020,8cyclictest8837-21sh21:44:311
753299301,26cyclictest23960-21sed19:45:241
117402300,2sleep2271rcuc/223:46:172
753499292,9cyclictest15218-21ssh21:25:123
753499292,11cyclictest640-21snmpd22:05:133
7532992923,4cyclictest28815-21ssh22:23:091
753199291,3cyclictest30629-21ssh22:24:590
250842290,12sleep228-21ksoftirqd/222:20:172
753499281,9cyclictest141rcu_preempt23:10:113
753499281,3cyclictest0-21swapper/322:15:253
7534992811,5cyclictest12629-21perl22:35:163
753399281,24cyclictest0-21swapper/221:26:382
753299281,25cyclictest29033-21apt-get20:00:131
753299280,3cyclictest14351-21ssh22:12:081
7531992821,5cyclictest20894-21fschecks_time19:40:130
753199281,25cyclictest0-21swapper/019:45:300
7534992722,3cyclictest27746-21ssh22:22:053
753499271,3cyclictest31058-21apt-get22:25:113
7533992722,3cyclictest12721-21ssh22:10:302
7533992715,7cyclictest52750irq/139-enp3s0-22:40:122
753399271,24cyclictest0-21swapper/219:44:112
753299271,25cyclictest0-21swapper/119:53:361
753299271,24cyclictest0-21swapper/122:25:011
753299271,24cyclictest0-21swapper/121:45:071
753199271,24cyclictest0-21swapper/020:08:130
753199271,24cyclictest0-21swapper/019:55:160
65812270,1sleep20-21swapper/221:42:162
61882270,1sleep30-21swapper/322:54:493
56662270,3chrt0-21swapper/223:41:552
293472270,2chrt0-21swapper/100:00:041
753499261,23cyclictest0-21swapper/319:51:353
7534992610,5cyclictest34-21ksoftirqd/319:20:103
753499260,25cyclictest0-21swapper/320:03:433
753399261,23cyclictest0-21swapper/221:39:562
753399261,23cyclictest0-21swapper/220:00:142
7532992624,1cyclictest0-21swapper/119:57:391
7532992624,1cyclictest0-21swapper/119:42:051
7532992613,10cyclictest560-21dbus-daemon21:10:001
753299261,24cyclictest0-21swapper/121:31:361
753299261,24cyclictest0-21swapper/120:07:311
7531992624,1cyclictest0-21swapper/021:27:040
7531992623,2cyclictest0-21swapper/019:39:520
753199261,24cyclictest0-21swapper/021:43:480
753199261,24cyclictest0-21swapper/019:51:340
753199261,1cyclictest0-21swapper/021:57:270
753199261,1cyclictest0-21swapper/021:31:400
753199261,1cyclictest0-21swapper/020:01:090
753199260,24cyclictest0-21swapper/022:25:090
266902260,4sleep2271rcuc/200:21:282
118522260,2sleep00-21swapper/000:10:230
753499251,23cyclictest0-21swapper/320:07:523
753499251,23cyclictest0-21swapper/319:45:103
753499250,23cyclictest0-21swapper/321:51:203
753399253,14cyclictest14716-21ssh23:25:112
7533992523,1cyclictest0-21swapper/220:07:292
7533992519,4cyclictest28-21ksoftirqd/200:00:002
753399251,23cyclictest0-21swapper/219:51:142
753399251,22cyclictest0-21swapper/219:57:402
753399251,1cyclictest0-21swapper/221:32:172
753399250,23cyclictest0-21swapper/222:25:012
7532992523,1cyclictest0-21swapper/121:57:201
7531992523,1cyclictest0-21swapper/021:38:240
7531992518,5cyclictest26852-21ssh23:57:480
7531992514,8cyclictest12125-21apt-get22:35:130
753199251,10cyclictest48932ssh23:17:220
232322250,2sleep00-21swapper/000:19:460
232322250,2sleep00-21swapper/000:19:450
166822250,10sleep334-21ksoftirqd/323:25:393
124552250,2sleep00-21swapper/021:22:400
7534992422,1cyclictest0-21swapper/321:57:073
7533992422,1cyclictest0-21swapper/221:17:492
7533992421,2cyclictest0-21swapper/222:02:362
7533992421,2cyclictest0-21swapper/222:02:362
753199241,8cyclictest14326-21sort23:25:000
143312240,2sleep30-21swapper/319:25:143
7534992321,1cyclictest0-21swapper/322:02:363
7534992321,1cyclictest0-21swapper/322:02:363
7533992316,5cyclictest28-21ksoftirqd/200:20:002
7533992316,5cyclictest28-21ksoftirqd/200:20:002
7533992313,6cyclictest11011-21grep21:45:232
753199231,7cyclictest22185-21chrt23:05:380
100412230,1sleep10-21swapper/122:09:311
7534992220,1cyclictest0-21swapper/320:10:023
7534992216,5cyclictest34-21ksoftirqd/323:24:593
753399222,14cyclictest6797-21apt-get22:55:102
7533992217,3cyclictest26787-21sh21:10:002
7533992216,4cyclictest2019-21sh23:40:002
7533992214,6cyclictest16981-21ls19:30:192
7532992211,9cyclictest8759-21latency_hist23:45:011
753299221,19cyclictest17753-21seq00:39:481
7531992214,6cyclictest3255-21users21:15:250
7531992210,7cyclictest29908-21ssh22:00:120
7531992210,7cyclictest29908-21ssh22:00:120
222122220,2sleep20-21swapper/221:54:282
118422220,1sleep20-21swapper/223:22:332
753499218,11cyclictest30265-21basename00:25:003
7534992113,6cyclictest30584-21pluginstate00:00:203
753399213,15cyclictest7807-21tail20:25:152
753399211,7cyclictest253072ssh23:08:462
7533992114,5cyclictest28-21ksoftirqd/221:20:012
753299213,13cyclictest17198-21apt-get00:15:131
753299213,13cyclictest17198-21apt-get00:15:131
7532992119,1cyclictest0-21swapper/122:02:141
7532992119,1cyclictest0-21swapper/122:02:131
7532992113,6cyclictest20616-21seq23:53:291
7532992113,6cyclictest13714-21seq19:24:311
7532992112,7cyclictest16704-21seq23:01:521
753299211,14cyclictest8348-21ssh21:20:031
753199212,13cyclictest18417-21apt-get22:40:030
753199212,12cyclictest25800-21apt-get22:45:120
753199212,12cyclictest25800-21apt-get22:45:110
7531992119,1cyclictest0-21swapper/021:45:350
7531992114,5cyclictest8127-21smartctl19:10:160
7531992114,5cyclictest102652sleep022:33:340
7531992113,6cyclictest22068-21grep20:55:260
7531992112,7cyclictest10698-21cat20:30:230
32192210,1sleep10-21swapper/122:51:491
92442200,3sleep19235-21perf20:29:591
7534992014,4cyclictest27778-21sh23:58:443
7534992014,4cyclictest14691-21sh21:48:393
753399201,6cyclictest0-21swapper/200:30:572
7533992015,3cyclictest20352-21sh20:55:002
7533992014,4cyclictest636-21runrttasks00:04:412
7533992013,4cyclictest636-21runrttasks21:10:072
7533992012,6cyclictest12438-21ls20:35:202
7533992011,7cyclictest560-21dbus-daemon21:04:592
7533992011,6cyclictest13981-21apt20:40:102
7532992013,4cyclictest30764-21sh21:12:131
7532992012,6cyclictest18485-21seq23:27:291
7532992012,6cyclictest1733-21perl23:15:161
7532992011,7cyclictest16362-21seq00:14:361
7532992010,7cyclictest10686-21sed20:30:231
753199209,6cyclictest53050irq/142-enp3s0-21:50:120
753199202,12cyclictest4958-21apt-get00:30:110
7531992013,4cyclictest16510-21sh23:25:290
7531992012,6cyclictest3965-21open_inodes20:15:210
7531992012,6cyclictest26797-21seq23:10:030
7531992012,6cyclictest23225-21tune2fs21:00:140
7531992012,6cyclictest11428-21seq19:17:530
7531992011,7cyclictest7410-21sort20:25:120
7531992011,6cyclictest14568-21seq23:49:080
305452200,7sleep012-21ksoftirqd/021:12:000
753499193,4cyclictest34-21ksoftirqd/321:15:453
753499191,9cyclictest20204-21sendmail19:40:003
7534991914,3cyclictest5438-21unixbench_singl00:05:253
7534991912,5cyclictest34-21ksoftirqd/323:08:583
7534991912,5cyclictest15080-21run-parts20:40:223
7534991911,6cyclictest7678-21forks20:25:143
7534991910,6cyclictest17114-21awk20:45:203
753399199,7cyclictest10040-21df19:15:132
753399193,11cyclictest75-21kswapd022:50:112
7533991913,4cyclictest17329-21sh00:39:222
7533991913,4cyclictest16809-21sh22:38:232
7533991913,3cyclictest15567-21sh23:00:432
7533991912,4cyclictest3963-21open_files20:15:212
7533991911,6cyclictest6471-21ntp_states22:30:222
7533991911,6cyclictest21354-21expr20:55:192
7533991910,7cyclictest1-21systemd00:10:212
7533991910,6cyclictest2068-21cat23:15:172
753299196,9cyclictest52850irq/140-enp3s0-22:35:121
753299192,11cyclictest18916-21apt-get22:40:031
753299191,3cyclictest26798-21diskmemload00:26:191
7532991913,3cyclictest31341-21sh23:36:551
7532991912,5cyclictest30350-21sed00:25:011
7532991912,5cyclictest11515-21seq19:19:011
7532991912,5cyclictest10225-21cut00:10:001
7532991911,6cyclictest9508-21expr23:45:131
7532991911,6cyclictest8147-21cat19:10:161
7532991911,6cyclictest2926-21cut20:15:001
753199192,10cyclictest26798-21diskmemload23:35:100
7531991913,4cyclictest4096-21sh00:29:320
7531991913,3cyclictest26798-21diskmemload22:50:340
7531991913,3cyclictest24474-21sh23:31:450
7531991912,5cyclictest2250-21sh00:03:530
7531991912,5cyclictest17999-21seq19:34:330
7531991911,6cyclictest560-21dbus-daemon23:05:010
7531991911,6cyclictest17901-21seq20:48:490
7531991911,6cyclictest14055-21cat20:40:130
7531991911,6cyclictest12336-21tune2fs19:20:140
753499189,6cyclictest21067-21tune2fs20:55:153
753499188,7cyclictest16478-21df19:30:133
753499186,9cyclictest11275-21ssh00:35:003
753499183,9cyclictest18467-21apt-get20:50:123
753499183,3cyclictest141rcu_preempt21:10:193
753499182,12cyclictest75-21kswapd000:35:113
753499181,8cyclictest11514-21ssh00:10:183
7534991813,3cyclictest23894-21mailstats21:00:213
7534991813,3cyclictest18650-21sh23:03:493
7534991813,3cyclictest10707-21sshd19:15:223
7534991811,5cyclictest7803-21expr19:10:123
7534991811,5cyclictest4497-21grep20:15:253
7534991811,5cyclictest2716-21expr23:40:113
7534991811,5cyclictest21647-21expr23:30:133
7534991811,5cyclictest11608-21basename20:35:103
7534991810,6cyclictest28721-21expr23:35:153
7534991810,6cyclictest10033-21sed23:45:183
753399189,6cyclictest6591-21unixbench_multi20:20:232
7533991812,4cyclictest859-21/usr/sbin/munin20:10:262
7533991812,3cyclictest4829-21sh00:05:202
7533991811,5cyclictest28-21ksoftirqd/222:50:002
7533991811,5cyclictest28-21ksoftirqd/222:50:002
7533991811,4cyclictest636-21runrttasks23:10:112
7533991811,4cyclictest26798-21diskmemload23:31:322
7533991810,6cyclictest17570-21unixbench_singl20:45:242
7532991813,3cyclictest17625-21tr21:50:211
7532991812,4cyclictest30975-21sh22:48:541
7532991812,4cyclictest30975-21sh22:48:531
7532991812,4cyclictest30219-21sh23:11:591
7532991812,4cyclictest21778-21swap20:55:241
7532991812,4cyclictest12683-21sh23:23:231
7532991812,4cyclictest1033-21sh21:38:301
7532991811,5cyclictest6950-21seq00:30:411
7532991811,5cyclictest6779-21find22:55:111
7532991811,5cyclictest16345-21expr20:45:121
7532991811,5cyclictest14185-21expr20:40:131
753299181,12cyclictest75-21kswapd022:30:121
753299181,12cyclictest18298-21apt-get19:35:121
7532991810,6cyclictest12234-21cat20:35:161
753199189,7cyclictest26260-21cat21:05:230
753199181,8cyclictest10056-21ssh22:56:570
7531991812,4cyclictest15714-21sh23:50:050
7531991811,5cyclictest5004-21gzip00:05:220
7531991811,5cyclictest3578-21tr23:40:200
7531991811,5cyclictest15088-21rt-features19:25:220
7531991811,5cyclictest1172-21expr20:10:130
7534991710,4cyclictest26798-21diskmemload23:15:043
753399179,6cyclictest9577-21grep20:30:122
753399178,7cyclictest15559-21seq19:25:422
753399172,10cyclictest4210-21apt-get22:05:132
7533991710,5cyclictest12365-21head19:20:152
7532991710,5cyclictest22510-21awk23:30:211
7532991710,5cyclictest18640-21ls20:50:131
753199179,6cyclictest20149-21seq20:52:300
7531991711,4cyclictest0-21swapper/020:35:160
753199171,12cyclictest5187-21apt-get20:20:120
753499169,5cyclictest25270-21cut21:05:143
7533991611,3cyclictest19009-21latency19:35:172
753299168,6cyclictest14202-21expr19:25:141
753299167,7cyclictest3855-21needreboot20:15:181
753299161,11cyclictest16155-21apt-get19:30:131
753299158,5cyclictest6704-21awk20:20:271
753299157,6cyclictest22734-21seq21:00:051
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional