You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-09 - 05:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sat May 09, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
141232221103,109sleep30-21swapper/319:06:123
141742144109,24sleep20-21swapper/219:06:522
140662142106,25sleep10-21swapper/119:05:271
142212138102,25sleep00-21swapper/019:07:260
220642660,3sleep221248-1kworker/u9:123:00:142
158452610,2sleep10-21swapper/100:30:261
236422590,2sleep00-21swapper/000:36:230
90382570,2sleep20-21swapper/200:01:462
272122570,2sleep00-21swapper/020:50:220
220182560,1sleep20-21swapper/200:35:212
120362560,1sleep00-21swapper/000:04:440
17122530,1sleep10-21swapper/122:45:111
78932520,2sleep10-21swapper/100:00:361
327642510,1sleep30-21swapper/321:31:583
218512500,1sleep30-21swapper/322:35:233
299452470,2sleep00-21swapper/021:54:140
172222470,1sleep117221-21ssh23:44:431
3652460,2sleep30-21swapper/321:05:143
16082460,2sleep30-21swapper/321:56:333
106752440,2sleep30-21swapper/321:15:253
58942430,3chrt0-21swapper/300:23:503
179622330,5sleep012-21ksoftirqd/023:20:220
1469299326,17cyclictest2417-21sh23:09:493
1469299324,11cyclictest6029-21grep23:35:233
1469299323,12cyclictest17625-21awk21:20:223
1469299313,11cyclictest14185-21rm22:54:293
1469299305,9cyclictest1956-21diskmemload23:29:173
1469299304,9cyclictest30424-21tune2fs19:45:163
1469299303,9cyclictest6921-21ssh23:12:343
1469299295,8cyclictest1956-21diskmemload00:15:373
1469299294,10cyclictest307-21ps23:55:213
1469299294,10cyclictest3023-21grep21:10:143
1469299293,9cyclictest4657-21ssh22:23:033
1469299293,9cyclictest14350-21ssh23:18:153
1469299293,8cyclictest3819-21ssh23:34:483
1469299292,9cyclictest16528-21/usr/sbin/munin23:20:243
1469299292,10cyclictest29179-21sed19:40:233
1469299291,5cyclictest0-21swapper/321:40:223
74842280,2sleep20-21swapper/222:25:132
188302280,1chrt0-21swapper/121:21:331
1469299284,11cyclictest15179-21grep19:10:153
1469299283,9cyclictest22501-21ssh21:48:463
1469299283,8cyclictest29613-21ssh23:53:383
1469299283,11cyclictest25688-21ssh23:02:283
1469299282,8cyclictest25076-21ssh22:14:383
1469299282,8cyclictest15416-21ssh23:42:533
1469299282,8cyclictest10620-21ssh00:03:203
1469299275,10cyclictest1956-21diskmemload00:13:173
1469299273,8cyclictest20875-21ssh00:35:083
1469299273,10cyclictest17796-21ssh00:08:463
1469299272,9cyclictest16232-21sh22:55:203
1469299272,9cyclictest13891-21cat20:20:173
1469299272,8cyclictest9521-21ssh21:39:223
1469299272,8cyclictest18077-21ssh22:09:233
1469299272,11cyclictest6664-21expr20:05:133
14692992716,5cyclictest7033-21apt-get22:25:103
14691992719,6cyclictest2174-21rm21:57:072
119282270,1sleep111929-21sh22:28:351
90522260,2chrt0-21swapper/121:38:531
1469299262,9cyclictest6362-21sh22:48:263
1469299262,8cyclictest16563-21ssh00:31:023
1469299262,11cyclictest18978-21ssh22:33:503
14691992618,6cyclictest13135-21sh21:17:402
1469099261,1cyclictest0-21swapper/119:14:001
1468999261,24cyclictest0-21swapper/019:14:030
315072250,2sleep20-21swapper/222:19:162
1469299253,9cyclictest31076-21sed21:00:183
1469299253,8cyclictest27308-21apt-get22:40:103
1469299252,8cyclictest26178-21ssh21:27:103
14689992517,6cyclictest23921-21ssh23:00:420
113832250,2chrt0-21swapper/121:15:551
1469299244,8cyclictest9319-21sh00:25:323
1469299244,8cyclictest9319-21sh00:25:313
1469299242,8cyclictest28355-21ssh22:16:093
1469299241,9cyclictest8809-21sh22:01:573
261182230,1sleep10-21swapper/100:38:521
182822230,3sleep012-21ksoftirqd/022:09:340
14692992316,5cyclictest25929-21sh21:50:243
1469299230,9cyclictest0-21swapper/320:25:153
1469199239,3cyclictest28-21ksoftirqd/223:35:162
1469199237,11cyclictest52950irq/141-enp3s0-00:05:112
1469199232,18cyclictest18753-21cron19:20:012
14691992321,1cyclictest0-21swapper/219:13:322
14691992316,5cyclictest30675-21ssh00:17:572
1469299227,12cyclictest19004-21ntp_states23:45:213
14692992212,7cyclictest5975-21seq20:01:103
1469299221,10cyclictest1468-21runrttasks19:51:103
1469299221,10cyclictest1468-21runrttasks19:51:093
14691992214,6cyclictest31297-21sed23:55:012
14690992215,4cyclictest32742-21sendmail00:20:001
14690992213,7cyclictest13649-21python322:05:231
14690992211,8cyclictest560-21dbus-daemon20:20:011
1468999221,15cyclictest3831-21ssh21:35:110
1468999220,7cyclictest18768-21sh19:20:000
193382210,1sleep00-21swapper/022:10:120
1469299218,7cyclictest25665-21apt-get19:35:113
1469299216,12cyclictest17945-21/usr/sbin/munin19:15:203
1469299213,6cyclictest21263-21runrttasks20:37:073
1469299212,4cyclictest0-21swapper/320:50:123
1469299210,13cyclictest25582-21taskset20:47:203
1469199211,9cyclictest258652sh23:26:122
14691992113,6cyclictest15292-21ntp_states00:30:222
14691992113,6cyclictest12845-21fw_packets22:05:152
14691992112,6cyclictest7954-21unixbench_multi20:05:242
1469199211,14cyclictest15952-21ssh21:20:102
1469199210,4cyclictest0-21swapper/220:02:262
1469099219,10cyclictest16184-21sh22:55:201
1469099217,2cyclictest141rcu_preempt23:14:581
1469099216,8cyclictest32495-21ssh23:07:301
1469099214,4cyclictest22-21ksoftirqd/121:40:121
1469099212,13cyclictest10246-21kworker/u8:219:30:111
1469099211,5cyclictest0-21swapper/122:50:111
14690992113,6cyclictest19951-21diskstats20:35:121
14690992112,7cyclictest23373-21threads23:00:251
14690992111,8cyclictest8851-21date21:15:001
14690992111,7cyclictest31852-21expr23:55:121
1468999213,13cyclictest1956-21diskmemload22:35:130
1468999212,14cyclictest17411-21ssh21:45:120
1468999211,9cyclictest15941-21grep00:30:250
1468999211,6cyclictest13325-21cut00:05:160
14689992113,6cyclictest626-21ls19:50:180
14689992113,6cyclictest626-21ls19:50:170
14689992113,6cyclictest27236-21seq19:36:070
217272200,3sleep228-21ksoftirqd/220:40:012
1469299209,8cyclictest8845-21/usr/sbin/munin20:10:123
1469299203,6cyclictest19208-21taskset20:33:123
14692992011,7cyclictest19503-21fw_packets19:20:153
1469199205,2cyclictest0-21swapper/221:49:392
1469199201,5cyclictest0-21swapper/222:34:492
14691992012,6cyclictest8513-21seq21:38:212
14691992012,6cyclictest18947-21ntp_kernel_err23:45:192
14691992011,7cyclictest15276-21cat20:25:002
14691992011,6cyclictest23677-21basename20:45:002
14691992011,6cyclictest23677-21basename20:44:592
14691992011,6cyclictest21530-21cat23:00:002
1469199201,16cyclictest11796-21grep23:40:152
1469099201,7cyclictest31153-21ntpq23:30:201
14690992012,6cyclictest8400-21seq20:09:531
14690992012,6cyclictest22520-21date20:40:171
14690992012,6cyclictest22520-21date20:40:161
14690992012,6cyclictest17798-21forks21:45:141
14690992012,5cyclictest278-21systemd-journal22:39:561
14690992012,4cyclictest1956-21diskmemload23:45:121
14690992011,7cyclictest32483-21apt-get21:05:101
14690992011,7cyclictest26763-21ls22:15:181
1469099200,7cyclictest141rcu_preempt22:22:581
1469099200,5cyclictest13777-21/usr/sbin/munin22:30:141
1468999208,4cyclictest31156-21ssh21:30:240
1468999207,4cyclictest141rcu_preempt00:10:140
1468999203,9cyclictest31468-21ssh23:55:040
1468999203,13cyclictest75-21kswapd023:30:120
1468999202,7cyclictest636-21runrttasks21:15:340
1468999201,6cyclictest28500-21processes22:40:190
14689992012,6cyclictest8603-21ls22:25:240
14689992012,6cyclictest3599-21cut21:10:210
14689992012,6cyclictest25310-21seq19:33:500
14689992011,7cyclictest8808-21ssh00:25:230
14689992011,7cyclictest8808-21ssh00:25:230
14689992011,7cyclictest7449-21seq22:00:360
271192190,3sleep012-21ksoftirqd/023:51:080
1469299197,10cyclictest21825-21/usr/sbin/munin19:25:173
1469299196,10cyclictest3151-21mailstats19:55:223
14692991911,3cyclictest28189-21apt-get20:55:113
14692991911,3cyclictest10815-21apt-get20:15:113
14692991910,6cyclictest24255-21cat19:30:173
1469199198,7cyclictest32310-21apt-get19:50:122
1469199198,7cyclictest32310-21apt-get19:50:122
1469199198,7cyclictest22706-21apt-get21:25:112
1469199191,6cyclictest0-21swapper/200:21:252
14691991915,3cyclictest7930-21fschecks_count00:25:142
14691991915,3cyclictest7930-21fschecks_count00:25:142
14691991913,4cyclictest9931-21tr23:15:122
14691991913,4cyclictest8870-21sh23:14:322
14691991913,4cyclictest22590-21sh22:12:092
14691991913,4cyclictest1667-21sh23:09:052
14691991912,4cyclictest1956-21diskmemload23:30:162
14691991911,6cyclictest8953-21sed22:50:142
14691991911,6cyclictest31544-21cat21:00:242
14691991911,6cyclictest16113-21cat20:25:182
14691991910,6cyclictest5600-21cat00:00:012
14691991910,6cyclictest30764-21mailstats21:30:222
1469099199,8cyclictest15312-21/usr/sbin/munin20:25:191
1469099199,6cyclictest52850irq/140-enp3s0-22:00:121
1469099199,6cyclictest10830-21sh23:15:201
1469099195,3cyclictest141rcu_preempt23:21:101
1469099192,12cyclictest23739-21apt-get21:50:041
14690991914,3cyclictest26231-21timerwakeupswit23:50:251
14690991912,5cyclictest29710-21seq20:55:271
14690991911,6cyclictest5840-21ls23:35:221
14690991911,6cyclictest21420-21cat19:25:131
14690991911,6cyclictest13384-21expr20:20:141
14690991910,6cyclictest20398-21cat19:20:231
1468999198,8cyclictest26790-21df_inode00:15:150
1468999192,12cyclictest25884-21apt-get22:15:120
14689991914,3cyclictest16853-21vmstat20:25:250
14689991913,4cyclictest16337-21sh22:31:130
14689991912,5cyclictest25588-21seq23:25:560
14689991912,5cyclictest20872-21cat20:35:230
14689991912,3cyclictest1956-21diskmemload23:05:240
14689991911,6cyclictest30811-21perl21:00:160
14689991911,6cyclictest17828-21awk20:30:140
14689991911,6cyclictest16301-21which22:55:210
14689991911,6cyclictest16040-21wc21:20:100
14689991911,5cyclictest4811-21diskstats23:35:130
14689991910,7cyclictest6938-21tune2fs20:05:150
14689991910,6cyclictest11733-21ntp_states20:15:200
1469299182,11cyclictest21773-21apt-get20:40:123
1469299182,11cyclictest21773-21apt-get20:40:123
1469199182,14cyclictest3935-21apt-get20:00:002
14691991813,3cyclictest9235-21sh22:02:222
14691991813,3cyclictest6989-21sh22:49:042
14691991813,3cyclictest23496-21sh00:12:292
14691991812,4cyclictest3451-21open_files21:10:202
14691991812,4cyclictest15909-21sh21:43:562
14691991811,5cyclictest560-21dbus-daemon19:30:022
14691991811,5cyclictest30813-21head19:45:202
14691991811,5cyclictest29263-21awk19:40:242
14691991811,5cyclictest20771-21ls22:35:132
14691991811,4cyclictest1956-21diskmemload23:21:412
14691991811,4cyclictest1956-21diskmemload21:51:552
14691991810,6cyclictest28390-21cat20:55:122
14691991810,6cyclictest26363-21cat19:35:162
14691991810,6cyclictest1139-21sed22:20:172
1469099189,6cyclictest27836-21grep19:40:121
1469099189,6cyclictest24494-21idleruntime20:45:161
1469099187,3cyclictest22-21ksoftirqd/100:20:161
1469099187,2cyclictest22-21ksoftirqd/120:30:241
1469099181,6cyclictest0-21swapper/100:28:081
1469099181,6cyclictest0-21swapper/100:28:081
1469099181,5cyclictest0-21swapper/100:12:571
14690991812,4cyclictest30253-21latency21:30:181
14690991812,4cyclictest22-21ksoftirqd/119:50:001
14690991811,5cyclictest31573-21sed21:00:241
14690991811,5cyclictest27564-21expr22:40:111
14690991810,6cyclictest23601-21expr23:25:121
14690991810,6cyclictest20608-21sed22:10:231
14690991810,5cyclictest17010-21grep19:15:111
1469099180,5cyclictest1956-21diskmemload21:27:101
1468999189,7cyclictest8360-21apt-get22:50:100
1468999182,11cyclictest32483-21apt-get21:05:110
1468999182,10cyclictest2037-21taskset19:55:120
1468999181,8cyclictest3226-21ssh00:21:100
14689991813,4cyclictest2103-21fschecks_time22:45:140
14689991813,3cyclictest32644-21/usr/sbin/munin22:20:140
14689991812,4cyclictest9370-21dump-pmu-power23:15:000
14689991812,4cyclictest32125-21apt-config19:50:010
14689991812,4cyclictest24889-21pmu-power20:45:210
14689991812,4cyclictest11265-21idleruntime21:40:160
14689991811,5cyclictest20612-21seq19:21:100
14689991811,5cyclictest18145-21expr23:45:130
14689991810,6cyclictest9201-21expr20:10:150
14689991810,6cyclictest4398-21awk20:00:110
14689991810,6cyclictest22984-21cat21:25:130
1469199179,6cyclictest1092-21processes21:05:202
1469199179,5cyclictest24381-21gltestperf20:45:142
1469199172,11cyclictest21175-21apt-get19:25:112
14691991711,5cyclictest28-21ksoftirqd/220:20:002
14691991711,4cyclictest29697-21sh22:41:072
14691991711,4cyclictest27935-21sh20:55:002
14691991710,5cyclictest28-21ksoftirqd/220:15:012
14691991710,5cyclictest21037-21sed19:25:012
1469099178,6cyclictest2017-21acpi19:55:111
1469099178,2cyclictest22-21ksoftirqd/120:50:101
14690991712,3cyclictest636-21runrttasks20:13:591
14690991712,3cyclictest13337-21grep00:05:151
1469099171,14cyclictest0-21swapper/119:35:021
14690991710,5cyclictest5694-21cat20:00:231
14690991710,5cyclictest31534-21grep21:55:141
1468999179,6cyclictest27908-21sort19:40:120
1468999179,6cyclictest22466-21grep20:40:170
1468999179,6cyclictest22466-21grep20:40:170
1468999171,7cyclictest9835-21/usr/sbin/munin23:15:110
14689991710,5cyclictest32030-21cat21:55:170
14689991710,5cyclictest28655-21gltestperf20:55:150
14689991710,5cyclictest21417-21expr19:25:120
14689991710,5cyclictest14401-21cat20:20:240
14689991710,4cyclictest1956-21diskmemload23:40:020
1469199168,6cyclictest18494-21ps20:30:222
1469099164,3cyclictest141rcu_preempt19:50:121
1469099164,3cyclictest141rcu_preempt19:50:111
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional