You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-15 - 04:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Mon Jun 15, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
111952237213,17sleep00-21swapper/019:08:590
11270217297,25sleep10-21swapper/119:09:571
746721570,1sleep30-21swapper/322:30:013
112272139104,24sleep20-21swapper/219:09:252
110912138102,25sleep30-21swapper/319:07:383
1888621060,1sleep20-21swapper/200:15:272
36842700,8sleep122-21ksoftirqd/121:13:381
139232680,2chrt0-21swapper/100:36:131
116762620,2sleep00-21swapper/021:19:590
246012600,2sleep00-21swapper/023:55:310
119862600,1sleep00-21swapper/023:46:120
41532560,2sleep10-21swapper/122:51:071
3472550,3sleep122-21ksoftirqd/123:13:261
161202550,2sleep00-21swapper/023:00:210
59132530,2sleep20-21swapper/222:28:402
246532530,1sleep30-21swapper/321:29:473
57892520,1sleep00-21swapper/000:05:540
321112490,1sleep20-21swapper/200:01:262
29422490,2sleep30-21swapper/322:25:363
222802450,2chrt0-21swapper/121:51:361
173592450,1sleep30-21swapper/319:20:253
199482430,2chrt0-21swapper/222:39:312
289682370,2chrt0-21swapper/222:20:502
225652370,1chrt0-21swapper/322:16:053
288432350,2sleep30-21swapper/323:59:513
1154899358,8cyclictest0-21swapper/321:59:593
11548993511,6cyclictest29385-21ls23:35:153
11548993511,5cyclictest31021-21grep00:00:283
11548993411,5cyclictest28192-21cat22:45:173
11548993411,5cyclictest22400-21date20:50:003
11548993411,5cyclictest20488-21expr19:30:143
11548993312,4cyclictest8011-21sh00:08:093
11548993310,5cyclictest12093-21grep19:10:153
11548993310,4cyclictest12604-21tr21:20:133
53932320,2sleep20-21swapper/220:10:112
244262320,2sleep30-21swapper/300:20:183
1154899329,6cyclictest18273-21cat20:40:133
1154899329,6cyclictest18273-21cat20:40:123
1154899329,4cyclictest25796-21switchtime20:55:233
1154899328,5cyclictest7944-21cron22:54:593
1154899328,5cyclictest27616-21cut22:20:193
1154899328,5cyclictest13073-21sed20:25:243
1154899322,7cyclictest5442-21apt-get21:15:133
11548993213,3cyclictest13456-21sh00:11:433
11548993211,4cyclictest30485-21sh21:33:573
11548993211,3cyclictest12229-21sh21:44:513
1154899318,5cyclictest9708-21cat23:45:143
1154899316,9cyclictest0-21swapper/322:06:553
11548993111,4cyclictest18920-21sh21:49:563
1154899309,3cyclictest22505-21sh21:51:493
11548993011,3cyclictest21131-21sh23:04:583
1154899301,10cyclictest13611-21apt-get22:10:133
11548993010,4cyclictest23817-21sh22:41:393
11545993026,3cyclictest131rcuc/022:20:300
1154899299,4cyclictest12553-21pmu-power00:35:213
1154899298,5cyclictest34-21ksoftirqd/321:05:143
1154899296,5cyclictest32516-21unixbench_multi19:55:253
1154899291,9cyclictest9429-21ntp_states22:55:213
1154899291,9cyclictest9429-21ntp_states22:55:213
1154899291,5cyclictest26356-21sh23:08:353
11548992914,6cyclictest1215-21uniq20:00:163
11548992912,3cyclictest3519-21sh21:13:293
11548992911,3cyclictest32241-21sh00:25:273
11548992911,2cyclictest6031-21rm23:17:273
1154599292,8cyclictest23384-21systemctl23:30:210
1154899289,4cyclictest17584-21cat23:50:243
1154599282,6cyclictest12-21ksoftirqd/023:18:310
1154599281,6cyclictest18153-21sh21:24:540
1154899272,6cyclictest31949-21apt-get21:35:133
1154899271,9cyclictest22115-21apt-get23:30:123
1154899271,10cyclictest32615-21apt-get22:00:123
1154899268,5cyclictest34-21ksoftirqd/319:40:103
1154899267,4cyclictest30819-21diskmemload22:35:463
1154899262,6cyclictest26610-21apt-get19:45:133
11547992617,6cyclictest9852-21sh23:20:182
1154799261,12cyclictest246162ssh23:06:462
1154599264,7cyclictest12-21ksoftirqd/000:00:010
1154599264,6cyclictest12-21ksoftirqd/000:18:060
1154599263,7cyclictest12-21ksoftirqd/021:44:100
1154599263,7cyclictest12-21ksoftirqd/019:15:000
1154599263,7cyclictest12-21ksoftirqd/000:21:510
11545992624,1cyclictest0-21swapper/019:12:460
11545992622,3cyclictest131rcuc/022:27:590
11545992622,2cyclictest131rcuc/022:42:070
1154599261,9cyclictest23635-21sh23:05:460
11545992617,6cyclictest26952-21grep21:30:260
1154599261,6cyclictest5264-21ssh00:30:150
239522250,1sleep20-21swapper/222:41:492
1154899251,10cyclictest2723-21apt-get20:05:033
1154799251,23cyclictest0-21swapper/219:12:152
1154699251,23cyclictest0-21swapper/119:12:281
1154599258,6cyclictest12-21ksoftirqd/021:35:080
1154599253,7cyclictest12-21ksoftirqd/000:35:480
1154599253,6cyclictest12-21ksoftirqd/022:39:240
1154599253,6cyclictest12-21ksoftirqd/021:51:550
11545992522,2cyclictest131rcuc/022:34:030
167052240,2sleep10-21swapper/120:35:191
1154799243,12cyclictest13614-21ssh22:10:112
1154799242,15cyclictest22805-21chrt23:55:132
11547992413,6cyclictest30819-21diskmemload22:50:132
11546992416,6cyclictest6070-21ls20:10:171
1154699241,10cyclictest234122ssh23:05:331
1154599243,6cyclictest12-21ksoftirqd/022:11:580
1154599243,6cyclictest12-21ksoftirqd/020:20:250
1154599243,5cyclictest0-21swapper/021:28:510
1154599242,7cyclictest12-21ksoftirqd/021:47:570
11545992420,3cyclictest131rcuc/023:28:400
11545992419,3cyclictest131rcuc/022:58:100
11545992419,3cyclictest131rcuc/022:58:090
1154599241,15cyclictest10302-21ssh00:10:030
1154899239,3cyclictest30819-21diskmemload23:13:473
1154799232,15cyclictest20217-21ssh22:15:132
11547992317,4cyclictest28-21ksoftirqd/222:00:002
11546992316,5cyclictest11049-21ssh22:32:091
1154599239,6cyclictest12-21ksoftirqd/023:52:040
1154599237,6cyclictest12-21ksoftirqd/022:54:250
1154599233,7cyclictest12-21ksoftirqd/021:14:200
1154599233,5cyclictest12-21ksoftirqd/021:56:020
11545992319,3cyclictest131rcuc/022:05:390
1154599231,8cyclictest388-21rm00:26:030
11545992318,3cyclictest131rcuc/023:44:420
11545992318,3cyclictest131rcuc/023:44:410
1154599230,6cyclictest13482-21taskset20:27:540
1154599230,6cyclictest0-21swapper/022:46:170
3472220,2sleep10-21swapper/100:02:101
181042220,6sleep334-21ksoftirqd/300:15:193
11547992214,6cyclictest14546-21users21:45:282
11547992214,6cyclictest12499-21open_files00:35:222
1154699229,4cyclictest141rcu_preempt21:20:001
11546992213,7cyclictest31840-21python300:25:241
1154599225,6cyclictest12-21ksoftirqd/022:17:000
1154599223,6cyclictest12-21ksoftirqd/023:37:160
1154599222,6cyclictest12-21ksoftirqd/023:10:470
1154599221,8cyclictest9053-21taskset20:16:220
11545992218,3cyclictest131rcuc/023:20:470
11545992218,3cyclictest131rcuc/022:02:490
11545992218,3cyclictest131rcuc/019:59:220
1154599220,6cyclictest26234-21taskset20:56:390
94812210,2sleep20-21swapper/222:30:332
1154899216,6cyclictest3012-21tr23:40:133
1154899216,6cyclictest3012-21tr23:40:123
11548992114,5cyclictest23637-21sensors_temp20:50:233
11547992113,6cyclictest29187-21sed22:45:242
11547992113,5cyclictest52750irq/139-enp3s0-22:05:132
11547992112,7cyclictest21847-21users20:45:242
11547992110,3cyclictest141rcu_preempt21:30:122
1154699212,14cyclictest13454-21ssh22:10:041
1154699211,5cyclictest0-21swapper/120:21:491
11546992114,5cyclictest27936-21open_files22:20:211
11546992113,6cyclictest4203-21seq23:15:331
11546992112,7cyclictest29129-21date00:00:001
11546992111,8cyclictest27696-21ntp_states21:00:201
1154699211,13cyclictest12852-21ssh21:45:131
1154599214,7cyclictest12-21ksoftirqd/020:52:510
1154599214,5cyclictest12-21ksoftirqd/019:50:160
1154599212,7cyclictest12-21ksoftirqd/021:00:050
1154599211,7cyclictest12-21ksoftirqd/021:06:340
1154599211,6cyclictest12-21ksoftirqd/019:36:200
11545992115,4cyclictest131rcuc/020:48:080
1154599210,8cyclictest17890-21runrttasks20:39:350
117542200,2sleep20-21swapper/223:45:572
1154899201,5cyclictest0-21swapper/320:30:473
11548992012,5cyclictest15158-21grep19:15:253
11548992011,6cyclictest5286-21seq20:10:023
1154899200,18cyclictest9785-21ssh00:33:473
1154799202,13cyclictest640-21snmpd19:55:022
1154799202,13cyclictest28146-21apt-get23:10:102
1154799202,12cyclictest20173-21apt-get19:30:142
11547992015,3cyclictest5733-21cron21:40:002
11547992012,6cyclictest32578-21seq23:37:372
11547992012,6cyclictest10473-21apt00:10:112
11547992011,7cyclictest22645-21df19:35:142
1154699209,7cyclictest18703-21apt-get21:25:131
1154699203,10cyclictest2951-21ssh23:40:131
1154699203,10cyclictest2951-21ssh23:40:131
11546992013,4cyclictest30819-21diskmemload21:40:201
11546992012,6cyclictest9501-21find23:45:111
11546992012,6cyclictest9178-21seq20:17:561
11546992012,6cyclictest29124-21awk23:35:141
11546992011,6cyclictest16671-21cat19:20:181
1154699201,15cyclictest4728-21kworker/u8:023:20:131
1154699201,12cyclictest23853-21ssh00:20:141
1154599201,7cyclictest12-21ksoftirqd/020:13:380
1154599201,7cyclictest12-21ksoftirqd/020:03:580
1154599201,7cyclictest12-21ksoftirqd/019:47:310
1154599201,6cyclictest19699-21seq19:27:210
11545992014,4cyclictest131rcuc/020:43:080
11545992014,4cyclictest131rcuc/020:43:070
11545992011,6cyclictest20894-21ls19:30:190
1154599200,8cyclictest5049-21runrttasks20:09:540
1154599200,8cyclictest17470-21runrttasks19:21:100
11548991912,5cyclictest16674-21pmu-power23:25:213
11548991910,7cyclictest8197-21cat20:15:193
1154799192,6cyclictest6773-21python321:15:232
1154799192,11cyclictest24622-21apt-get20:55:132
11547991913,4cyclictest9658-21sendmail_mailtr22:55:232
11547991913,4cyclictest9658-21sendmail_mailtr22:55:222
11547991912,5cyclictest16260-21expr20:35:142
11547991912,4cyclictest30819-21diskmemload21:21:202
11547991911,6cyclictest27606-21ls21:00:192
11547991911,6cyclictest25701-21date19:40:232
11547991911,6cyclictest19072-21df21:25:142
11547991911,6cyclictest17109-21sed19:20:242
11547991911,6cyclictest15965-21awk23:25:152
11547991911,6cyclictest15129-21cat20:30:252
1154699199,7cyclictest278-21systemd-journal23:05:001
1154699192,14cyclictest16140-21apt-get23:50:131
1154699192,12cyclictest75-21kswapd022:45:121
1154699192,12cyclictest0-21swapper/122:55:121
1154699192,12cyclictest0-21swapper/122:55:111
11546991913,4cyclictest3559-21sh00:05:101
11546991912,5cyclictest32034-21open_inodes19:55:211
11546991912,5cyclictest28745-21apt-get19:50:121
11546991912,5cyclictest23220-21sed20:50:171
11546991912,5cyclictest16978-21seq22:36:291
11546991912,5cyclictest11942-21cstates20:25:141
11546991911,6cyclictest2461-21seq20:01:481
11546991911,6cyclictest10309-21seq22:07:151
11546991911,5cyclictest22656-21grep23:30:151
11546991911,4cyclictest20744-21gltestperf22:15:171
11546991910,7cyclictest26683-21cat19:45:131
1154599191,9cyclictest12-21ksoftirqd/019:41:460
11545991915,3cyclictest131rcuc/020:30:080
122432180,1chrt0-21swapper/100:10:291
11548991813,3cyclictest26757-21/usr/sbin/munin21:00:143
11548991811,5cyclictest9873-21cat23:20:193
11548991811,5cyclictest10410-21cat20:20:193
11548991811,4cyclictest29913-21swap19:50:233
1154799189,7cyclictest560-21dbus-daemon19:20:002
1154799186,3cyclictest141rcu_preempt23:40:162
1154799186,3cyclictest141rcu_preempt23:40:152
1154799182,11cyclictest832-21apt-get20:00:132
1154799182,11cyclictest3606-21apt-get00:05:132
1154799181,8cyclictest1356-21ssh00:27:012
11547991813,3cyclictest6162-21/usr/sbin/munin21:40:122
11547991813,3cyclictest4492-21sh21:14:292
11547991813,3cyclictest3162-21/usr/sbin/munin20:05:252
11547991812,4cyclictest21169-21idleruntime-cro23:05:012
11547991812,3cyclictest5986-21sh23:17:242
11547991811,5cyclictest6142-21cat00:30:222
11547991811,5cyclictest22807-21cat23:30:152
11547991810,6cyclictest29699-21ls19:50:212
11547991810,6cyclictest11028-21unixbench_multi20:20:242
1154699188,7cyclictest18626-21cat00:15:241
1154699187,3cyclictest209912sh23:29:151
1154699183,8cyclictest13732-21apt-get19:15:121
1154699181,8cyclictest29526-21ssh21:57:161
11546991812,4cyclictest4611-21sh00:30:051
11546991811,5cyclictest629-21grep22:00:151
11546991811,5cyclictest20268-21cut20:45:011
11546991811,5cyclictest20268-21cut20:45:001
11546991811,4cyclictest30819-21diskmemload21:37:311
11546991810,6cyclictest3514-21df20:05:151
11546991810,6cyclictest26977-21tail21:30:261
11546991810,6cyclictest18811-21cat19:25:151
11548991712,3cyclictest18006-21/usr/sbin/munin19:25:133
11548991712,3cyclictest17026-21pmu-power20:35:223
1154799173,8cyclictest640-21snmpd19:25:092
11547991713,2cyclictest28920-21apt-get21:05:122
11547991712,3cyclictest16663-21gltestperf23:50:162
11547991712,3cyclictest1343-21grep22:00:202
11547991711,3cyclictest30819-21diskmemload00:20:362
11547991710,5cyclictest22675-21expr20:50:132
11547991710,5cyclictest12236-21fw_packets20:25:152
1154699179,6cyclictest14811-21ntpq20:30:201
1154699177,8cyclictest560-21dbus-daemon20:50:001
1154699171,7cyclictest1222-21cstates22:25:131
11546991712,3cyclictest16589-21sh21:23:161
11546991710,5cyclictest23836-21seq19:35:311
11546991710,5cyclictest20428-21expr19:30:141
1154899169,5cyclictest22323-21grep19:35:143
1154799169,5cyclictest26964-21grep19:45:152
1154799169,5cyclictest18814-21if_enp3s020:40:172
1154799169,5cyclictest18814-21if_enp3s020:40:162
1154799165,9cyclictest0-21swapper/221:50:172
1154799161,11cyclictest7466-21apt-get20:15:142
1154699169,5cyclictest26068-21seq19:41:291
1154699164,3cyclictest30819-21diskmemload22:40:071
1154699161,6cyclictest31017-21latency_hist21:10:021
11546991610,4cyclictest24956-21df20:55:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional