You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-23 - 13:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Thu Jul 23, 2026 00:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
187502176100,26sleep20-21swapper/219:07:522
18594216995,24sleep00-21swapper/019:05:500
187262158106,29sleep30-21swapper/319:07:333
188112146110,25sleep10-21swapper/119:08:391
285872730,16sleep028584-21ssh21:50:170
116622720,2sleep00-21swapper/023:40:160
259762710,2sleep30-21swapper/322:38:423
304332670,2sleep20-21swapper/223:54:272
136972670,1sleep00-21swapper/021:39:450
232842610,1sleep10-21swapper/123:00:201
298142600,2sleep30-21swapper/320:49:593
54162590,2sleep30-21swapper/321:33:133
27102580,1sleep00-21swapper/023:33:380
135392580,4sleep30-21swapper/323:17:123
244272570,1sleep30-21swapper/321:23:183
304342550,2sleep20-21swapper/221:27:342
198682550,2sleep10-21swapper/100:10:431
155762530,1sleep00-21swapper/022:28:450
224292520,2sleep30-21swapper/300:13:183
229382510,2sleep10-21swapper/121:45:271
209442490,2sleep3331rcuc/300:35:293
320552470,1sleep20-21swapper/220:55:122
319212470,2sleep20-21swapper/222:42:562
319212470,2sleep20-21swapper/222:42:552
66562450,2sleep00-21swapper/022:48:230
35242440,1sleep30-21swapper/322:20:013
51702430,1sleep30-21swapper/321:56:373
267772430,2sleep30-21swapper/300:15:423
206412430,2sleep30-21swapper/319:10:253
121242410,2sleep20-21swapper/223:15:462
290262310,3sleep20-21swapper/223:05:112
1918799319,10cyclictest22-21ksoftirqd/121:34:421
311082300,3chrt0-21swapper/023:05:450
269922300,2sleep00-21swapper/023:27:120
269922300,2sleep00-21swapper/023:27:120
1918799309,12cyclictest22-21ksoftirqd/122:45:011
1918799309,12cyclictest22-21ksoftirqd/122:45:011
1918799307,9cyclictest22-21ksoftirqd/100:05:171
246922290,3chrt0-21swapper/100:15:091
1918799287,4cyclictest22-21ksoftirqd/122:17:051
1918799281,25cyclictest0-21swapper/119:15:111
274702270,8sleep334-21ksoftirqd/321:25:203
1918799277,4cyclictest3831-21ls21:55:261
1918799277,12cyclictest141rcu_preempt00:36:541
1918799276,9cyclictest22-21ksoftirqd/123:12:191
1918799275,11cyclictest22-21ksoftirqd/100:31:341
19187992710,5cyclictest22-21ksoftirqd/123:51:381
1918699271,24cyclictest0-21swapper/019:15:150
149022270,1sleep30-21swapper/322:04:363
135002270,1sleep00-21swapper/023:17:080
244072260,2sleep00-21swapper/022:10:270
1918799268,4cyclictest22-21ksoftirqd/100:01:101
1918799267,4cyclictest22-21ksoftirqd/123:16:211
1918699261,11cyclictest22198-21ntp_kernel_pll_21:45:200
106752260,2sleep10-21swapper/123:39:531
20992250,1sleep30-21swapper/300:21:453
19189992520,3cyclictest21698-21df19:15:143
19189992519,3cyclictest5821-21diskmemload21:40:143
1918899251,23cyclictest0-21swapper/219:15:262
1918799256,8cyclictest22-21ksoftirqd/122:34:021
1918799255,10cyclictest22-21ksoftirqd/121:40:181
1918799254,8cyclictest22-21ksoftirqd/121:24:271
1918799253,13cyclictest22-21ksoftirqd/123:45:031
19187992519,5cyclictest22-21ksoftirqd/100:00:001
19187992518,5cyclictest987-21ssh23:08:051
19187992510,4cyclictest22-21ksoftirqd/121:39:201
175992250,1sleep00-21swapper/022:55:420
1918999243,15cyclictest8735-21ssh22:50:043
19188992422,1cyclictest0-21swapper/219:21:052
19188992412,8cyclictest648-21ntpd22:45:222
1918899241,1cyclictest0-21swapper/219:14:532
1918799248,4cyclictest22-21ksoftirqd/123:20:161
1918799248,3cyclictest141rcu_preempt22:59:401
1918799247,4cyclictest22-21ksoftirqd/121:28:301
1918799247,2cyclictest141rcu_preempt22:45:551
1918799246,7cyclictest22-21ksoftirqd/121:15:571
1918799246,4cyclictest22-21ksoftirqd/123:40:001
1918799245,10cyclictest22-21ksoftirqd/100:20:171
19187992417,4cyclictest52950irq/141-enp3s0-00:25:191
1918799241,6cyclictest22-21ksoftirqd/122:20:261
1918799240,12cyclictest22-21ksoftirqd/123:28:131
1918799240,12cyclictest22-21ksoftirqd/123:28:131
19186992417,5cyclictest20475-21ssh22:08:120
19186992413,2cyclictest31442-21/usr/sbin/munin00:20:260
226342230,1sleep30-21swapper/322:10:103
1918999235,15cyclictest21929-21kernelversion21:45:173
1918899232,13cyclictest5821-21diskmemload00:01:432
19188992314,7cyclictest20023-21ls00:35:192
19188992314,5cyclictest5821-21diskmemload23:40:142
1918799239,2cyclictest141rcu_preempt22:37:051
1918799238,9cyclictest22-21ksoftirqd/121:52:061
1918799235,8cyclictest22-21ksoftirqd/122:27:261
1918799234,8cyclictest22-21ksoftirqd/122:02:441
1918799234,11cyclictest22-21ksoftirqd/122:07:081
19187992321,1cyclictest0-21swapper/119:20:061
1918799231,5cyclictest22-21ksoftirqd/122:10:141
19186992312,8cyclictest1867-21netstat21:30:200
1918999227,10cyclictest21901-21kworker/u8:021:50:133
19189992211,7cyclictest53050irq/142-enp3s0-23:40:203
19188992214,6cyclictest18288-21ls20:20:212
19188992214,6cyclictest1555-21idleruntime21:30:182
19188992214,6cyclictest13715-21ntp_kernel_err00:05:222
1918899221,13cyclictest5821-21diskmemload23:25:492
1918899221,13cyclictest5821-21diskmemload23:25:482
19188992210,9cyclictest4589-21grep23:10:222
1918799228,8cyclictest22-21ksoftirqd/119:50:251
1918799228,3cyclictest22-21ksoftirqd/122:53:471
1918799225,9cyclictest22-21ksoftirqd/123:33:331
1918699226,3cyclictest141rcu_preempt23:57:590
1918699222,15cyclictest29423-21ssh22:15:100
19186992214,5cyclictest21142-21sensors_temp21:20:230
111592220,2sleep20-21swapper/220:05:172
1918999213,12cyclictest3358-21sh23:10:123
1918999212,13cyclictest29069-21ssh23:05:113
19189992114,4cyclictest3012-21switchtime19:45:253
19189992113,6cyclictest17133-21sed22:55:243
19189992113,5cyclictest605-21switchtime23:55:253
19189992112,6cyclictest53050irq/142-enp3s0-22:25:113
19189992111,7cyclictest53050irq/142-enp3s0-00:25:213
19189992111,7cyclictest2612-21sed22:45:173
1918899219,11cyclictest13980-21perf21:40:002
1918899212,12cyclictest5821-21diskmemload22:00:332
1918899211,3cyclictest141rcu_preempt23:55:132
19188992113,6cyclictest2302-21fw_packets19:45:172
19188992113,6cyclictest15171-21cron22:54:592
1918799215,7cyclictest22-21ksoftirqd/121:12:581
19187992115,5cyclictest22-21ksoftirqd/119:40:011
19187992113,6cyclictest31156-21ls20:50:271
19187992112,7cyclictest26627-21pluginstate19:25:211
1918799211,11cyclictest22-21ksoftirqd/120:34:251
1918699211,9cyclictest75-21kswapd022:50:030
19186992113,6cyclictest23175-21seq00:37:430
1918699210,18cyclictest28042-21expr20:45:100
1918999209,8cyclictest560-21dbus-daemon20:30:013
1918999201,9cyclictest22078-21munin-run22:35:003
19189992012,5cyclictest14630-21cron20:15:003
19189992011,7cyclictest8928-21tr21:35:243
19189992011,7cyclictest30290-21df19:35:153
19189992011,6cyclictest24307-21missed_timers19:20:193
19189992011,6cyclictest14111-21ntp_states21:15:223
19189992010,7cyclictest26199-21head19:25:153
1918899209,8cyclictest25676-21apt-get19:25:122
19188992012,6cyclictest6286-21grep21:10:132
19188992011,6cyclictest32073-21date19:40:002
1918899201,14cyclictest24617-21apt-get00:15:042
1918799208,2cyclictest141rcu_preempt20:10:211
1918799202,13cyclictest640-21snmpd20:05:131
1918799202,12cyclictest3939-21apt-get21:05:181
1918799202,12cyclictest3939-21apt-get21:05:181
1918799201,5cyclictest2997-21python319:45:251
19187992014,4cyclictest22-21ksoftirqd/120:05:001
19187992014,4cyclictest211rcuc/120:30:001
19187992013,5cyclictest22-21ksoftirqd/120:40:001
19187992011,7cyclictest32553-21perl19:40:151
19187992011,7cyclictest2143-21awk21:00:131
1918699201,5cyclictest13043-21apt-get21:15:130
19186992012,6cyclictest560-21dbus-daemon22:34:590
19186992012,6cyclictest29423-21seq19:31:370
19186992011,7cyclictest31519-21seq20:51:340
19186992011,7cyclictest20746-21cron21:44:590
19186992011,6cyclictest18067-21cat23:45:130
1918699201,14cyclictest4545-21ssh23:35:120
1918699201,10cyclictest246912ssh00:15:090
261252190,10sleep012-21ksoftirqd/022:38:520
1918999198,9cyclictest6884-21hddtemp_smartct21:10:173
1918999193,3cyclictest141rcu_preempt00:10:003
1918999192,12cyclictest27814-21apt-get19:30:123
1918999191,4cyclictest0-21swapper/300:30:123
19189991912,5cyclictest524-21ntpq20:55:213
19189991912,5cyclictest5190-21sed22:20:233
19189991912,5cyclictest23855-21sed23:00:263
19189991912,5cyclictest10529-21sed20:05:003
19189991912,4cyclictest5821-21diskmemload23:29:033
19189991912,4cyclictest5821-21diskmemload23:29:023
19189991911,6cyclictest3795-21date21:05:003
19189991911,6cyclictest16683-21awk22:05:183
1918999191,15cyclictest5821-21diskmemload23:51:413
1918999191,14cyclictest1585-21basename22:45:003
1918999191,14cyclictest1585-21basename22:44:593
19189991910,6cyclictest5101-21smartctl21:05:293
19189991910,6cyclictest5101-21smartctl21:05:283
1918899199,7cyclictest29784-21df22:15:122
1918899199,7cyclictest23089-21df_inode22:10:162
1918899198,9cyclictest0-21swapper/221:53:162
1918899197,6cyclictest28-21ksoftirqd/219:50:252
1918899192,6cyclictest14018-21ssh00:30:242
19188991914,3cyclictest24828-21sh21:23:432
19188991914,3cyclictest12042-21sh00:30:012
19188991912,5cyclictest28703-21ntp_kernel_pll_19:30:192
19188991912,5cyclictest12952-21expr20:10:132
19188991912,5cyclictest1057-21seq00:20:412
19188991912,4cyclictest5821-21diskmemload22:27:512
1918899191,1cyclictest0-21swapper/223:30:132
19188991911,6cyclictest5721-21seq21:57:112
19188991911,6cyclictest3916-21find21:05:172
19188991911,6cyclictest3916-21find21:05:162
19188991911,6cyclictest25944-21expr20:40:132
19188991911,6cyclictest24811-21cat20:35:272
19188991911,6cyclictest1936-21grep21:00:122
19188991911,5cyclictest28862-21cut23:05:002
19188991910,7cyclictest14801-21cat21:40:142
19188991910,6cyclictest4113-21cat22:20:142
1918899190,12cyclictest5821-21diskmemload22:33:452
1918799199,7cyclictest786-21python320:55:231
1918799197,3cyclictest141rcu_preempt20:20:171
19187991911,6cyclictest22-21ksoftirqd/120:00:001
19187991911,5cyclictest27174-21seq20:40:381
1918699197,3cyclictest12-21ksoftirqd/000:11:120
19186991913,4cyclictest12268-21sh00:30:020
19186991912,5cyclictest2922-21basename21:55:170
19186991912,5cyclictest26537-21ntp_kernel_pll_20:40:200
19186991912,5cyclictest16520-21seq20:16:510
19186991912,5cyclictest16520-21seq20:16:500
19186991911,6cyclictest6440-21cat00:25:140
19186991911,6cyclictest32688-21interrupts20:55:160
19186991911,6cyclictest25173-21seq19:21:580
19186991911,6cyclictest18709-21seq20:22:120
19186991911,6cyclictest14521-21seq20:13:370
19186991911,5cyclictest2660-21expr19:45:200
1918699191,15cyclictest23226-21ssh23:00:200
19186991910,6cyclictest22690-21timerwakeupswit20:30:250
19186991910,6cyclictest11338-21grep20:05:190
1918999189,6cyclictest6779-21sed19:55:193
1918999188,7cyclictest31799-21cron20:55:003
1918999184,10cyclictest18398-21ssh23:20:273
1918999181,8cyclictest9933-21ssh23:39:093
1918999181,4cyclictest141rcu_preempt23:30:213
1918999181,13cyclictest17973-21ssh23:45:123
19189991810,6cyclictest6026-21sort00:00:123
1918899189,6cyclictest8823-21cat20:00:152
1918899188,6cyclictest52750irq/139-enp3s0-22:05:112
1918899187,9cyclictest0-21swapper/221:16:552
19188991813,3cyclictest21159-21sh23:46:522
19188991812,4cyclictest22682-21sh22:35:242
19188991812,4cyclictest19565-21sh00:10:242
19188991811,5cyclictest4738-21expr23:35:142
19188991811,5cyclictest32094-21find19:40:122
19188991811,5cyclictest30120-21expr20:50:172
19188991811,5cyclictest28096-21df20:45:122
19188991811,5cyclictest21449-21grep20:30:132
19188991811,5cyclictest17977-21which23:20:242
19188991811,5cyclictest16633-21ntpq22:55:202
19188991811,5cyclictest14878-21find20:15:112
19188991811,5cyclictest14878-21find20:15:112
19188991810,6cyclictest21349-21ls21:45:132
1918799181,6cyclictest22-21ksoftirqd/119:10:171
1918799181,5cyclictest0-21swapper/119:30:221
19187991813,4cyclictest22-21ksoftirqd/120:20:001
19187991813,4cyclictest22-21ksoftirqd/120:20:001
1918799180,4cyclictest0-21swapper/120:47:241
1918699189,7cyclictest2079-21ls21:00:130
1918699186,10cyclictest0-21swapper/022:20:160
1918699181,5cyclictest8019-21taskset00:00:440
19186991813,3cyclictest7024-21ntp_offset19:55:210
19186991811,5cyclictest7173-21ls21:10:190
19186991811,5cyclictest3382-21cpu23:10:130
19186991811,5cyclictest31333-21unixbench-2d19:35:250
19186991811,5cyclictest13555-21ls00:05:200
19186991811,4cyclictest5821-21diskmemload23:20:100
1918699181,14cyclictest52850irq/140-enp3s0-22:00:120
19186991810,6cyclictest5932-21sort21:10:010
19186991810,6cyclictest5932-21sort21:10:000
19186991810,6cyclictest32640-21head19:40:170
120212180,3sleep30-21swapper/320:05:263
1918999178,7cyclictest25612-21/usr/sbin/munin20:40:203
1918999178,6cyclictest18499-21grep20:20:233
19188991712,3cyclictest19277-21cpuspeed_turbos20:25:142
19188991710,5cyclictest6730-21cat19:55:172
1918699179,6cyclictest20853-21seq19:11:460
1918699171,2cyclictest19200-21apt-get20:25:130
19186991712,3cyclictest24129-21interrupts20:35:190
19186991711,4cyclictest8649-21cstates20:00:130
19186991711,4cyclictest29443-21sh21:26:350
19186991711,4cyclictest27643-21sh22:40:090
19186991711,4cyclictest27643-21sh22:40:080
19186991710,5cyclictest26354-21tail23:50:260
19186991710,5cyclictest26098-21tune2fs19:25:140
1918999168,6cyclictest15384-21grep20:15:163
1918999168,6cyclictest15384-21grep20:15:153
19189991611,3cyclictest32091-21/usr/sbin/munin19:40:173
1918999161,12cyclictest5799-21sshd19:54:233
19189991610,4cyclictest0-21swapper/320:35:013
19186991611,3cyclictest636-21runrttasks19:51:130
1918999159,4cyclictest25354-21cron20:40:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional