You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-08 - 22:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Mon Jun 08, 2026 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8552187111,28sleep00-21swapper/007:06:190
9492171102,62sleep20-21swapper/207:07:352
9932170101,28sleep10-21swapper/107:08:091
10672162112,27sleep30-21swapper/307:09:073
209642800,2chrt0-21swapper/009:33:340
112882700,8sleep122-21ksoftirqd/108:45:141
196432670,2sleep20-21swapper/209:55:552
126812610,2sleep30-21swapper/309:50:383
278342600,2sleep10-21swapper/109:14:531
63452580,2sleep20-21swapper/210:10:192
69002540,2sleep1211rcuc/111:47:101
61772510,1sleep00-21swapper/010:58:450
105492500,2sleep2560-21dbus-daemon08:44:592
35562490,2sleep334-21ksoftirqd/311:20:223
265702480,2sleep30-21swapper/311:14:003
238272480,2sleep10-21swapper/111:11:141
213642480,2sleep10-21swapper/111:34:251
11412470,2sleep10-21swapper/112:30:481
35692440,1sleep1211rcuc/109:44:581
193832440,2sleep20-21swapper/211:56:132
17812440,2sleep10-21swapper/110:30:291
294772420,1sleep30-21swapper/312:28:213
285832420,1sleep30-21swapper/310:03:103
10622420,2chrt0-21swapper/111:43:051
227632400,2chrt0-21swapper/309:10:213
151832400,2sleep122-21ksoftirqd/111:05:171
157572390,1sleep20-21swapper/211:54:222
244762330,2chrt0-21swapper/210:24:112
41912320,2sleep10-21swapper/110:09:291
85582310,2chrt0-21swapper/309:48:133
1417993012,11cyclictest53050irq/142-enp3s0-10:15:023
141599281,25cyclictest0-21swapper/107:40:291
141499281,25cyclictest0-21swapper/007:35:190
1417992719,6cyclictest24777-21/usr/sbin/munin08:00:213
1417992719,6cyclictest24777-21/usr/sbin/munin08:00:203
141799271,24cyclictest0-21swapper/307:39:273
141699271,24cyclictest0-21swapper/208:18:052
141699270,24cyclictest15143-21cat07:40:142
141599271,24cyclictest0-21swapper/108:17:081
141599271,24cyclictest0-21swapper/107:45:451
141599271,24cyclictest0-21swapper/107:37:211
141599271,19cyclictest0-21swapper/107:30:221
141499272,17cyclictest636-21runrttasks11:25:100
141499271,25cyclictest0-21swapper/008:19:080
141499271,24cyclictest0-21swapper/007:45:500
299212260,2sleep228-21ksoftirqd/212:28:492
1417992617,6cyclictest9953-21sed08:40:253
141799261,23cyclictest0-21swapper/308:19:033
141799261,23cyclictest0-21swapper/307:45:423
141799261,23cyclictest0-21swapper/307:40:133
141699261,23cyclictest0-21swapper/207:37:302
141699261,23cyclictest0-21swapper/207:30:202
1414992624,1cyclictest0-21swapper/008:26:210
1414992624,1cyclictest0-21swapper/007:41:090
263942250,1sleep20-21swapper/211:37:432
141799251,22cyclictest0-21swapper/308:26:053
141699251,23cyclictest0-21swapper/208:26:292
1415992523,1cyclictest0-21swapper/107:26:181
141599251,1cyclictest0-21swapper/108:25:451
1414992523,1cyclictest0-21swapper/007:30:470
1414992518,5cyclictest16859-21ssh09:54:520
1414992514,8cyclictest22830-21gdbus09:35:010
141499251,1cyclictest0-21swapper/007:29:500
89202240,3sleep08916-21ssh11:00:210
1416992422,1cyclictest0-21swapper/207:46:192
141499249,6cyclictest10096-21find11:50:090
1414992414,6cyclictest52850irq/140-enp3s0-12:33:150
177872230,2sleep00-21swapper/010:43:030
141799237,9cyclictest141rcu_preempt12:39:593
141699238,3cyclictest2326-21proc_pri10:55:222
1415992313,7cyclictest7098-21df_inode10:35:151
1414992315,5cyclictest636-21runrttasks09:45:220
11382230,1sleep00-21swapper/009:18:530
320172220,1sleep20-21swapper/211:41:372
141799226,8cyclictest0-21swapper/311:46:133
1417992213,7cyclictest1983-21ssh10:30:403
141799221,12cyclictest34-21ksoftirqd/312:10:423
1416992220,1cyclictest0-21swapper/207:26:382
1416992214,6cyclictest5763-21users09:45:262
141499228,5cyclictest12-21ksoftirqd/010:05:430
1414992212,7cyclictest7422-21ntp_states08:35:210
141799218,6cyclictest21006-21wc11:10:013
141799217,3cyclictest34-21ksoftirqd/312:30:123
141799216,3cyclictest66050lldpd08:07:013
141799216,11cyclictest15251-21kworker/u8:009:35:033
141799213,10cyclictest141rcu_preempt11:32:103
1417992113,6cyclictest12132-21seq11:26:553
1417992112,7cyclictest11850-21users11:50:243
1417992112,6cyclictest560-21dbus-daemon09:10:003
1417992111,8cyclictest34-21ksoftirqd/312:20:003
141799211,16cyclictest21267-21diskmemload10:40:123
141699212,13cyclictest31553-21ssh12:30:112
141699212,13cyclictest18070-21apt-get12:20:042
1416992114,5cyclictest2301-21sendmail09:20:012
1415992113,6cyclictest18091-21smartctl09:55:171
1415992113,6cyclictest14930-21ls10:40:251
1415992112,7cyclictest4933-21diskstats12:10:131
1415992112,7cyclictest14370-21cat11:05:011
141599211,18cyclictest814-21ssh12:06:411
141599211,10cyclictest23046-21/usr/sbin/munin09:35:111
141499219,10cyclictest20336-21cstates10:45:130
1414992114,4cyclictest636-21runrttasks11:56:010
1414992113,6cyclictest26355-21ls12:25:240
1414992112,6cyclictest20610-21grep10:20:240
141499211,14cyclictest11299-21ssh12:15:120
311572200,1sleep30-21swapper/309:40:233
244632200,2chrt0-21swapper/212:00:192
244632200,2chrt0-21swapper/212:00:192
141799208,10cyclictest29119-21ssh11:40:123
141799207,8cyclictest23582-21ssh12:00:133
141799207,8cyclictest23582-21ssh12:00:123
141799204,3cyclictest141rcu_preempt09:59:533
141799203,13cyclictest17575-21apt-get12:20:043
1417992018,1cyclictest0-21swapper/307:30:393
141799201,5cyclictest0-21swapper/310:20:493
1417992013,4cyclictest11670-21sh10:38:423
1417992012,6cyclictest8847-21ls11:00:203
1417992012,6cyclictest29882-21pluginstate08:15:013
1417992012,6cyclictest15031-21date08:55:013
1417992012,5cyclictest9817-21sed07:25:263
1417992011,6cyclictest17915-21ntp_states11:55:223
1417992010,8cyclictest7497-21rm09:23:303
1417992010,8cyclictest30807-21ssh11:16:323
1417992010,8cyclictest19129-21basename09:05:013
1417992010,8cyclictest1573-21ssh09:19:183
1417992010,8cyclictest10389-21rm10:13:463
141699207,4cyclictest21267-21diskmemload11:21:482
141699202,15cyclictest5156-21head11:45:262
141699202,13cyclictest5839-21apt-get12:35:032
1416992015,3cyclictest2630-21switchtime07:10:242
1416992015,3cyclictest2630-21switchtime07:10:232
1416992014,4cyclictest16140-21sh10:17:352
1416992014,3cyclictest28049-21sh09:38:452
1416992013,5cyclictest22053-21ssh10:45:362
1416992013,4cyclictest3690-21sh10:32:252
1416992013,4cyclictest29057-21sh10:03:392
1416992013,4cyclictest23579-21sh09:10:342
1416992013,4cyclictest15473-21sh09:29:462
1416992012,6cyclictest22304-21ls07:55:192
1416992012,6cyclictest10239-21sensors_temp11:25:212
1416992012,5cyclictest14143-21sed08:50:232
1416992011,6cyclictest26733-21awk08:05:212
141699201,13cyclictest27988-21ssh11:15:102
1416992010,6cyclictest17173-21grep11:30:262
1415992016,3cyclictest29751-21ssh12:04:551
1415992016,3cyclictest29751-21ssh12:04:551
1415992014,4cyclictest9010-21sh09:25:001
1415992014,4cyclictest6293-21cpu07:20:121
1415992012,6cyclictest9589-21expr09:25:121
1415992012,6cyclictest25234-21cat12:25:141
1415992012,6cyclictest11524-21seq12:39:141
141499202,15cyclictest15214-21tr08:55:130
1414992015,3cyclictest15706-21latency_hist09:30:010
1414992013,4cyclictest9594-21sh12:13:440
1414992012,6cyclictest6655-21cat12:35:160
1414992012,6cyclictest17702-21fw_packets09:00:150
1414992010,7cyclictest15996-21df_inode11:30:140
141499200,5cyclictest0-21swapper/007:53:150
141799198,6cyclictest53050irq/142-enp3s0-10:45:123
141799195,4cyclictest34-21ksoftirqd/310:05:183
141799195,3cyclictest34-21ksoftirqd/310:51:203
141799193,11cyclictest25702-21ssh10:25:123
141799193,11cyclictest25702-21ssh10:25:113
1417991914,3cyclictest7370-21sendmail_mailtr07:20:233
1417991911,6cyclictest4337-21find08:30:103
1417991911,6cyclictest10633-21processes09:25:213
1417991910,7cyclictest22380-21awk11:35:123
141699199,7cyclictest560-21dbus-daemon09:05:002
141699199,7cyclictest11192-21uniq09:50:142
141699199,7cyclictest10800-21apt-get08:45:112
141699192,12cyclictest2716-21ssh09:20:112
141699192,12cyclictest25687-21apt-get10:25:112
141699192,12cyclictest25687-21apt-get10:25:102
1416991913,4cyclictest8780-21sh10:35:452
1416991913,4cyclictest22416-21munin-plugin-st09:34:592
1416991913,4cyclictest16394-21sh12:18:502
1416991913,4cyclictest1348-21sh09:42:432
1416991913,3cyclictest32242-21sh12:05:402
1416991913,3cyclictest10162-21sh11:00:592
1416991912,5cyclictest31033-21cat10:05:142
1416991912,5cyclictest19651-21cat09:05:142
1416991912,4cyclictest29502-21sh10:51:222
1416991911,6cyclictest6089-21date07:20:002
1416991911,6cyclictest1424-21seq08:20:502
141599194,3cyclictest50122irqstats08:30:171
141599192,12cyclictest4042-21apt-get07:15:041
141599192,12cyclictest10150-21ssh11:50:121
141599191,9cyclictest1597550ssh10:17:241
141599191,8cyclictest19006-21kernelversion12:20:191
1415991914,3cyclictest16766-21hwlatdetect09:30:151
1415991913,4cyclictest30869-21sh11:16:351
1415991913,4cyclictest10383-21sh10:13:461
1415991912,5cyclictest21266-21ls10:45:221
1415991911,6cyclictest8967-21apt11:25:091
1415991911,6cyclictest27892-21uname10:50:211
1415991911,6cyclictest26873-21cut10:25:221
1415991911,6cyclictest26873-21cut10:25:211
1415991911,6cyclictest24949-21threads08:00:231
1415991911,6cyclictest24949-21threads08:00:221
1415991911,6cyclictest22886-21cat07:55:261
1415991911,6cyclictest19145-21expr10:20:121
1415991911,6cyclictest1877-21cut07:10:161
1415991911,6cyclictest1877-21cut07:10:151
1415991911,6cyclictest1648-21seq08:23:441
1415991911,6cyclictest14973-21cat08:55:011
141599191,14cyclictest52950irq/141-enp3s0-10:55:131
1415991910,7cyclictest15410-21df_inode08:55:141
141499199,6cyclictest17994-21grep09:55:170
141499192,12cyclictest75-21kswapd011:05:120
141499192,12cyclictest29099-21apt-get11:40:130
141499192,12cyclictest19376-21apt-get09:05:120
1414991914,3cyclictest8637-21/usr/sbin/munin08:40:170
1414991913,4cyclictest23933-21sh12:00:140
1414991913,4cyclictest23933-21sh12:00:130
1414991913,4cyclictest15534-21sh10:16:580
1414991911,6cyclictest31405-21pluginstate12:05:220
1414991911,6cyclictest28970-21awk11:15:190
1414991911,6cyclictest24664-21seq11:12:060
1414991911,6cyclictest1714-21expr07:10:140
1414991911,6cyclictest1714-21expr07:10:130
1414991911,6cyclictest14863-21cut08:55:010
141799182,8cyclictest141rcu_preempt09:30:193
141799182,11cyclictest30291-21apt-get12:05:133
1417991813,3cyclictest6479-21/usr/sbin/munin08:35:213
1417991812,4cyclictest3651-21sh10:56:133
1417991811,5cyclictest2852-21grep07:10:263
1417991811,5cyclictest2852-21grep07:10:253
1416991813,3cyclictest6298-21cpuspeed_turbos07:20:132
1416991812,4cyclictest6735-21sh12:10:532
1416991812,4cyclictest16898-21sh11:06:002
1416991812,4cyclictest16226-21sh10:41:302
1416991812,3cyclictest21407-21apt-config07:55:012
1416991811,5cyclictest23883-21ls08:00:122
1416991811,5cyclictest23883-21ls08:00:112
1416991811,5cyclictest21528-21cat11:10:122
1416991810,6cyclictest6590-21grep08:35:132
141599189,6cyclictest52950irq/141-enp3s0-09:50:121
141599187,9cyclictest0-21swapper/110:00:271
1415991813,3cyclictest32172-21sh09:17:281
1415991813,3cyclictest25284-21sh11:36:371
1415991812,4cyclictest5617-21sh11:22:001
1415991811,5cyclictest29629-21seq08:12:041
1415991811,4cyclictest21267-21diskmemload09:45:211
141599181,12cyclictest11278-21apt-get12:15:131
1415991810,6cyclictest18977-21seq09:03:161
141499189,6cyclictest52850irq/140-enp3s0-12:20:140
141499185,6cyclictest10895-21cpuspeed_turbos08:45:130
1414991814,2cyclictest3423-21/usr/sbin/munin11:45:200
1414991813,3cyclictest5751-21sh11:22:090
1414991813,3cyclictest31049-21sh10:52:550
1414991813,3cyclictest30309-21sh10:28:200
1414991813,3cyclictest30309-21sh10:28:200
1414991812,4cyclictest31197-21timerandwakeup09:40:230
1414991812,3cyclictest21267-21diskmemload10:30:160
1414991811,5cyclictest6556-21/usr/sbin/munin10:10:220
1414991811,5cyclictest25831-21sort08:05:120
1414991810,6cyclictest6148-21sed07:20:000
141799179,6cyclictest12741-21cat08:50:003
1417991713,2cyclictest341ksoftirqd/309:00:003
1417991712,3cyclictest4731-21irqstats07:15:183
1417991710,5cyclictest21623-21cat07:55:143
1417991710,5cyclictest19294-21expr07:50:123
1416991710,5cyclictest28640-21kernelversion08:10:162
1416991710,5cyclictest17068-21cut09:00:012
141599179,6cyclictest9255-21idleruntime08:40:171
141599179,6cyclictest7159-21kernelversion08:35:171
141599179,6cyclictest21017-21seq07:54:131
141599171,9cyclictest20537-21python309:05:221
1415991712,3cyclictest17895-21sh11:55:211
141499179,6cyclictest25602-21pluginstate08:05:010
141499179,6cyclictest25602-21pluginstate08:05:010
141499176,9cyclictest0-21swapper/011:37:370
141499172,10cyclictest3906-21apt-get08:30:030
1414991713,2cyclictest6688-21/usr/sbin/munin10:35:170
1414991712,3cyclictest21267-21diskmemload09:20:180
1414991712,3cyclictest1767-21latency_hist08:25:010
1414991711,3cyclictest21267-21diskmemload09:10:030
141799169,5cyclictest32546-21expr08:20:133
141699168,6cyclictest5668-21cat08:30:232
1415991611,3cyclictest26780-21sh08:05:211
141499169,5cyclictest28751-21wc08:10:180
141499169,5cyclictest22857-21sed07:55:260
141499162,8cyclictest6242-21apt-get07:20:130
141499161,6cyclictest24089-21/usr/sbin/munin10:00:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional