You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-29 - 02:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Fri Aug 28, 2026 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3972217497,26sleep20-21swapper/207:06:472
4055217299,25sleep30-21swapper/307:07:513
41522160147,7sleep10-21swapper/107:09:071
38722140105,24sleep00-21swapper/007:05:280
3832590,3sleep228-21ksoftirqd/209:40:152
215292580,2sleep30-21swapper/311:55:203
266622570,2sleep20-21swapper/211:10:402
31072560,2sleep10-21swapper/111:41:291
125892530,3sleep34491-21cyclictest07:25:253
82992490,3sleep00-21swapper/010:57:420
169032480,1sleep10-21swapper/111:51:441
27152460,1sleep20-21swapper/211:41:062
282882450,2sleep30-21swapper/312:00:173
292452420,2sleep00-21swapper/011:13:150
69042390,3chrt0-21swapper/312:32:033
224482390,1sleep00-21swapper/011:31:530
173512380,1chrt0-21swapper/011:52:110
118822360,2chrt0-21swapper/012:11:530
80152340,1sleep30-21swapper/310:10:013
63442290,3sleep26302-21gltestperf11:20:152
42232280,3chrt0-21swapper/112:30:171
127102280,2chrt0-21swapper/107:25:271
84472270,2sleep00-21swapper/009:45:250
190102270,3sleep30-21swapper/310:41:143
449499261,24cyclictest0-21swapper/107:14:181
449399261,2cyclictest0-21swapper/010:40:120
449399261,24cyclictest0-21swapper/007:13:370
449599252,16cyclictest515-21irqbalance07:30:122
449599251,23cyclictest0-21swapper/207:13:202
4495992510,13cyclictest8565-21sed11:45:222
168462250,2chrt0-21swapper/009:28:410
4496992422,1cyclictest0-21swapper/307:13:243
4496992414,7cyclictest17782-21df08:55:123
43712240,3sleep223654-21diskmemload10:30:232
292862240,1sleep10-21swapper/109:37:251
265132240,11sleep334-21ksoftirqd/312:22:443
12342240,2sleep10-21swapper/112:04:441
4496992314,7cyclictest20081-21perl09:00:163
449699231,10cyclictest161492cut07:35:183
449599234,11cyclictest24078-21ssh09:10:122
4494992311,7cyclictest52850irq/140-enp3s0-10:40:131
449399232,13cyclictest5247-21ssh10:31:000
175692230,12sleep228-21ksoftirqd/210:40:222
4496992214,6cyclictest28120-21cron10:25:003
4496992213,7cyclictest30940-21users12:25:253
4496992213,7cyclictest29596-21smartctl08:05:233
4496992213,7cyclictest29596-21smartctl08:05:223
4496992213,6cyclictest3259-21df_inode10:30:133
4496992212,7cyclictest27462-21df10:00:143
449599222,13cyclictest640-21snmpd09:00:132
4495992210,8cyclictest6150-21ntpq09:20:192
449499227,5cyclictest0-21swapper/107:35:171
449499223,14cyclictest15678-21ssh12:15:121
449499222,4cyclictest141rcu_preempt11:00:101
449499221,8cyclictest141rcu_preempt10:49:231
4494992214,5cyclictest10631-21sendmail11:00:001
4494992211,8cyclictest15037-21http12:15:011
449399229,8cyclictest12-21ksoftirqd/012:15:240
4493992214,6cyclictest2900-21date10:05:240
4493992213,7cyclictest29594-21python308:05:230
4493992213,7cyclictest29594-21python308:05:230
449399221,13cyclictest23654-21diskmemload10:23:430
43222220,2sleep20-21swapper/212:30:182
317202220,1sleep30-21swapper/309:39:503
250762220,3sleep125238-21head11:10:161
449699212,14cyclictest5142-21ssh09:20:123
449699212,14cyclictest14371-21kworker/u8:111:40:113
449699212,13cyclictest26036-21ssh11:35:133
4496992113,6cyclictest53050irq/142-enp3s0-10:49:153
4496992113,6cyclictest18064-21diskstats07:40:143
4496992112,6cyclictest560-21dbus-daemon09:30:003
4496992112,6cyclictest25327-21which09:10:223
449599213,10cyclictest640-21snmpd11:50:142
449599212,12cyclictest8199-21ssh10:10:082
4495992114,5cyclictest12227-21processes07:25:212
4495992113,6cyclictest29261-21ls08:05:192
4495992113,6cyclictest29261-21ls08:05:182
4495992112,7cyclictest15719-21cut08:50:162
4495992111,7cyclictest13584-21tune2fs08:45:162
449499219,8cyclictest31429-21/usr/sbin/munin11:15:231
449499215,4cyclictest141rcu_preempt12:35:181
449499213,5cyclictest0-21swapper/112:23:221
449499212,3cyclictest0-21swapper/110:15:171
449499212,16cyclictest22-21ksoftirqd/109:45:211
4494992115,4cyclictest22-21ksoftirqd/110:05:011
4494992114,5cyclictest21976-21unixbench_singl09:55:251
4494992113,6cyclictest11089-21ls10:35:231
4494992111,7cyclictest20079-21grep11:30:181
449399217,11cyclictest0-21swapper/010:50:010
449399213,16cyclictest17346-21latency_hist11:05:000
449399212,13cyclictest15469-21apt-get07:35:130
4493992116,3cyclictest8813-21grep10:10:120
4493992116,3cyclictest8261-21turbostat.cron12:10:000
4493992113,6cyclictest30080-21date10:25:240
4493992110,6cyclictest12-21ksoftirqd/010:51:010
449399210,13cyclictest23654-21diskmemload12:30:450
4496992013,5cyclictest1477-21ls09:40:243
4496992013,4cyclictest3047-21sh11:17:433
4496992012,6cyclictest25719-21ntpq11:10:203
4496992011,7cyclictest11781-21seq09:48:433
4496992011,7cyclictest11397-21df11:00:133
4496992011,6cyclictest53050irq/142-enp3s0-11:50:153
4496992011,6cyclictest53050irq/142-enp3s0-11:25:113
4496992011,6cyclictest11297-21awk10:35:253
449699200,6cyclictest1255850ssh10:12:593
449599206,5cyclictest28-21ksoftirqd/212:25:162
449599203,10cyclictest640-21snmpd08:40:112
4495992013,5cyclictest30918-21grep10:50:192
4495992012,6cyclictest13933-21tune2fs09:50:152
4495992011,7cyclictest26300-21apt-get08:00:112
449599201,16cyclictest32285-21latency_hist11:40:002
4495992011,6cyclictest278-21systemd-journal07:45:002
449499209,2cyclictest22-21ksoftirqd/112:30:001
449499208,8cyclictest30805-21apt-get09:15:121
449499207,3cyclictest141rcu_preempt10:30:001
449499206,2cyclictest141rcu_preempt10:50:131
449499205,3cyclictest141rcu_preempt08:55:151
449499202,14cyclictest2914-21ssh10:30:101
449499202,13cyclictest22003-21apt-get07:50:121
449499201,6cyclictest20606-21head09:00:221
449499201,4cyclictest0-21swapper/111:35:391
4494992014,4cyclictest12929-21sh08:45:001
4494992014,3cyclictest18734-21needreboot11:05:211
4494992014,3cyclictest18734-21needreboot11:05:201
4494992010,6cyclictest52850irq/140-enp3s0-10:05:231
449399209,5cyclictest12-21ksoftirqd/011:41:010
449399201,5cyclictest0-21swapper/009:19:590
4493992013,5cyclictest52950irq/141-enp3s0-09:30:100
4493992012,6cyclictest6201-21open_inodes09:20:190
4493992012,6cyclictest21155-21seq09:00:400
4493992011,7cyclictest1225-21python309:40:220
4493992010,8cyclictest7196-21threads11:20:230
4493992010,7cyclictest22694-21latency07:50:180
304332200,4sleep0131rcuc/012:25:220
280682200,1sleep00-21swapper/012:24:180
196782201,16sleep10-21swapper/109:54:501
127172200,2sleep20-21swapper/209:25:182
449699192,11cyclictest17774-21apt-get11:05:123
449699192,11cyclictest17774-21apt-get11:05:123
449699191,4cyclictest0-21swapper/308:15:213
4496991913,4cyclictest53050irq/142-enp3s0-11:23:003
4496991913,4cyclictest20734-21sh09:30:373
4496991913,4cyclictest15251-21sh12:38:383
4496991913,3cyclictest3637-21uptime12:05:253
4496991911,6cyclictest7506-21ls07:15:193
4496991911,6cyclictest31371-21expr09:15:163
4496991911,6cyclictest28283-21date08:05:013
4496991911,6cyclictest20203-21expr11:30:183
4496991911,6cyclictest15647-21df08:50:153
4496991911,6cyclictest15205-21cron07:35:003
4496991911,6cyclictest10953-21seq12:10:583
4496991911,6cyclictest10764-21cron08:40:003
4496991910,6cyclictest53050irq/142-enp3s0-09:50:103
4496991910,6cyclictest31153-21grep08:10:163
4496991910,3cyclictest29982-21cpu10:50:133
449599192,10cyclictest23200-21ssh10:45:122
449599191,9cyclictest17871-21ssh12:15:552
449599191,6cyclictest640-21snmpd11:55:542
449599191,6cyclictest113212rm09:48:152
4495991913,4cyclictest32579-21sh12:03:402
4495991911,6cyclictest3253-21python312:05:222
4495991911,6cyclictest25740-21expr09:35:102
4495991911,6cyclictest23438-21seq09:07:332
4495991910,6cyclictest278-21systemd-journal11:20:002
4495991910,6cyclictest10220-21cat12:10:242
449499198,9cyclictest17806-21ssh11:28:591
449499198,7cyclictest52850irq/140-enp3s0-11:45:091
449499198,3cyclictest22-21ksoftirqd/110:12:561
449499192,5cyclictest0-21swapper/109:32:331
449499191,5cyclictest0-21swapper/108:20:211
449499191,4cyclictest0-21swapper/108:32:431
4494991913,3cyclictest25696-21sh11:58:471
4494991911,6cyclictest22850-21which09:05:211
4494991911,6cyclictest18714-21awk07:40:211
4494991910,6cyclictest6188-21awk09:20:201
4494991910,6cyclictest24407-21ls07:55:131
449399199,6cyclictest52950irq/141-enp3s0-10:35:110
449399197,4cyclictest12-21ksoftirqd/011:25:150
449399195,9cyclictest52950irq/141-enp3s0-09:55:050
449399193,4cyclictest0-21swapper/007:25:160
449399193,10cyclictest12-21ksoftirqd/008:55:240
449399192,11cyclictest15324-21apt-get08:50:130
449399191,9cyclictest0-21swapper/011:16:080
449399191,4cyclictest10280-21apt-get12:35:110
4493991914,3cyclictest23916-21sh11:57:000
4493991914,3cyclictest19468-21unixbench_singl11:05:270
4493991914,3cyclictest19468-21unixbench_singl11:05:260
4493991913,4cyclictest14357-21switchtime08:45:240
4493991913,4cyclictest14157-21needreboot07:30:180
449399191,2cyclictest0-21swapper/010:17:430
4493991912,5cyclictest4867-21expr08:25:150
4493991912,5cyclictest31347-21seq12:02:270
4493991912,4cyclictest30073-21sh10:01:390
4493991912,4cyclictest14354-21sh09:50:180
4493991911,6cyclictest7043-21apt11:45:090
4493991911,6cyclictest18291-21smartctl07:40:160
4493991911,5cyclictest7921-21find07:15:230
4493991910,7cyclictest9124-21cat08:35:140
4493991910,7cyclictest2805-21cat08:20:150
4493991910,6cyclictest24461-21df07:55:120
449699189,6cyclictest20443-21hwlatdetect07:45:163
449699188,8cyclictest21373-21open_files09:55:213
449699182,11cyclictest4612-21apt-get08:25:143
449699182,11cyclictest28561-21apt-get10:25:133
449699182,11cyclictest11047-21apt-get08:40:133
449699181,7cyclictest15459-21taskset12:15:023
4496991812,4cyclictest9493-21kernelversion07:20:173
4496991811,5cyclictest7131-21cat11:45:113
4496991811,5cyclictest6976-21expr08:30:113
4496991811,5cyclictest4676-21sort10:55:133
4496991811,5cyclictest22506-21load09:05:173
4496991811,4cyclictest17304-21sh10:15:563
449599189,7cyclictest11403-21df_inode11:00:132
449599189,6cyclictest18943-21grep08:55:252
449599188,7cyclictest20799-21awk07:45:212
449599185,3cyclictest23654-21diskmemload11:27:592
4495991813,3cyclictest9816-21pmu-power07:20:222
4495991813,3cyclictest25045-21sh11:34:292
4495991813,3cyclictest23219-21switchtime10:20:242
4495991813,3cyclictest22659-21sh09:56:032
4495991812,4cyclictest22095-21sh09:31:582
4495991811,5cyclictest15841-21tune2fs07:35:152
449599181,14cyclictest16655-21apt12:39:592
449599181,12cyclictest6684-21chrt07:15:122
449599181,11cyclictest640-21snmpd10:00:132
4495991810,6cyclictest21271-21seq11:07:132
4495991810,6cyclictest21271-21seq11:07:132
4495991810,5cyclictest30234-21awk10:25:262
449499188,2cyclictest22-21ksoftirqd/108:05:231
449499188,2cyclictest22-21ksoftirqd/108:05:231
449499182,3cyclictest141rcu_preempt12:05:101
449499182,12cyclictest24825-21ssh09:10:181
4494991812,4cyclictest23654-21diskmemload11:20:321
4494991812,3cyclictest6512-21idleruntime-cro08:30:011
4494991812,3cyclictest25965-21sh10:22:501
4494991811,5cyclictest10889-21sed08:40:001
4494991811,4cyclictest15212-21sh07:35:011
4494991810,6cyclictest8560-21cron07:24:541
4494991810,5cyclictest1579-21grep09:40:231
4494991810,5cyclictest13367-21timerwakeupswit09:25:231
4493991813,3cyclictest7376-21if_err_enp3s008:30:150
4493991813,3cyclictest28443-21sh09:36:330
4493991813,3cyclictest25019-21sh09:10:200
449399181,2cyclictest0-21swapper/011:35:140
4493991811,5cyclictest21505-21seq07:47:000
4493991811,5cyclictest11330-21expr08:40:130
4493991810,6cyclictest9744-21ntpq07:20:200
449399180,12cyclictest30629-21/usr/sbin/munin08:10:130
449699178,6cyclictest22463-21perl07:50:153
449699176,8cyclictest2790-21df_inode08:20:133
4496991712,3cyclictest24114-21/usr/sbin/munin07:55:203
449599179,6cyclictest950-21cut08:15:172
449599179,6cyclictest8862-21/usr/sbin/munin08:35:222
449599179,5cyclictest5957-21timerwakeupswit08:25:252
449599176,8cyclictest560-21dbus-daemon08:25:012
4495991712,3cyclictest6720-21/usr/sbin/munin08:30:182
4495991712,3cyclictest21977-21/usr/sbin/munin07:50:142
449599171,12cyclictest31875-21ntp_states09:15:222
449599171,11cyclictest1479-21apt-get10:05:132
4495991710,5cyclictest24036-21date12:20:242
4495991710,5cyclictest15330-21expr10:15:162
4495991710,5cyclictest10453-21cat10:35:162
449499178,6cyclictest1724-21grep08:15:251
449499177,7cyclictest560-21dbus-daemon07:20:001
449499176,2cyclictest22-21ksoftirqd/108:45:131
4494991710,5cyclictest30954-21grep08:10:141
449499170,5cyclictest0-21swapper/108:50:151
449399179,6cyclictest26703-21grep08:00:150
4493991710,5cyclictest22475-21ls09:05:160
449699161,10cyclictest13186-21apt-get08:45:123
449599161,12cyclictest4239-21/usr/sbin/munin10:55:172
449599161,10cyclictest24139-21apt-get07:55:122
449499169,5cyclictest21194-21sed07:45:251
4493991611,3cyclictest684-21fschecks_count08:15:140
449599157,6cyclictest31988-21grep08:10:252
449499154,9cyclictest27683-21sed08:00:231
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional