You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-07 - 12:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Jun 07, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6652237163,24sleep20-21swapper/219:09:102
309862237164,23sleep00-21swapper/019:05:010
6912202166,24sleep30-21swapper/319:09:323
6892201166,23sleep10-21swapper/119:09:291
537321610,5sleep086299cyclictest00:10:220
136821370,4sleep287799cyclictest00:00:262
171092600,4sleep317110-21grep20:55:253
87982560,4sleep086299cyclictest23:05:180
160512450,1sleep20-21swapper/222:10:012
86299379,12cyclictest3443-21idleruntime-cro22:55:020
86299345,14cyclictest13806-21munin-run22:05:000
86299337,13cyclictest29187-21munin-run22:40:020
88499321,16cyclictest30842-21irqstats23:55:193
86899327,2cyclictest0-21swapper/121:05:271
86899327,2cyclictest0-21swapper/121:05:271
86899321,16cyclictest0-21swapper/119:49:321
86299328,11cyclictest10991-21awk20:45:010
884993011,9cyclictest419-21chrt21:34:103
88499300,21cyclictest0-21swapper/319:21:413
88499300,12cyclictest0-21swapper/322:35:193
86899301,23cyclictest0-21swapper/100:14:081
86299302,13cyclictest26382-21awk21:20:000
88499291,12cyclictest27564-21users20:05:273
88499291,10cyclictest0-21swapper/319:42:593
86899291,17cyclictest30197-21grep21:25:251
86299294,12cyclictest21361-21awk19:55:010
88499281,18cyclictest0-21swapper/323:52:593
88499281,12cyclictest0-21swapper/300:20:233
88499281,11cyclictest0-21swapper/323:18:463
88499281,10cyclictest0-21swapper/321:26:043
88499280,11cyclictest0-21swapper/323:12:543
88499271,11cyclictest0-21swapper/323:35:033
88499270,13cyclictest0-21swapper/321:52:273
88499270,13cyclictest0-21swapper/320:53:503
87799278,10cyclictest0-21swapper/221:15:232
86899274,3cyclictest87550irq/128-enp1s0-23:55:261
86899271,9cyclictest0-21swapper/122:20:391
86299279,16cyclictest17262-21munin-run00:40:010
88499261,11cyclictest0-21swapper/319:53:173
88499261,10cyclictest0-21swapper/319:57:433
88499261,10cyclictest0-21swapper/319:57:433
88499260,13cyclictest0-21swapper/323:43:353
88499260,12cyclictest0-21swapper/321:40:153
88499260,12cyclictest0-21swapper/320:20:163
88499260,12cyclictest0-21swapper/319:46:393
88499260,11cyclictest0-21swapper/322:06:073
88499260,10cyclictest0-21swapper/322:40:273
88499260,10cyclictest0-21swapper/319:32:513
86899268,10cyclictest0-21swapper/123:47:511
86899263,4cyclictest0-21swapper/121:59:101
86899261,6cyclictest0-21swapper/121:47:021
86299264,11cyclictest7-21ksoftirqd/021:35:010
88499251,9cyclictest0-21swapper/319:36:273
88499251,5cyclictest0-21swapper/322:03:133
88499251,13cyclictest22693-21grep21:10:143
88499251,10cyclictest24407-21users22:25:273
88499250,9cyclictest0-21swapper/321:03:343
88499250,9cyclictest0-21swapper/319:11:463
88499250,6cyclictest0-21swapper/322:11:373
88499250,18cyclictest0-21swapper/320:00:313
88499250,11cyclictest0-21swapper/322:22:113
868992523,1cyclictest0-21swapper/120:05:151
86899251,22cyclictest0-21swapper/100:30:001
86899251,16cyclictest0-21swapper/123:31:111
86899251,16cyclictest0-21swapper/123:31:111
296422250,3sleep2311ktimersoftd/221:25:192
88499241,8cyclictest313-21fschecks_time00:00:153
88499241,10cyclictest0-21swapper/323:49:273
88499241,10cyclictest0-21swapper/319:18:243
88499240,11cyclictest0-21swapper/300:36:353
88499240,11cyclictest0-21swapper/300:30:013
88499240,10cyclictest0-21swapper/322:34:433
86899249,9cyclictest0-21swapper/123:08:231
86899243,2cyclictest87550irq/128-enp1s0-21:52:251
86899243,15cyclictest0-21swapper/100:01:531
86899242,17cyclictest3708-21kworker/u8:000:21:291
868992420,3cyclictest0-21swapper/121:39:211
868992420,3cyclictest0-21swapper/119:13:101
86899241,16cyclictest0-21swapper/121:22:501
86899241,16cyclictest0-21swapper/121:15:061
86899241,12cyclictest17183-21df23:25:141
86899241,11cyclictest6833-21kworker/u8:022:50:211
86899240,16cyclictest0-21swapper/121:10:361
224602241,19sleep122503-21smart_sda23:35:241
88499239,2cyclictest0-21swapper/300:10:273
88499237,3cyclictest0-21swapper/321:46:033
88499231,9cyclictest0-21swapper/323:07:323
88499231,9cyclictest0-21swapper/321:06:343
88499231,9cyclictest0-21swapper/321:06:343
88499231,9cyclictest0-21swapper/320:37:183
88499231,9cyclictest0-21swapper/320:28:183
88499231,14cyclictest0-21swapper/320:30:243
88499231,12cyclictest0-21swapper/320:40:543
88499230,10cyclictest0-21swapper/323:23:383
87799239,2cyclictest5636-21awk23:00:012
86899239,7cyclictest1-21systemd22:17:351
868992320,2cyclictest0-21swapper/119:19:511
868992319,3cyclictest0-21swapper/100:32:551
868992318,3cyclictest4392-21munin-run20:30:021
868992317,3cyclictest4299-21sed22:55:161
86899231,6cyclictest0-21swapper/100:07:271
86899231,17cyclictest0-21swapper/123:50:381
86899231,16cyclictest0-21swapper/122:49:491
86899231,11cyclictest11249-21gltestperf19:30:151
86899231,11cyclictest10440-21cstates23:10:121
86899231,10cyclictest0-21swapper/123:41:431
86899231,10cyclictest0-21swapper/122:40:231
868992310,11cyclictest26723-21kworker/u8:122:35:061
86899230,21cyclictest0-21swapper/122:10:001
86899230,18cyclictest6833-21kworker/u8:022:26:491
86899230,16cyclictest0-21swapper/120:10:511
86899230,15cyclictest0-21swapper/121:41:541
86299230,11cyclictest10734-21kworker/0:220:57:350
72482230,5sleep086299cyclictest00:15:170
88499225,2cyclictest87350irq/126-enp1s0-21:16:433
88499221,9cyclictest0-21swapper/321:37:473
88499221,14cyclictest0-21swapper/322:58:223
88499220,9cyclictest18118-21/usr/sbin/munin23:25:253
88499220,9cyclictest0-21swapper/322:50:143
87799229,4cyclictest107402pmu-power21:55:252
87799224,7cyclictest0-21swapper/220:20:112
87799221,7cyclictest18014-21ps23:25:232
87799221,6cyclictest9536-21sed23:05:272
86899229,3cyclictest23597-21latency_hist20:00:001
86899229,3cyclictest23597-21latency_hist20:00:001
86899226,2cyclictest87350irq/126-enp1s0-23:17:441
86899224,11cyclictest21302-21sshd19:54:341
86899222,10cyclictest0-21swapper/122:00:191
86899221,9cyclictest15971-21strings19:40:181
868992219,2cyclictest0-21swapper/121:04:341
868992218,3cyclictest0-21swapper/120:41:461
86899221,6cyclictest0-21swapper/122:31:331
86899221,15cyclictest0-21swapper/121:33:011
86899221,11cyclictest0-21swapper/119:39:541
86899221,11cyclictest0-21swapper/100:39:291
86899221,10cyclictest25231-21/usr/sbin/munin20:00:251
86899221,10cyclictest15103-21chrt20:51:431
86899221,10cyclictest0-21swapper/120:16:531
86299223,16cyclictest26992-21munin-run22:35:020
86299223,16cyclictest23215-21munin-run23:40:000
86299222,9cyclictest8373-21kworker/0:000:20:270
86299222,14cyclictest20296-21kworker/0:322:26:270
88499216,3cyclictest0-21swapper/319:28:333
88499213,3cyclictest27201-21head21:20:173
88499211,9cyclictest0-21swapper/322:49:083
88499210,9cyclictest0-21swapper/300:17:493
88499210,7cyclictest0-21swapper/300:30:273
87799215,4cyclictest4660-21/usr/sbin/munin00:10:162
87799215,4cyclictest12050irq/123-ahci[0000:00:022
87799215,3cyclictest12050irq/123-ahci[0019:20:022
87799213,6cyclictest5227-21/usr/sbin/munin21:45:162
877992118,2cyclictest0-21swapper/220:51:162
86899217,3cyclictest32753-21cstates20:20:131
86899214,4cyclictest15607-21missed_timers23:20:191
86899212,11cyclictest0-21swapper/119:20:451
86899212,10cyclictest0-21swapper/120:31:041
86899212,10cyclictest0-21swapper/100:15:141
86899211,9cyclictest0-21swapper/123:04:161
86899211,12cyclictest12721-21awk20:45:261
86899211,10cyclictest17995-21sshd22:12:341
86299213,10cyclictest10250-21/usr/sbin/munin23:10:190
86299210,3cyclictest13695-21kworker/u8:122:07:290
86299210,10cyclictest12421-21kworker/0:019:48:300
88499209,3cyclictest28172-21/usr/sbin/munin20:10:213
88499209,3cyclictest2094-21/usr/sbin/munin00:05:243
88499204,3cyclictest0-21swapper/322:19:113
88499204,3cyclictest0-21swapper/320:17:433
88499203,2cyclictest87350irq/126-enp1s0-20:45:143
88499201,6cyclictest0-21swapper/323:30:463
88499201,6cyclictest0-21swapper/323:30:463
884992015,3cyclictest10308-21smartctl21:55:193
88499201,11cyclictest0-21swapper/323:01:023
87799204,4cyclictest31692-21cpuspeed_turbos22:45:122
87799203,4cyclictest27702-21pluginstate21:20:242
87799203,4cyclictest25365-21kworker/u8:200:17:522
87799202,7cyclictest11350-21if_enp2s019:30:172
87799202,4cyclictest14019-21ls00:30:182
87799201,7cyclictest2418-21/usr/sbin/munin20:25:232
87799201,6cyclictest30323-21memory22:40:192
87799201,6cyclictest27561-21diskstats22:35:132
87799201,6cyclictest15712-21ntp_states23:20:212
87799201,6cyclictest0-21swapper/222:29:152
87799201,5cyclictest32081-21open_inodes21:30:242
87799201,5cyclictest0-21swapper/223:35:232
877992011,3cyclictest18424-21/usr/sbin/munin22:15:242
86899202,17cyclictest0-21swapper/120:57:421
86899202,14cyclictest9943-21grep19:25:241
86899201,10cyclictest0-21swapper/120:36:501
86299203,15cyclictest8183-21basename19:25:010
86299202,10cyclictest9371-21kworker/0:122:11:310
86299202,10cyclictest16847-21/usr/sbin/munin23:25:130
86299201,6cyclictest0-21swapper/023:16:470
86299201,5cyclictest29806-21munin-run23:55:010
862992015,3cyclictest8934-21grep21:50:270
86299201,11cyclictest4618-21/usr/sbin/munin20:30:160
86299201,11cyclictest22395-21sshd22:22:360
86299201,11cyclictest19563-21sshd21:02:250
86299201,10cyclictest22199-21/usr/sbin/munin21:10:150
86299201,10cyclictest0-21swapper/020:28:350
87799197,2cyclictest87550irq/128-enp1s0-00:23:292
87799196,4cyclictest12050irq/123-ahci[0023:15:272
87799195,3cyclictest87350irq/126-enp1s0-21:50:172
87799195,3cyclictest18412-21grep19:45:232
87799195,3cyclictest15347-21/usr/sbin/munin19:40:152
87799194,4cyclictest30-21rcuc/200:25:452
87799194,4cyclictest28236-21cpuspeed_turbos20:10:122
87799192,6cyclictest6810-21/usr/sbin/munin20:35:212
87799191,9cyclictest0-21swapper/223:02:502
87799191,7cyclictest909-21sssd_nss20:31:342
87799191,7cyclictest0-21swapper/222:14:382
87799191,7cyclictest0-21swapper/220:15:192
87799191,6cyclictest23849-21cat20:00:122
87799191,6cyclictest0-21swapper/221:01:322
877991914,3cyclictest26345-21grep20:05:142
877991913,3cyclictest9457-21ntp_states19:25:212
87799190,11cyclictest0-21swapper/219:10:122
86299191,6cyclictest0-21swapper/023:44:300
862991914,3cyclictest8041-21find20:35:230
862991914,3cyclictest29381-21grep21:25:150
862991914,3cyclictest12859-21latency_hist00:30:010
86299191,11cyclictest5227-21/usr/sbin/munin21:45:120
76902191,2sleep00-21swapper/023:03:220
206172191,3sleep20-21swapper/222:20:092
87799185,2cyclictest87550irq/128-enp1s0-20:40:172
87799185,2cyclictest87350irq/126-enp1s0-19:39:032
87799184,3cyclictest87650irq/129-enp1s0-23:50:362
87799184,3cyclictest87650irq/129-enp1s0-21:35:212
87799183,3cyclictest0-21swapper/223:30:182
87799183,3cyclictest0-21swapper/223:30:182
87799181,6cyclictest6918-21basename19:20:172
87799181,5cyclictest26604-21users22:30:242
87799181,5cyclictest23437-21/usr/sbin/munin23:40:152
87799181,5cyclictest21058-21ntpq21:05:222
87799181,5cyclictest21058-21ntpq21:05:222
87799181,5cyclictest19904-21/usr/sbin/munin19:50:152
87799181,5cyclictest11513-21smart_sda23:10:232
87799181,4cyclictest579-21systemd-journal22:05:012
87799181,4cyclictest25981-21diskstats23:45:142
87799181,4cyclictest0-21swapper/221:42:352
87799181,3cyclictest15388-21cut00:35:122
86299185,2cyclictest12999-21kworker/0:020:50:050
86299182,5cyclictest23547-21munin-run20:00:010
86299182,5cyclictest23547-21munin-run20:00:010
86299181,7cyclictest0-21swapper/022:58:180
86299181,5cyclictest8564-1kworker/u9:122:47:080
86299181,4cyclictest25635-21/usr/sbin/munin23:45:240
86299181,3cyclictest14772-21basename19:40:000
86299181,15cyclictest28566-21munin-run21:25:010
86299181,11cyclictest0-21swapper/000:00:390
86299181,10cyclictest0-21swapper/023:23:240
86299181,10cyclictest0-21swapper/020:23:360
86299180,8cyclictest3803-21kworker/0:000:07:320
86299180,12cyclictest0-21swapper/019:30:110
241192181,3sleep20-21swapper/221:14:322
87799171,9cyclictest0-21swapper/200:08:212
87799171,3cyclictest0-21swapper/222:50:162
87799171,13cyclictest16120-21fschecks_time20:55:152
86299176,2cyclictest29770-21kworker/0:023:55:080
86299172,6cyclictest8564-1kworker/u9:121:40:420
86299171,8cyclictest0-21swapper/000:30:500
86299171,6cyclictest0-21swapper/023:34:350
86299171,6cyclictest0-21swapper/023:34:350
86299171,3cyclictest0-21swapper/022:18:040
862991711,3cyclictest9627-21/usr/sbin/munin21:55:260
86299171,13cyclictest23772-21/usr/sbin/munin20:00:130
294052170,1sleep00-21swapper/020:10:240
877991611,3cyclictest22398-21kernelversion19:55:182
877991611,3cyclictest22398-21kernelversion19:55:182
86299161,6cyclictest0-21swapper/019:40:150
86299161,5cyclictest0-21swapper/021:36:110
86299161,3cyclictest0-21swapper/020:17:500
862991612,3cyclictest0-21swapper/019:26:210
862991612,3cyclictest0-21swapper/019:20:010
862991611,2cyclictest20004-21/usr/sbin/munin21:05:180
862991611,2cyclictest20004-21/usr/sbin/munin21:05:180
862991610,3cyclictest12002-21interrupts20:45:170
87799158,4cyclictest11852-21head20:45:152
86299159,3cyclictest2161-21ntpq19:10:200
862991511,3cyclictest0-21swapper/020:07:020
862991510,3cyclictest30477-21sh22:40:230
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional