You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-06 - 06:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Mon Apr 06, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
187952236163,23sleep10-21swapper/119:06:141
189222203167,24sleep20-21swapper/219:07:522
171262201163,25sleep30-21swapper/319:05:073
171282200163,25sleep00-21swapper/019:05:090
2326621050,5sleep21917299cyclictest00:10:192
324762520,1sleep30-21swapper/300:30:223
19172995148,2cyclictest0-21swapper/200:26:192
253842500,2sleep00-21swapper/020:35:140
19172994113,25cyclictest26994-21df19:25:132
1917299381,21cyclictest13520-21awk23:50:012
1917299376,21cyclictest17068-21ls21:30:122
1917299361,4cyclictest87350irq/126-enp1s0-19:10:222
17892360,4sleep31917899cyclictest20:55:123
1917299359,22cyclictest6818-21grep19:50:242
1917299351,5cyclictest15727-21nfs4_client21:25:202
1917299348,22cyclictest3480-21/usr/sbin/munin21:00:012
1917299341,24cyclictest32480-21chrt23:17:342
1917299341,22cyclictest26885-21chrt20:38:102
19172993410,21cyclictest13177-21hddtemp_smartct21:20:162
1917299334,3cyclictest87350irq/126-enp1s0-22:45:202
1917299332,5cyclictest4675-21diskstats22:15:142
1917299332,3cyclictest12050irq/123-ahci[0022:05:012
1917299331,7cyclictest15219-21cat22:40:022
1917299331,4cyclictest9040-21sed21:10:202
1917299331,29cyclictest4346-21cut21:00:172
1917299321,5cyclictest29787-21kworker/u8:022:08:232
1917299321,5cyclictest26551-21cat19:25:012
1917299321,5cyclictest19754-21sshd00:00:292
1917299321,5cyclictest0-21swapper/223:40:122
1917299321,5cyclictest0-21swapper/223:40:122
1917299321,4cyclictest11986-21ntp_states22:30:212
1917299321,4cyclictest11986-21ntp_states22:30:212
19172993214,15cyclictest2114-21awk23:20:252
19172993212,17cyclictest17287-21sort23:55:242
1917299319,19cyclictest29636-21cat20:45:112
1917299319,18cyclictest25354-21gltestperf00:15:152
1917299314,3cyclictest87350irq/126-enp1s0-23:50:252
1917299314,3cyclictest87350irq/126-enp1s0-00:20:232
1917299311,6cyclictest0-21swapper/220:30:242
1917299311,5cyclictest9447-21latency_hist20:00:002
1917299311,5cyclictest2762-21awk00:35:272
1917299311,3cyclictest87350irq/126-enp1s0-20:20:012
1917299311,3cyclictest87350irq/126-enp1s0-20:20:012
1917299311,3cyclictest1089-21cstates19:40:112
1917299306,21cyclictest4174-21timerwakeupswit23:25:252
1917299305,22cyclictest0-21swapper/200:10:012
1917299304,13cyclictest9980-21cstates20:00:122
1917299302,3cyclictest21293-21/usr/sbin/munin21:40:192
1917299301,4cyclictest7063-21grep22:20:142
1917299301,4cyclictest31665-21/usr/sbin/munin20:50:132
1917299301,4cyclictest20357-21/usr/sbin/munin21:35:232
1917299301,3cyclictest827-21runrttasks22:28:432
1917299301,3cyclictest19392-21memory20:20:172
1917299301,3cyclictest15237-21/usr/sbin/munin20:10:232
1917299301,3cyclictest1032-21latency_hist00:35:012
19172993011,4cyclictest21643-21missed_timers20:25:192
1917299299,17cyclictest29841-21cut19:30:182
1917299293,5cyclictest21854-21smart_sda19:15:032
1917299291,6cyclictest0-21swapper/222:50:192
1917299291,5cyclictest3109-21grep19:45:122
1917299291,5cyclictest0-21swapper/221:50:172
1917299291,4cyclictest7159-21/usr/sbin/munin23:35:112
1917299291,4cyclictest2833-21/usr/sbin/munin22:10:162
1917299291,4cyclictest23400-21/usr/sbin/munin22:55:262
1917299291,4cyclictest15314-21/usr/sbin/munin22:40:122
1917299291,4cyclictest10644-21df21:15:142
1917299291,3cyclictest23494-21/usr/sbin/munin21:45:222
19172992913,13cyclictest24797-21hwlatdetect23:00:162
19172992912,14cyclictest5912-21/usr/sbin/munin21:05:142
19172992910,16cyclictest13865-21basename20:10:002
19172992910,15cyclictest27563-21cat20:40:122
1916599290,22cyclictest0-21swapper/122:00:071
1916099292,14cyclictest21694-21munin-run22:55:010
1917299286,18cyclictest26938-21gltestperf23:05:162
1917299282,3cyclictest87650irq/129-enp1s0-19:35:172
1917299281,4cyclictest29194-21/usr/sbin/munin23:10:172
1917299281,3cyclictest27886-21/usr/sbin/munin21:55:122
1916099280,13cyclictest16066-21awk20:15:010
75592270,2sleep11916599cyclictest23:35:141
1917299271,3cyclictest6564-21chrt23:30:382
1916599271,6cyclictest0-21swapper/122:24:061
1917899261,11cyclictest0-21swapper/322:47:293
1916599261,11cyclictest0-21swapper/121:20:251
19165992610,3cyclictest21615-21smart_sda00:05:221
1916099261,12cyclictest24250-21awk19:20:010
1917899251,6cyclictest28567-21smartctl21:55:143
1916599252,6cyclictest31228-40dnf20:47:291
1916599251,21cyclictest29492-21df_inode00:25:141
1916599251,11cyclictest27037-21latency_hist20:40:021
1916599241,5cyclictest0-21swapper/121:38:241
1916599241,4cyclictest0-21swapper/123:49:481
1916599241,11cyclictest26943-21expr19:25:131
1916599241,11cyclictest15637-21cat22:40:121
1916599241,11cyclictest0-21swapper/123:32:111
1916599240,8cyclictest0-21swapper/122:27:041
1916599240,3cyclictest0-21swapper/121:54:511
19160992415,3cyclictest1676-1kworker/u9:200:28:000
87312230,2sleep00-21swapper/023:35:250
1917899231,9cyclictest0-21swapper/323:16:533
1917899231,5cyclictest0-21swapper/300:37:503
1917899231,14cyclictest0-21swapper/320:10:243
19178992311,10cyclictest17912-40updatedb00:00:023
1916599238,8cyclictest0-21swapper/121:15:441
1916599231,6cyclictest0-21swapper/121:43:051
1916599231,16cyclictest0-21swapper/100:39:261
1916599231,10cyclictest735-21expr23:20:111
1916599230,7cyclictest0-21swapper/100:00:021
1916599230,6cyclictest0-21swapper/123:53:181
1916599230,6cyclictest0-21swapper/100:30:491
1916599230,5cyclictest0-21swapper/121:14:071
1917899221,6cyclictest0-21swapper/321:39:213
1917899221,6cyclictest0-21swapper/321:22:473
1917899221,5cyclictest0-21swapper/323:05:463
1917899221,5cyclictest0-21swapper/322:32:303
1917899221,5cyclictest0-21swapper/322:32:303
1917899221,11cyclictest0-21swapper/323:23:193
1917899220,16cyclictest0-21swapper/320:40:163
1916599227,4cyclictest18658-21proc_pri22:45:211
19165992218,2cyclictest17334-21ntpq20:15:221
19165992218,2cyclictest17334-21ntpq20:15:221
1916599221,15cyclictest0-21swapper/121:30:441
1916599221,13cyclictest0-21swapper/123:41:411
1916599221,13cyclictest0-21swapper/123:41:411
1916599221,11cyclictest0-21swapper/122:15:541
1916599220,5cyclictest0-21swapper/122:06:561
1916599220,5cyclictest0-21swapper/121:58:091
1916599220,5cyclictest0-21swapper/100:24:401
1916599220,3cyclictest0-21swapper/119:58:211
1916099224,15cyclictest29872-21awk22:00:020
1916099221,6cyclictest25238-21chrt21:47:140
1916099221,3cyclictest0-21swapper/000:07:220
19160992211,3cyclictest25254-21head23:00:210
1917899219,3cyclictest9174-21ntp_states21:10:223
1917899217,5cyclictest0-21swapper/322:36:123
1917899216,6cyclictest0-21swapper/300:24:023
1917899215,5cyclictest0-21swapper/322:17:133
1917899212,2cyclictest0-21swapper/323:54:103
1917899211,9cyclictest0-21swapper/321:25:193
1917899211,5cyclictest0-21swapper/323:41:283
1917899211,5cyclictest0-21swapper/323:41:283
1917899211,5cyclictest0-21swapper/322:57:453
1917899211,4cyclictest0-21swapper/323:12:593
1917899211,17cyclictest401ktimersoftd/322:50:113
1916599215,5cyclictest0-21swapper/100:19:051
1916599212,15cyclictest20638-21smart_sda19:10:231
19165992115,3cyclictest16271-21df23:55:131
1916599211,18cyclictest14171-21ntpq22:35:211
1916599211,18cyclictest0-21swapper/123:10:101
1916599211,10cyclictest0-21swapper/123:03:481
1916599210,6cyclictest0-21swapper/122:11:441
1916099211,6cyclictest0-21swapper/021:35:310
1916099211,5cyclictest17048-21cpuspeed_turbos21:30:130
1916099211,13cyclictest0-21swapper/022:10:260
1916099211,11cyclictest0-21swapper/022:22:000
1917899202,10cyclictest30089-21/usr/sbin/munin22:00:263
1917899201,6cyclictest0-21swapper/322:14:573
1917899201,4cyclictest4461-21kworker/u8:023:29:413
1917899201,2cyclictest0-21swapper/322:05:253
1917899201,13cyclictest0-21swapper/323:45:173
1917899201,11cyclictest25074-21/usr/sbin/munin20:35:183
1917899201,11cyclictest0-21swapper/300:11:413
1917899201,11cyclictest0-21swapper/300:09:043
1917899200,7cyclictest0-21swapper/323:31:173
1916599209,2cyclictest18495-21cpu20:20:111
1916599205,4cyclictest24850-21munin-run20:35:021
1916599201,7cyclictest27307-21strings23:05:191
19165992017,2cyclictest0-21swapper/120:52:131
19165992016,3cyclictest0-21swapper/119:51:391
1916599201,4cyclictest0-21swapper/100:10:471
1916599201,18cyclictest0-21swapper/121:05:241
1916599201,18cyclictest0-21swapper/120:14:301
1916599201,10cyclictest0-21swapper/122:50:321
1916599200,5cyclictest0-21swapper/123:18:361
1916099209,3cyclictest19075-21gltestperf20:20:140
1916099206,6cyclictest755-21rngd22:38:440
1916099203,9cyclictest13860-21cat21:20:250
1916099203,15cyclictest7269-21idleruntime-cro19:55:010
1916099201,7cyclictest0-21swapper/023:45:140
1916099201,6cyclictest0-21swapper/023:41:360
1916099201,6cyclictest0-21swapper/023:41:360
1916099201,6cyclictest0-21swapper/023:15:450
1916099201,4cyclictest6914-21ls23:35:000
1916099201,4cyclictest18086-21perl22:45:130
1916099201,12cyclictest0-21swapper/023:14:260
1917899199,2cyclictest23278-21awk21:45:023
1917899198,3cyclictest9678-21/usr/sbin/munin20:00:233
1917899196,3cyclictest0-21swapper/321:34:083
1917899194,2cyclictest0-21swapper/300:25:343
1917899191,9cyclictest0-21swapper/323:35:043
1917899191,6cyclictest0-21swapper/322:20:513
1917899191,5cyclictest0-21swapper/300:03:473
19178991913,3cyclictest27268-21gltestperf19:25:163
19178991912,4cyclictest7879-21dump-pmu-power21:10:013
1917899191,10cyclictest7885-21diskmemload21:51:073
1917899191,10cyclictest4142-21ntpq19:45:203
1917899191,10cyclictest29768-21memory19:30:183
1917899191,10cyclictest15313-21/usr/sbin/munin22:40:263
1917899191,10cyclictest0-21swapper/323:01:473
1917899191,10cyclictest0-21swapper/300:17:423
1916599196,6cyclictest0-21swapper/122:32:341
1916599196,6cyclictest0-21swapper/122:32:341
1916599191,9cyclictest0-21swapper/121:48:401
19165991915,3cyclictest0-21swapper/120:27:191
19165991915,3cyclictest0-21swapper/119:36:371
19165991914,3cyclictest15069-21grep21:25:151
1916599191,17cyclictest0-21swapper/122:57:541
1916599191,16cyclictest0-21swapper/120:40:261
1916599191,16cyclictest0-21swapper/119:49:021
1916599191,16cyclictest0-21swapper/119:31:181
1916599191,16cyclictest0-21swapper/119:17:141
1916599190,5cyclictest0-21swapper/123:27:071
1916599190,17cyclictest0-21swapper/119:40:231
1916099195,9cyclictest0-21swapper/022:15:170
1916099195,7cyclictest15592-21ls22:40:120
1916099195,3cyclictest12050irq/123-ahci[0020:00:020
1916099192,9cyclictest7887-21munin-run21:10:000
1916099191,7cyclictest27086-21cut00:20:120
1916099191,7cyclictest0-21swapper/022:56:070
1916099191,7cyclictest0-21swapper/022:30:180
1916099191,7cyclictest0-21swapper/022:30:180
1916099191,6cyclictest3933-21kernelversion19:45:190
1916099191,6cyclictest0-21swapper/022:28:440
1916099191,5cyclictest29808-21diskstats20:45:120
1916099191,4cyclictest0-21swapper/021:25:060
19160991912,4cyclictest27867-21grep20:40:140
1916099191,11cyclictest23868-21needreboot20:30:180
1916099191,11cyclictest0-21swapper/000:38:520
1916099191,10cyclictest26019-21smart_sda00:15:240
19160991910,4cyclictest0-21swapper/020:15:190
19160991910,4cyclictest0-21swapper/020:15:190
1917899188,4cyclictest0-21swapper/321:45:253
1917899186,10cyclictest401ktimersoftd/320:50:283
1917899181,8cyclictest32147-21open_inodes19:35:223
1917899181,8cyclictest0-21swapper/320:25:173
1917899181,7cyclictest0-21swapper/320:20:243
1917899181,4cyclictest11884-21/usr/sbin/munin20:05:223
1917899181,4cyclictest0-21swapper/321:15:473
19178991813,3cyclictest2490-21vmstat19:40:263
1917899181,15cyclictest0-21swapper/319:55:243
1917899181,10cyclictest0-21swapper/322:28:223
19165991814,3cyclictest0-21swapper/121:00:011
1916599181,16cyclictest0-21swapper/120:56:291
1916599181,16cyclictest0-21swapper/119:22:401
1916599180,16cyclictest0-21swapper/120:00:021
1916099187,3cyclictest4509-21sshd23:27:280
1916099186,2cyclictest3687-21sed21:00:010
1916099183,2cyclictest87650irq/129-enp1s0-21:10:200
1916099181,9cyclictest29240-21fschecks_count19:30:140
1916099181,9cyclictest18507-21fschecks_count00:00:150
1916099181,8cyclictest0-21swapper/022:06:530
1916099181,5cyclictest0-21swapper/023:09:240
19160991812,3cyclictest11638-21sed21:15:260
1916099181,10cyclictest30852-21if_enp2s022:00:200
1916099181,10cyclictest21898-21gltestperf21:40:160
1916099181,10cyclictest0-21swapper/023:58:590
19160991810,2cyclictest0-21swapper/021:50:250
1917899171,7cyclictest0-21swapper/319:50:113
1917899171,3cyclictest0-21swapper/319:19:313
1917899171,2cyclictest20978-21chrt19:10:253
19178991712,3cyclictest17322-21ntpq20:15:213
19178991712,3cyclictest17322-21ntpq20:15:213
1917899171,13cyclictest4339-21cut21:00:173
19165991715,1cyclictest0-21swapper/120:05:291
1916099172,2cyclictest87650irq/129-enp1s0-19:35:190
1916099171,5cyclictest0-21swapper/023:54:090
1916099171,5cyclictest0-21swapper/019:20:110
1916099171,4cyclictest0-21swapper/000:30:210
1916099171,4cyclictest0-21swapper/000:10:350
19160991712,3cyclictest27759-21open_inodes19:25:220
1916099171,11cyclictest0-21swapper/021:00:190
1916099171,11cyclictest0-21swapper/019:40:150
1916099170,4cyclictest0-21swapper/020:25:120
19178991611,3cyclictest29462-21/usr/sbin/munin20:45:213
19178991610,3cyclictest24463-21/usr/sbin/munin19:20:253
1916099161,8cyclictest20765-21timerandwakeup19:10:240
1916099161,6cyclictest0-21swapper/020:50:120
1916099161,6cyclictest0-21swapper/020:05:270
1916099161,5cyclictest0-21swapper/023:20:140
1916099161,12cyclictest11372-21sshd20:01:270
1917899151,8cyclictest0-21swapper/320:30:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional