You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-19 - 20:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Tue May 19, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
49582233160,60sleep00-21swapper/007:09:020
48132230153,24sleep10-21swapper/107:07:101
50272210179,21sleep20-21swapper/207:09:582
47942202166,24sleep30-21swapper/307:06:573
2082221400,4sleep1512299cyclictest08:55:221
1727021400,3sleep1512299cyclictest10:00:241
5127999288,3cyclictest0-21swapper/208:43:002
45222580,4sleep339-21rcuc/308:20:133
5127995754,2cyclictest0-21swapper/208:48:112
513499433,32cyclictest0-21swapper/310:25:593
513499431,30cyclictest21612-21cut10:10:213
513499431,30cyclictest21612-21cut10:10:213
513499405,23cyclictest0-21swapper/309:55:243
513499403,30cyclictest23063-40dnf07:46:263
513499403,30cyclictest23063-40dnf07:46:263
513499391,28cyclictest0-21swapper/309:35:113
513499391,23cyclictest17040-21latency_hist12:30:013
5117993912,13cyclictest29614-21munin-run11:45:010
513499381,26cyclictest23873-21sed11:30:143
513499381,20cyclictest956-21kworker/u8:210:49:533
513499378,20cyclictest0-21swapper/312:35:273
513499376,22cyclictest25788-21/usr/sbin/munin07:55:203
513499371,26cyclictest18621-21latency_hist11:20:013
513499364,20cyclictest26178-21diskmemload12:21:433
513499362,23cyclictest12050irq/123-ahci[0008:30:233
513499361,26cyclictest26178-21diskmemload11:00:573
513499361,25cyclictest8489-21/usr/sbin/munin12:10:253
513499361,21cyclictest28608-21/usr/sbin/munin09:15:133
513499351,24cyclictest10867-21cstates12:15:143
513499351,21cyclictest20077-21cat07:40:173
513499351,20cyclictest827-21runrttasks11:20:513
513499351,20cyclictest827-21runrttasks11:20:513
513499351,19cyclictest23696-1kworker/3:0H09:05:233
513499351,16cyclictest31286-21cat10:35:023
513499350,27cyclictest0-21swapper/309:05:023
513499350,22cyclictest1901-21/usr/sbin/munin09:25:233
513499346,18cyclictest87350irq/126-enp1s0-09:35:013
513499346,18cyclictest0-21swapper/309:00:203
513499342,22cyclictest755-21rngd09:45:253
513499342,21cyclictest5501-21cstates07:10:133
513499341,25cyclictest0-21swapper/312:00:143
513499341,24cyclictest27640-21/usr/sbin/munin11:40:133
513499341,24cyclictest17912-21expr08:50:133
513499341,22cyclictest7237-21/usr/sbin/munin09:40:133
513499341,21cyclictest20203-21latency_hist10:10:013
513499341,20cyclictest87350irq/126-enp1s0-11:25:273
513499341,20cyclictest16429-21turbostat11:15:003
513499341,20cyclictest10408-21/usr/sbin/munin07:20:243
513499341,20cyclictest0-21swapper/308:15:023
513499340,24cyclictest0-21swapper/309:20:123
513499335,13cyclictest16029-21/usr/sbin/munin10:00:263
513499333,17cyclictest26602-21proc_pri11:35:213
513499332,19cyclictest12265-21/usr/sbin/munin11:05:273
513499331,28cyclictest26178-21diskmemload10:15:273
513499331,24cyclictest8215-21/usr/sbin/munin07:15:203
513499331,24cyclictest7864-21/usr/sbin/munin10:55:233
513499331,24cyclictest12887-21sshd08:38:543
513499331,23cyclictest1644-21latency_hist11:55:023
513499331,23cyclictest1103-21cat10:40:023
513499331,21cyclictest3740-21taskset11:58:503
513499331,21cyclictest17274-21/usr/sbin/munin12:30:173
513499331,18cyclictest4193-21latency_hist08:20:023
513499330,22cyclictest0-21swapper/308:25:183
513499323,13cyclictest0-21swapper/310:22:213
513499321,28cyclictest5663-21/usr/sbin/munin10:50:203
513499321,22cyclictest11636-21/usr/sbin/munin09:50:123
513499321,20cyclictest19808-21/usr/sbin/munin08:55:173
513499321,19cyclictest31389-21processes08:05:223
513499321,19cyclictest27869-21cat09:10:243
513499321,18cyclictest0-21swapper/308:00:163
513499320,21cyclictest0-21swapper/311:50:023
513499320,21cyclictest0-21swapper/310:45:023
513499320,20cyclictest0-21swapper/308:42:143
243182320,3sleep10-21swapper/111:30:211
513499314,18cyclictest87650irq/129-enp1s0-12:05:233
513499313,15cyclictest0-21swapper/307:50:223
513499311,27cyclictest0-21swapper/308:47:553
513499301,26cyclictest87350irq/126-enp1s0-07:35:183
513499301,26cyclictest12605-21/usr/sbin/munin07:25:133
513499291,25cyclictest16018-21/usr/sbin/munin07:30:233
5127992611,8cyclictest18023-21awk10:05:012
512299266,18cyclictest0-21swapper/109:37:281
512299261,4cyclictest0-21swapper/111:04:421
511799261,12cyclictest2837-21/usr/sbin/munin09:30:140
511799261,12cyclictest20823-21munin-run11:25:010
511799261,12cyclictest20823-21munin-run11:25:010
512799252,6cyclictest0-21swapper/210:12:462
512799252,6cyclictest0-21swapper/210:12:462
512799251,4cyclictest0-21swapper/208:13:312
512299252,4cyclictest755-21rngd10:59:061
5122992515,3cyclictest0-21swapper/111:23:051
5122992515,3cyclictest0-21swapper/111:23:051
512299251,10cyclictest7288-21latency_hist07:15:011
5122992510,7cyclictest0-21swapper/111:12:281
511799253,2cyclictest87550irq/128-enp1s0-10:08:530
5117992510,12cyclictest29979-21awk08:05:000
512299244,16cyclictest5257-40dnf10:47:551
512299241,15cyclictest0-21swapper/109:32:091
511799249,3cyclictest28608-21/usr/sbin/munin09:15:230
511799245,4cyclictest5807-21sshd12:01:530
511799242,11cyclictest24210-21/usr/sbin/munin09:05:140
511799241,14cyclictest0-21swapper/012:22:100
511799241,10cyclictest6629-21/usr/sbin/munin08:25:160
511799241,10cyclictest17465-21irqstats11:15:190
511799241,10cyclictest15004-21sshd08:42:550
512799234,10cyclictest20211-21ls10:10:012
512299233,6cyclictest87550irq/128-enp1s0-09:48:161
512299231,4cyclictest23988-21munin-run09:05:011
512299231,4cyclictest23988-21munin-run09:05:011
512299231,3cyclictest0-21swapper/112:16:321
5122992312,3cyclictest0-21swapper/108:13:531
512299231,19cyclictest25804-21cpu07:55:111
512299231,17cyclictest0-21swapper/111:51:401
512299231,14cyclictest0-21swapper/112:03:081
511799231,8cyclictest0-21swapper/007:15:060
511799231,6cyclictest0-21swapper/011:13:280
511799231,14cyclictest0-21swapper/012:39:540
511799231,13cyclictest0-21swapper/010:58:040
511799231,13cyclictest0-21swapper/008:55:040
511799231,11cyclictest27236-21sshd11:37:530
511799231,10cyclictest270022head09:10:160
512799221,5cyclictest0-21swapper/211:09:232
512799221,11cyclictest31285-21timerwakeupswit11:45:252
512299224,9cyclictest12598-21df11:05:151
512299222,4cyclictest1712350sshd08:46:541
512299222,12cyclictest774-21avahi-daemon10:15:141
5122992215,4cyclictest9685-21grep07:15:271
512299221,4cyclictest0-21swapper/112:14:531
512299221,3cyclictest26504-21ntp_states11:35:201
512299221,19cyclictest0-21swapper/109:43:301
512299221,18cyclictest0-21swapper/110:42:031
512299221,14cyclictest0-21swapper/109:54:081
512299221,13cyclictest29678-21latency_hist11:45:021
512299220,6cyclictest0-21swapper/109:13:171
512299220,4cyclictest0-21swapper/109:29:061
512299220,3cyclictest0-21swapper/112:25:561
511799221,8cyclictest0-21swapper/010:50:250
511799221,3cyclictest0-21swapper/012:17:040
511799221,13cyclictest0-21swapper/009:03:090
511799221,13cyclictest0-21swapper/009:03:090
512799217,6cyclictest827-21runrttasks10:57:202
512799216,3cyclictest12050irq/123-ahci[0010:50:012
512799215,3cyclictest87350irq/126-enp1s0-11:35:202
512799214,4cyclictest3224-21sh10:45:012
512799211,10cyclictest0-21swapper/209:55:052
512299215,5cyclictest22411-21munin-run10:15:001
512299215,5cyclictest22411-21munin-run10:15:001
512299214,7cyclictest0-21swapper/111:57:171
512299213,2cyclictest0-21swapper/110:27:231
5122992111,3cyclictest0-21swapper/109:15:131
512299211,12cyclictest30209-21needreboot10:30:171
512299211,12cyclictest13215-21df12:20:121
512299210,15cyclictest0-21swapper/111:47:261
512299210,15cyclictest0-21swapper/110:07:021
511799211,8cyclictest0-21swapper/011:01:110
511799211,8cyclictest0-21swapper/009:55:010
511799211,8cyclictest0-21swapper/008:09:060
511799211,7cyclictest0-21swapper/009:45:090
511799211,7cyclictest0-21swapper/008:32:050
511799211,6cyclictest0-21swapper/010:39:170
511799211,6cyclictest0-21swapper/008:46:090
511799211,13cyclictest0-21swapper/012:27:090
511799211,12cyclictest0-21swapper/007:52:110
269992210,4sleep230-21rcuc/209:10:152
512799207,4cyclictest11702-21hwlatdetect08:35:162
512799202,9cyclictest6219-21perl10:50:152
512799202,9cyclictest12660-21perf12:20:002
512799201,7cyclictest21784-21awk09:00:002
512799201,7cyclictest10461-21awk12:15:012
512799201,6cyclictest0-21swapper/209:04:412
512799201,6cyclictest0-21swapper/209:04:412
512799201,5cyclictest579-21systemd-journal08:10:012
512799201,4cyclictest0-21swapper/211:57:512
512799201,11cyclictest5601-21cstates07:10:122
512799201,10cyclictest28326-21df_inode08:00:142
512799200,6cyclictest0-21swapper/211:15:272
512299206,3cyclictest0-21swapper/112:05:571
512299201,4cyclictest16990-21df11:15:151
512299201,17cyclictest0-21swapper/109:57:181
512299201,17cyclictest0-21swapper/108:23:511
512299201,10cyclictest25040-21expr10:20:121
512299201,10cyclictest17949-21hddtemp_smartct12:30:171
5122992010,4cyclictest0-21swapper/110:54:551
512299200,14cyclictest0-21swapper/112:37:111
512299200,10cyclictest0-21swapper/109:24:061
511799202,9cyclictest0-21swapper/010:16:550
511799202,2cyclictest10795-21fschecks_count07:20:150
511799201,8cyclictest0-21swapper/010:04:050
511799201,8cyclictest0-21swapper/007:56:030
511799201,7cyclictest0-21swapper/008:39:220
511799201,5cyclictest0-21swapper/011:07:020
512799196,9cyclictest87550irq/128-enp1s0-09:30:222
512799195,4cyclictest30483-21/usr/sbin/munin10:30:222
512799193,9cyclictest0-21swapper/212:25:032
512799193,7cyclictest32-21ksoftirqd/208:25:132
512799192,7cyclictest20141-21cat11:20:242
512799192,7cyclictest20141-21cat11:20:242
512799191,7cyclictest0-21swapper/211:11:492
512799191,6cyclictest0-21swapper/209:27:362
512799191,2cyclictest87550irq/128-enp1s0-09:20:512
5127991912,4cyclictest4782-21sort12:00:182
512799190,1cyclictest0-21swapper/212:06:352
512299199,3cyclictest24473-21cat09:05:131
512299191,7cyclictest0-21swapper/108:32:121
5122991917,1cyclictest0-21swapper/108:25:131
512299191,5cyclictest0-21swapper/110:38:121
5122991914,3cyclictest25179-21users07:50:251
512299191,3cyclictest0-21swapper/107:40:141
512299191,3cyclictest0-21swapper/107:36:341
5122991913,3cyclictest10741-21df07:20:141
512299191,17cyclictest0-21swapper/108:09:141
512299191,17cyclictest0-21swapper/107:27:221
5122991911,5cyclictest0-21swapper/108:15:141
511799193,10cyclictest26178-21diskmemload10:47:560
511799192,5cyclictest9137-21kworker/0:109:50:550
511799192,5cyclictest7-21ksoftirqd/011:50:540
511799191,9cyclictest0-21swapper/011:46:020
511799191,8cyclictest0-21swapper/012:09:060
511799191,8cyclictest0-21swapper/008:17:080
511799191,7cyclictest0-21swapper/008:23:050
511799191,6cyclictest16995-21/usr/sbin/munin07:35:130
511799191,6cyclictest0-21swapper/011:26:080
511799191,3cyclictest0-21swapper/010:12:230
511799191,3cyclictest0-21swapper/010:12:230
5117991913,4cyclictest601-21hddtemp_smartct08:10:170
511799190,11cyclictest0-21swapper/009:25:240
512799189,2cyclictest87550irq/128-enp1s0-12:23:112
512799187,8cyclictest23913-21switchtime10:15:252
512799187,3cyclictest14064-21unixbench-2d07:25:262
512799184,3cyclictest87350irq/126-enp1s0-07:50:132
512799184,2cyclictest87350irq/126-enp1s0-10:35:262
512799184,2cyclictest21159-21crond07:45:012
512799183,5cyclictest0-21swapper/209:43:482
512799182,8cyclictest10764-21idleruntime11:00:172
512799182,4cyclictest18968-21munin-run07:40:012
512799181,7cyclictest18836-21grep08:50:222
512799181,5cyclictest30569-21crond09:20:012
512799181,5cyclictest0-21swapper/210:25:272
512799181,4cyclictest0-21swapper/209:54:282
512799181,11cyclictest0-21swapper/211:41:502
512799180,9cyclictest0-21swapper/211:54:552
512799180,10cyclictest8092-21timerwakeupswit07:15:042
5122991814,3cyclictest0-21swapper/107:46:511
5122991814,3cyclictest0-21swapper/107:46:511
5122991814,3cyclictest0-21swapper/107:34:001
512299181,3cyclictest21381-21df11:25:131
512299181,16cyclictest0-21swapper/108:52:361
512299181,16cyclictest0-21swapper/108:43:101
511799187,3cyclictest0-21swapper/011:55:200
511799185,3cyclictest87550irq/128-enp1s0-10:40:180
511799185,3cyclictest0-21swapper/012:32:100
511799183,3cyclictest32158-21sed09:20:230
511799182,13cyclictest29002-21munin-run10:30:020
511799181,7cyclictest827-21runrttasks09:35:250
511799181,7cyclictest5319-21/usr/sbin/munin07:10:190
511799181,7cyclictest0-21swapper/007:26:080
511799181,5cyclictest0-21swapper/011:31:550
511799181,5cyclictest0-21swapper/010:30:200
5117991811,4cyclictest16763-21idleruntime-cro07:35:000
511799181,10cyclictest0-21swapper/012:10:120
512799176,3cyclictest87350irq/126-enp1s0-08:20:132
512799175,3cyclictest87350irq/126-enp1s0-12:35:132
512799173,3cyclictest10835-21latency_hist08:35:002
512799172,7cyclictest0-21swapper/212:33:542
512799172,4cyclictest2388-1kworker/u9:111:30:342
512799171,9cyclictest0-21swapper/210:20:192
512799171,7cyclictest0-21swapper/209:05:262
512799171,6cyclictest0-21swapper/211:25:202
512799171,6cyclictest0-21swapper/207:56:482
512799171,4cyclictest2411-21cstates08:15:142
512799171,4cyclictest0-21swapper/209:35:172
512799170,6cyclictest0-21swapper/207:45:162
512799170,6cyclictest0-21swapper/207:45:162
512299171,4cyclictest28190-21cstates08:00:121
5122991714,2cyclictest0-21swapper/108:37:071
511799171,7cyclictest25973-21pmu-power10:20:220
511799171,7cyclictest0-21swapper/007:44:040
511799171,6cyclictest0-21swapper/009:41:210
511799171,13cyclictest17607-21/usr/sbin/munin08:50:120
512799163,5cyclictest11365-21sed07:20:202
5127991611,3cyclictest14797-21/usr/sbin/munin07:30:202
512799160,9cyclictest0-21swapper/209:45:162
511799161,6cyclictest0-21swapper/007:46:100
511799161,6cyclictest0-21swapper/007:46:100
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional