You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-05 - 23:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Apr 05, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
97212240162,26sleep00-21swapper/007:06:050
97212240162,26sleep00-21swapper/007:06:040
98522239164,24sleep10-21swapper/107:07:471
98522239164,24sleep10-21swapper/107:07:471
98232236164,22sleep20-21swapper/207:07:242
98232236164,22sleep20-21swapper/207:07:242
99312202165,24sleep30-21swapper/307:08:503
99312202165,24sleep30-21swapper/307:08:493
1098321600,4sleep11011099cyclictest07:10:151
3190721530,5sleep21011599cyclictest11:35:272
127021510,3sleep11011099cyclictest08:00:161
1476321030,5sleep31012099cyclictest10:59:203
101159910197,3cyclictest0-21swapper/212:10:302
10115999995,3cyclictest0-21swapper/208:42:262
164162720,1sleep10-21swapper/108:35:151
128622580,4sleep2311ktimersoftd/209:40:172
204692480,4sleep341-21ksoftirqd/308:45:133
115372480,1sleep00-21swapper/007:10:220
99062460,4sleep21011599cyclictest08:20:152
10099993711,12cyclictest17375-21munin-run07:25:010
1011099361,3cyclictest9417-21sh12:00:141
1011099342,14cyclictest0-21swapper/111:13:251
1009999305,12cyclictest12635-21munin-run10:55:010
10099993011,16cyclictest24578-21munin-run08:55:010
1009999291,13cyclictest1652-21awk10:30:000
10115992811,3cyclictest827-21runrttasks12:28:302
1011099280,13cyclictest495-21jbd2/dm-0-807:48:571
1009999289,16cyclictest21433-21munin-run11:15:020
1009999282,14cyclictest2262-21munin-run11:45:010
1009999282,14cyclictest2262-21munin-run11:45:010
1011099272,7cyclictest0-21swapper/110:44:061
1011099271,16cyclictest30923-21cpuspeed_turbos07:55:131
1011099271,11cyclictest0-21swapper/110:37:461
1011099270,11cyclictest12555-21/usr/sbin/munin12:05:261
1009999272,12cyclictest11399-21munin-run08:25:000
1011099261,11cyclictest11941-21switchtime10:50:231
1011099251,9cyclictest13590-21munin-run08:30:011
1011099251,16cyclictest21536-21taskset07:32:151
1011099251,11cyclictest0-21swapper/109:55:341
10120992414,3cyclictest12805-21sed12:05:273
1011099241,9cyclictest0-21swapper/111:54:431
1011099241,7cyclictest0-21swapper/112:31:421
1011099241,16cyclictest0-21swapper/112:13:261
1012099231,9cyclictest4657-21latency_hist11:50:023
10120992311,3cyclictest0-21swapper/311:50:203
1011099231,8cyclictest13153-21grep07:15:041
1011099231,19cyclictest24149-21memory10:05:191
1011099231,16cyclictest22020-21fschecks_count11:15:141
1011099231,16cyclictest0-21swapper/110:27:181
1011099231,16cyclictest0-21swapper/110:18:461
1011099231,16cyclictest0-21swapper/107:26:581
1011099231,10cyclictest0-21swapper/110:33:431
1011099231,10cyclictest0-21swapper/110:33:431
1011099230,14cyclictest0-21swapper/109:45:021
1012099222,13cyclictest0-21swapper/312:15:473
1012099221,11cyclictest1959-40dnf11:41:213
1012099221,11cyclictest1959-40dnf11:41:213
1011599227,4cyclictest0-21swapper/211:26:442
1011599221,6cyclictest0-21swapper/212:00:012
1011099223,11cyclictest31181-21diskmemload09:45:151
1011099221,9cyclictest0-21swapper/110:55:111
10110992214,2cyclictest0-21swapper/108:16:411
1011099221,3cyclictest17701-21ntp_states09:50:211
10110992213,3cyclictest0-21swapper/111:45:151
1011099221,14cyclictest0-21swapper/110:21:291
1011099221,12cyclictest0-21swapper/111:04:571
1011099221,11cyclictest1580-21df_inode09:15:111
1011099220,8cyclictest0-21swapper/110:00:341
1011099220,16cyclictest0-21swapper/108:40:161
213772210,1sleep00-21swapper/010:00:150
211912210,4sleep011-21rcuc/008:45:170
1012099217,5cyclictest405-1kworker/u9:011:35:083
1012099216,5cyclictest0-21swapper/312:33:273
1012099215,3cyclictest0-21swapper/312:10:183
1012099214,3cyclictest4669-21grep10:35:143
1012099213,12cyclictest29626-21kworker/3:109:53:263
1012099213,12cyclictest0-21swapper/311:58:153
10120992111,4cyclictest23844-21grep10:05:153
10120992111,3cyclictest30483-21switchtime09:05:253
1012099211,13cyclictest20930-21latency12:25:183
1012099211,11cyclictest0-21swapper/310:50:123
1011599217,5cyclictest17031-21crond11:05:022
1011599216,4cyclictest0-21swapper/212:36:152
1011599213,7cyclictest32-21ksoftirqd/210:30:172
1011599213,7cyclictest32-21ksoftirqd/210:30:172
1011599212,7cyclictest0-21swapper/211:45:132
1011599212,6cyclictest0-21swapper/211:41:242
1011599212,6cyclictest0-21swapper/211:41:232
1011599211,6cyclictest0-21swapper/211:06:282
1011599211,4cyclictest0-21swapper/212:19:492
1011099213,3cyclictest87550irq/128-enp1s0-10:45:131
1011099212,3cyclictest0-21swapper/112:35:021
1011099211,8cyclictest0-21swapper/111:20:351
1011099211,16cyclictest0-21swapper/111:27:271
1011099211,15cyclictest0-21swapper/108:06:561
1011099211,15cyclictest0-21swapper/107:54:511
1011099211,14cyclictest0-21swapper/107:40:041
1011099211,11cyclictest25028-21expr08:55:121
1011099211,10cyclictest7484-21gltestperf11:55:151
1011099210,7cyclictest0-21swapper/111:05:161
1009999214,15cyclictest19584-21sleep07:30:020
1009999213,2cyclictest87350irq/126-enp1s0-08:15:170
1009999212,10cyclictest4056-21/usr/sbin/munin10:35:190
1009999211,7cyclictest0-21swapper/010:30:250
1009999211,7cyclictest0-21swapper/010:30:250
1009999211,12cyclictest7838-21/usr/sbin/munin09:30:250
1009999211,12cyclictest24803-21/usr/sbin/munin08:55:150
1012099204,9cyclictest31181-21diskmemload09:46:153
1012099201,5cyclictest9328-21latency10:45:183
1012099201,12cyclictest0-21swapper/311:20:203
1012099201,12cyclictest0-21swapper/309:15:433
1012099201,10cyclictest827-21runrttasks09:35:213
10120992010,2cyclictest87550irq/128-enp1s0-10:17:443
1011599205,5cyclictest14838-21perf11:00:022
1011599203,11cyclictest17922-21taskset08:39:222
1011599201,7cyclictest2233-21head09:15:162
1011599201,7cyclictest0-21swapper/209:28:422
1011599201,6cyclictest499-21missed_timers10:25:212
1011099209,3cyclictest8752-21memory09:30:181
1011099209,3cyclictest6303-21gltestperf09:25:161
1011099205,3cyclictest14523-21ls08:30:161
1011099202,10cyclictest28738-21grep11:30:141
10110992016,3cyclictest0-21swapper/108:50:221
10110992013,4cyclictest10055-21sed08:20:161
10110992012,3cyclictest0-21swapper/108:47:421
1011099201,18cyclictest0-21swapper/109:24:511
1011099201,18cyclictest0-21swapper/109:24:511
1011099201,17cyclictest0-21swapper/109:10:301
1011099201,16cyclictest6097-21ntp_states08:10:231
1011099201,14cyclictest0-21swapper/112:28:491
1011099201,11cyclictest0-21swapper/112:24:361
1011099200,9cyclictest30078-21kworker/u8:011:40:111
1011099200,9cyclictest30078-21kworker/u8:011:40:111
1009999207,2cyclictest87350irq/126-enp1s0-12:10:150
1009999203,2cyclictest87550irq/128-enp1s0-12:03:020
1009999203,10cyclictest0-21swapper/011:24:030
1009999202,10cyclictest26385-21/usr/sbin/munin07:45:140
1009999201,6cyclictest0-21swapper/011:39:130
1009999201,13cyclictest0-21swapper/012:27:450
262432191,3sleep00-21swapper/012:37:390
250192191,2sleep20-21swapper/210:07:482
144882190,2sleep00-21swapper/010:55:480
1012099197,3cyclictest6257-21/usr/sbin/munin10:40:173
1012099196,7cyclictest87350irq/126-enp1s0-09:59:413
1012099196,3cyclictest28582-21/usr/sbin/munin07:50:133
1012099195,7cyclictest0-21swapper/310:00:373
1012099195,4cyclictest18951-21ntp_states12:20:223
1012099193,3cyclictest0-21swapper/311:11:343
1012099192,3cyclictest0-21swapper/309:32:263
1012099191,3cyclictest4579-21tr09:20:223
1012099191,3cyclictest4579-21tr09:20:213
1012099191,3cyclictest0-21swapper/307:10:243
1012099191,15cyclictest16118-21ntp_states11:00:203
1012099191,10cyclictest808-21snmpd08:18:033
1012099191,10cyclictest16015-21/usr/sbin/munin08:35:253
1012099190,3cyclictest0-21swapper/311:05:113
1012099190,14cyclictest0-21swapper/312:36:383
1012099190,10cyclictest5207-21kworker/u8:110:24:313
1011599195,5cyclictest22418-21cat08:50:022
1011599194,5cyclictest9044-21sshd08:18:202
1011599191,8cyclictest0-21swapper/210:00:262
1011599191,7cyclictest0-21swapper/212:24:312
1011599191,6cyclictest0-21swapper/211:12:422
1011599191,6cyclictest0-21swapper/209:58:192
10115991915,2cyclictest30221-21awk11:35:012
1011599191,4cyclictest0-21swapper/210:22:292
1011599191,15cyclictest808-21snmpd08:01:382
1011599190,4cyclictest4087-21kworker/u8:211:50:192
1011099198,2cyclictest10082-21cpu09:35:121
1011099191,15cyclictest0-21swapper/110:10:261
1011099191,10cyclictest0-21swapper/109:00:231
1011099190,9cyclictest0-21swapper/107:35:291
1011099190,4cyclictest0-21swapper/112:16:391
1011099190,14cyclictest0-21swapper/111:36:301
1011099190,14cyclictest0-21swapper/107:20:371
1009999199,3cyclictest21990-21/usr/sbin/munin07:35:250
1009999194,9cyclictest0-21swapper/011:55:590
1009999192,10cyclictest27709-21kworker/0:111:30:020
1009999191,6cyclictest0-21swapper/009:40:220
10099991913,4cyclictest5031-21/usr/sbin/munin11:50:150
1009999191,16cyclictest21757-21idleruntime-cro07:35:020
1009999191,12cyclictest0-21swapper/009:25:530
1009999191,10cyclictest32451-21sshd07:56:200
1009999191,10cyclictest18830-21/usr/sbin/munin09:55:160
1009999191,10cyclictest17879-21/usr/sbin/munin12:20:180
1009999191,10cyclictest17263-21/usr/sbin/munin11:05:260
272602180,2sleep20-21swapper/210:13:212
110532181,2sleep211194-21sed12:05:012
1012099188,7cyclictest441-21sshd10:25:203
1012099187,3cyclictest32480-21ntp_states09:10:233
1012099185,3cyclictest19384-21ps08:40:223
1012099185,2cyclictest87350irq/126-enp1s0-11:25:103
1012099182,3cyclictest22415-21ethtool11:15:173
1012099182,3cyclictest0-21swapper/310:31:533
1012099182,3cyclictest0-21swapper/310:31:533
1012099181,7cyclictest0-21swapper/312:00:273
10120991815,2cyclictest0-21swapper/309:25:153
10120991811,4cyclictest6537-21tail08:10:263
1012099181,12cyclictest0-21swapper/311:33:143
1012099180,12cyclictest24339-21kworker/u8:010:10:363
1011599183,2cyclictest87550irq/128-enp1s0-09:05:262
1011599181,8cyclictest0-21swapper/209:11:132
1011599181,7cyclictest18605-21munin-run09:55:022
1011599181,6cyclictest0-21swapper/210:37:162
1011599181,6cyclictest0-21swapper/209:24:462
1011599181,6cyclictest0-21swapper/209:24:462
1011599181,5cyclictest27103-21basename07:45:152
1011599181,3cyclictest23266-21strings12:30:182
10115991813,3cyclictest6725-21fschecks_time10:40:162
10115991811,4cyclictest579-21systemd-journal07:15:022
1011599181,10cyclictest0-21swapper/210:50:032
1009999189,2cyclictest7-21ksoftirqd/008:05:020
1009999188,8cyclictest11-21rcuc/009:20:020
1009999184,7cyclictest87550irq/128-enp1s0-12:05:210
1009999184,3cyclictest87550irq/128-enp1s0-10:10:170
1009999183,2cyclictest87350irq/126-enp1s0-10:45:160
1009999181,8cyclictest10705-21sed09:35:160
1009999181,7cyclictest0-21swapper/008:25:170
10099991813,3cyclictest6491-21grep08:10:260
10099991812,3cyclictest16135-21tr08:35:120
1009999181,11cyclictest0-21swapper/009:00:120
1009999180,3cyclictest87350irq/126-enp1s0-12:15:130
302632170,2sleep00-21swapper/007:51:250
160592171,2sleep20-21swapper/209:45:432
1012099177,2cyclictest11619-21/usr/sbin/munin08:25:143
1012099172,12cyclictest12237-21/usr/sbin/munin09:40:203
1012099171,9cyclictest30780-21/usr/sbin/munin07:55:173
1012099171,8cyclictest26385-21/usr/sbin/munin07:45:223
1012099171,8cyclictest0-21swapper/307:35:263
1012099171,3cyclictest808-21snmpd08:52:153
1012099171,3cyclictest0-21swapper/307:20:263
10120991712,3cyclictest13265-21cpu07:15:123
1012099171,13cyclictest27527-21fschecks_time09:00:153
10120991711,3cyclictest18483-21cat07:25:183
1012099171,10cyclictest0-21swapper/308:55:223
1012099171,10cyclictest0-21swapper/308:21:173
1011599175,3cyclictest87350irq/126-enp1s0-07:55:152
1011599174,3cyclictest87650irq/129-enp1s0-07:35:112
1011599172,12cyclictest25024-21proc_pri11:20:212
1011599171,6cyclictest30099-21tail07:50:242
1011599171,6cyclictest0-21swapper/211:15:192
1011599171,5cyclictest13882-21hddtemp_smartct07:15:162
1011599171,5cyclictest11474-21cstates12:05:142
1011599171,5cyclictest0-21swapper/210:46:482
1011599171,5cyclictest0-21swapper/210:15:192
1011599171,5cyclictest0-21swapper/209:34:302
10115991713,3cyclictest0-21swapper/209:03:012
10115991712,4cyclictest0-21swapper/209:35:142
10115991711,4cyclictest4108-21kworker/u8:007:30:022
1011599171,14cyclictest0-21swapper/208:30:382
10115991711,3cyclictest18963-21timerandwakeup07:25:242
10115991710,4cyclictest26364-21grep08:55:262
1011099178,3cyclictest0-21swapper/109:05:171
1009999172,14cyclictest0-21swapper/011:05:010
1009999171,7cyclictest0-21swapper/009:12:160
1009999171,6cyclictest0-21swapper/010:20:180
1009999171,6cyclictest0-21swapper/010:09:070
1009999171,5cyclictest0-21swapper/011:15:120
1009999171,4cyclictest27626-21/usr/sbin/munin10:15:180
1009999171,3cyclictest0-21swapper/012:32:200
1009999171,3cyclictest0-21swapper/009:21:420
1009999171,3cyclictest0-21swapper/009:21:420
10099991713,3cyclictest7-21ksoftirqd/008:31:040
10099991713,3cyclictest7-21ksoftirqd/008:09:420
10099991712,3cyclictest14234-21awk07:15:200
1009999171,13cyclictest6353-21grep10:40:120
1009999170,5cyclictest31191-21awk09:10:010
1012099163,3cyclictest20869-21ntpq07:30:223
1012099161,3cyclictest0-21swapper/307:40:143
10120991612,3cyclictest0-21swapper/308:08:233
1012099161,14cyclictest0-21swapper/308:03:573
1012099161,12cyclictest13817-21/usr/sbin/munin08:30:163
1011599165,3cyclictest87350irq/126-enp1s0-08:25:132
1011599161,9cyclictest0-21swapper/208:05:172
10115991610,3cyclictest24405-21expr07:40:132
1009999161,6cyclictest0-21swapper/008:40:170
1009999161,14cyclictest0-21swapper/009:51:080
10099991611,3cyclictest29294-21ntpq11:30:200
10099991611,3cyclictest14804-21grep09:45:140
1009999160,9cyclictest0-21swapper/007:40:160
1011599158,4cyclictest23838-21sendmail_mailqu08:50:232
1011599155,8cyclictest5122-21grep08:10:142
10115991510,3cyclictest16410-21netstat07:20:192
1009999153,3cyclictest87550irq/128-enp1s0-11:45:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional