You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-28 - 16:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Mar 28, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
323072236162,24sleep10-21swapper/107:06:281
322942232161,58sleep30-21swapper/307:06:193
323952203166,25sleep20-21swapper/207:07:382
323982200163,25sleep00-21swapper/007:07:400
32678999895,2cyclictest0-21swapper/308:24:253
32678998986,2cyclictest0-21swapper/307:34:213
32678998582,2cyclictest0-21swapper/310:16:353
32678998381,1cyclictest0-21swapper/309:25:293
32678998279,2cyclictest0-21swapper/308:17:063
117272760,5sleep011729-21latency_hist11:15:010
32678997571,3cyclictest0-21swapper/309:10:023
32678997571,3cyclictest0-21swapper/309:10:023
32678995956,2cyclictest0-21swapper/311:43:183
259202540,5sleep13266599cyclictest08:05:161
111772510,4sleep339-21rcuc/308:45:153
40182470,2sleep10-21swapper/110:55:201
3267199352,29cyclictest829-21sed09:35:142
32671993511,4cyclictest851-21sssd_nss08:25:202
32671993511,3cyclictest16575-21/usr/sbin/munin07:45:122
3265599355,14cyclictest3790-21munin-run08:30:010
32671993411,4cyclictest851-21sssd_nss10:45:232
32671993411,3cyclictest11216-21/usr/sbin/munin10:00:152
32671993410,4cyclictest851-21sssd_nss10:25:232
32671993410,4cyclictest851-21sssd_nss10:25:232
138372340,1sleep00-21swapper/012:30:260
3267899331,23cyclictest6103-21kworker/3:107:51:263
3267899331,23cyclictest6103-21kworker/3:107:10:403
3267199333,4cyclictest27355-21cpuspeed_turbos11:50:112
32671993314,15cyclictest21595-21grep07:55:162
32671993311,3cyclictest14430-21latency_hist12:35:012
32671993310,4cyclictest16374-21tr11:25:132
32671993310,3cyclictest4016-21/usr/sbin/munin08:30:252
32671993310,3cyclictest21911-21sshd10:24:122
3267899321,22cyclictest6103-21kworker/3:110:54:213
3267199323,6cyclictest5459-40dnf12:12:502
3267199322,7cyclictest9981-21df11:10:132
3267199322,5cyclictest16628-21sh12:40:012
3267199321,6cyclictest21450-21irqstats11:35:182
3267199321,6cyclictest21450-21irqstats11:35:182
32671993211,3cyclictest3540-21pluginstate09:40:222
32671993211,3cyclictest10316-21expr07:30:152
32671993210,4cyclictest851-21sssd_nss07:35:232
32671993210,2cyclictest833-21runrttasks08:52:582
3267199319,4cyclictest0-21swapper/210:15:132
3267199319,4cyclictest0-21swapper/208:20:152
3267199319,3cyclictest14139-21sort11:20:142
3267199319,2cyclictest833-21runrttasks08:15:462
3267199311,6cyclictest6599-21latency_hist09:50:022
3267199311,6cyclictest0-21swapper/210:30:232
32671993113,4cyclictest15456-21ntp_states07:40:202
3267199311,27cyclictest24636-21grep08:00:262
3267199311,27cyclictest24636-21grep08:00:262
32671993112,3cyclictest3188-21cat07:15:042
32671993111,3cyclictest6398-21proc_pri11:00:232
32671993111,3cyclictest6234-21grep08:35:112
3267199310,5cyclictest0-21swapper/212:20:212
3267899301,22cyclictest6103-21kworker/3:111:00:083
3267199309,3cyclictest6619-21cat12:15:182
3267199309,3cyclictest17280-21cpuspeed_turbos09:00:132
3267199306,6cyclictest0-21swapper/211:55:142
3267199305,3cyclictest86850irq/129-enp1s0-09:50:242
3267199304,8cyclictest10601-21/usr/sbin/munin08:45:192
3267199301,5cyclictest24043-21kworker/u8:210:35:122
32671993012,3cyclictest26112-21grep09:20:132
32671993011,3cyclictest19167-21dump-pmu-power09:09:562
32671993011,3cyclictest13421-21/usr/sbin/munin10:05:192
32671993011,2cyclictest833-21runrttasks08:06:592
32671993010,3cyclictest833-21runrttasks09:12:532
32671993010,3cyclictest833-21runrttasks09:12:532
32671993010,3cyclictest17021-21timerwakeupswit10:10:252
3267899291,17cyclictest6103-21kworker/3:110:09:283
3267199299,3cyclictest2323-21latency12:05:182
3267199291,7cyclictest0-21swapper/211:45:292
3267199291,6cyclictest21371-21diskmemload11:18:292
3267199291,6cyclictest0-21swapper/211:10:022
3267199291,5cyclictest10082-21ntp_states09:55:192
3267199291,4cyclictest0-21swapper/210:43:522
32671992912,3cyclictest19462-21hwlatdetect07:50:172
32671992910,3cyclictest1082-21expr10:50:132
3265599294,12cyclictest20746-21munin-run07:55:010
3267899282,8cyclictest6103-21kworker/3:111:19:333
3267199289,4cyclictest10601-21df12:25:132
3267199289,3cyclictest783-21expr07:10:132
3267199282,11cyclictest24950-21tr09:15:242
3267199281,7cyclictest21371-21diskmemload09:34:112
3267199280,2cyclictest86850irq/129-enp1s0-11:40:412
3267199280,11cyclictest7097-21unixbench_multi07:20:262
3266599282,18cyclictest0-21swapper/110:52:221
3267899272,8cyclictest6103-21kworker/3:110:27:523
3267899272,8cyclictest6103-21kworker/3:110:27:523
3267899270,21cyclictest6103-21kworker/3:109:52:573
3267199271,5cyclictest3830-21if_err_enp2s010:55:172
3267199271,5cyclictest0-21swapper/212:00:342
3267199271,2cyclictest0-21swapper/207:29:092
3267199271,14cyclictest0-21swapper/208:59:392
3267899262,9cyclictest20117-21kworker/3:212:32:393
3267899262,16cyclictest6103-21kworker/3:110:41:203
3267899262,16cyclictest6103-21kworker/3:109:05:503
3267199269,3cyclictest28751-21fschecks_time09:25:132
3267199269,3cyclictest18966-21fschecks_time11:30:152
3265599261,13cyclictest31756-21awk08:20:010
3267899252,16cyclictest6103-21kworker/3:109:57:023
3267899251,9cyclictest6103-21kworker/3:107:55:433
3267899251,9cyclictest6103-21kworker/3:107:45:133
3267899251,17cyclictest20117-21kworker/3:212:14:253
3267899251,11cyclictest6103-21kworker/3:108:03:393
3267899251,11cyclictest6103-21kworker/3:108:03:393
3267899251,10cyclictest6103-21kworker/3:108:11:103
3267899250,15cyclictest6103-21kworker/3:109:20:273
3267199252,2cyclictest0-21swapper/208:40:172
3266599252,12cyclictest6609-21munin-run09:50:011
32665992519,3cyclictest24811-21perf11:45:011
3265599251,11cyclictest6222-21vmstat09:45:270
42152241,4sleep34316-21memory07:15:183
3267899242,9cyclictest6103-21kworker/3:111:22:053
3267899242,8cyclictest20117-21kworker/3:211:45:153
3267899242,11cyclictest6103-21kworker/3:110:14:513
3267899241,9cyclictest6103-21kworker/3:111:05:043
3267899241,16cyclictest20117-21kworker/3:212:17:153
3267899241,15cyclictest20117-21kworker/3:212:25:113
32665992417,4cyclictest0-21swapper/112:05:191
32665992415,2cyclictest15112-21if_enp1s007:40:161
3266599241,19cyclictest13818-21munin-run11:20:001
3267899232,9cyclictest6103-21kworker/3:111:30:053
3267899232,8cyclictest6103-21kworker/3:107:40:443
3267899232,7cyclictest6103-21kworker/3:109:44:553
3267899232,11cyclictest6103-21kworker/3:110:24:223
3267899230,8cyclictest6103-21kworker/3:109:00:323
3266599234,6cyclictest23262-40dnf09:11:151
3266599234,6cyclictest23262-40dnf09:11:151
3266599234,14cyclictest5459-40dnf12:12:511
3266599231,19cyclictest31757-21if_enp2s010:45:181
3266599231,16cyclictest0-21swapper/107:38:381
3266599231,10cyclictest31748-21munin-run08:20:011
286982230,1sleep00-21swapper/010:40:010
3267899229,4cyclictest25476-21ntp_states10:30:223
3267899229,2cyclictest6103-21kworker/3:108:06:473
3267899222,9cyclictest6103-21kworker/3:108:25:403
3267899222,8cyclictest6103-21kworker/3:109:16:083
3267899222,8cyclictest20117-21kworker/3:212:22:313
3267899222,8cyclictest20117-21kworker/3:211:36:323
3267899222,8cyclictest20117-21kworker/3:211:36:313
3267899221,8cyclictest6103-21kworker/3:109:45:093
3267899221,8cyclictest6103-21kworker/3:108:42:323
3267899221,8cyclictest6103-21kworker/3:107:29:083
3266599222,15cyclictest0-21swapper/107:49:531
32665992218,3cyclictest0-21swapper/111:11:031
32665992215,2cyclictest0-21swapper/107:33:211
3266599221,3cyclictest9139-21ntp_states12:20:221
3266599221,18cyclictest0-21swapper/109:56:461
3266599221,16cyclictest0-21swapper/110:40:151
3265599220,12cyclictest0-21swapper/010:28:380
3265599220,12cyclictest0-21swapper/010:28:380
3267899212,9cyclictest6103-21kworker/3:110:47:493
3267899212,8cyclictest16971-21munin-run09:00:013
3267899212,10cyclictest20117-21kworker/3:212:36:273
3267899211,9cyclictest20117-21kworker/3:212:01:353
3267899211,8cyclictest6103-21kworker/3:109:36:483
3267899211,8cyclictest20117-21kworker/3:212:08:233
3267899211,7cyclictest6103-21kworker/3:110:01:373
3266599214,8cyclictest0-21swapper/110:07:421
3266599212,15cyclictest0-21swapper/108:31:021
32665992117,3cyclictest0-21swapper/111:58:121
32665992113,3cyclictest0-21swapper/108:51:191
32665992113,3cyclictest0-21swapper/108:27:361
32665992113,3cyclictest0-21swapper/108:21:021
3266599211,18cyclictest0-21swapper/112:25:171
3266599211,18cyclictest0-21swapper/111:20:061
3266599211,18cyclictest0-21swapper/110:34:051
3266599211,18cyclictest0-21swapper/109:23:001
3266599211,18cyclictest0-21swapper/107:29:071
3266599211,16cyclictest0-21swapper/107:51:491
3266599211,15cyclictest0-21swapper/108:48:541
3266599211,15cyclictest0-21swapper/108:35:031
3266599211,15cyclictest0-21swapper/108:00:261
3266599211,15cyclictest0-21swapper/108:00:261
3266599210,15cyclictest0-21swapper/112:36:201
3265599217,5cyclictest0-21swapper/010:09:560
324652211,3sleep032581-21missed_timers12:00:170
205412210,4sleep011-21rcuc/009:05:220
3267899208,2cyclictest20117-21kworker/3:211:56:053
3267899207,2cyclictest6103-21kworker/3:107:22:243
3267899203,2cyclictest6103-21kworker/3:108:50:363
3267899201,9cyclictest6103-21kworker/3:111:11:203
3267899201,9cyclictest6103-21kworker/3:109:34:163
3267899201,8cyclictest6103-21kworker/3:108:32:563
3267899201,11cyclictest6103-21kworker/3:110:58:593
3267899201,10cyclictest26606-21/usr/sbin/munin10:35:143
3267899201,10cyclictest0-21swapper/307:35:193
3267899200,11cyclictest20117-21kworker/3:211:53:513
3267899200,10cyclictest6103-21kworker/3:111:28:273
3267199205,3cyclictest28255-21hddtemp_smartct08:10:182
3266599202,16cyclictest0-21swapper/110:15:081
3266599202,10cyclictest18711-21expr11:30:121
3266599201,8cyclictest0-21swapper/111:08:541
3266599201,7cyclictest0-21swapper/109:34:231
32665992014,3cyclictest17832-21gltestperf09:00:171
32665992013,2cyclictest0-21swapper/107:14:031
3266599201,17cyclictest0-21swapper/109:16:571
3266599201,17cyclictest0-21swapper/108:41:171
3266599201,16cyclictest21371-21diskmemload11:49:051
3266599201,15cyclictest0-21swapper/108:56:451
3266599201,14cyclictest0-21swapper/112:34:561
3266599201,14cyclictest0-21swapper/107:22:031
3266599201,14cyclictest0-21swapper/107:17:221
3266599201,11cyclictest27566-21df11:50:121
3266599201,11cyclictest21322-21diskstats07:55:151
3266599201,11cyclictest1400-21cat12:05:021
3266599200,4cyclictest0-21swapper/110:35:211
3266599200,18cyclictest0-21swapper/110:25:171
3266599200,18cyclictest0-21swapper/110:25:171
3266599200,18cyclictest0-21swapper/109:25:161
3266599200,15cyclictest0-21swapper/108:14:291
3265599207,3cyclictest0-21swapper/010:30:210
3265599206,5cyclictest0-21swapper/011:02:530
3265599202,10cyclictest25376-21/usr/sbin/munin08:05:240
3265599202,10cyclictest25033-21/usr/sbin/munin11:45:180
3265599202,10cyclictest11216-21/usr/sbin/munin10:00:160
3265599202,10cyclictest0-21swapper/009:12:120
3265599202,10cyclictest0-21swapper/009:12:120
3265599201,16cyclictest7-21ksoftirqd/008:20:060
3265599201,11cyclictest20633-21/usr/sbin/munin11:35:230
3265599201,11cyclictest20633-21/usr/sbin/munin11:35:230
3265599201,11cyclictest0-21swapper/010:52:530
3265599201,10cyclictest21371-21diskmemload12:10:140
3265599201,10cyclictest0-21swapper/007:25:150
316182200,5sleep03265599cyclictest09:30:220
3266599194,4cyclictest0-21swapper/109:36:581
3266599191,8cyclictest0-21swapper/110:13:111
32665991915,3cyclictest0-21swapper/112:16:061
32665991915,3cyclictest0-21swapper/111:25:591
3266599191,17cyclictest0-21swapper/109:41:081
3266599191,16cyclictest0-21swapper/109:51:371
3266599191,16cyclictest0-21swapper/109:07:291
3266599191,10cyclictest20849-21expr11:35:121
3266599191,10cyclictest20849-21expr11:35:121
3265599198,3cyclictest25770-21munin-run09:20:010
3265599198,2cyclictest16140-21pmu-power08:55:240
3265599197,3cyclictest0-21swapper/011:57:240
3265599196,2cyclictest21371-21diskmemload09:35:290
3265599191,9cyclictest0-21swapper/011:25:110
3265599191,9cyclictest0-21swapper/009:20:110
3265599191,6cyclictest0-21swapper/010:43:270
3265599191,6cyclictest0-21swapper/009:41:180
3265599191,11cyclictest21371-21diskmemload11:20:210
3265599191,11cyclictest0-21swapper/011:50:160
3265599191,10cyclictest0-21swapper/012:35:550
3265599191,10cyclictest0-21swapper/012:23:500
3265599191,10cyclictest0-21swapper/011:34:590
3265599191,10cyclictest0-21swapper/011:06:410
3265599191,10cyclictest0-21swapper/009:25:350
218232191,3sleep00-21swapper/010:23:060
3267899186,2cyclictest6103-21kworker/3:108:36:223
3266599181,7cyclictest0-21swapper/111:01:571
3266599181,16cyclictest0-21swapper/110:01:521
32665991811,4cyclictest851-21sssd_nss10:20:251
3265599188,3cyclictest0-21swapper/012:05:240
3265599181,9cyclictest10839-21taskset09:58:130
3265599181,5cyclictest0-21swapper/010:15:530
32655991813,3cyclictest6228-21bc08:35:120
32655991812,3cyclictest17193-21/usr/sbin/munin09:00:160
32655991811,4cyclictest6243-21/usr/sbin/munin12:15:160
32655991811,4cyclictest0-21swapper/010:55:250
3265599181,11cyclictest0-21swapper/012:27:440
32655991810,2cyclictest833-21runrttasks08:14:030
3265599180,7cyclictest7533-21kworker/0:007:49:280
3265599180,5cyclictest0-21swapper/011:40:400
3265599180,10cyclictest5603-21/usr/sbin/munin07:20:180
3265599175,3cyclictest0-21swapper/010:45:370
3265599173,12cyclictest0-21swapper/007:40:150
32655991712,3cyclictest403-21/usr/sbin/munin07:10:120
3265599171,15cyclictest0-21swapper/010:11:060
3265599171,14cyclictest0-21swapper/007:39:200
32655991711,3cyclictest3129-21if_err_enp2s007:15:040
32655991711,3cyclictest21875-21cat07:55:190
32655991711,3cyclictest10020-21grep07:30:130
147282171,2sleep00-21swapper/008:54:410
3265599161,9cyclictest0-21swapper/011:15:250
3265599161,3cyclictest5008-21cut08:30:210
32655991612,3cyclictest0-21swapper/008:48:250
32655991611,3cyclictest9776-21timerandwakeup08:40:260
32655991611,3cyclictest6599-21latency_hist09:50:020
3265599161,12cyclictest23171-21/usr/sbin/munin08:00:120
3265599161,12cyclictest23171-21/usr/sbin/munin08:00:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional