You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-04 - 14:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Fri Sep 04, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
220682238163,24sleep20-21swapper/207:07:072
221902235162,60sleep00-21swapper/007:08:420
221292230163,54sleep10-21swapper/107:07:541
219472203167,24sleep30-21swapper/307:05:323
1409321570,5sleep32238999cyclictest09:15:133
744621270,4sleep02236899cyclictest07:45:210
22384998076,3cyclictest0-21swapper/211:45:562
202092620,12sleep30-21swapper/311:55:233
136782500,4sleep011-21rcuc/009:15:100
107932470,1sleep10-21swapper/107:55:091
11502440,2sleep30-21swapper/310:00:133
2236899379,12cyclictest29313-21awk11:05:010
2238999351,30cyclictest0-21swapper/307:49:093
2236899324,13cyclictest13918-21basename10:30:010
2238999311,12cyclictest0-21swapper/310:56:233
234012300,2sleep30-21swapper/312:05:023
2238499308,2cyclictest27715-21latency_hist12:15:022
183362300,1sleep00-21swapper/009:25:140
22389992916,3cyclictest0-21swapper/307:32:393
2236899291,16cyclictest87650irq/129-enp1s0-07:25:150
84012280,5sleep230-21rcuc/209:00:242
2237999280,21cyclictest0-21swapper/110:27:411
2236899281,10cyclictest22264-21kworker/0:007:13:370
22389992716,3cyclictest0-21swapper/308:36:433
22389992712,9cyclictest0-21swapper/311:03:073
2238999271,13cyclictest0-21swapper/311:43:433
2238999271,11cyclictest0-21swapper/310:25:203
2238499273,8cyclictest30865-21latency_hist09:55:032
2238499271,10cyclictest0-21swapper/212:04:572
2237999271,13cyclictest24551-21latency_hist07:15:011
2238999267,3cyclictest0-21swapper/308:19:513
22389992618,3cyclictest0-21swapper/311:16:593
22389992613,3cyclictest0-21swapper/310:11:443
22389992611,3cyclictest0-21swapper/310:17:003
2238999261,12cyclictest0-21swapper/310:40:433
2238499269,5cyclictest30199-21cstates07:25:142
2238499268,2cyclictest16329-21/usr/sbin/munin10:35:172
22384992615,6cyclictest25152-40dnf12:05:282
2236899260,18cyclictest10392-21kworker/0:207:55:210
2238999259,8cyclictest0-21swapper/307:18:473
22389992517,3cyclictest0-21swapper/310:32:353
22389992516,3cyclictest0-21swapper/311:33:073
22389992516,3cyclictest0-21swapper/311:33:073
2238999251,4cyclictest4082-21missed_timers10:05:183
22389992514,3cyclictest0-21swapper/309:53:513
2238999251,2cyclictest0-21swapper/311:50:203
22389992511,3cyclictest0-21swapper/308:07:283
2238999251,11cyclictest22354-21hddtemp_smartct08:20:153
2238999251,10cyclictest0-21swapper/310:39:393
2238499259,3cyclictest2577-21kworker/u8:010:17:252
2238499252,8cyclictest2968-21date10:05:012
22368992510,3cyclictest16087-21kworker/0:210:35:220
2238999248,3cyclictest0-21swapper/308:42:033
2238999246,3cyclictest0-21swapper/308:56:113
22389992410,8cyclictest0-21swapper/309:21:273
22389992410,3cyclictest31416-21swap12:20:253
2238499249,3cyclictest0-21swapper/212:35:532
2238499248,7cyclictest32063-21/usr/sbin/munin07:30:152
2238499247,8cyclictest30142-21/usr/sbin/munin12:20:192
2238499247,8cyclictest10350-21/usr/sbin/munin11:35:252
2238499245,6cyclictest7117-21runrttasks10:12:412
2238499245,11cyclictest27940-21taskset12:15:082
2238499244,11cyclictest16841-21pmu-power09:20:232
2238499243,7cyclictest3545-21awk11:20:012
2238499240,7cyclictest26472-21latency_hist09:45:022
2237999241,18cyclictest0-21swapper/109:53:431
2236899249,3cyclictest14713-21kworker/0:308:16:130
2236899247,5cyclictest2596-21kworker/0:110:08:050
2236899247,3cyclictest12067-21kworker/0:311:55:210
2236899246,15cyclictest19262-21munin-run08:15:010
2236899241,10cyclictest32021-21kworker/0:212:34:140
2238999239,8cyclictest0-21swapper/309:45:553
2238999238,3cyclictest11086-21diskmemload12:29:113
2238999237,3cyclictest0-21swapper/311:48:203
2238999231,20cyclictest0-21swapper/310:50:153
2238999230,14cyclictest24829-21switchtime12:05:233
2238499238,2cyclictest827-21runrttasks10:31:082
2238499236,8cyclictest20244-21pmu-power11:55:232
2238499235,7cyclictest87350irq/126-enp1s0-08:08:022
2238499234,7cyclictest4315-21perl11:20:162
2238499234,6cyclictest15068-21unixbench_singl09:15:242
2238499231,10cyclictest0-21swapper/208:38:032
2238499230,15cyclictest11248-21fschecks_time07:55:122
2238499230,12cyclictest2282-21latency_hist08:50:012
2238499230,11cyclictest0-21swapper/208:33:032
22379992317,3cyclictest8877-21grep07:50:141
2237999231,16cyclictest0-21swapper/112:36:541
2237999231,15cyclictest0-21swapper/110:43:061
22379992310,11cyclictest3777-21kworker/u8:209:30:081
2237999230,17cyclictest0-21swapper/110:05:401
2236899237,4cyclictest23640-21kworker/0:208:27:180
2236899235,3cyclictest12067-21kworker/0:312:00:290
2236899232,10cyclictest24538-21kworker/0:107:41:410
2238999229,8cyclictest0-21swapper/310:48:593
2238999229,3cyclictest24230-21df_inode08:25:133
2238999229,3cyclictest0-21swapper/307:35:073
2238999227,3cyclictest0-21swapper/307:58:043
2238999221,9cyclictest0-21swapper/307:11:193
2238999221,6cyclictest0-21swapper/311:20:153
22389992214,3cyclictest0-21swapper/309:11:513
22389992214,3cyclictest0-21swapper/309:11:513
22389992213,3cyclictest0-21swapper/309:42:313
22389992211,7cyclictest0-21swapper/312:36:233
2238999220,12cyclictest0-21swapper/307:22:193
2238499225,8cyclictest11703-21latency_hist10:25:012
2238499225,8cyclictest11703-21latency_hist10:25:012
2238499225,3cyclictest0-21swapper/208:02:332
2238499225,3cyclictest0-21swapper/208:02:332
2238499222,6cyclictest11086-21diskmemload11:08:552
2238499221,5cyclictest0-21swapper/209:10:352
2238499221,5cyclictest0-21swapper/209:10:352
22384992215,4cyclictest32224-21ls09:55:212
2237999221,19cyclictest0-21swapper/107:17:391
2237999221,14cyclictest0-21swapper/112:08:121
2236899229,3cyclictest21404-21kworker/0:108:21:060
2236899229,3cyclictest16087-21kworker/0:210:44:260
2236899229,2cyclictest12067-21kworker/0:311:45:170
2236899228,4cyclictest8159-21kworker/0:107:51:530
2236899228,3cyclictest31436-21kworker/0:111:11:450
2236899228,3cyclictest27210-21kworker/0:008:53:140
2236899228,2cyclictest7084-21kworker/0:010:22:490
2236899228,2cyclictest7084-21kworker/0:010:22:490
2236899228,2cyclictest24750-21kworker/0:311:05:210
2236899227,3cyclictest32021-21kworker/0:212:27:100
2236899226,4cyclictest24538-21kworker/0:107:38:170
2236899226,4cyclictest17450-21kworker/0:109:32:170
2236899226,2cyclictest23305-21kworker/0:012:22:220
2236899223,12cyclictest87650irq/129-enp1s0-11:15:230
22368992213,2cyclictest87650irq/129-enp1s0-10:53:320
2236899221,18cyclictest32452-21munin-run08:45:020
2236899221,10cyclictest2596-21kworker/0:110:16:250
22368992210,3cyclictest2596-21kworker/0:110:10:210
105772220,2sleep00-21swapper/011:35:130
2238999218,3cyclictest0-21swapper/310:23:363
2238999218,3cyclictest0-21swapper/310:23:363
2238999216,3cyclictest0-21swapper/311:14:273
2238999216,3cyclictest0-21swapper/308:03:243
2238999216,3cyclictest0-21swapper/308:03:243
2238999216,3cyclictest0-21swapper/307:43:233
2238999213,7cyclictest0-21swapper/309:34:113
2238999211,14cyclictest0-21swapper/312:34:033
2238999211,14cyclictest0-21swapper/312:15:393
2238499216,2cyclictest23886-21/usr/sbin/munin08:25:252
2238499215,3cyclictest11086-21diskmemload10:05:132
2238499214,8cyclictest30476-21/usr/sbin/munin08:40:122
2238499213,6cyclictest808-21snmpd12:30:312
2238499210,11cyclictest0-21swapper/211:00:152
2237999217,11cyclictest28688-21timerandwakeup11:00:251
2237999212,11cyclictest1996-21ntp_states10:00:211
22379992113,3cyclictest0-21swapper/112:20:051
22379992112,2cyclictest808-21snmpd11:55:071
2237999211,17cyclictest618-21if_enp1s012:25:171
2237999211,14cyclictest0-21swapper/111:50:431
2237999211,10cyclictest29223-21switchtime12:15:241
22379992110,3cyclictest565-21/usr/sbin/munin08:45:131
22379992110,3cyclictest0-21swapper/109:10:021
2236899219,2cyclictest10744-21kworker/0:009:10:210
2236899219,2cyclictest10744-21kworker/0:009:10:210
2236899218,4cyclictest27511-21irqstats09:45:160
2236899218,3cyclictest24538-21kworker/0:107:30:570
2236899218,3cyclictest17450-21kworker/0:109:41:330
2236899217,2cyclictest31436-21kworker/0:111:21:210
2236899216,2cyclictest7701-21kworker/0:211:29:530
2236899212,11cyclictest20717-21/usr/sbin/munin10:45:260
2236899211,10cyclictest29944-21kworker/0:208:36:460
2238999206,3cyclictest0-21swapper/309:02:193
2238999205,3cyclictest0-21swapper/311:25:033
2238999202,3cyclictest0-21swapper/309:08:113
2238999201,6cyclictest0-21swapper/309:38:373
22389992013,4cyclictest18058-21tr09:25:113
2238999201,11cyclictest10350-21/usr/sbin/munin11:35:193
2238499206,3cyclictest0-21swapper/207:10:132
2238499204,7cyclictest12035-21cpuspeed_turbos10:25:122
2238499203,14cyclictest25632-21tr07:15:122
2238499203,10cyclictest11086-21diskmemload11:51:022
2238499201,9cyclictest0-21swapper/209:25:222
2238499201,6cyclictest2513-21/usr/sbin/munin08:50:142
2238499201,13cyclictest22307-21users10:45:252
2238499201,12cyclictest0-21swapper/211:25:212
2237999207,4cyclictest17750-21latency_hist09:25:011
22379992017,2cyclictest0-21swapper/109:42:441
22379992016,3cyclictest0-21swapper/110:33:541
2237999201,3cyclictest7013-21taskset10:11:211
2237999201,18cyclictest0-21swapper/108:02:111
2237999201,18cyclictest0-21swapper/108:02:111
2237999201,17cyclictest0-21swapper/108:25:221
2237999201,17cyclictest0-21swapper/107:40:151
2237999201,15cyclictest0-21swapper/112:34:131
2237999201,15cyclictest0-21swapper/111:46:591
2237999201,13cyclictest5935-21kworker/1:008:15:491
2237999201,11cyclictest22448-21pmu-power12:00:221
2237999201,10cyclictest8273-21perf07:50:001
2237999201,10cyclictest31695-21cat11:10:021
2237999201,10cyclictest23672-21idleruntime-cro08:25:011
2237999201,10cyclictest13955-21timerwakeupswit11:40:261
22379992010,7cyclictest6046-21cpuspeed_turbos11:25:111
2237999200,18cyclictest0-21swapper/109:11:081
2237999200,18cyclictest0-21swapper/109:11:071
2237999200,18cyclictest0-21swapper/108:05:211
2236899208,3cyclictest9958-21kworker/0:111:40:370
2236899207,4cyclictest12630-21kworker/0:008:02:010
2236899207,4cyclictest12630-21kworker/0:008:02:010
2236899207,2cyclictest5828-21kworker/0:112:39:100
2236899206,3cyclictest10744-21kworker/0:009:08:450
2236899206,2cyclictest5471-21kworker/0:011:33:210
2236899206,2cyclictest5471-21kworker/0:011:33:210
2236899203,2cyclictest21087-21kworker/0:112:05:490
2236899202,10cyclictest27945-21/usr/sbin/munin12:15:170
2236899201,5cyclictest0-21swapper/010:04:150
2236899201,11cyclictest27668-21/usr/sbin/munin07:20:190
22368992010,5cyclictest87650irq/129-enp1s0-10:34:190
179972200,4sleep32238999cyclictest08:10:163
267342191,2sleep00-21swapper/010:55:330
249682190,4sleep230-21rcuc/210:55:012
226352191,3sleep10-21swapper/110:49:241
2238999196,3cyclictest0-21swapper/308:49:033
2238999193,3cyclictest0-21swapper/312:12:353
2238999193,3cyclictest0-21swapper/307:53:473
2238999191,6cyclictest0-21swapper/307:29:153
22389991913,3cyclictest2513-21/usr/sbin/munin08:50:173
2238499194,3cyclictest0-21swapper/207:39:182
2238499192,8cyclictest0-21swapper/207:40:192
2238499191,4cyclictest0-21swapper/211:40:352
2238499191,4cyclictest0-21swapper/208:20:172
22384991912,4cyclictest23514-21processes09:35:222
2238499191,12cyclictest0-21swapper/209:49:432
2238499190,7cyclictest808-21snmpd07:50:132
2237999198,5cyclictest0-21swapper/111:24:161
2237999193,15cyclictest221ktimersoftd/109:49:581
22379991916,2cyclictest0-21swapper/110:20:191
22379991916,2cyclictest0-21swapper/110:20:191
22379991915,3cyclictest0-21swapper/107:25:311
2237999191,4cyclictest29726-21cat07:25:021
22379991913,3cyclictest9749-21users11:30:301
22379991913,3cyclictest9749-21users11:30:301
22379991913,3cyclictest2367-21fschecks_time07:35:131
2237999191,17cyclictest0-21swapper/109:18:271
2237999191,17cyclictest0-21swapper/108:35:181
2237999191,17cyclictest0-21swapper/108:11:201
2237999191,16cyclictest0-21swapper/108:56:001
2237999191,12cyclictest0-21swapper/109:57:551
2237999191,11cyclictest32295-21cstates07:30:111
2237999191,10cyclictest8073-21ps09:00:221
2237999191,10cyclictest395-21pmu-power11:10:271
2237999191,10cyclictest24299-21sed10:50:281
2237999191,10cyclictest18761-21expr09:25:171
2237999191,10cyclictest0-21swapper/110:19:021
2237999190,17cyclictest0-21swapper/110:55:221
2237999190,17cyclictest0-21swapper/109:35:111
2236899199,6cyclictest0-21swapper/007:19:570
2236899199,3cyclictest22300-21/usr/sbin/munin09:35:100
2236899196,4cyclictest27210-21kworker/0:008:32:540
2236899191,9cyclictest11086-21diskmemload12:10:160
22368991914,3cyclictest31464-21grep09:55:140
2236899191,10cyclictest4709-21/usr/sbin/munin08:55:220
19842190,3sleep22238499cyclictest12:30:012
106952190,8sleep123-21ksoftirqd/111:35:141
2238999182,3cyclictest0-21swapper/308:32:153
2238999180,7cyclictest0-21swapper/311:09:033
2238499184,6cyclictest28582-21memory07:20:182
2238499181,6cyclictest0-21swapper/210:56:042
2238499181,15cyclictest0-21swapper/207:46:272
22384991811,4cyclictest18522-21/usr/sbin/munin10:40:162
2238499181,11cyclictest0-21swapper/211:33:592
2238499181,11cyclictest0-21swapper/211:33:592
2238499180,9cyclictest0-21swapper/211:10:292
2238499180,4cyclictest22071-21latency_hist09:35:032
2237999181,9cyclictest0-21swapper/108:30:181
22379991814,3cyclictest0-21swapper/110:36:321
2237999181,16cyclictest0-21swapper/111:19:551
2237999181,16cyclictest0-21swapper/108:51:111
2237999181,16cyclictest0-21swapper/108:42:361
2237999181,11cyclictest0-21swapper/112:11:581
2236899183,5cyclictest0-21swapper/011:50:140
2236899181,6cyclictest0-21swapper/009:22:430
22368991812,3cyclictest320-21cpuspeed_turbos08:45:130
2236899181,11cyclictest0-21swapper/009:51:360
2238999171,12cyclictest0-21swapper/309:55:593
2238499173,11cyclictest17405-21tr08:10:122
2238499171,6cyclictest6674-21latency_hist09:00:012
2238499171,6cyclictest0-21swapper/209:07:172
2236899176,3cyclictest165050irq/12-ttyS009:00:190
2236899171,3cyclictest0-21swapper/008:05:100
2238499169,4cyclictest19833-21diskstats08:15:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional