You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-25 - 04:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Sat Jul 25, 2026 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1184823040,15sleep30-21swapper/319:05:123
134222237163,24sleep10-21swapper/119:09:261
133452236163,22sleep20-21swapper/219:08:262
133312202167,24sleep00-21swapper/019:08:170
2927521470,4sleep21356099cyclictest20:55:222
2250321390,2sleep20-21swapper/221:55:132
2577021160,2sleep01355099cyclictest19:35:140
13560999390,2cyclictest0-21swapper/221:22:132
13560999390,2cyclictest0-21swapper/220:12:412
13560998784,2cyclictest0-21swapper/223:34:442
40412670,1sleep10-21swapper/120:00:031
13560996662,3cyclictest0-21swapper/219:10:552
13560994441,2cyclictest0-21swapper/220:05:032
209432360,1sleep30-21swapper/323:05:133
13550993410,11cyclictest5708-21awk23:45:000
1355599281,2cyclictest87550irq/128-enp1s0-23:30:071
314132270,3sleep011-21rcuc/022:15:140
1356599271,6cyclictest0-21swapper/322:23:523
1356599261,11cyclictest13864-21grep21:35:153
1355599262,3cyclictest0-21swapper/121:53:031
1355599261,17cyclictest0-21swapper/122:45:401
178472250,1sleep20-21swapper/221:45:012
1356599251,2cyclictest87350irq/126-enp1s0-22:08:483
1356599250,11cyclictest0-21swapper/322:33:213
1355599258,4cyclictest18778-21sshd00:13:221
1355599252,15cyclictest0-21swapper/121:10:121
1355599251,17cyclictest0-21swapper/122:38:261
1355599251,13cyclictest0-21swapper/123:35:141
1355599251,10cyclictest0-21swapper/123:41:001
1355599250,7cyclictest0-21swapper/123:50:241
1355099251,6cyclictest0-21swapper/022:33:040
1355099251,5cyclictest0-21swapper/000:28:320
114362250,4sleep339-21rcuc/321:30:143
241562240,19sleep00-21swapper/019:30:200
1356599248,3cyclictest23238-21/usr/sbin/munin19:30:133
1356599241,5cyclictest0-21swapper/323:44:323
1356599241,3cyclictest0-21swapper/321:20:403
1356599241,20cyclictest18547-21hddtemp_smartct21:45:153
1356599241,15cyclictest0-21swapper/323:53:473
1356599241,11cyclictest0-21swapper/323:16:023
1355599247,6cyclictest21777-21kworker/u8:222:05:011
1355599244,10cyclictest0-21swapper/123:45:061
1355599242,3cyclictest15728-21munin-run19:15:021
1355599242,15cyclictest0-21swapper/119:51:591
1355599241,8cyclictest0-21swapper/121:41:331
1355599241,8cyclictest0-21swapper/121:41:331
1355599241,19cyclictest23627-21munin-run20:45:021
1355599241,17cyclictest0-21swapper/122:14:181
1355599241,10cyclictest0-21swapper/100:35:441
1355099241,5cyclictest28502-21kworker/u8:122:36:320
13550992413,3cyclictest10122chrt23:31:460
1356599239,3cyclictest14102-21/usr/sbin/munin22:50:283
1356599239,3cyclictest14102-21/usr/sbin/munin22:50:283
1356599231,11cyclictest16585-21chrt00:08:253
1356599231,11cyclictest0-21swapper/322:38:313
1356599231,10cyclictest17880-21sed00:10:203
1356599230,5cyclictest0-21swapper/322:46:563
1355599238,9cyclictest0-21swapper/123:22:391
1355599237,4cyclictest0-21swapper/100:19:121
1355599234,2cyclictest87550irq/128-enp1s0-21:15:201
1355599231,6cyclictest0-21swapper/121:48:021
13555992314,3cyclictest0-21swapper/122:20:511
1355599231,3cyclictest31805-21perf19:50:011
1355599231,19cyclictest16080-21perf22:55:021
1355599231,19cyclictest16080-21perf22:55:021
1355599231,16cyclictest0-21swapper/100:03:111
1355599231,16cyclictest0-21swapper/100:03:111
1355599231,15cyclictest808-21snmpd20:36:111
1355599231,12cyclictest0-21swapper/123:15:561
1355099236,9cyclictest87550irq/128-enp1s0-19:50:250
1355099231,3cyclictest0-21swapper/000:30:240
1355099231,13cyclictest0-21swapper/000:09:310
1355099231,12cyclictest0-21swapper/023:49:140
1355099231,11cyclictest0-21swapper/022:10:270
1356599225,3cyclictest1055-21sshd23:32:203
1356599221,6cyclictest0-21swapper/321:10:403
1356599221,18cyclictest20749-21hddtemp_smartct21:50:163
1356599221,12cyclictest0-21swapper/323:46:473
1356599221,11cyclictest32139-21ntp_states22:15:213
1356599221,11cyclictest0-21swapper/323:24:243
1356599221,11cyclictest0-21swapper/300:24:383
1356599221,10cyclictest24867-21latency_hist23:15:013
1356599221,10cyclictest0-21swapper/323:38:033
1356099221,5cyclictest0-21swapper/221:28:462
1355599223,11cyclictest909-21sssd_nss22:05:191
1355599222,15cyclictest0-21swapper/121:04:391
1355599221,6cyclictest0-21swapper/122:17:151
1355599221,6cyclictest0-21swapper/100:23:311
13555992213,3cyclictest0-21swapper/120:28:321
1355599221,19cyclictest0-21swapper/123:29:191
1355599221,16cyclictest0-21swapper/122:40:591
1355599221,16cyclictest0-21swapper/121:32:181
1355599221,16cyclictest0-21swapper/100:07:531
1355599221,15cyclictest0-21swapper/121:08:051
1355599221,15cyclictest0-21swapper/120:57:321
1355599221,15cyclictest0-21swapper/120:12:221
1355599221,15cyclictest0-21swapper/119:21:461
1355599221,15cyclictest0-21swapper/119:21:461
13555992211,3cyclictest17217-21memory22:55:201
1355599221,11cyclictest0-21swapper/121:26:281
13555992210,3cyclictest0-21swapper/121:55:101
1355099229,10cyclictest12304-21idleruntime-cro00:00:010
1355099226,4cyclictest18911-21idleruntime-cro00:15:010
1355099225,9cyclictest87550irq/128-enp1s0-00:20:170
1355099225,4cyclictest7-21ksoftirqd/021:20:220
1355099225,3cyclictest266712/usr/sbin/munin22:05:160
1355099225,10cyclictest7-21ksoftirqd/021:50:190
1355099222,10cyclictest32425-21munin-run21:05:010
1355099221,7cyclictest0-21swapper/023:28:430
1355099221,7cyclictest0-21swapper/021:39:490
1355099221,7cyclictest0-21swapper/000:37:400
1355099221,6cyclictest0-21swapper/023:22:020
1355099221,6cyclictest0-21swapper/023:14:220
1355099221,6cyclictest0-21swapper/021:28:520
1355099221,6cyclictest0-21swapper/021:17:380
13550992210,3cyclictest24671-21kernelversion20:45:170
305162211,16sleep330551-21ntp_states23:25:203
261132211,2sleep326221-21if_err_enp2s019:35:173
1356599219,2cyclictest6664-21awk21:20:013
1356599212,7cyclictest15674-21/usr/sbin/munin21:40:233
1356599212,7cyclictest15674-21/usr/sbin/munin21:40:223
1356599212,3cyclictest0-21swapper/322:29:193
1356599211,5cyclictest20513-21sshd19:21:183
1356599211,5cyclictest20513-21sshd19:21:183
1356599211,10cyclictest0-21swapper/322:42:443
1356599211,10cyclictest0-21swapper/320:14:183
1356599211,10cyclictest0-21swapper/300:25:303
1356099217,6cyclictest18245-1kworker/u9:200:11:152
1356099214,3cyclictest87350irq/126-enp1s0-22:15:012
1356099211,6cyclictest0-21swapper/222:41:182
1356099211,6cyclictest0-21swapper/221:46:592
1356099211,13cyclictest0-21swapper/221:09:592
1356099211,10cyclictest0-21swapper/222:02:372
1355599219,3cyclictest24600-21ntp_states00:25:221
1355599216,4cyclictest0-21swapper/123:01:261
1355599215,7cyclictest0-21swapper/122:27:171
1355599211,9cyclictest0-21swapper/123:12:141
13555992114,2cyclictest0-21swapper/119:30:571
13555992113,3cyclictest0-21swapper/123:56:071
13555992113,3cyclictest0-21swapper/120:07:241
1355599211,15cyclictest0-21swapper/120:54:221
1355599211,15cyclictest0-21swapper/120:04:191
1355599211,15cyclictest0-21swapper/119:42:561
1355599211,11cyclictest7686-21irqstats21:20:171
1355599211,11cyclictest0-21swapper/123:08:461
13555992110,3cyclictest18525-21smart_sda20:30:251
13555992110,3cyclictest18525-21smart_sda20:30:251
1355599210,6cyclictest0-21swapper/122:31:421
1355099216,7cyclictest0-21swapper/022:51:420
1355099216,7cyclictest0-21swapper/022:51:420
1355099212,11cyclictest16302-21/usr/sbin/munin22:55:200
1355099211,6cyclictest29110-21grep19:40:240
1355099211,5cyclictest0-21swapper/023:51:050
1355099211,4cyclictest13647-21kworker/u8:222:49:210
13550992114,4cyclictest18686-21irqstats21:45:170
1355099211,12cyclictest0-21swapper/022:44:470
1355099211,12cyclictest0-21swapper/021:33:320
1355099211,11cyclictest16647-21/usr/sbin/munin19:15:270
1355099211,10cyclictest18819-21head20:30:260
1355099211,10cyclictest18819-21head20:30:260
1355099211,10cyclictest0-21swapper/020:25:000
1355099210,10cyclictest20603-21tr20:35:220
1356599204,3cyclictest10739-21cat20:15:133
1356599203,4cyclictest24170-21sshd21:59:183
1356599201,7cyclictest0-21swapper/300:40:023
1356599201,6cyclictest16302-21/usr/sbin/munin22:55:193
1356599201,6cyclictest0-21swapper/319:49:183
1356599201,3cyclictest25722-21sensors_temp22:00:233
1356599201,2cyclictest2469-21grep19:55:163
1356599201,10cyclictest0-21swapper/321:29:443
1356099201,8cyclictest0-21swapper/222:29:452
1356099201,7cyclictest0-21swapper/220:40:192
1356099201,6cyclictest0-21swapper/223:20:302
1356099201,6cyclictest0-21swapper/200:25:122
1356099201,5cyclictest27929-21/usr/sbin/munin00:35:212
1356099201,5cyclictest17300-21taskset21:40:392
1356099201,4cyclictest0-21swapper/222:51:422
1356099201,4cyclictest0-21swapper/222:51:422
1356099201,3cyclictest27343-40dnf19:39:232
1356099201,3cyclictest0-21swapper/221:19:542
1355599209,3cyclictest11010-21diskstats20:15:161
13555992013,2cyclictest0-21swapper/100:32:571
1355599201,14cyclictest0-21swapper/120:46:271
1355599201,14cyclictest0-21swapper/120:22:571
1355599201,11cyclictest15445-21latency_hist21:40:011
1355599201,10cyclictest21057-21cpu19:25:131
1355099207,4cyclictest8539-21cpuspeed_turbos20:10:120
1355099204,4cyclictest22060-21timerandwakeup23:05:250
1355099203,9cyclictest4233-21pmu-power22:25:220
1355099202,10cyclictest2487-21/usr/sbin/munin21:10:260
1355099201,9cyclictest2114-21head23:35:130
1355099201,8cyclictest0-21swapper/000:15:060
1355099201,6cyclictest18245-1kworker/u9:222:23:580
1355099201,6cyclictest0-21swapper/023:01:160
1355099201,6cyclictest0-21swapper/000:02:200
1355099201,6cyclictest0-21swapper/000:02:200
1355099201,5cyclictest0-21swapper/023:16:580
1355099201,4cyclictest17034-21munin-run20:30:010
1355099201,3cyclictest0-21swapper/021:42:180
1355099201,3cyclictest0-21swapper/021:42:180
1355099201,11cyclictest13752-21/usr/sbin/munin19:10:230
1355099201,11cyclictest0-21swapper/022:00:190
1355099201,10cyclictest0-21swapper/021:55:120
261812190,8sleep3401ktimersoftd/320:50:123
1356599197,3cyclictest0-21swapper/321:03:593
1356599196,3cyclictest0-21swapper/320:46:553
1356599195,3cyclictest0-21swapper/323:02:053
1356599193,3cyclictest10317-21/usr/sbin/munin23:55:153
1356599191,7cyclictest0-21swapper/320:43:043
1356599191,5cyclictest7611-21timerandwakeup20:05:233
1356599191,5cyclictest21040-21/usr/sbin/munin19:25:133
1356599191,3cyclictest0-21swapper/300:02:413
1356599191,3cyclictest0-21swapper/300:02:413
1356599191,10cyclictest28989-21cat19:40:233
1356099195,3cyclictest0-21swapper/220:25:152
1356099193,7cyclictest0-21swapper/222:30:502
1356099193,3cyclictest87650irq/129-enp1s0-20:35:172
1356099193,3cyclictest87550irq/128-enp1s0-23:35:132
1356099191,9cyclictest31001-21unixbench_multi23:25:252
1356099191,5cyclictest0-21swapper/222:35:592
1356099191,5cyclictest0-21swapper/221:50:162
1356099191,3cyclictest0-21swapper/223:53:222
1356099191,3cyclictest0-21swapper/222:21:482
13560991913,3cyclictest27630-21/usr/sbin/munin19:40:242
1356099191,15cyclictest21139-21cut19:25:132
1356099191,11cyclictest0-21swapper/219:33:582
1356099191,10cyclictest0-21swapper/222:09:002
1356099190,7cyclictest0-21swapper/220:45:212
1355599192,15cyclictest0-21swapper/119:16:011
13555991912,2cyclictest0-21swapper/119:39:001
1355099194,8cyclictest87550irq/128-enp1s0-19:45:180
1355099193,4cyclictest2468-21grep19:55:160
1355099192,9cyclictest26964-21memory20:50:190
1355099191,9cyclictest5418-21cat20:00:240
1355099191,8cyclictest7612-21timerandwakeup20:05:230
1355099191,6cyclictest21381-21df_inode19:25:150
1355099191,5cyclictest30208-21sh21:00:010
13550991912,4cyclictest524-21df21:05:120
1355099191,10cyclictest21650-21/usr/sbin/munin20:40:200
1356599184,11cyclictest15062-21/usr/sbin/munin20:25:143
1356599182,4cyclictest1189-21/usr/sbin/munin19:50:253
1356599181,6cyclictest0-21swapper/320:21:543
1356599181,5cyclictest13752-21/usr/sbin/munin19:10:163
1356599181,4cyclictest32648-21/usr/sbin/munin21:05:213
13565991812,3cyclictest20291-21tr00:15:213
1356599181,15cyclictest401ktimersoftd/320:55:133
13565991811,4cyclictest26404-21sh00:30:163
1356599181,10cyclictest0-21swapper/322:11:043
1356599181,10cyclictest0-21swapper/319:15:193
1356099183,3cyclictest0-21swapper/223:45:212
1356099183,2cyclictest87550irq/128-enp1s0-19:55:122
1356099181,6cyclictest0-21swapper/221:30:122
1356099181,5cyclictest909-21sssd_nss23:59:212
1356099181,5cyclictest22600-21sensors_temp00:20:262
1356099181,5cyclictest10662-21/usr/sbin/munin20:15:152
1356099181,5cyclictest0-21swapper/223:42:002
1356099181,4cyclictest21773-21ntp_states23:05:212
1356099181,4cyclictest0-21swapper/223:17:382
1356099181,4cyclictest0-21swapper/200:16:582
1356099181,3cyclictest87650irq/129-enp1s0-22:55:182
13560991813,3cyclictest24877-21awk23:15:012
1356099181,15cyclictest15564-21kernelversion00:05:212
1356099181,14cyclictest31433-21grep22:15:142
1356099181,11cyclictest0-21swapper/220:52:192
1356099181,10cyclictest0-21swapper/200:03:042
1356099181,10cyclictest0-21swapper/200:03:042
317552171,2sleep20-21swapper/219:49:362
13565991710,4cyclictest17444-21cstates20:30:123
13565991710,4cyclictest17444-21cstates20:30:123
1356099176,9cyclictest26392-21gltestperf00:30:152
1356099173,3cyclictest87650irq/129-enp1s0-20:00:192
1356099171,5cyclictest0-21swapper/223:03:282
1356099171,5cyclictest0-21swapper/222:46:392
1356099171,5cyclictest0-21swapper/220:30:252
1356099171,5cyclictest0-21swapper/220:30:252
1356099171,4cyclictest3736-21mailstats21:10:232
13560991711,4cyclictest19447-21grep19:20:142
13560991711,4cyclictest19447-21grep19:20:142
1356099171,10cyclictest0-21swapper/219:15:212
1356099170,3cyclictest0-21swapper/221:38:172
1355099178,2cyclictest0-21swapper/020:15:260
13550991712,3cyclictest20430-21users19:20:250
13550991712,3cyclictest20430-21users19:20:240
1356599161,5cyclictest0-21swapper/320:00:173
13565991611,3cyclictest19456-21/usr/sbin/munin20:35:193
13560991610,4cyclictest1631-21taskset19:55:012
13560991610,3cyclictest30645-21expr21:00:152
1356099150,4cyclictest14208-21timerandwakeup20:20:242
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional