You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-25 - 22:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Thu Dec 25, 2025 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
152352252191,49sleep00-21swapper/007:07:390
152352252191,49sleep00-21swapper/007:07:390
153512202163,26sleep10-21swapper/107:09:091
153512202163,26sleep10-21swapper/107:09:091
151352201165,24sleep30-21swapper/307:06:223
151352201165,24sleep30-21swapper/307:06:223
150942201165,24sleep20-21swapper/207:05:482
150942201165,24sleep20-21swapper/207:05:482
81822710,3sleep38185-21missed_timers10:30:203
15512996864,3cyclictest0-21swapper/210:07:202
15512994946,2cyclictest0-21swapper/207:29:482
1551899400,27cyclictest0-21swapper/312:15:183
1551899371,24cyclictest0-21swapper/309:50:043
1550399378,13cyclictest10191-21munin-run08:10:010
1551899360,19cyclictest0-21swapper/312:25:183
1551899341,21cyclictest0-21swapper/308:59:573
1551899341,21cyclictest0-21swapper/307:53:213
1550399349,12cyclictest1369-21munin-run07:50:000
1551899331,14cyclictest0-21swapper/310:54:383
1551899331,10cyclictest2497-21ps11:30:243
1551899321,20cyclictest0-21swapper/307:36:473
1551899321,15cyclictest0-21swapper/309:25:573
1551899321,15cyclictest0-21swapper/309:25:563
1551899321,15cyclictest0-21swapper/309:13:353
1551899321,14cyclictest0-21swapper/311:01:263
1551899321,12cyclictest0-21swapper/312:21:543
1551899320,13cyclictest0-21swapper/311:56:503
1551899320,13cyclictest0-21swapper/311:56:503
15518993113,3cyclictest1055-21ntp_kernel_pll_09:00:193
1551899311,22cyclictest0-21swapper/311:26:573
1551899311,11cyclictest0-21swapper/310:40:233
1551899311,11cyclictest0-21swapper/310:40:233
1551899310,23cyclictest0-21swapper/309:41:243
1551899301,9cyclictest0-21swapper/308:46:353
1551899301,19cyclictest0-21swapper/308:20:213
1551899301,13cyclictest0-21swapper/311:18:403
1551899301,13cyclictest0-21swapper/310:47:403
1551899301,12cyclictest0-21swapper/312:36:463
1551899301,12cyclictest0-21swapper/310:03:293
1551899301,12cyclictest0-21swapper/307:40:313
1551899301,11cyclictest0-21swapper/309:46:073
1551899301,11cyclictest0-21swapper/308:19:253
1551899301,11cyclictest0-21swapper/308:19:253
15518992916,3cyclictest0-21swapper/307:34:073
1551899291,18cyclictest0-21swapper/308:06:033
1551899291,14cyclictest0-21swapper/311:49:303
1551899291,14cyclictest0-21swapper/311:10:573
1551899291,14cyclictest0-21swapper/309:22:433
1551899291,13cyclictest0-21swapper/310:13:033
1551899291,13cyclictest0-21swapper/310:05:293
1551899291,13cyclictest0-21swapper/307:45:453
1551899291,12cyclictest0-21swapper/311:51:403
1551899291,10cyclictest0-21swapper/310:22:313
1551899290,18cyclictest0-21swapper/312:32:443
1551899290,13cyclictest0-21swapper/311:44:483
15512992926,2cyclictest0-21swapper/208:56:562
1550399296,11cyclictest11439-21perf10:40:010
15503992910,16cyclictest20569-21munin-run07:20:010
1551899281,14cyclictest0-21swapper/308:32:493
1551899281,12cyclictest0-21swapper/309:30:533
1551899281,12cyclictest0-21swapper/308:11:193
1551899281,11cyclictest0-21swapper/312:14:043
1551899281,11cyclictest0-21swapper/311:23:483
1551899281,11cyclictest0-21swapper/309:56:043
1551299288,6cyclictest0-21swapper/209:55:122
15508992810,9cyclictest18342-21kworker/u8:112:21:071
1551899271,17cyclictest0-21swapper/311:09:163
1551899271,13cyclictest0-21swapper/309:05:153
1551899271,12cyclictest0-21swapper/310:25:373
1551899271,11cyclictest0-21swapper/310:17:323
1551899271,11cyclictest0-21swapper/308:50:553
1551899270,9cyclictest0-21swapper/307:19:473
1551899270,13cyclictest0-21swapper/312:08:103
1551899270,12cyclictest0-21swapper/307:25:393
1551899270,12cyclictest0-21swapper/307:23:393
1550399272,12cyclictest0-21swapper/011:20:010
1551899261,8cyclictest0-21swapper/310:36:123
15518992617,3cyclictest0-21swapper/308:26:493
1551899261,10cyclictest7069-21ntpq08:00:213
1551899261,10cyclictest0-21swapper/311:38:323
1551899261,10cyclictest0-21swapper/309:36:173
1551899261,10cyclictest0-21swapper/307:12:553
1551899260,11cyclictest0-21swapper/310:56:443
1550399263,11cyclictest6425-21awk09:15:010
1551899251,9cyclictest0-21swapper/308:40:033
1551899251,9cyclictest0-21swapper/308:38:353
1551899251,11cyclictest0-21swapper/312:02:163
1550899252,19cyclictest13623-21munin-run10:45:011
1550899252,19cyclictest13623-21munin-run10:45:011
15508992517,7cyclictest5570-21fschecks_time10:25:141
15508992514,3cyclictest21872-21cut08:35:131
1550899251,19cyclictest0-21swapper/110:45:441
1550899250,18cyclictest0-21swapper/111:35:231
1550399251,5cyclictest0-21swapper/012:26:540
303522240,3sleep030356-21pmu-power11:20:220
1551899241,13cyclictest5311-21sed07:55:233
1551899240,11cyclictest0-21swapper/309:15:313
1550399241,6cyclictest10532-40dnf09:21:410
1550399241,6cyclictest0-21swapper/012:23:160
1550899239,12cyclictest6934-21kworker/u8:010:57:101
15508992314,3cyclictest0-21swapper/108:51:421
1550899231,16cyclictest0-21swapper/108:43:581
1550899231,16cyclictest0-21swapper/108:08:471
1550899231,13cyclictest0-21swapper/109:21:411
15508992311,10cyclictest32019-21kworker/u8:111:27:091
129592231,2sleep113161-21tune2fs08:15:151
129592231,2sleep113161-21tune2fs08:15:151
201922220,3sleep00-21swapper/008:30:180
15512992211,3cyclictest29362-21awk07:40:012
1551299221,10cyclictest0-21swapper/210:54:302
1550899227,4cyclictest0-21swapper/110:31:111
1550899221,6cyclictest0-21swapper/111:08:121
15508992216,3cyclictest19266-21sed07:15:151
15508992214,3cyclictest0-21swapper/111:17:421
1550899221,16cyclictest0-21swapper/112:09:241
1550899221,15cyclictest0-21swapper/108:55:401
1550899221,11cyclictest20300-21cut09:45:151
1550899221,11cyclictest0-21swapper/109:13:471
15508992210,10cyclictest12646-21kworker/u8:210:16:111
1550899220,4cyclictest32632-21kworker/u8:110:24:361
1550899220,16cyclictest0-21swapper/112:34:341
1550399222,5cyclictest0-21swapper/011:49:030
1550399221,6cyclictest6934-21kworker/u8:010:46:300
1550399221,6cyclictest0-21swapper/009:43:520
1551299211,7cyclictest0-21swapper/211:29:482
1551299211,6cyclictest0-21swapper/210:10:152
15512992114,4cyclictest5779-21idleruntime-cro08:00:012
1550899217,4cyclictest0-21swapper/110:53:141
1550899213,10cyclictest4221-21diskmemload11:10:251
1550899212,14cyclictest0-21swapper/109:01:451
1550899212,14cyclictest0-21swapper/107:55:321
1550899212,11cyclictest11466-21timerwakeupswit11:50:261
1550899211,3cyclictest6712-21sort09:15:121
1550899211,18cyclictest0-21swapper/109:31:591
1550899211,15cyclictest0-21swapper/112:15:231
1550899211,15cyclictest0-21swapper/108:02:591
1550899211,14cyclictest0-21swapper/111:42:181
1550899211,14cyclictest0-21swapper/108:22:361
1550899211,14cyclictest0-21swapper/107:13:551
1550899211,11cyclictest31385-21grep12:35:251
1550899211,11cyclictest10563-21grep08:10:121
1550899211,10cyclictest0-21swapper/109:57:491
15508992110,3cyclictest26034-21grep07:30:201
1550899210,5cyclictest0-21swapper/109:36:151
1550899210,16cyclictest0-21swapper/108:28:291
1550899210,15cyclictest87150irq/128-enp1s0-09:40:471
1550899210,11cyclictest0-21swapper/111:24:171
1550399215,5cyclictest16047-21/usr/sbin/munin10:50:150
1550399212,11cyclictest6639-21/usr/sbin/munin09:15:230
1550399211,4cyclictest0-21swapper/009:31:330
15503992114,4cyclictest14905-21tr08:20:140
1550399211,11cyclictest20793-21sshd07:20:090
199992200,5sleep230-21rcuc/208:30:162
1551299208,3cyclictest0-21swapper/210:40:122
1551299208,3cyclictest0-21swapper/210:40:122
1551299204,8cyclictest0-21swapper/208:45:392
1551299201,13cyclictest0-21swapper/211:14:272
1550899207,12cyclictest29519-21kworker/u8:207:35:101
1550899206,6cyclictest26444-21needreboot12:25:201
1550899202,17cyclictest0-21swapper/109:07:131
15508992015,3cyclictest29889-21smartctl10:05:251
15508992012,3cyclictest0-21swapper/108:34:331
1550899201,17cyclictest0-21swapper/110:35:101
1550899201,17cyclictest0-21swapper/109:50:191
1550899201,14cyclictest0-21swapper/111:46:171
1550899201,14cyclictest0-21swapper/108:46:561
1550899201,11cyclictest12400-21/usr/sbin/munin09:25:251
1550899201,11cyclictest12400-21/usr/sbin/munin09:25:251
1550899201,10cyclictest2101-21kernelversion11:30:181
1550899200,5cyclictest0-21swapper/112:01:141
1550899200,13cyclictest32632-21kworker/u8:111:00:391
1550399209,3cyclictest27385-21/usr/sbin/munin07:35:190
1550399203,9cyclictest32384-21missed_timers11:25:180
1550399201,5cyclictest18991-21perf08:30:010
1550399201,3cyclictest0-21swapper/011:04:070
1550399201,11cyclictest2861-21/usr/sbin/munin10:20:180
1550399201,11cyclictest21069-21/usr/sbin/munin12:15:170
1550399201,10cyclictest8322-21ntp_states10:30:230
1550399201,10cyclictest32042-21rt-features10:10:240
1550399201,10cyclictest10664-21grep11:50:160
134022201,2sleep013505-21latency08:15:180
134022201,2sleep013505-21latency08:15:180
1551299197,3cyclictest32-21ksoftirqd/208:05:102
1551299191,7cyclictest1568-21grep09:00:252
1551299191,7cyclictest10206-21grep09:20:242
1551299191,6cyclictest0-21swapper/209:14:542
1551299191,4cyclictest0-21swapper/211:31:582
15512991912,4cyclictest11506-21sort09:25:152
15512991912,4cyclictest11506-21sort09:25:152
1551299191,12cyclictest0-21swapper/208:10:142
1550899198,3cyclictest29666-21cpuspeed_turbos07:40:111
1550899196,3cyclictest0-21swapper/110:11:041
1550899195,4cyclictest0-21swapper/110:03:451
1550899191,3cyclictest0-21swapper/107:27:421
15508991912,2cyclictest0-21swapper/112:10:361
15508991912,2cyclictest0-21swapper/107:20:571
1550899190,17cyclictest0-21swapper/107:45:091
1550399198,3cyclictest1256-21/usr/sbin/munin11:30:120
1550399197,4cyclictest800-21sshd07:40:110
1550399191,6cyclictest12600-21df11:55:130
1550399191,6cyclictest12600-21df11:55:130
1550399191,15cyclictest835-21runrttasks12:32:190
1550399191,10cyclictest6014-21/usr/sbin/munin08:00:260
1550399191,10cyclictest31631-21sshd12:37:390
1550399191,10cyclictest25189-21/usr/sbin/munin07:30:210
1550399191,10cyclictest14816-21seq12:00:150
69572181,3sleep00-21swapper/010:29:140
320622181,3sleep00-21swapper/008:58:520
303532181,3sleep00-21swapper/010:07:030
264932181,2sleep00-21swapper/011:11:000
260902181,2sleep00-21swapper/009:58:520
198572180,2sleep00-21swapper/010:55:350
1551299189,3cyclictest0-21swapper/212:35:272
1551299185,4cyclictest7180-21ps08:00:232
1551299182,9cyclictest0-21swapper/209:54:002
1551299181,7cyclictest7920-21head10:30:172
1551299181,7cyclictest19834-21users10:55:272
1551299181,7cyclictest0-21swapper/208:35:122
1551299181,6cyclictest0-21swapper/210:35:442
1551299181,6cyclictest0-21swapper/209:30:262
1551299181,5cyclictest0-21swapper/211:37:112
1551299181,3cyclictest23866-21ps11:05:232
15512991813,3cyclictest25809-21grep07:30:182
15512991813,3cyclictest15767-21df09:35:132
1551299181,2cyclictest86950irq/126-enp1s0-12:22:512
15512991812,3cyclictest7686-21ntpq09:15:202
15512991812,3cyclictest27961-21grep10:00:272
1551299181,15cyclictest28995-21turbostat.cron11:19:562
1551299181,15cyclictest0-21swapper/212:07:162
1551299181,11cyclictest0-21swapper/212:10:402
1551299181,11cyclictest0-21swapper/211:47:092
15512991810,5cyclictest0-21swapper/210:15:162
1550899184,9cyclictest0-21swapper/111:58:451
1550899184,9cyclictest0-21swapper/111:58:451
15508991811,2cyclictest0-21swapper/107:53:041
1550399189,6cyclictest0-21swapper/010:40:140
1550399189,6cyclictest0-21swapper/010:40:140
1550399181,9cyclictest27495-21sshd08:46:380
1550399181,7cyclictest0-21swapper/011:37:010
1550399181,6cyclictest16409-21/usr/sbin/munin09:35:190
1550399181,5cyclictest0-21swapper/008:37:390
15503991812,3cyclictest22650-21/usr/sbin/munin11:05:160
1550399181,1cyclictest0-21swapper/010:00:210
164812171,13sleep215491-21cyclictest08:21:142
1551299174,4cyclictest13241-21needreboot11:55:182
1551299174,4cyclictest13241-21needreboot11:55:182
1551299171,9cyclictest0-21swapper/212:04:392
1551299171,7cyclictest31778-21/usr/sbin/munin07:45:172
1551299171,7cyclictest0-21swapper/210:49:562
1551299171,6cyclictest3792-21cat07:55:012
1551299171,6cyclictest26529-21/usr/sbin/munin12:25:222
1551299171,6cyclictest0-21swapper/211:44:352
1551299171,6cyclictest0-21swapper/209:40:202
1551299171,3cyclictest0-21swapper/209:46:112
15512991712,3cyclictest3502-21gltestperf10:20:182
15512991712,3cyclictest21069-21/usr/sbin/munin12:15:142
15512991712,3cyclictest17235-21grep07:10:242
1551299171,15cyclictest0-21swapper/211:05:002
15512991711,4cyclictest0-21swapper/209:10:012
15512991710,4cyclictest6091-21grep10:25:192
15512991710,4cyclictest24071-21grep08:40:152
1550399173,4cyclictest6738-21ntp_states11:40:190
1550399172,4cyclictest5799-21idleruntime-cro08:00:010
1550399172,12cyclictest563-21/usr/sbin/munin10:15:250
1550399171,9cyclictest0-21swapper/012:05:050
1550399171,9cyclictest0-21swapper/009:46:110
1550399171,6cyclictest17820-21cut07:15:010
1550399171,6cyclictest0-21swapper/008:40:250
1550399171,2cyclictest0-21swapper/009:53:240
1551299168,3cyclictest30085-21grep07:40:132
1551299165,3cyclictest87250irq/129-enp1s0-11:20:172
1551299165,3cyclictest86950irq/126-enp1s0-08:15:272
1551299165,3cyclictest86950irq/126-enp1s0-08:15:272
1551299161,7cyclictest0-21swapper/212:32:262
1551299161,5cyclictest17013-21/usr/sbin/munin08:25:192
1551299161,3cyclictest0-21swapper/211:53:292
15512991611,3cyclictest19027-21cut07:15:142
15512991610,3cyclictest21045-21expr07:20:142
1550399167,3cyclictest0-21swapper/007:25:180
1550399161,6cyclictest32389-21/usr/sbin/munin09:00:140
1550399161,6cyclictest27994-21/usr/sbin/munin08:50:190
1550399161,3cyclictest2240-21/usr/sbin/munin09:05:160
15503991611,3cyclictest11692-21sed09:25:160
15503991611,3cyclictest11692-21sed09:25:160
1550399161,12cyclictest1592-21/usr/sbin/munin07:50:210
1550399161,10cyclictest0-21swapper/012:10:130
15503991610,3cyclictest10567-21expr08:10:120
15512991510,2cyclictest27994-21/usr/sbin/munin08:50:262
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional