You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-14 - 14:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Thu May 14, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
140812235161,61sleep00-21swapper/007:09:420
138362203167,24sleep20-21swapper/207:06:302
138652202166,24sleep10-21swapper/107:06:541
138372202166,24sleep30-21swapper/307:06:313
216421730,3sleep01418899cyclictest10:20:160
2062521580,4sleep01418899cyclictest11:00:250
629221250,5sleep31420699cyclictest10:30:143
1710421200,9sleep230-21rcuc/210:55:132
679621150,4sleep230-21rcuc/209:15:252
3272221130,2sleep01418899cyclictest09:00:540
14200999082,2cyclictest0-21swapper/211:35:102
233792640,4sleep123-21ksoftirqd/112:20:241
253722630,4sleep011-21rcuc/007:30:240
14200996360,2cyclictest0-21swapper/208:40:152
212802600,4sleep01418899cyclictest07:22:540
174872500,4sleep123-21ksoftirqd/112:10:011
219722490,1sleep00-21swapper/009:50:220
25092430,1sleep30-21swapper/307:55:113
1420699431,33cyclictest17559-21/usr/sbin/munin12:10:243
14188993910,13cyclictest30858-21munin-run09:00:010
14188993910,13cyclictest30858-21munin-run09:00:010
1420699371,6cyclictest0-21swapper/309:10:183
1420699371,10cyclictest0-21swapper/311:40:183
1420699369,3cyclictest29514-21/usr/sbin/munin10:10:173
14206993610,23cyclictest7250-21df_inode08:05:133
1420099361,19cyclictest0-21swapper/207:52:582
1420699351,21cyclictest25107-21/usr/sbin/munin10:00:173
1418899359,12cyclictest31499-21idleruntime-cro10:15:010
1420699346,3cyclictest13495-21/usr/sbin/munin08:20:153
1420699345,3cyclictest0-21swapper/310:28:173
1420699342,4cyclictest14113-21/usr/sbin/munin09:35:243
1420699341,7cyclictest27031-21chrt08:50:133
14206993412,19cyclictest22623-21sort11:05:243
1420099341,14cyclictest0-21swapper/207:42:222
1420099341,14cyclictest0-21swapper/207:42:222
1420699338,3cyclictest24153-21/usr/sbin/munin12:25:233
1420699335,3cyclictest12050irq/123-ahci[0008:45:013
1420699335,16cyclictest7880-21kworker/u8:110:39:503
1420699332,4cyclictest910-21systemd-logind07:15:023
1420699331,7cyclictest0-21swapper/309:40:123
1420699331,5cyclictest13311-21cat08:20:023
1420699331,29cyclictest24086-21ps09:55:233
1420699330,6cyclictest0-21swapper/312:36:353
14200993330,2cyclictest0-21swapper/208:34:402
1420699329,20cyclictest4698-21/usr/sbin/munin08:00:163
1420699329,20cyclictest32665-21/usr/sbin/munin07:50:223
1420699327,3cyclictest2905-21diskmemload09:15:123
1420699326,3cyclictest87350irq/126-enp1s0-12:15:113
1420699321,9cyclictest0-21swapper/311:50:463
1420699321,7cyclictest10090-21kworker/u8:011:32:073
1420699321,6cyclictest0-21swapper/309:30:023
14206993211,18cyclictest827-21runrttasks09:02:293
14206993211,18cyclictest10954-21/usr/sbin/munin11:55:233
1420099321,21cyclictest0-21swapper/211:15:502
1419699321,10cyclictest0-21swapper/110:01:291
1420699319,19cyclictest827-21runrttasks08:34:083
1420699319,19cyclictest3345-21processes11:35:223
1420699319,19cyclictest32161-21cut10:15:153
1420699319,19cyclictest27751-21timerwakeupswit12:30:233
1420699319,19cyclictest27751-21timerwakeupswit12:30:233
1420699319,19cyclictest14734-21/usr/sbin/munin10:50:163
1420699317,3cyclictest1742-21irqstats09:05:183
1420699315,3cyclictest0-21swapper/311:45:203
1420699312,3cyclictest24132-21cat12:25:013
1420699311,9cyclictest0-21swapper/310:55:273
1420699311,8cyclictest28680-21timerandwakeup10:05:223
1420699311,26cyclictest2905-21diskmemload09:50:233
14206993112,16cyclictest25625-21grep08:45:223
1420699311,11cyclictest26714-21tail07:35:153
1420699311,11cyclictest0-21swapper/311:16:473
1420699310,6cyclictest0-21swapper/309:47:593
1420099310,22cyclictest0-21swapper/207:30:222
1420699308,3cyclictest17287-21/usr/sbin/munin07:15:153
1420699307,3cyclictest13149-21/usr/sbin/munin12:00:253
1420699307,3cyclictest11310-21vmstat09:25:273
1420699306,2cyclictest5750irq/9-acpi07:30:143
1420699302,4cyclictest19829-21diskstats07:20:123
14200993018,3cyclictest0-21swapper/207:39:022
1420099301,17cyclictest0-21swapper/207:25:202
1420099301,14cyclictest0-21swapper/209:04:502
1420099301,12cyclictest20182-21ntp_offset11:00:202
1420699296,2cyclictest87350irq/126-enp1s0-10:40:213
1420699295,3cyclictest309882latency11:25:173
1420699295,3cyclictest0-21swapper/311:03:493
1420699292,4cyclictest9440-21df_inode08:10:123
1420699292,4cyclictest7870-21diskstats09:20:133
1420699291,25cyclictest2905-21diskmemload10:21:203
14206992910,16cyclictest31922-21cat07:45:233
14188992910,16cyclictest17338-21munin-run12:10:010
1420699289,16cyclictest22624-21missed_timers07:25:183
1420699287,3cyclictest29897-21latency_hist11:25:013
1420699287,3cyclictest29897-21latency_hist11:25:013
1420699286,6cyclictest0-21swapper/312:05:143
1420699282,4cyclictest13595-21ntp_states10:45:213
1420699282,3cyclictest23879-21diskstats11:10:123
1420099281,14cyclictest0-21swapper/208:23:022
1420099281,13cyclictest0-21swapper/208:28:462
1420699272,5cyclictest30146-21smart_sda08:55:253
1420699272,5cyclictest30146-21smart_sda08:55:253
1420699272,4cyclictest16048-21diskstats08:25:143
14206992710,14cyclictest28540-21expr07:40:133
14206992710,14cyclictest28540-21expr07:40:133
1420099271,14cyclictest0-21swapper/210:28:462
14200992711,3cyclictest29663-21taskset11:22:062
14200992711,3cyclictest29663-21taskset11:22:062
1420099271,13cyclictest0-21swapper/210:46:422
1420099271,13cyclictest0-21swapper/209:05:142
1420099271,13cyclictest0-21swapper/207:55:182
1420099271,12cyclictest827-21runrttasks09:26:302
1420099271,12cyclictest0-21swapper/210:05:142
1419699275,15cyclictest87650irq/129-enp1s0-09:28:541
1419699271,17cyclictest0-21swapper/112:32:091
1419699271,17cyclictest0-21swapper/112:32:091
1419699270,20cyclictest0-21swapper/111:10:171
1419699270,11cyclictest10090-21kworker/u8:011:41:141
1420699265,3cyclictest20094-21/usr/sbin/munin08:35:223
1420099269,3cyclictest0-21swapper/207:17:322
1420099268,9cyclictest0-21swapper/212:17:302
1420099268,8cyclictest0-21swapper/208:56:322
1420099268,8cyclictest0-21swapper/208:56:322
1420099261,15cyclictest14202-21ntp_kernel_pll_12:00:202
1420099261,13cyclictest0-21swapper/212:29:522
1420099261,10cyclictest0-21swapper/211:49:002
1418899263,11cyclictest2290-21awk07:55:010
1420099259,3cyclictest0-21swapper/208:15:262
1420099258,8cyclictest0-21swapper/208:01:582
1420099251,9cyclictest0-21swapper/210:15:162
1420099251,17cyclictest0-21swapper/211:31:122
1420099251,13cyclictest0-21swapper/211:50:502
1420099251,11cyclictest0-21swapper/208:38:502
1420099251,10cyclictest0-21swapper/209:37:442
1419699252,12cyclictest0-21swapper/109:50:111
14196992514,2cyclictest0-21swapper/107:10:201
1419699251,17cyclictest0-21swapper/109:43:111
1419699251,11cyclictest7499-21missed_timers11:45:191
14196992510,4cyclictest30290-21cat08:55:261
14196992510,4cyclictest30290-21cat08:55:261
1419699250,5cyclictest0-21swapper/107:45:201
1420099249,2cyclictest0-21swapper/207:10:542
1420099245,2cyclictest0-21swapper/208:50:242
1420099241,12cyclictest0-21swapper/208:46:222
1420099241,12cyclictest0-21swapper/208:13:582
1420099241,11cyclictest0-21swapper/212:38:022
1419699241,20cyclictest17537-21grep10:55:171
1419699241,16cyclictest0-21swapper/111:08:191
1419699241,11cyclictest0-21swapper/110:37:551
1419699241,10cyclictest0-21swapper/110:28:231
14196992410,3cyclictest18947-21grep09:45:151
14196992410,12cyclictest10090-21kworker/u8:011:34:241
1419699240,6cyclictest0-21swapper/111:35:111
1419699240,17cyclictest0-21swapper/110:40:541
1419699240,12cyclictest0-21swapper/110:10:511
1418899249,8cyclictest87550irq/128-enp1s0-09:10:260
1418899246,2cyclictest13232-21kworker/0:208:53:220
1418899241,3cyclictest0-21swapper/009:27:090
1418899240,4cyclictest29194-21kworker/u8:010:17:540
1420099239,3cyclictest0-21swapper/210:41:382
1420099239,3cyclictest0-21swapper/210:37:262
1420099231,17cyclictest27515-21ps12:30:202
1420099231,17cyclictest27515-21ps12:30:202
1420099231,14cyclictest0-21swapper/212:10:352
1420099231,14cyclictest0-21swapper/210:50:122
1420099231,10cyclictest2905-21diskmemload10:32:182
1419699231,16cyclictest0-21swapper/110:48:461
1419699231,16cyclictest0-21swapper/109:33:501
1419699231,15cyclictest0-21swapper/111:16:131
1419699231,13cyclictest0-21swapper/112:35:151
1419699231,11cyclictest0-21swapper/112:18:071
1419699230,22cyclictest0-21swapper/108:31:471
1418899236,3cyclictest2482-21grep09:05:250
1418899231,7cyclictest0-21swapper/011:36:370
95202221,2sleep0909-21sssd_nss08:10:140
1420099229,3cyclictest0-21swapper/211:41:142
1420099228,10cyclictest0-21swapper/209:31:522
1420099221,3cyclictest0-21swapper/211:29:512
1420099221,11cyclictest0-21swapper/212:23:422
1420099221,11cyclictest0-21swapper/209:52:262
1420099220,10cyclictest0-21swapper/211:55:022
1419699227,4cyclictest0-21swapper/111:27:531
1419699222,11cyclictest28619-21diskstats07:40:141
1419699222,11cyclictest28619-21diskstats07:40:141
1419699221,4cyclictest0-21swapper/111:52:191
1419699221,4cyclictest0-21swapper/111:04:191
1419699221,3cyclictest27021-21df08:50:131
14196992211,3cyclictest514-21smart_sda10:15:231
1419699221,13cyclictest0-21swapper/109:38:181
1419699221,12cyclictest15435-21basename10:50:161
1419699221,11cyclictest11556-21gltestperf11:55:161
1419699221,11cyclictest0-21swapper/112:14:271
1419699221,10cyclictest0-21swapper/112:26:501
1418899221,5cyclictest0-21swapper/011:32:560
1420099211,5cyclictest0-21swapper/211:14:262
1420099211,13cyclictest0-21swapper/209:49:522
1419699215,4cyclictest0-21swapper/109:24:331
1419699214,9cyclictest0-21swapper/110:06:361
1419699212,17cyclictest0-21swapper/107:21:131
1419699212,10cyclictest7205-21smart_sda10:30:241
14196992115,3cyclictest3517-21latency_hist10:25:011
1419699211,3cyclictest808-21snmpd07:28:581
1419699211,18cyclictest18748-21unixbench-2d07:15:251
1419699211,18cyclictest0-21swapper/108:04:441
1419699211,11cyclictest0-21swapper/112:01:361
1419699210,15cyclictest0-21swapper/109:10:301
1419699210,12cyclictest87550irq/128-enp1s0-09:17:281
1418899216,14cyclictest0-21swapper/012:15:010
1418899211,3cyclictest0-21swapper/009:55:100
1418899211,10cyclictest0-21swapper/011:55:190
1420099208,3cyclictest0-21swapper/210:03:542
1420099206,4cyclictest23314-21perf11:10:012
1420099205,3cyclictest0-21swapper/207:20:582
1420099201,16cyclictest0-21swapper/208:06:342
14200992010,3cyclictest87550irq/128-enp1s0-10:24:332
1419699201,17cyclictest0-21swapper/111:20:151
1419699201,17cyclictest0-21swapper/111:20:151
1419699201,17cyclictest0-21swapper/109:55:221
1419699201,17cyclictest0-21swapper/108:28:021
1419699201,10cyclictest22066-21munin-run08:40:011
1419699201,10cyclictest1860-21sed09:05:191
1419699201,10cyclictest11415-21expr08:15:111
1418899209,3cyclictest808-21snmpd11:15:580
1418899204,5cyclictest87350irq/126-enp1s0-10:35:220
1418899203,2cyclictest0-21swapper/010:00:120
1418899202,3cyclictest0-21swapper/009:23:340
1418899201,2cyclictest0-21swapper/007:15:020
1418899201,17cyclictest18293-21munin-run09:45:010
1418899201,11cyclictest0-21swapper/010:58:440
14188992010,3cyclictest3844-21cpuspeed_turbos10:25:130
1418899200,6cyclictest0-21swapper/009:47:090
14200991913,3cyclictest16837-21unixbench-2d12:05:242
1420099191,14cyclictest0-21swapper/209:55:142
1419699197,3cyclictest28146-21latency_hist07:40:011
1419699196,3cyclictest0-21swapper/107:55:221
1419699192,16cyclictest0-21swapper/108:46:041
1419699191,8cyclictest14100-21head08:20:151
14196991915,3cyclictest0-21swapper/108:41:061
14196991912,4cyclictest23897-21grep07:30:111
1418899196,3cyclictest9743-21grep11:50:190
1418899196,3cyclictest31633-21ps07:45:200
1418899195,4cyclictest29573-21switchtime07:40:240
1418899195,4cyclictest29573-21switchtime07:40:240
1418899194,7cyclictest87550irq/128-enp1s0-12:35:200
1418899191,5cyclictest0-21swapper/012:16:340
1418899191,11cyclictest0-21swapper/011:27:370
1418899191,11cyclictest0-21swapper/011:11:350
1418899191,10cyclictest28995-21ntp_states11:20:200
1418899191,10cyclictest28995-21ntp_states11:20:200
1418899191,10cyclictest14335-21sshd10:47:540
1418899191,10cyclictest14113-21/usr/sbin/munin09:35:240
1420099188,3cyclictest0-21swapper/209:13:562
1420099181,8cyclictest7643-21cpuspeed_turbos09:20:122
1420099181,13cyclictest0-21swapper/207:48:382
1420099181,11cyclictest0-21swapper/210:10:342
1420099181,11cyclictest0-21swapper/209:42:362
1419699185,4cyclictest32068-21needreboot09:00:201
1419699181,3cyclictest7902-21needreboot08:05:191
14196991813,2cyclictest679-21fschecks_count07:50:141
1419699180,16cyclictest0-21swapper/108:11:141
1418899187,3cyclictest11298-21/usr/sbin/munin08:15:120
1418899186,3cyclictest24174-21egrep12:25:110
1418899184,7cyclictest755-21rngd08:06:580
1418899184,7cyclictest15901-21ps10:50:210
1418899183,7cyclictest15817-21kworker/u8:210:06:250
1418899182,5cyclictest0-21swapper/011:05:190
1418899182,13cyclictest24487-21/usr/sbin/munin08:45:230
1418899181,8cyclictest0-21swapper/009:15:160
1418899181,4cyclictest4478-21munin-run08:00:010
1418899181,4cyclictest18748-21latency08:30:170
1418899181,3cyclictest5319-21tail08:00:140
1418899181,15cyclictest15016-21grep08:20:240
1418899181,11cyclictest0-21swapper/010:33:260
1418899174,3cyclictest808-21snmpd08:29:360
1418899173,8cyclictest0-21swapper/011:48:010
1418899172,6cyclictest26903-21perl12:30:130
1418899172,6cyclictest26903-21perl12:30:130
1418899172,5cyclictest0-21swapper/012:00:510
1418899171,5cyclictest839-21lldpd08:36:420
1418899171,5cyclictest5034-21hddtemp_smartct11:40:160
1418899171,5cyclictest12722-21irqstats09:30:170
1418899171,5cyclictest0-21swapper/012:20:150
1418899171,3cyclictest0-21swapper/008:40:220
14188991712,3cyclictest17060-21memory07:15:040
1418899171,14cyclictest0-21swapper/010:40:070
14188991710,4cyclictest22703-21netstat07:25:190
14188991611,3cyclictest26074-21/usr/sbin/munin07:35:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional