You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-11 - 13:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Sun Jan 11, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
324652236162,24sleep00-21swapper/019:09:360
322742205167,25sleep30-21swapper/319:07:103
324712202165,24sleep10-21swapper/119:09:411
323372200163,25sleep20-21swapper/219:07:572
1378321620,4sleep13258799cyclictest23:20:021
1305021500,4sleep13258799cyclictest20:50:131
1159621160,3sleep23259399cyclictest20:45:242
73232610,4sleep03258199cyclictest21:50:150
304992490,1sleep20-21swapper/223:55:222
76112460,1sleep20-21swapper/223:05:142
214952450,1sleep30-21swapper/321:10:013
94382440,3sleep29440-21pmu-power20:40:232
32581993910,13cyclictest22866-21munin-run20:00:010
3259899341,21cyclictest0-21swapper/320:26:363
3259899341,20cyclictest0-21swapper/323:27:563
3259899331,14cyclictest0-21swapper/320:22:083
3258199338,12cyclictest7788-21munin-run00:20:010
3259899321,25cyclictest0-21swapper/300:15:153
3259899311,27cyclictest0-21swapper/319:10:263
3259899311,19cyclictest0-21swapper/320:51:483
3259899311,19cyclictest0-21swapper/320:14:203
3259899311,14cyclictest0-21swapper/323:10:043
3259899311,13cyclictest0-21swapper/320:40:203
3259899311,13cyclictest0-21swapper/319:40:073
3259899301,21cyclictest0-21swapper/320:39:043
3259899301,21cyclictest0-21swapper/320:08:203
3259899301,20cyclictest3948-21users21:40:263
3259899301,20cyclictest0-21swapper/319:56:523
3259899301,18cyclictest0-21swapper/323:02:103
3259899301,17cyclictest3103-21diskstats19:15:043
3259899301,13cyclictest0-21swapper/323:23:243
3259899301,13cyclictest0-21swapper/321:25:363
3259899301,13cyclictest0-21swapper/321:25:363
3259899301,12cyclictest0-21swapper/321:30:483
3258199304,12cyclictest7170-21perf23:05:000
32581993011,16cyclictest25076-21munin-run20:05:010
3259899291,19cyclictest0-21swapper/320:47:283
3259899291,18cyclictest0-21swapper/319:32:163
3259899291,18cyclictest0-21swapper/300:02:003
3259899291,15cyclictest0-21swapper/320:55:563
3259899291,14cyclictest0-21swapper/323:18:523
3259899291,13cyclictest0-21swapper/322:03:483
3259899291,13cyclictest0-21swapper/321:36:203
3259899291,12cyclictest10553-21df_inode00:25:133
3259899291,12cyclictest0-21swapper/322:45:403
3259899291,12cyclictest0-21swapper/321:48:123
3259899291,12cyclictest0-21swapper/321:17:043
3259899291,12cyclictest0-21swapper/319:39:173
3259899291,11cyclictest21289-21diskmemload22:55:323
32581992910,16cyclictest2750-21munin-run22:55:010
310452290,4sleep232-21ksoftirqd/220:15:272
3259899281,12cyclictest0-21swapper/322:20:443
3259899281,12cyclictest0-21swapper/322:05:283
3259899281,12cyclictest0-21swapper/319:52:003
3259899281,12cyclictest0-21swapper/319:24:283
3259899281,10cyclictest0-21swapper/319:25:283
3259899280,20cyclictest0-21swapper/300:06:583
3259399283,13cyclictest9602-21sed21:55:152
3258799281,18cyclictest21823-21smart_sda23:35:231
3258799281,18cyclictest0-21swapper/122:21:001
3258199289,16cyclictest20671-21munin-run19:55:010
3259899271,3cyclictest0-21swapper/321:51:183
32598992712,3cyclictest21289-21diskmemload23:32:283
3259899271,13cyclictest0-21swapper/300:13:363
3259899271,12cyclictest0-21swapper/322:29:243
3259899271,12cyclictest0-21swapper/300:31:233
3259899271,11cyclictest0-21swapper/321:03:303
3259899271,11cyclictest0-21swapper/320:32:203
3259899271,11cyclictest0-21swapper/300:36:363
3259899271,10cyclictest0-21swapper/323:56:363
3259899271,10cyclictest0-21swapper/319:46:373
3259399275,11cyclictest25632-21hddtemp_smartct23:45:152
3258799271,19cyclictest0-21swapper/122:05:361
3258799271,14cyclictest24388-21hddtemp_smartct21:15:151
3258199272,12cyclictest19110-21basename21:05:010
3259899261,8cyclictest0-21swapper/322:36:463
3259899261,8cyclictest0-21swapper/322:36:463
3259899261,14cyclictest0-21swapper/321:11:443
3259899261,11cyclictest0-21swapper/322:50:483
3259899261,10cyclictest0-21swapper/322:44:163
3259399261,14cyclictest8112-21cut00:20:122
3258199263,12cyclictest11-21rcuc/019:25:010
32598992515,3cyclictest0-21swapper/322:15:193
3259899251,12cyclictest0-21swapper/321:20:483
3259899251,12cyclictest0-21swapper/320:02:083
3259899251,12cyclictest0-21swapper/300:24:233
3259899251,11cyclictest0-21swapper/323:40:163
3259899251,11cyclictest0-21swapper/321:55:243
3259899251,10cyclictest0-21swapper/323:06:083
3259399254,8cyclictest25360-21awk20:05:122
3258799259,5cyclictest87350irq/128-enp1s0-21:13:251
3258799254,10cyclictest21289-21diskmemload22:37:351
3258799254,10cyclictest21289-21diskmemload22:37:351
3258199251,12cyclictest18150-21awk23:30:010
11132251,20sleep11178-21ntp_kernel_pll_21:35:211
3259899241,11cyclictest0-21swapper/322:13:443
3259399247,4cyclictest3835-21latency22:55:182
3259399247,2cyclictest7816-21ntp_states21:50:212
3259399244,7cyclictest15374-21uname19:40:212
32593992413,6cyclictest5564-40dnf00:14:532
3258799249,5cyclictest25070-21awk20:05:011
3258799241,3cyclictest16282-21perf19:45:021
3258799241,18cyclictest0-21swapper/100:29:371
3258799241,17cyclictest0-21swapper/120:37:381
3258799241,14cyclictest9298-21diskstats21:55:141
3258799240,17cyclictest0-21swapper/121:44:221
3258799240,16cyclictest12220-21expr19:35:111
3258199246,9cyclictest26579-21gltestperf21:20:160
3259899231,15cyclictest0-21swapper/323:54:043
3259899231,15cyclictest0-21swapper/323:54:043
3259899231,13cyclictest0-21swapper/320:18:203
3259399239,6cyclictest20232-21chrt23:33:512
3259399233,8cyclictest0-21swapper/222:50:242
3259399233,5cyclictest26089-21cat21:20:132
3259399233,12cyclictest19402-21idleruntime19:50:162
3259399233,11cyclictest27493-21/usr/sbin/munin20:10:172
3259399233,11cyclictest10897-21sed23:10:252
3259399232,10cyclictest21299-21turbostat21:10:012
3259399231,14cyclictest28184-21taskset22:36:032
3259399231,14cyclictest28184-21taskset22:36:032
3259399230,14cyclictest13969-21/usr/sbin/munin23:20:232
3258799231,6cyclictest0-21swapper/122:00:251
3258799231,17cyclictest0-21swapper/123:30:441
3258799231,16cyclictest0-21swapper/123:10:231
3258799231,16cyclictest0-21swapper/119:25:241
3259899221,9cyclictest0-21swapper/323:37:123
3259899221,9cyclictest0-21swapper/322:31:243
3259899221,10cyclictest0-21swapper/323:46:223
3259399226,8cyclictest24538-21chrt23:42:422
3259399224,5cyclictest15545-21/usr/sbin/munin22:10:232
3259399223,2cyclictest87150irq/126-enp1s0-19:35:112
3259399222,8cyclictest16729-21cat00:40:012
32593992211,3cyclictest21424-21fschecks_time19:55:162
3259399221,12cyclictest19138-21cat22:15:262
3259399221,11cyclictest0-21swapper/221:10:022
3259399220,13cyclictest13213-21ls23:15:262
3258799228,9cyclictest0-21swapper/123:51:051
3258799228,9cyclictest0-21swapper/123:51:051
3258799228,6cyclictest0-21swapper/123:22:031
3258799227,3cyclictest0-21swapper/100:11:311
3258799222,14cyclictest0-21swapper/123:56:001
3258799222,10cyclictest22860-21sh20:00:011
3258799221,9cyclictest0-21swapper/100:35:141
3258799221,19cyclictest0-21swapper/123:26:171
3258799221,19cyclictest0-21swapper/123:06:141
3258799221,18cyclictest1030-21grep19:10:151
3258799221,18cyclictest0-21swapper/122:50:261
3258799221,16cyclictest0-21swapper/123:43:061
3258799221,14cyclictest0-21swapper/121:29:461
3258799221,14cyclictest0-21swapper/121:29:461
3258799220,16cyclictest0-21swapper/120:06:061
3258799220,13cyclictest0-21swapper/123:02:091
3258199221,4cyclictest0-21swapper/023:37:010
3258199221,4cyclictest0-21swapper/022:23:140
3259399217,2cyclictest27150-21/usr/sbin/munin23:50:262
3259399217,2cyclictest27150-21/usr/sbin/munin23:50:262
3259399216,7cyclictest12917-21chrt22:02:292
3259399215,8cyclictest13012-21expr00:30:152
3259399214,8cyclictest0-21swapper/220:05:012
3259399213,6cyclictest13785-21ntp_states20:50:222
3259399212,5cyclictest4921-21fschecks_count21:45:132
3259399211,7cyclictest0-21swapper/223:04:042
3259399211,6cyclictest16131-21ps20:55:222
3259399211,6cyclictest1164-21awk00:05:012
3258799218,4cyclictest0-21swapper/100:32:441
3258799217,6cyclictest0-21swapper/122:12:001
32587992113,3cyclictest0-21swapper/120:56:511
3258799211,2cyclictest0-21swapper/122:55:181
3258799211,15cyclictest0-21swapper/121:03:131
3258799211,15cyclictest0-21swapper/120:46:271
3258799211,11cyclictest7563-21irqstats21:50:181
3258799211,10cyclictest29464-21munin-run20:15:011
32587992110,3cyclictest6153-21df00:15:131
3258799210,16cyclictest0-21swapper/120:28:451
3258799210,12cyclictest8592-21kworker/u8:022:15:151
3258199218,3cyclictest23362-21gltestperf23:40:160
3258199214,15cyclictest11562-21awk23:15:010
3258199211,6cyclictest0-21swapper/022:27:550
3258199211,11cyclictest18363-21/usr/sbin/munin23:30:130
3259399204,2cyclictest2908-21grep00:05:272
3259399201,2cyclictest87150irq/126-enp1s0-19:19:522
3259399201,17cyclictest29649-21missed_timers22:40:202
3259399201,12cyclictest0-21swapper/200:27:502
3259399201,10cyclictest0-21swapper/222:06:452
3259399201,10cyclictest0-21swapper/220:35:162
3258799201,7cyclictest0-21swapper/122:47:091
32587992013,2cyclictest0-21swapper/121:07:491
3258799201,14cyclictest0-21swapper/120:30:051
3258799201,14cyclictest0-21swapper/120:16:481
3258799201,14cyclictest0-21swapper/119:22:081
3258799201,11cyclictest1883-21sort00:05:161
3258799201,11cyclictest1492-21awk20:25:011
3258799201,11cyclictest10480-21cut19:30:151
3258799200,13cyclictest0-21swapper/123:45:091
3258199201,9cyclictest18884-21tr22:15:230
3258199201,10cyclictest25683-21fschecks_count20:05:150
3258199201,10cyclictest13285-21tail23:15:260
3259399195,8cyclictest87150irq/126-enp1s0-21:36:042
3259399195,5cyclictest17569-21ntp_states19:45:212
3259399193,7cyclictest87150irq/126-enp1s0-00:15:482
3259399193,13cyclictest20927-21sort23:35:132
3259399192,9cyclictest311ktimersoftd/221:15:122
32593991911,6cyclictest23180-21ntp_kernel_pll_22:25:212
32593991911,5cyclictest3928-21/usr/sbin/munin20:30:282
32593991911,5cyclictest3764-21wc21:40:232
3259399191,10cyclictest0-21swapper/223:25:582
32593991910,6cyclictest32023-21cpuspeed_turbos20:20:122
3259399190,5cyclictest0-21swapper/222:45:222
3258799199,2cyclictest29806-21ntp_states22:40:231
3258799197,5cyclictest23-21ksoftirqd/119:50:131
3258799197,3cyclictest0-21swapper/122:33:441
3258799192,9cyclictest23205-21ntp_states22:25:211
32587991916,2cyclictest0-21swapper/121:32:391
32587991912,2cyclictest0-21swapper/120:40:321
3258799191,17cyclictest0-21swapper/119:49:001
3258799191,10cyclictest9083-21ntp_states00:20:211
3258199195,10cyclictest87350irq/128-enp1s0-21:47:160
3258199192,4cyclictest16283-21awk19:45:020
3258199191,6cyclictest0-21swapper/022:14:140
3258199191,6cyclictest0-21swapper/021:38:200
32581991914,3cyclictest9689-21munin-run19:30:010
3258199191,10cyclictest1710-21/usr/sbin/munin20:25:280
3258199191,10cyclictest13969-21/usr/sbin/munin23:20:120
3258199191,10cyclictest0-21swapper/022:08:220
32581991910,2cyclictest87350irq/128-enp1s0-23:48:030
3258199190,10cyclictest4759-21pmu-power00:10:220
3259399181,8cyclictest0-21swapper/221:28:382
3259399181,8cyclictest0-21swapper/221:28:382
3259399181,6cyclictest0-21swapper/221:32:172
3259399181,15cyclictest0-21swapper/222:30:172
32593991811,4cyclictest2623-21cat20:25:192
32593991810,5cyclictest2218-40dnf19:12:522
32587991814,3cyclictest0-21swapper/100:00:021
3258799181,16cyclictest0-21swapper/121:20:201
3258799181,10cyclictest4895-21df_inode21:45:131
3258199188,3cyclictest87350irq/128-enp1s0-22:47:350
3258199185,4cyclictest16675-21sshd20:57:260
3258199182,6cyclictest11-21rcuc/023:55:250
3258199182,13cyclictest7-21ksoftirqd/020:50:190
3258199181,8cyclictest0-21swapper/020:20:550
3258199181,8cyclictest0-21swapper/019:11:560
3258199181,7cyclictest30955-21sed21:30:160
3258199181,7cyclictest0-21swapper/000:01:520
3258199181,6cyclictest0-21swapper/000:28:440
3258199181,4cyclictest0-21swapper/023:07:220
3258199181,3cyclictest0-21swapper/000:30:210
32581991812,3cyclictest9908-21/usr/sbin/munin19:30:220
32581991812,3cyclictest3447-21awk21:40:210
3258199181,15cyclictest29114-21crond23:55:010
3258199181,15cyclictest29114-21crond23:55:010
3258199181,11cyclictest0-21swapper/021:10:110
3258199181,11cyclictest0-21swapper/020:10:220
3258199181,10cyclictest9708-21if_enp2s021:55:170
202832180,4sleep20-21swapper/222:20:112
3259399174,10cyclictest10257-21diskstats19:30:142
32593991713,3cyclictest0-21swapper/219:28:052
3259399170,5cyclictest19189-21cat21:05:012
32587991714,2cyclictest0-21swapper/119:15:071
3258199176,3cyclictest0-21swapper/021:17:520
3258199175,4cyclictest28935-21cat22:40:130
3258199175,3cyclictest87350irq/128-enp1s0-22:30:220
3258199171,8cyclictest19709-21fschecks_count21:05:150
3258199171,7cyclictest7-21ksoftirqd/020:35:150
3258199171,7cyclictest15180-21grep00:35:160
3258199171,7cyclictest0-21swapper/021:25:240
3258199171,7cyclictest0-21swapper/021:25:240
3258199171,5cyclictest0-21swapper/000:20:370
3258199171,2cyclictest27303-21if_enp2s022:35:230
3258199171,2cyclictest27303-21if_enp2s022:35:230
3258199171,14cyclictest0-21swapper/020:15:070
32581991711,3cyclictest3029-21ntpq19:15:030
3259399168,6cyclictest5520-21/usr/sbin/munin19:20:092
3258199165,3cyclictest87350irq/128-enp1s0-22:00:240
3258199162,4cyclictest4891-21missed_timers20:30:190
3258199161,6cyclictest0-21swapper/019:45:120
32581991611,3cyclictest8325-21/usr/sbin/munin20:40:140
32581991611,3cyclictest2478-21ntp_states00:05:230
32581991611,3cyclictest12102-21/usr/sbin/munin19:35:190
32581991611,3cyclictest10521-21/usr/sbin/munin20:45:220
3258199161,12cyclictest0-21swapper/022:55:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional