You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-23 - 12:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat May 23, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27632247164,21sleep20-21swapper/219:05:272
31282237161,22sleep00-21swapper/019:09:470
29712204168,23sleep30-21swapper/319:07:453
30932201165,24sleep10-21swapper/119:09:201
1659021740,3sleep2323699cyclictest00:25:202
2621921450,3sleep2323699cyclictest23:35:122
3236997571,3cyclictest0-21swapper/221:21:202
264162530,1sleep10-21swapper/120:00:121
264162530,1sleep10-21swapper/120:00:121
172792490,2sleep1323099cyclictest00:27:241
3236993936,2cyclictest0-21swapper/221:06:242
323699351,16cyclictest1-21systemd20:24:312
323699329,4cyclictest23689-21awk19:55:012
323699321,15cyclictest29369-21ntp_states20:05:212
323699311,10cyclictest18070-21diskstats00:30:132
322299301,14cyclictest0-21swapper/023:09:560
324399298,9cyclictest21698-21ntp_kernel_err22:10:193
323699292,6cyclictest21035-40dnf23:22:252
323699291,14cyclictest0-21swapper/222:30:212
323699291,13cyclictest0-21swapper/219:10:292
323699291,12cyclictest6666-21df_inode19:15:132
322299296,11cyclictest0-21swapper/019:55:010
323699281,9cyclictest0-21swapper/222:00:212
323699281,4cyclictest9642-21idleruntime00:10:162
323699281,13cyclictest0-21swapper/223:34:412
323699281,13cyclictest0-21swapper/223:34:412
323699281,13cyclictest0-21swapper/219:55:372
323699281,12cyclictest2307-21munin-run20:20:012
54662270,4sleep1323099cyclictest19:15:011
324399271,11cyclictest0-21swapper/320:41:013
323699271,13cyclictest0-21swapper/223:07:092
323699271,12cyclictest0-21swapper/220:43:252
323699271,11cyclictest28420-21diskstats23:40:132
323699270,14cyclictest0-21swapper/223:04:332
323099271,13cyclictest0-21swapper/123:39:271
3243992615,3cyclictest0-21swapper/320:53:073
324399261,17cyclictest0-21swapper/320:10:113
324399261,12cyclictest0-21swapper/300:20:173
324399261,10cyclictest15996-21diskstats20:45:133
3243992610,5cyclictest87350irq/126-enp1s0-20:21:113
323699269,2cyclictest0-21swapper/220:25:232
323699261,11cyclictest0-21swapper/222:20:052
323699261,11cyclictest0-21swapper/222:20:052
323699261,11cyclictest0-21swapper/221:48:092
323699261,11cyclictest0-21swapper/220:30:572
323699260,11cyclictest0-21swapper/200:36:572
323099261,17cyclictest0-21swapper/121:20:401
323099260,3cyclictest87350irq/126-enp1s0-00:18:071
322299261,13cyclictest17511-21basename00:30:010
324399258,3cyclictest0-21swapper/323:38:353
324399251,10cyclictest0-21swapper/322:43:053
324399251,10cyclictest0-21swapper/319:38:053
324399251,10cyclictest0-21swapper/319:25:173
323699251,12cyclictest0-21swapper/220:45:052
323699251,11cyclictest0-21swapper/222:13:492
323699250,14cyclictest0-21swapper/222:40:172
323699250,12cyclictest0-21swapper/220:00:172
323699250,12cyclictest0-21swapper/220:00:172
324399248,4cyclictest0-21swapper/300:25:013
324399248,3cyclictest0-21swapper/322:45:533
324399248,3cyclictest0-21swapper/322:29:573
324399248,3cyclictest0-21swapper/321:30:133
324399247,8cyclictest0-21swapper/320:19:033
324399247,2cyclictest0-21swapper/322:30:153
324399245,3cyclictest22537-21ntpq23:25:193
3243992414,2cyclictest0-21swapper/323:30:113
3243992414,2cyclictest0-21swapper/323:30:113
324399241,13cyclictest0-21swapper/322:37:453
324399241,10cyclictest0-21swapper/323:13:493
324399241,10cyclictest0-21swapper/319:41:533
324399240,9cyclictest0-21swapper/300:13:013
323699244,2cyclictest18226-21pmu-power23:15:212
323099245,2cyclictest87650irq/129-enp1s0-19:45:081
323099242,8cyclictest17464-21kworker/u8:100:32:171
323099242,15cyclictest0-21swapper/121:31:131
323099241,14cyclictest0-21swapper/122:43:311
323099240,19cyclictest0-21swapper/121:11:391
323099240,19cyclictest0-21swapper/121:11:391
324399239,3cyclictest0-21swapper/321:10:173
324399239,3cyclictest0-21swapper/321:10:173
324399238,4cyclictest6775-21systemd20:25:073
324399238,3cyclictest0-21swapper/321:00:213
324399236,8cyclictest31184-21missed_timers23:45:183
324399231,11cyclictest0-21swapper/322:00:173
324399231,11cyclictest0-21swapper/320:36:333
3243992310,3cyclictest10021-21df_inode21:45:153
324399230,4cyclictest0-21swapper/300:35:153
323699238,3cyclictest0-21swapper/219:26:332
323699230,11cyclictest0-21swapper/200:21:012
323099239,4cyclictest2849-21munin-run21:30:011
323099239,4cyclictest0-21swapper/123:02:501
323099237,9cyclictest0-21swapper/121:59:461
323099231,7cyclictest13364-21hddtemp_smartct23:05:161
323099231,19cyclictest0-21swapper/120:20:091
323099231,15cyclictest0-21swapper/123:31:481
323099231,15cyclictest0-21swapper/123:31:481
322299231,12cyclictest1-21systemd20:25:080
311082230,5sleep339-21rcuc/321:20:123
324399229,3cyclictest17277-21expr23:15:113
324399229,1cyclictest0-21swapper/323:00:153
324399228,3cyclictest26413-21diskmemload00:18:313
324399228,3cyclictest0-21swapper/323:50:293
324399228,3cyclictest0-21swapper/320:02:293
324399228,3cyclictest0-21swapper/320:02:293
324399227,9cyclictest0-21swapper/300:30:153
324399227,8cyclictest0-21swapper/319:52:513
324399225,6cyclictest0-21swapper/321:28:523
324399221,11cyclictest0-21swapper/321:39:143
324399221,11cyclictest0-21swapper/300:02:373
324399221,10cyclictest0-21swapper/322:19:143
324399220,11cyclictest0-21swapper/322:24:253
324399220,11cyclictest0-21swapper/322:24:253
323699225,3cyclictest19516-21sshd20:51:152
323699220,11cyclictest0-21swapper/221:43:492
323099227,9cyclictest0-21swapper/122:26:281
323099223,7cyclictest25500-21kworker/u8:123:50:361
323099223,14cyclictest87550irq/128-enp1s0-21:42:521
3230992216,3cyclictest15921-21irqstats19:35:151
323099221,4cyclictest24920-21munin-run22:20:011
323099221,18cyclictest15473-21gltestperf23:10:141
323099221,15cyclictest0-21swapper/100:21:191
323099221,12cyclictest21969-21fschecks_time23:25:131
323099221,11cyclictest0-21swapper/100:37:591
323099221,10cyclictest0-21swapper/122:47:361
323099220,17cyclictest0-21swapper/122:37:331
322299221,8cyclictest10029-21/usr/sbin/munin21:45:160
322299221,10cyclictest4303-21kworker/u8:200:05:240
324399214,3cyclictest0-21swapper/319:48:593
324399211,8cyclictest0-21swapper/321:51:293
3243992115,3cyclictest18820-21diskstats22:05:143
324399211,4cyclictest14805-21sshd19:34:083
324399211,11cyclictest9449-21/usr/sbin/munin21:45:013
3243992110,3cyclictest0-21swapper/319:21:253
324399210,7cyclictest0-21swapper/323:57:153
323699214,3cyclictest6404-21chrt00:03:572
323699212,9cyclictest15575-21unixbench_multi21:55:252
323699211,3cyclictest0-21swapper/223:26:092
323699211,2cyclictest87650irq/129-enp1s0-00:18:312
323699211,14cyclictest0-21swapper/222:26:212
323099219,4cyclictest18407-21perl20:50:161
323099219,2cyclictest0-21swapper/123:47:481
323099218,3cyclictest0-21swapper/123:44:091
323099214,3cyclictest25500-21kworker/u8:123:55:181
3230992115,3cyclictest24062-21cpuspeed_turbos19:55:121
323099211,15cyclictest0-21swapper/120:19:031
323099211,12cyclictest10605-21taskset00:11:191
323099211,11cyclictest20932-21cat20:55:171
322299216,4cyclictest0-21swapper/023:14:050
322299211,11cyclictest30503-21/usr/sbin/munin20:10:120
322299211,11cyclictest30268-21/usr/sbin/munin23:45:250
322299211,11cyclictest16267-21/usr/sbin/munin22:00:160
324399206,3cyclictest0-21swapper/319:58:353
324399205,3cyclictest0-21swapper/319:11:393
324399204,4cyclictest0-21swapper/322:55:173
3243992015,3cyclictest29259-21grep21:15:153
324399201,3cyclictest15802-21sshd21:57:083
324399201,12cyclictest0-21swapper/319:17:153
324399201,11cyclictest839-21lldpd21:05:293
324399201,11cyclictest28306-21/usr/sbin/munin20:05:193
324399201,11cyclictest0-21swapper/323:40:043
324399201,10cyclictest19283-21/usr/sbin/munin23:20:273
324399201,10cyclictest0-21swapper/322:50:123
323699206,8cyclictest488-21kworker/u8:221:27:062
323699205,4cyclictest32-21ksoftirqd/223:55:242
323699201,15cyclictest0-21swapper/220:10:292
323699201,14cyclictest13212-21timerandwakeup21:50:252
323699201,11cyclictest2093-21idleruntime-cro23:55:012
323699201,10cyclictest0-21swapper/222:18:092
323099208,3cyclictest0-21swapper/123:23:401
323099207,4cyclictest0-21swapper/100:05:291
323099203,7cyclictest6333-21runrttasks00:03:031
323099202,15cyclictest19142-21head22:05:161
3230992013,2cyclictest0-21swapper/122:13:531
3230992013,2cyclictest0-21swapper/121:18:251
323099201,17cyclictest0-21swapper/123:19:281
323099201,17cyclictest0-21swapper/119:15:421
323099201,14cyclictest0-21swapper/122:54:331
323099201,14cyclictest0-21swapper/120:48:541
323099201,14cyclictest0-21swapper/120:33:201
323099201,11cyclictest18251-21munin-run22:05:011
323099200,2cyclictest26413-21diskmemload22:30:531
322299209,2cyclictest21477-21/usr/sbin/munin23:25:220
322299207,3cyclictest808-21snmpd21:34:510
322299207,3cyclictest23675-21/usr/sbin/munin23:30:210
322299207,3cyclictest23675-21/usr/sbin/munin23:30:210
322299205,3cyclictest31446-21kworker/0:223:15:210
322299202,5cyclictest12469-21munin-run23:05:010
322299201,7cyclictest0-21swapper/022:38:310
322299201,5cyclictest24655-1kworker/u9:022:19:370
3222992015,2cyclictest7-21ksoftirqd/020:24:440
3222992011,3cyclictest87550irq/128-enp1s0-21:35:160
322299201,12cyclictest0-21swapper/021:44:580
322299201,11cyclictest30495-21sshd21:16:080
322299201,10cyclictest8523-21/usr/sbin/munin19:20:210
322299201,10cyclictest19125-21sshd19:43:080
322299201,10cyclictest0-21swapper/023:37:510
322299201,10cyclictest0-21swapper/022:55:250
324399195,4cyclictest11185-21cat20:35:023
324399191,9cyclictest12971-21cstates23:05:133
324399191,6cyclictest7843-21cut00:05:223
323699196,3cyclictest12278-21/usr/sbin/munin20:35:172
323699196,3cyclictest0-21swapper/223:11:552
323699193,8cyclictest4751-21sshd21:31:172
323699193,2cyclictest87550irq/128-enp1s0-22:45:032
323699192,11cyclictest0-21swapper/222:07:242
323699191,15cyclictest0-21swapper/219:33:372
323699191,14cyclictest0-21swapper/219:43:332
323699191,13cyclictest5625-21diskstats21:35:132
323699191,11cyclictest0-21swapper/219:35:272
323699191,10cyclictest6515-21uniq22:50:142
323099195,4cyclictest0-21swapper/121:49:001
323099194,3cyclictest17827-21kworker/1:121:35:211
3230991915,3cyclictest0-21swapper/119:33:491
323099191,17cyclictest0-21swapper/121:02:451
323099191,17cyclictest0-21swapper/120:09:501
323099191,16cyclictest0-21swapper/121:05:481
323099191,11cyclictest0-21swapper/121:50:591
323099191,10cyclictest31764-21smartctl20:10:231
323099191,10cyclictest13801-21diskstats20:40:131
323099190,17cyclictest0-21swapper/120:37:191
323099190,14cyclictest0-21swapper/119:43:351
322299197,3cyclictest87550irq/128-enp1s0-21:52:440
322299191,9cyclictest0-21swapper/000:01:000
322299191,7cyclictest0-21swapper/023:54:200
322299191,7cyclictest0-21swapper/022:47:500
322299191,6cyclictest10275-21timerandwakeup00:10:250
322299191,6cyclictest0-21swapper/022:26:250
3222991914,3cyclictest7-21ksoftirqd/000:00:010
322299191,12cyclictest0-21swapper/021:10:180
322299191,12cyclictest0-21swapper/021:10:180
322299191,10cyclictest0-21swapper/022:40:420
324399182,5cyclictest21087-21ntp_kernel_pll_20:55:193
323699184,5cyclictest755-21rngd22:35:042
323699182,2cyclictest87350irq/126-enp1s0-20:59:512
323699181,5cyclictest31330-21ntp_states23:45:202
3236991813,3cyclictest6728-21/usr/sbin/munin00:05:262
323699181,1cyclictest0-21swapper/221:10:382
323699181,1cyclictest0-21swapper/221:10:372
323699181,15cyclictest0-21swapper/219:20:152
323699181,14cyclictest0-21swapper/221:16:372
323699181,14cyclictest0-21swapper/221:02:412
323699181,14cyclictest0-21swapper/219:45:132
323099181,4cyclictest8638-21diskstats22:55:131
3230991814,3cyclictest0-21swapper/122:20:121
3230991814,3cyclictest0-21swapper/122:20:121
323099181,16cyclictest0-21swapper/119:24:151
323099181,15cyclictest0-21swapper/119:25:081
323099180,17cyclictest0-21swapper/119:50:201
322299182,8cyclictest29600-21/usr/sbin/munin22:30:160
322299181,7cyclictest0-21swapper/023:20:270
322299181,6cyclictest0-21swapper/021:59:360
322299181,6cyclictest0-21swapper/021:05:170
322299181,5cyclictest0-21swapper/022:07:020
322299181,5cyclictest0-21swapper/021:26:300
3222991812,3cyclictest15206-21grep19:35:100
322299181,15cyclictest0-21swapper/000:20:240
3222991810,2cyclictest12050irq/123-ahci[0020:30:130
323699171,2cyclictest0-21swapper/222:55:252
323099178,2cyclictest910-21systemd-logind20:25:021
322299179,5cyclictest8295-21munin-run19:20:010
322299172,3cyclictest0-21swapper/019:59:310
322299171,7cyclictest18197-21fschecks_time00:30:140
322299171,6cyclictest0-21swapper/021:00:180
322299171,6cyclictest0-21swapper/000:18:370
322299171,4cyclictest0-21swapper/022:21:090
322299171,4cyclictest0-21swapper/022:21:090
322299171,4cyclictest0-21swapper/022:11:130
3222991713,3cyclictest7-21ksoftirqd/020:45:140
3222991711,3cyclictest31689-21gltestperf21:20:160
3222991711,3cyclictest17953-21grep20:50:130
3222991711,3cyclictest13452-21/usr/sbin/munin20:40:130
3222991710,4cyclictest29555-21grep23:40:240
148152171,3sleep00-21swapper/019:34:160
322299167,6cyclictest11361-21tr20:35:100
322299167,3cyclictest30488-21kworker/0:320:15:170
322299161,6cyclictest6093-21/usr/sbin/munin22:50:200
322299161,5cyclictest0-21swapper/000:35:230
322299161,2cyclictest0-21swapper/020:58:140
322299161,14cyclictest0-21swapper/019:26:590
3222991611,3cyclictest28860-21perl20:05:170
3222991611,3cyclictest26440-21sh20:00:140
3222991611,3cyclictest26440-21sh20:00:130
322299161,12cyclictest3436-21/usr/sbin/munin19:10:170
322299161,12cyclictest19509-21/usr/sbin/munin19:45:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional