You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-06 - 18:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Sat Jun 06, 2026 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
259392243167,24sleep20-21swapper/219:07:362
260642205176,20sleep00-21swapper/019:09:150
259702204167,24sleep30-21swapper/319:08:013
259262202167,23sleep10-21swapper/119:07:271
234092540,3sleep123412-21ntp_offset22:40:201
125682500,2sleep121-21rcuc/119:50:131
187582490,3sleep32621599cyclictest23:45:123
2619799431,30cyclictest1257-21expr23:05:120
2619799421,6cyclictest4559-21awk22:00:010
2619799421,30cyclictest11986-21/usr/sbin/munin23:30:150
14622420,1sleep31466-21pmu-power21:50:223
2619799409,18cyclictest10240-21ntp_states22:10:200
2619799381,26cyclictest21680-21cat21:25:020
26197993810,13cyclictest10530-21awk21:00:010
26210993733,3cyclictest0-21swapper/220:29:022
2619799361,24cyclictest3535-21/usr/sbin/munin19:30:240
2619799351,5cyclictest18347-21latency_hist23:45:010
2619799351,22cyclictest23965-21sshd22:40:310
2619799349,22cyclictest10125-21/usr/sbin/munin19:45:240
2619799346,25cyclictest26536-21munin-run22:50:010
2619799346,18cyclictest0-21swapper/023:26:490
2619799346,18cyclictest0-21swapper/023:26:490
26197993410,21cyclictest27390-21/usr/sbin/munin00:05:130
2619799339,21cyclictest1951-21/usr/sbin/munin20:40:230
2619799335,13cyclictest1090-21munin-run19:25:010
2619799331,29cyclictest17348-21/usr/sbin/munin21:15:270
26197993311,19cyclictest25517-21/usr/sbin/munin20:20:140
2619799326,22cyclictest25762-21tr00:00:140
2619799324,19cyclictest540-21kworker/u8:023:18:210
2619799323,3cyclictest27040-21cat21:35:170
2619799322,7cyclictest0-21swapper/020:06:320
2619799322,3cyclictest25913-21munin-run21:35:010
2619799322,1cyclictest0-21swapper/022:05:570
2619799321,4cyclictest9735-21sed00:35:250
2619799321,21cyclictest7870-21sshd00:33:310
2619799321,21cyclictest7870-21sshd00:33:310
2619799319,19cyclictest4770-21/usr/sbin/munin22:00:130
2619799315,2cyclictest87350irq/126-enp1s0-22:20:080
2619799315,2cyclictest87350irq/126-enp1s0-22:20:080
2619799311,5cyclictest31737-21cat00:15:010
2619799311,4cyclictest29990-21/usr/sbin/munin22:55:200
2619799311,4cyclictest19555-21taskset22:30:270
2619799311,3cyclictest22207-21fschecks_count21:25:140
2619799311,3cyclictest2154-21tune2fs00:20:130
2619799311,26cyclictest28336-21/usr/sbin/munin21:40:260
26197993110,17cyclictest21125-21/usr/sbin/munin20:10:160
2619799310,10cyclictest0-21swapper/023:52:260
2621099302,15cyclictest0-21swapper/200:02:392
2619799308,19cyclictest1-21systemd19:15:010
2619799308,14cyclictest0-21swapper/023:00:150
2619799307,20cyclictest28520-21munin-node20:45:110
2619799302,3cyclictest31601-21ntp_states21:45:220
2619799301,4cyclictest13348-21/usr/sbin/munin22:20:010
26197993013,14cyclictest21268-21ls22:35:230
2619799301,19cyclictest827-21runrttasks00:25:560
26197993010,17cyclictest9634-21cat23:25:010
26197993010,17cyclictest827-21runrttasks22:26:010
26197993010,17cyclictest827-21runrttasks20:54:020
2619799299,17cyclictest23141-21ls23:55:120
2619799299,17cyclictest14651-21expr19:55:120
2619799299,17cyclictest14651-21expr19:55:120
2619799299,17cyclictest14286-21threads21:05:250
2619799299,17cyclictest14286-21threads21:05:240
2619799298,18cyclictest9062-21pmu-power19:40:200
2619799298,18cyclictest827-21runrttasks23:35:110
2619799298,18cyclictest28825-21ls20:25:210
2619799295,3cyclictest87550irq/128-enp1s0-20:35:240
2619799293,3cyclictest17627-21/usr/sbin/munin20:00:190
2619799292,4cyclictest3870-21hddtemp_smartct23:10:160
2619799291,8cyclictest14929-21diskmemload22:54:460
2619799291,4cyclictest30728-21irqstats20:30:170
2619799291,4cyclictest19521-21/usr/sbin/munin23:45:180
2619799291,3cyclictest32382-21/usr/sbin/munin00:15:150
2619799291,3cyclictest28661-21diskstats19:15:020
26197992911,15cyclictest14226-21seq19:54:170
2619799290,19cyclictest0-21swapper/019:35:130
2619799281,5cyclictest626-21/usr/sbin/munin21:50:130
2619799281,24cyclictest15151-21/usr/sbin/munin21:10:120
26197992811,14cyclictest3113-21seq19:27:400
2621599279,5cyclictest827-21runrttasks22:47:253
2621599271,11cyclictest0-21swapper/319:42:353
2619799272,4cyclictest25292-21munin-run20:20:010
2619799271,4cyclictest11954-21/usr/sbin/munin21:00:210
2621599261,12cyclictest31496-21/usr/sbin/munin19:20:123
2621599261,11cyclictest24014-21sshd22:41:083
2620299261,17cyclictest30541-21users22:55:261
2621599253,11cyclictest9172-21/usr/sbin/munin22:10:173
2621599252,11cyclictest14521-21/usr/sbin/munin19:55:163
2621599252,11cyclictest14521-21/usr/sbin/munin19:55:163
2621599251,11cyclictest8225-21/usr/sbin/munin00:35:123
2621599251,11cyclictest3835-21/usr/sbin/munin00:25:193
2621099257,8cyclictest0-21swapper/200:38:532
2621099251,12cyclictest0-21swapper/223:11:592
2621099251,10cyclictest0-21swapper/223:55:532
2620299251,4cyclictest19134-21wc22:30:231
2621599241,7cyclictest0-21swapper/300:30:183
2621599241,7cyclictest0-21swapper/300:30:183
2621599241,16cyclictest0-21swapper/320:25:263
2621099247,3cyclictest0-21swapper/221:35:592
26210992421,2cyclictest0-21swapper/223:07:482
26202992414,2cyclictest20612-40dnf20:06:311
2620299240,19cyclictest0-21swapper/122:09:001
2621599231,6cyclictest0-21swapper/300:00:523
2621599231,10cyclictest30536-21/usr/sbin/munin21:45:123
2621099231,8cyclictest0-21swapper/220:55:112
2620299231,8cyclictest0-21swapper/122:39:121
26202992315,3cyclictest0-21swapper/100:00:251
26202992314,2cyclictest808-21snmpd21:49:581
26202992314,2cyclictest24895-21df_inode22:45:131
2620299231,3cyclictest1733-21munin-run20:40:001
26202992313,3cyclictest808-21snmpd19:21:441
2620299231,16cyclictest0-21swapper/120:22:531
2621599221,15cyclictest0-21swapper/323:51:293
2621599221,10cyclictest27390-21/usr/sbin/munin00:05:163
2621099229,3cyclictest0-21swapper/222:02:492
2621099225,5cyclictest14948-21/usr/sbin/munin23:35:152
2621099221,8cyclictest0-21swapper/200:13:382
26210992218,3cyclictest32-21ksoftirqd/222:30:012
2621099221,7cyclictest0-21swapper/221:55:152
2620299223,10cyclictest21966-21processes23:50:231
2620299221,6cyclictest0-21swapper/121:43:431
2620299221,3cyclictest72742sleep123:19:011
2620299221,16cyclictest0-21swapper/123:34:171
2620299221,16cyclictest0-21swapper/119:59:401
2620299221,16cyclictest0-21swapper/119:59:401
2620299221,16cyclictest0-21swapper/119:32:291
2620299221,15cyclictest0-21swapper/119:15:041
2620299221,14cyclictest14929-21diskmemload23:05:071
2620299221,13cyclictest0-21swapper/121:15:281
2620299221,11cyclictest9895-21sh19:45:011
2620299221,10cyclictest0-21swapper/122:27:281
2620299220,15cyclictest14929-21diskmemload21:32:401
2621599217,3cyclictest0-21swapper/321:21:263
2621599215,9cyclictest0-21swapper/319:45:133
2621599211,5cyclictest0-21swapper/323:07:213
2621099211,7cyclictest0-21swapper/223:17:512
2621099211,10cyclictest0-21swapper/223:26:392
2621099211,10cyclictest0-21swapper/223:26:392
2620299219,3cyclictest692-21munin-run23:05:011
2620299211,9cyclictest0-21swapper/123:25:381
2620299211,9cyclictest0-21swapper/123:25:381
2620299211,3cyclictest0-21swapper/123:41:081
26202992113,3cyclictest0-21swapper/122:17:561
2620299211,18cyclictest0-21swapper/120:57:111
2620299211,15cyclictest0-21swapper/123:59:211
2620299211,15cyclictest0-21swapper/123:36:351
2620299211,15cyclictest0-21swapper/123:21:051
2620299211,15cyclictest0-21swapper/122:11:591
2620299211,15cyclictest0-21swapper/121:57:231
2620299211,15cyclictest0-21swapper/120:19:481
2620299211,14cyclictest0-21swapper/122:00:131
2620299211,11cyclictest18755-21cat20:05:011
26202992110,7cyclictest0-21swapper/100:27:511
2621599201,8cyclictest0-21swapper/321:35:223
2621599201,5cyclictest87650irq/129-enp1s0-23:18:203
2621599201,5cyclictest0-21swapper/323:23:453
2621599201,4cyclictest0-21swapper/321:41:293
2621599201,4cyclictest0-21swapper/321:17:043
2621599201,4cyclictest0-21swapper/300:00:003
2621599201,13cyclictest0-21swapper/322:58:363
2621599201,12cyclictest0-21swapper/322:24:193
2621599201,12cyclictest0-21swapper/322:24:193
2621599201,12cyclictest0-21swapper/322:07:423
2621099207,3cyclictest0-21swapper/221:16:332
2621099206,5cyclictest24655-1kworker/u9:022:09:012
2621099206,3cyclictest32-21ksoftirqd/223:20:202
2621099201,9cyclictest0-21swapper/200:30:522
2621099201,9cyclictest0-21swapper/200:30:522
2621099201,8cyclictest0-21swapper/223:53:382
26210992013,4cyclictest27914-21grep19:10:252
26210992013,4cyclictest22810-21ntp_states21:25:212
26210992013,2cyclictest24655-1kworker/u9:022:53:032
2621099201,12cyclictest0-21swapper/222:55:242
2621099201,11cyclictest0-21swapper/222:48:132
2621099201,10cyclictest755-21rngd21:20:292
2620299206,3cyclictest0-21swapper/119:49:011
2620299206,12cyclictest1371-21cpu19:25:121
2620299204,7cyclictest21-21rcuc/121:05:241
2620299204,7cyclictest21-21rcuc/121:05:241
2620299204,7cyclictest0-21swapper/121:10:051
2620299201,3cyclictest0-21swapper/123:10:281
2620299201,18cyclictest0-21swapper/121:54:491
2620299201,14cyclictest0-21swapper/121:39:431
2620299201,14cyclictest0-21swapper/121:22:571
2620299201,14cyclictest0-21swapper/121:00:081
2620299201,14cyclictest0-21swapper/120:54:091
2620299201,13cyclictest0-21swapper/100:32:031
2620299201,13cyclictest0-21swapper/100:32:031
2620299201,11cyclictest6117-21latency_hist20:50:011
2620299201,11cyclictest23708-21latency_hist21:30:021
2620299200,15cyclictest0-21swapper/100:36:251
2620299200,15cyclictest0-21swapper/100:15:361
2620299200,15cyclictest0-21swapper/100:06:141
2620299200,14cyclictest0-21swapper/120:44:321
2620299200,14cyclictest0-21swapper/100:12:081
2621599198,2cyclictest87350irq/126-enp1s0-22:15:133
2621599196,5cyclictest41-21ksoftirqd/322:30:223
2621599196,4cyclictest41-21ksoftirqd/323:00:173
2621599193,5cyclictest755-21rngd00:16:193
2621599192,9cyclictest4770-21/usr/sbin/munin22:00:213
2621599191,5cyclictest0-21swapper/323:30:313
2621599191,5cyclictest0-21swapper/322:37:093
2621599191,5cyclictest0-21swapper/321:32:173
2621599191,5cyclictest0-21swapper/300:13:433
26215991913,3cyclictest28298-21grep22:50:253
2621599191,16cyclictest87350irq/126-enp1s0-19:30:213
2621599191,15cyclictest6426-21idleruntime19:35:153
2621599190,9cyclictest39-21rcuc/320:10:263
2621599190,6cyclictest41-21ksoftirqd/321:14:593
2621099192,5cyclictest20833-21hddtemp_smartct22:35:162
2621099191,9cyclictest15960-21irqstats21:10:172
2621099191,6cyclictest29175-21kernelversion21:40:172
2621099191,6cyclictest12085-21tr23:30:122
2621099191,5cyclictest2062-21sshd21:51:292
26210991915,2cyclictest32-21ksoftirqd/219:30:002
2621099191,11cyclictest0-21swapper/222:33:172
2621099191,11cyclictest0-21swapper/200:20:172
26210991910,3cyclictest24655-1kworker/u9:023:45:102
2621099190,4cyclictest0-21swapper/221:45:152
2621099190,12cyclictest0-21swapper/222:41:202
2620299198,3cyclictest2549-21memory00:20:171
26202991912,2cyclictest0-21swapper/120:13:221
2620299191,16cyclictest0-21swapper/120:31:071
2620299191,10cyclictest6272-21cut19:35:141
2620299191,10cyclictest27739-21egrep20:25:111
2620299191,10cyclictest19390-21irqstats23:45:171
2620299190,4cyclictest19782-21kworker/u8:022:50:441
2620299190,14cyclictest0-21swapper/119:11:181
2621599187,3cyclictest16720-21/usr/sbin/munin20:00:163
2621599184,6cyclictest2143-21taskset19:25:173
2621599181,7cyclictest0-21swapper/323:25:133
2621599181,7cyclictest0-21swapper/323:25:133
2621599181,6cyclictest0-21swapper/322:26:423
2621599181,6cyclictest0-21swapper/321:56:283
2621599181,6cyclictest0-21swapper/321:05:193
2621599181,6cyclictest0-21swapper/321:05:193
2621599181,6cyclictest0-21swapper/320:45:203
26215991812,3cyclictest1951-21/usr/sbin/munin20:40:203
26215991812,3cyclictest13396-21ntp_states19:50:213
2621599181,15cyclictest0-21swapper/319:17:313
2621099181,7cyclictest0-21swapper/222:14:122
2621099181,6cyclictest0-21swapper/221:31:352
2621099181,5cyclictest0-21swapper/222:23:262
2621099181,5cyclictest0-21swapper/222:23:262
2621099181,4cyclictest10125-21/usr/sbin/munin19:45:182
26210991814,2cyclictest30-21rcuc/221:00:112
2621099181,3cyclictest87650irq/129-enp1s0-22:16:002
26210991812,4cyclictest29930-21grep19:15:152
2621099181,15cyclictest5095-21grep19:30:252
2621099181,15cyclictest0-21swapper/219:41:412
2621099181,14cyclictest17038-21gltestperf23:40:152
2621099181,10cyclictest0-21swapper/200:28:002
2621099181,10cyclictest0-21swapper/200:15:032
2620299180,13cyclictest0-21swapper/122:21:101
2620299180,13cyclictest0-21swapper/122:21:101
19492181,13sleep31980-21diskstats00:20:123
141772181,3sleep30-21swapper/323:35:093
2621599175,3cyclictest6364-21egrep20:50:113
2621599173,2cyclictest87350irq/126-enp1s0-23:10:123
2621599171,9cyclictest0-21swapper/320:20:233
2621599171,6cyclictest21742-21/usr/sbin/munin21:25:203
2621599171,3cyclictest0-21swapper/321:02:223
2621599171,14cyclictest0-21swapper/323:40:103
2621599171,14cyclictest0-21swapper/320:35:133
2621599171,10cyclictest0-21swapper/320:30:253
2621099179,3cyclictest4330-21cstates20:45:122
2621099171,9cyclictest0-21swapper/219:23:302
2621099171,4cyclictest0-21swapper/220:02:302
2621099171,14cyclictest0-21swapper/220:51:112
2621099171,14cyclictest0-21swapper/220:20:082
2621099171,14cyclictest0-21swapper/219:55:452
2621099171,14cyclictest0-21swapper/219:55:442
26210991711,3cyclictest31961-21irqstats23:00:162
26210991711,3cyclictest27748-21entropy00:05:142
26210991711,3cyclictest19267-21df_inode20:05:132
2621599168,5cyclictest27971-21grep19:10:263
26215991611,3cyclictest41-21ksoftirqd/320:15:203
26215991610,4cyclictest9568-21netstat20:55:203
26210991611,3cyclictest827-21runrttasks21:08:372
26210991611,3cyclictest827-21runrttasks21:08:372
26210991611,3cyclictest3018-21ntp_states20:40:212
26210991611,3cyclictest23180-21cut20:15:012
14662161,2sleep20-21swapper/220:37:062
26215991510,3cyclictest19913-21tr20:05:193
2621099159,3cyclictest23504-21cstates20:15:122
2621099152,3cyclictest12923-21kworker/u8:219:51:352
26210991510,3cyclictest6460-21if_enp1s019:35:162
26210991510,3cyclictest30623-21idleruntime20:30:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional