You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-22 - 05:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Dec 22, 2025 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
251332231164,54sleep10-21swapper/119:09:591
251332231164,54sleep10-21swapper/119:09:591
250502202165,25sleep30-21swapper/319:08:573
250502202165,25sleep30-21swapper/319:08:563
231752201165,25sleep20-21swapper/219:05:092
231752201165,25sleep20-21swapper/219:05:092
250222200162,26sleep00-21swapper/019:08:340
250222200162,26sleep00-21swapper/019:08:340
3064621800,4sleep22522699cyclictest21:45:222
176742660,5sleep32523299cyclictest21:15:253
37432470,3sleep03747-21unixbench_singl23:10:280
187912460,1sleep20-21swapper/220:05:172
14582460,1sleep10-21swapper/119:25:231
2523299393,9cyclictest0-21swapper/319:11:303
2523299370,29cyclictest0-21swapper/323:54:373
2522699371,9cyclictest32736-21kworker/2:123:41:132
319852360,5sleep32523299cyclictest20:35:203
2523299361,7cyclictest2546-21/usr/sbin/munin19:30:133
2523299356,3cyclictest5354-21users22:00:263
2523299352,11cyclictest15372-21/usr/sbin/munin23:40:253
2523299351,6cyclictest0-21swapper/321:05:143
25232993512,3cyclictest25881-21if_enp1s021:35:213
2523299351,21cyclictest835-21runrttasks23:37:273
2522699351,6cyclictest14242-21ntp_states23:35:232
2521599358,12cyclictest2943-21awk20:45:010
2523299349,8cyclictest23945-21awk00:00:003
2523299341,10cyclictest0-21swapper/322:33:363
2522699342,12cyclictest32736-21kworker/2:121:28:332
2522699342,12cyclictest32736-21kworker/2:121:28:332
2522699341,4cyclictest20593-21kernelversion23:50:202
25226993410,3cyclictest27950-21/usr/sbin/munin22:55:252
2522699340,2cyclictest0-21swapper/220:25:162
2523299335,3cyclictest87250irq/129-enp1s0-20:00:143
2523299334,3cyclictest12050irq/123-ahci[0021:50:013
2523299332,3cyclictest30146-21/usr/sbin/munin23:00:243
2523299331,5cyclictest0-21swapper/321:44:223
2523299331,15cyclictest25368-21sshd22:48:103
25232993311,4cyclictest28457-21tr19:15:143
2522699339,4cyclictest25987-21/usr/sbin/munin21:35:212
2522699331,8cyclictest30974-21uname23:00:182
2522699331,4cyclictest14388-21taskset22:23:162
25226993312,3cyclictest19154-21/usr/sbin/munin22:35:262
25232993215,10cyclictest0-21swapper/323:15:103
2523299321,2cyclictest0-21swapper/321:20:223
25232993212,4cyclictest21442-21sshd20:10:283
2523299321,15cyclictest1920-21sed19:25:283
2523299321,10cyclictest0-21swapper/323:05:103
2523299321,10cyclictest0-21swapper/321:13:263
2522699329,4cyclictest13304-21/usr/sbin/munin19:55:012
2522699329,3cyclictest835-21runrttasks19:39:132
2522699329,3cyclictest30793-21/usr/sbin/munin00:15:222
2522699328,3cyclictest12237-21smart_sda23:30:232
2522699322,26cyclictest25012-21grep22:45:262
2522699321,4cyclictest0-21swapper/219:45:222
25226993211,3cyclictest26724-21df00:05:142
25226993210,3cyclictest31194-21hwlatdetect19:20:172
25226993210,3cyclictest13218-21swap21:05:262
2522299321,18cyclictest0-21swapper/120:19:581
2523299313,19cyclictest87050irq/127-enp1s0-21:57:433
2523299311,9cyclictest0-21swapper/322:38:063
2523299311,6cyclictest0-21swapper/322:40:333
2523299311,4cyclictest0-21swapper/323:21:133
2523299311,3cyclictest26404-21grep00:05:133
2523299311,10cyclictest0-21swapper/320:25:293
2523299311,10cyclictest0-21swapper/300:24:063
2523299310,9cyclictest0-21swapper/320:56:503
2522699319,3cyclictest2546-21/usr/sbin/munin19:30:252
2522699319,3cyclictest18312-21latency_hist21:20:012
2522699311,5cyclictest0-21swapper/221:53:192
25226993110,4cyclictest0-21swapper/223:10:122
25226993110,3cyclictest835-21runrttasks22:26:082
25226993110,3cyclictest11722-21latency_hist21:05:012
2522299311,3cyclictest0-21swapper/120:51:261
2522299311,10cyclictest0-21swapper/123:43:181
2523299307,3cyclictest0-21swapper/320:15:233
2523299301,4cyclictest1269-21df20:40:153
2523299300,10cyclictest0-21swapper/320:45:113
2522699308,3cyclictest24161-21/usr/sbin/munin00:00:262
2522699307,3cyclictest5497-21sort00:30:162
2522699301,6cyclictest0-21swapper/222:05:222
2522699301,14cyclictest0-21swapper/220:50:282
2522699301,10cyclictest5889-21grep22:05:022
2522699300,19cyclictest32736-21kworker/2:122:51:282
2522299301,12cyclictest87150irq/128-enp1s0-23:21:541
2523299291,10cyclictest0-21swapper/323:33:473
2523299290,2cyclictest13290-21if_enp1s022:20:183
2523299290,10cyclictest0-21swapper/321:52:423
2522699299,3cyclictest5119-21latency_hist20:50:022
2522699299,3cyclictest24359-21latency_hist20:20:012
2522699299,3cyclictest2389-21wc20:40:272
2522699298,3cyclictest28661-21diskstats19:15:162
2522699297,3cyclictest942-21df00:20:152
25226992911,3cyclictest22208-21cat20:15:012
25226992910,3cyclictest17339-21cut22:30:152
2522299291,14cyclictest0-21swapper/119:23:021
2522299291,13cyclictest0-21swapper/120:11:341
2522299291,11cyclictest0-21swapper/122:43:571
2522299291,10cyclictest0-21swapper/119:18:291
25215992910,16cyclictest4798-21munin-run00:30:010
25232992815,11cyclictest0-21swapper/322:05:043
2523299281,12cyclictest24161-21/usr/sbin/munin00:00:293
25232992811,11cyclictest0-21swapper/322:25:203
2523299280,9cyclictest0-21swapper/300:35:183
2522699288,3cyclictest27518-21cstates21:40:152
2522699287,3cyclictest5085-21idleruntime23:15:152
2522699286,3cyclictest25415-21kworker/2:219:40:452
2522699281,5cyclictest0-21swapper/221:31:442
25226992810,3cyclictest23471-21taskset22:45:022
2522299288,2cyclictest87150irq/128-enp1s0-21:20:571
2522299281,13cyclictest0-21swapper/100:34:381
2522299281,12cyclictest0-21swapper/121:37:381
2522299281,11cyclictest0-21swapper/120:20:561
2522299281,10cyclictest0-21swapper/119:12:061
25222992810,6cyclictest0-21swapper/120:42:481
2521599283,12cyclictest29924-21munin-run23:00:010
2521599282,10cyclictest18617-21kworker/0:222:48:490
2521599280,13cyclictest16743-21awk22:30:000
2522699279,3cyclictest9515-21latency_hist21:00:022
2522699278,3cyclictest26954-21unixbench_singl19:10:262
2522699278,3cyclictest0-21swapper/221:10:242
2522699278,3cyclictest0-21swapper/220:30:132
2522699271,6cyclictest0-21swapper/223:07:582
2522699271,5cyclictest0-21swapper/222:19:112
2522699271,5cyclictest0-21swapper/222:11:412
2522699271,3cyclictest0-21swapper/200:36:272
2522699271,2cyclictest87150irq/128-enp1s0-23:22:102
25226992712,3cyclictest0-21swapper/223:57:562
2522699271,14cyclictest0-21swapper/223:46:502
2522699271,14cyclictest0-21swapper/223:46:502
2522699271,14cyclictest0-21swapper/200:13:272
2522699271,10cyclictest15320-21taskset19:57:552
25226992710,3cyclictest19558-21netstat21:20:182
2522299271,12cyclictest23591-21kworker/1:122:14:341
2522299271,12cyclictest0-21swapper/123:49:261
2522299271,12cyclictest0-21swapper/123:49:261
2522299271,12cyclictest0-21swapper/122:15:261
2522299271,11cyclictest0-21swapper/120:31:501
2522299271,10cyclictest0-21swapper/123:09:501
2522299271,10cyclictest0-21swapper/121:08:531
2522299271,10cyclictest0-21swapper/119:30:541
2521599270,13cyclictest17716-21awk20:05:010
2523299268,3cyclictest0-21swapper/321:01:083
2523299261,16cyclictest0-21swapper/300:30:163
2522699268,3cyclictest2220-21gltestperf21:55:152
2522699263,3cyclictest86950irq/126-enp1s0-20:35:152
2522699261,4cyclictest8782-21/usr/sbin/munin23:25:162
2522699261,4cyclictest0-21swapper/200:29:512
2522699260,6cyclictest0-21swapper/220:20:162
2522699260,14cyclictest0-21swapper/220:01:452
2522299261,3cyclictest0-21swapper/123:12:541
2522299261,11cyclictest0-21swapper/123:30:581
2522299261,11cyclictest0-21swapper/122:31:021
2522299260,12cyclictest0-21swapper/119:42:541
2522299260,11cyclictest0-21swapper/120:03:341
2521599261,5cyclictest0-21swapper/021:50:150
2521599260,12cyclictest0-21swapper/000:00:010
2522299251,7cyclictest0-21swapper/119:53:181
2522299251,15cyclictest14340-21kworker/u8:122:50:441
2522299251,11cyclictest0-21swapper/122:22:381
2522299251,11cyclictest0-21swapper/119:49:061
2522299251,10cyclictest10000-21cstates21:00:141
2522299251,10cyclictest0-21swapper/121:56:141
2522299251,10cyclictest0-21swapper/120:26:061
2522299250,17cyclictest0-21swapper/122:07:261
2522299250,13cyclictest0-21swapper/123:50:101
2522299250,11cyclictest0-21swapper/122:57:171
2522299250,11cyclictest0-21swapper/119:56:381
2523299241,14cyclictest0-21swapper/300:18:063
2522299245,7cyclictest0-21swapper/121:16:461
2522299242,3cyclictest0-21swapper/120:55:091
2522299242,13cyclictest31989-21kworker/u8:123:18:411
2522299241,16cyclictest9853-21ntpq23:25:221
2522299241,16cyclictest23303-21timerandwakeup23:55:251
25222992411,3cyclictest0-21swapper/121:30:141
2522299241,12cyclictest0-21swapper/100:07:281
2522299241,11cyclictest0-21swapper/100:28:061
2522299241,10cyclictest0-21swapper/122:37:411
2522299241,10cyclictest0-21swapper/120:35:281
2522299240,10cyclictest0-21swapper/120:46:461
2521599249,14cyclictest0-21swapper/023:45:010
2521599247,15cyclictest6373-21awk23:20:020
2521599242,10cyclictest26511-21perf20:25:010
2521599241,3cyclictest0-21swapper/021:27:530
2521599241,3cyclictest0-21swapper/021:27:530
2523299238,3cyclictest907-21systemd-logind19:41:293
2523299231,11cyclictest0-21swapper/321:30:233
25222992314,2cyclictest29655-21ntpq00:10:231
2522299231,17cyclictest2137-21kworker/u8:221:47:071
2522299231,15cyclictest0-21swapper/121:44:301
2522299231,14cyclictest0-21swapper/100:18:511
2522299231,10cyclictest0-21swapper/122:01:451
2522299230,9cyclictest0-21swapper/123:02:091
2522299230,9cyclictest0-21swapper/121:53:581
2522299230,6cyclictest32709-21kworker/u8:000:20:371
2521599231,6cyclictest0-21swapper/021:15:280
2521599231,5cyclictest0-21swapper/000:37:490
2521599231,11cyclictest18941-21awk22:35:000
5852221,17sleep0598-21diskstats19:25:150
2523299223,3cyclictest0-21swapper/321:25:083
2523299223,3cyclictest0-21swapper/321:25:083
2523299221,13cyclictest0-21swapper/319:57:493
2523299221,11cyclictest0-21swapper/300:10:143
2523299221,10cyclictest25751-21/usr/sbin/munin22:50:233
2523299221,10cyclictest0-21swapper/322:58:083
2523299221,10cyclictest0-21swapper/322:10:033
2523299221,10cyclictest0-21swapper/320:52:473
25232992210,4cyclictest0-21swapper/323:10:223
2522299226,3cyclictest0-21swapper/120:08:221
2522299222,12cyclictest87150irq/128-enp1s0-00:01:321
2522299221,16cyclictest0-21swapper/122:48:431
2522299221,14cyclictest801-21snmpd21:13:131
2522299221,13cyclictest12806-21kworker/u8:023:39:001
2521599225,14cyclictest11738-21awk21:05:000
2521599224,16cyclictest10753-21munin-run23:30:010
2521599221,8cyclictest22536-21kworker/0:121:36:400
2521599221,5cyclictest0-21swapper/021:11:100
2521599221,4cyclictest12806-21kworker/u8:023:36:090
2523299216,8cyclictest0-21swapper/300:27:203
2523299211,11cyclictest87050irq/127-enp1s0-19:52:493
2522299211,7cyclictest0-21swapper/121:29:311
2522299211,7cyclictest0-21swapper/121:29:301
2522299211,3cyclictest0-21swapper/122:28:291
2522299211,15cyclictest0-21swapper/119:35:081
2522299210,15cyclictest0-21swapper/100:39:561
2521599211,5cyclictest0-21swapper/023:52:430
2521599211,4cyclictest0-21swapper/023:23:140
2521599211,4cyclictest0-21swapper/000:02:220
25215992111,3cyclictest87150irq/128-enp1s0-22:50:180
2523299209,3cyclictest0-21swapper/323:29:223
2523299202,12cyclictest5959-21rt-features19:35:243
2521599202,5cyclictest0-21swapper/000:33:350
2521599202,11cyclictest28595-21/usr/sbin/munin00:10:230
2521599201,3cyclictest0-21swapper/023:08:520
2521599201,11cyclictest30146-21/usr/sbin/munin23:00:250
2523299198,3cyclictest0-21swapper/323:47:573
2523299198,3cyclictest0-21swapper/323:47:573
2523299198,3cyclictest0-21swapper/320:05:073
2523299197,3cyclictest0-21swapper/320:31:403
2523299191,7cyclictest10719-21diskstats22:15:183
2523299191,5cyclictest24854-21df20:20:133
2521599194,3cyclictest0-21swapper/000:15:180
2521599191,6cyclictest0-21swapper/021:58:390
2521599191,5cyclictest4519-21munin-run19:35:010
25215991913,4cyclictest9278-21ls22:10:230
25215991913,3cyclictest6723-21awk19:40:020
2521599191,10cyclictest11330-21/usr/sbin/munin19:50:230
25215991910,3cyclictest87150irq/128-enp1s0-21:30:170
41052181,3sleep00-21swapper/022:00:140
2523299182,8cyclictest0-21swapper/319:49:293
2522699182,8cyclictest32705-21/usr/sbin/munin19:25:272
2521599189,3cyclictest87150irq/128-enp1s0-21:20:550
2521599183,5cyclictest87150irq/128-enp1s0-22:23:320
2521599181,8cyclictest0-21swapper/023:45:310
2521599181,8cyclictest0-21swapper/023:45:310
2521599181,7cyclictest0-21swapper/022:15:270
2521599181,6cyclictest663-21grep00:20:130
88992171,1sleep00-21swapper/019:45:020
2523299171,2cyclictest87150irq/128-enp1s0-19:20:203
2521599178,3cyclictest0-21swapper/000:05:230
2521599178,2cyclictest986-1kworker/u9:122:06:120
2521599175,4cyclictest2137-21kworker/u8:221:45:260
2521599171,7cyclictest7552-21/usr/sbin/munin20:55:210
2521599171,7cyclictest13929-21diskmemload23:33:260
2521599171,5cyclictest2137-21kworker/u8:221:44:430
2521599171,14cyclictest27404-21awk19:15:000
25215991711,3cyclictest13529-21/usr/sbin/munin19:55:270
2521599166,2cyclictest12050irq/123-ahci[0020:15:010
2521599163,8cyclictest0-21swapper/022:35:040
2521599162,5cyclictest21698-21taskset22:40:130
2521599161,8cyclictest801-21snmpd20:30:020
2521599161,6cyclictest22323-21/usr/sbin/munin20:15:260
2521599161,5cyclictest0-21swapper/019:15:190
25215991611,3cyclictest3280-21cpuspeed_turbos20:45:130
25215991611,3cyclictest30508-21/usr/sbin/munin19:20:200
25215991610,3cyclictest32187-21ntpq20:35:240
25215991610,3cyclictest26721-21/usr/sbin/munin20:25:170
2521599153,5cyclictest9136-21/usr/sbin/munin19:45:150
2521599152,5cyclictest801-21snmpd21:05:170
2521599151,5cyclictest0-21swapper/020:50:050
25215991510,3cyclictest18295-21fschecks_count20:05:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional