You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-24 - 07:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Wed Jun 24, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
77082232163,56sleep20-21swapper/219:08:562
77042231156,24sleep30-21swapper/319:08:533
75562204166,26sleep00-21swapper/019:06:580
74582201166,23sleep10-21swapper/119:05:411
624921790,4sleep2788499cyclictest20:15:212
58021710,4sleep3788999cyclictest23:45:153
58021710,4sleep3788999cyclictest23:45:153
2061621360,4sleep0787299cyclictest22:03:070
7884991101,107cyclictest0-21swapper/221:38:082
7884996764,2cyclictest0-21swapper/220:46:382
325352580,4sleep121-21rcuc/123:45:011
287732510,1sleep30-21swapper/319:55:113
95322490,2sleep30-21swapper/321:37:023
7872994010,14cyclictest27989-21munin-run23:35:000
787299378,14cyclictest2233-21idleruntime-cro23:50:010
787299378,14cyclictest2233-21idleruntime-cro23:50:010
7872993610,13cyclictest19196-40dnf23:15:010
787299357,13cyclictest12593-21munin-run23:00:020
115832340,5sleep3788999cyclictest00:10:123
788999321,15cyclictest0-21swapper/321:10:223
788999320,15cyclictest0-21swapper/322:16:103
788999311,13cyclictest0-21swapper/323:15:293
788999310,16cyclictest0-21swapper/322:55:093
788999301,15cyclictest0-21swapper/321:09:103
788999301,12cyclictest0-21swapper/323:58:053
7889992914,9cyclictest0-21swapper/300:31:483
788999291,14cyclictest0-21swapper/320:27:533
788999291,13cyclictest0-21swapper/320:56:463
788999291,13cyclictest0-21swapper/320:40:293
788999291,13cyclictest0-21swapper/319:25:343
7872992913,2cyclictest2944-21kworker/0:121:25:230
788999281,14cyclictest0-21swapper/320:01:333
788999281,13cyclictest0-21swapper/321:47:103
788999281,13cyclictest0-21swapper/321:04:263
788999281,13cyclictest0-21swapper/320:19:533
788999281,10cyclictest0-21swapper/320:12:493
788999281,10cyclictest0-21swapper/320:12:493
787299281,12cyclictest26127-21munin-run19:50:010
787299281,11cyclictest12878-21kworker/0:000:20:110
788999271,15cyclictest0-21swapper/323:32:313
788999271,13cyclictest0-21swapper/323:13:333
788999271,13cyclictest0-21swapper/322:43:413
788999271,13cyclictest0-21swapper/322:38:253
788999271,13cyclictest0-21swapper/322:04:303
788999271,12cyclictest15360-21df23:05:133
788999271,12cyclictest0-21swapper/320:33:413
788999271,12cyclictest0-21swapper/319:23:223
788999271,12cyclictest0-21swapper/300:21:183
788999271,12cyclictest0-21swapper/300:06:243
788999270,9cyclictest0-21swapper/320:45:173
788999270,11cyclictest0-21swapper/321:40:063
787899270,6cyclictest0-21swapper/122:41:021
787299271,11cyclictest26680-21kworker/0:321:14:070
788999261,9cyclictest28558-21diskstats23:35:133
7889992611,9cyclictest0-21swapper/323:20:073
788999261,13cyclictest0-21swapper/322:20:423
7889992611,3cyclictest0-21swapper/321:56:063
788999261,13cyclictest0-21swapper/321:53:063
788999261,13cyclictest0-21swapper/319:36:263
788999261,12cyclictest0-21swapper/323:03:093
788999261,12cyclictest0-21swapper/319:49:013
788999261,12cyclictest0-21swapper/300:00:063
788999261,11cyclictest0-21swapper/322:47:093
788999261,10cyclictest0-21swapper/322:28:133
788999260,13cyclictest0-21swapper/319:42:583
788999260,12cyclictest0-21swapper/322:51:093
788999260,12cyclictest0-21swapper/320:36:493
7878992613,3cyclictest22006-21systemd-journal20:50:491
787899261,17cyclictest12334-21ntp_states00:10:201
787899261,10cyclictest0-21swapper/122:50:101
787299262,8cyclictest19047-21kworker/0:323:24:430
787299261,9cyclictest26680-21kworker/0:321:20:070
787299261,9cyclictest26680-21kworker/0:321:20:070
787299261,10cyclictest21962-21kworker/0:200:38:070
79202250,2sleep10-21swapper/100:00:221
788999251,12cyclictest0-21swapper/321:22:063
788999251,12cyclictest0-21swapper/321:22:063
788999251,12cyclictest0-21swapper/300:19:463
788999251,11cyclictest0-21swapper/319:51:373
788999250,12cyclictest0-21swapper/321:18:303
788999250,12cyclictest0-21swapper/300:37:303
787899252,16cyclictest28951-21diskmemload23:18:101
787299259,3cyclictest4606-21kworker/0:320:13:480
787299259,3cyclictest4606-21kworker/0:320:13:480
787299259,13cyclictest4423-21munin-run23:55:010
787299257,2cyclictest24867-21kworker/0:322:40:120
7872992512,2cyclictest27653-21kworker/0:023:35:190
787299251,12cyclictest11978-21munin-run21:45:020
154862251,20sleep115542-21cut21:50:361
788999248,3cyclictest0-21swapper/322:33:213
788999248,3cyclictest0-21swapper/322:33:213
788999241,9cyclictest0-21swapper/320:51:213
788999241,12cyclictest0-21swapper/321:32:223
788999241,11cyclictest0-21swapper/322:11:463
787899242,3cyclictest14698-21memory20:35:201
7878992414,5cyclictest4647-40dnf20:13:431
7878992414,5cyclictest4647-40dnf20:13:431
787299249,5cyclictest0-21swapper/023:15:030
787299246,4cyclictest2413-21kworker/0:220:17:560
787299242,10cyclictest30197-21kworker/0:219:59:400
787299241,10cyclictest19047-21kworker/0:323:27:190
788999231,11cyclictest0-21swapper/319:18:573
788999231,10cyclictest0-21swapper/323:44:453
7878992317,3cyclictest7193-21users21:30:281
787899231,16cyclictest0-21swapper/120:15:211
787299237,3cyclictest9031-21kworker/0:020:26:520
787299237,2cyclictest2944-21kworker/0:121:55:200
787299236,4cyclictest24867-21kworker/0:322:11:200
787299231,14cyclictest23887-21kworker/0:120:05:360
787299230,5cyclictest0-21swapper/021:35:090
160952231,2sleep216156-21ls23:05:222
788999229,3cyclictest0-21swapper/300:27:103
788999228,6cyclictest25645-21sshd23:28:213
788999220,10cyclictest0-21swapper/320:06:013
788499221,11cyclictest20082-21gltestperf23:15:152
787899221,2cyclictest2565-21munin-run20:10:011
787899221,18cyclictest9112-21awk19:10:201
787899221,15cyclictest0-21swapper/123:55:201
787899221,15cyclictest0-21swapper/119:35:261
787899221,15cyclictest0-21swapper/100:31:351
787899221,10cyclictest0-21swapper/122:29:411
787899220,3cyclictest4117-21sshd23:51:191
787899220,16cyclictest0-21swapper/122:07:481
787299229,2cyclictest23887-21kworker/0:120:04:200
787299226,3cyclictest15035-21kworker/0:019:24:510
787299226,3cyclictest14574-21kworker/0:020:37:010
787299222,12cyclictest6566-21kworker/0:200:05:270
787299221,9cyclictest29880-21kworker/0:223:41:550
787299221,7cyclictest5716-21kworker/0:122:46:080
7872992210,4cyclictest204662idleruntime19:35:180
4812220,4sleep2788499cyclictest22:30:212
4812220,4sleep2788499cyclictest22:30:212
187762220,2sleep10-21swapper/100:25:191
788999215,3cyclictest0-21swapper/320:23:073
788999211,14cyclictest0-21swapper/321:27:103
788499211,8cyclictest23818-21/usr/sbin/munin23:25:142
788499211,7cyclictest0-21swapper/222:56:512
7884992115,3cyclictest14399-21munin-node21:50:112
787899219,4cyclictest26997-21cpu21:05:121
787899215,3cyclictest23-21ksoftirqd/119:15:071
787899213,2cyclictest2194-21kworker/u8:000:38:241
787899212,5cyclictest4084-21hddtemp_smartct21:25:161
7878992117,3cyclictest0-21swapper/123:13:441
7878992116,3cyclictest12723-21tr20:30:241
787899211,3cyclictest9878-21irqstats00:05:211
787899211,18cyclictest0-21swapper/123:06:101
787899211,15cyclictest0-21swapper/123:32:591
787899211,14cyclictest0-21swapper/122:58:391
787899211,14cyclictest0-21swapper/120:45:451
7878992110,3cyclictest20295-21unixbench_multi22:00:251
787899210,15cyclictest0-21swapper/123:36:191
787299219,3cyclictest17258-21kworker/0:219:40:320
787299216,5cyclictest23887-21kworker/0:119:50:160
787299215,4cyclictest15234-21kworker/0:100:30:080
787299211,9cyclictest3464-21kworker/0:022:36:440
7872992116,3cyclictest21933-21missed_timers22:05:190
787299211,3cyclictest27104-21kworker/0:022:20:040
787299211,12cyclictest8077-21/usr/sbin/munin19:10:130
787299211,11cyclictest22574-21/usr/sbin/munin20:55:220
48332211,16sleep24935-21memory22:40:182
266752210,5sleep121-21rcuc/119:50:121
788999200,13cyclictest0-21swapper/319:11:533
788499203,15cyclictest21460-21runrttasks19:36:402
788499201,6cyclictest0-21swapper/222:54:562
788499201,5cyclictest0-21swapper/200:34:442
788499201,10cyclictest10207-21/usr/sbin/munin20:25:202
787899208,4cyclictest0-21swapper/123:22:081
787899201,9cyclictest0-21swapper/123:29:241
787899201,3cyclictest0-21swapper/122:10:541
787899201,2cyclictest0-21swapper/119:57:061
787899201,16cyclictest0-21swapper/122:31:021
787899201,16cyclictest0-21swapper/122:31:021
787899201,14cyclictest0-21swapper/120:04:341
787899201,11cyclictest27375-21awk22:20:001
787899201,11cyclictest26252-21cat19:50:011
787899201,11cyclictest18156-21grep19:30:151
787899201,11cyclictest10461-21ntp_states20:25:251
787299207,2cyclictest26680-21kworker/0:321:19:230
787299206,3cyclictest5315-21hddtemp_smartct23:55:170
787299205,3cyclictest17258-21kworker/0:219:33:560
787299202,3cyclictest15683-21kworker/0:221:00:230
787299202,2cyclictest2413-21kworker/0:220:20:160
787299202,11cyclictest9769-21kworker/0:219:16:430
787299202,10cyclictest5716-21kworker/0:123:00:240
787299201,9cyclictest12878-21kworker/0:000:12:310
787299201,8cyclictest2944-21kworker/0:121:46:160
7872992015,3cyclictest15441-21awk00:20:010
788999191,13cyclictest19151-21users19:30:263
788999190,10cyclictest0-21swapper/322:05:103
788499192,14cyclictest22271-21cut23:20:162
788499191,8cyclictest0-21swapper/221:30:222
788499191,7cyclictest0-21swapper/222:26:222
788499191,6cyclictest0-21swapper/223:37:082
788499191,5cyclictest18245-1kworker/u9:220:36:342
788499191,14cyclictest2357-21fschecks_time22:35:142
788499191,14cyclictest13051-21grep20:30:262
7884991911,3cyclictest0-21swapper/200:22:162
788499190,6cyclictest22353-21tr00:35:122
787899199,2cyclictest13454-21mailstats21:45:231
7878991915,3cyclictest0-21swapper/122:47:531
7878991915,3cyclictest0-21swapper/121:44:141
7878991915,3cyclictest0-21swapper/121:36:481
7878991915,3cyclictest0-21swapper/121:01:301
7878991915,3cyclictest0-21swapper/119:20:141
787899191,17cyclictest0-21swapper/121:56:341
787899191,17cyclictest0-21swapper/120:58:001
787899191,16cyclictest0-21swapper/122:21:521
787899191,14cyclictest0-21swapper/119:28:361
787899191,14cyclictest0-21swapper/100:25:001
7878991911,3cyclictest0-21swapper/120:41:291
787899191,10cyclictest31679-21cat21:15:141
787899191,10cyclictest0-21swapper/121:20:261
787899191,10cyclictest0-21swapper/121:20:261
787899191,10cyclictest0-21swapper/119:40:231
787299196,2cyclictest15683-21kworker/0:220:51:190
787299194,7cyclictest6566-21kworker/0:200:01:110
787299193,4cyclictest15683-21kworker/0:221:07:070
787299193,2cyclictest14574-21kworker/0:020:40:350
787299192,8cyclictest7957-21kworker/0:022:50:200
787299191,11cyclictest0-21swapper/022:30:220
787299191,11cyclictest0-21swapper/022:30:220
787299191,10cyclictest5603-21/usr/sbin/munin21:30:210
787299190,6cyclictest0-21swapper/021:51:040
788999181,14cyclictest0-21swapper/323:52:013
788499181,9cyclictest0-21swapper/222:05:402
788499181,7cyclictest9415-21diskstats00:05:192
788499181,7cyclictest3173-21fschecks_count20:10:152
788499181,7cyclictest3173-21fschecks_count20:10:152
788499181,7cyclictest24369-21grep22:10:212
788499181,7cyclictest19304-21sed23:15:012
788499181,7cyclictest0-21swapper/223:48:202
788499181,7cyclictest0-21swapper/223:48:202
788499181,7cyclictest0-21swapper/223:42:522
788499181,7cyclictest0-21swapper/222:00:212
788499181,5cyclictest7569-21if_enp1s000:00:182
788499181,5cyclictest0-21swapper/221:40:252
7884991814,3cyclictest0-21swapper/219:11:232
788499181,3cyclictest13852-21ntpq23:00:192
7884991812,3cyclictest6558-21sh22:45:132
788499181,15cyclictest0-21swapper/219:25:442
788499181,11cyclictest0-21swapper/219:57:512
788499180,15cyclictest30895-21grep20:00:132
787899187,3cyclictest9202-21cat20:25:021
787899182,13cyclictest29513-21df21:10:151
787899181,9cyclictest13487-21grep00:15:111
787899181,8cyclictest14407-21users23:00:261
787899181,3cyclictest0-21swapper/122:39:571
787899181,2cyclictest28951-21diskmemload23:45:481
787899181,2cyclictest28951-21diskmemload23:45:481
787299182,4cyclictest18737-21latency00:25:180
787299181,7cyclictest14574-21kworker/0:020:45:150
787299181,6cyclictest0-21swapper/023:07:580
7872991813,3cyclictest26049-21head22:15:170
24232181,3sleep20-21swapper/220:08:242
138162180,2sleep20-21swapper/219:20:162
788499177,3cyclictest87350irq/126-enp1s0-23:30:202
788499176,3cyclictest30244-21ntp_states21:10:232
788499176,3cyclictest14226-21if_enp2s000:15:162
788499174,4cyclictest17975-21cpuspeed_turbos00:25:122
788499173,7cyclictest0-21swapper/221:27:592
788499173,6cyclictest30-21rcuc/223:55:202
788499171,7cyclictest22354-21awk20:55:012
788499171,7cyclictest0-21swapper/222:23:392
788499171,6cyclictest0-21swapper/221:46:072
788499171,4cyclictest0-21swapper/221:22:492
788499171,4cyclictest0-21swapper/221:22:492
7884991713,3cyclictest0-21swapper/200:11:372
7884991712,3cyclictest12186-21cut19:15:242
788499171,14cyclictest0-21swapper/219:40:042
7884991711,3cyclictest3151-21sort23:50:172
7884991711,3cyclictest17362-21mii-tool21:55:202
7884991710,4cyclictest24540-21dump-pmu-power21:00:012
788499170,15cyclictest0-21swapper/221:15:112
787299173,3cyclictest4606-21kworker/0:320:30:060
787299172,9cyclictest24867-21kworker/0:322:28:040
788499161,4cyclictest0-21swapper/222:15:242
7884991612,3cyclictest0-21swapper/221:05:082
7884991612,3cyclictest0-21swapper/221:02:012
788499161,14cyclictest0-21swapper/220:21:202
788499161,14cyclictest0-21swapper/219:52:562
7884991611,3cyclictest16158-21cat20:40:132
7884991611,2cyclictest17561-21/usr/sbin/munin19:30:172
787299161,13cyclictest16436-21ntp_states19:25:220
788499154,3cyclictest87650irq/129-enp1s0-19:45:112
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional