You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-12 - 08:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Fri Dec 12, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
34352238161,24sleep00-21swapper/019:05:370
37232234162,24sleep30-21swapper/319:09:183
37612204166,25sleep10-21swapper/119:09:501
35802203166,24sleep20-21swapper/219:07:282
1300321430,5sleep2387299cyclictest23:09:462
38729910796,10cyclictest0-21swapper/219:42:482
3872996951,8cyclictest0-21swapper/219:24:512
3872995148,2cyclictest0-21swapper/222:22:262
251552490,1sleep011-21rcuc/021:10:020
83002450,1sleep30-21swapper/321:45:143
385999389,13cyclictest16798-21munin-run22:05:010
385999378,13cyclictest13324-21munin-run19:30:020
386799362,6cyclictest0-21swapper/123:05:251
386799331,6cyclictest0-21swapper/119:48:291
385999337,12cyclictest13966-21awk20:45:020
386799321,15cyclictest16860-21timerwakeupswit23:15:251
386799320,6cyclictest0-21swapper/120:55:171
385999329,10cyclictest674-21munin-run20:15:010
387299311,5cyclictest19415-21expr22:10:132
386799311,5cyclictest0-21swapper/121:59:091
386799311,23cyclictest0-21swapper/122:17:331
386799311,13cyclictest8855-21/usr/sbin/munin23:00:271
386799311,12cyclictest0-21swapper/100:10:211
386799301,9cyclictest0-21swapper/123:32:531
386799291,9cyclictest0-21swapper/122:45:411
386799291,9cyclictest0-21swapper/122:45:411
386799291,6cyclictest0-21swapper/122:20:211
386799291,18cyclictest3388-21grep00:00:171
386799291,12cyclictest0-21swapper/123:50:371
386799291,10cyclictest24956-21diskmemload23:21:091
386799291,10cyclictest20192-21df23:25:151
386799281,23cyclictest26152-21df22:25:141
386799281,14cyclictest4266-21ntp_states20:20:231
386799281,13cyclictest0-21swapper/122:42:291
272292280,4sleep230-21rcuc/221:15:012
387899271,15cyclictest0-21swapper/323:45:143
386799272,12cyclictest28065-21hddtemp_smartct21:15:181
3867992716,3cyclictest19241-21/usr/sbin/munin22:10:121
386799271,11cyclictest0-21swapper/100:06:531
387899268,7cyclictest24963-21munin-run21:10:023
386799261,9cyclictest0-21swapper/121:21:391
386799261,17cyclictest7745-21sshd21:42:011
386799261,17cyclictest7745-21sshd21:42:011
386799261,16cyclictest5306-21unixbench-2d21:35:251
386799260,18cyclictest0-21swapper/121:48:531
386799260,11cyclictest0-21swapper/100:27:171
386799260,10cyclictest0-21swapper/100:16:531
387899251,16cyclictest14634-21snmpd20:50:263
386799251,9cyclictest0-21swapper/123:35:491
386799251,16cyclictest0-21swapper/120:27:011
386799251,11cyclictest0-21swapper/122:04:371
386799251,10cyclictest0-21swapper/119:25:171
386799250,12cyclictest0-21swapper/119:20:211
386799250,11cyclictest0-21swapper/122:51:251
386799250,10cyclictest0-21swapper/123:56:371
386799250,10cyclictest0-21swapper/123:56:371
3859992519,3cyclictest5653-21expr20:25:170
385999251,13cyclictest361-21awk23:55:020
3878992419,2cyclictest17440-21idleruntime-cro23:20:013
387899240,5cyclictest12323-21kworker/u8:022:11:273
387299248,6cyclictest0-21swapper/222:50:012
387299248,6cyclictest0-21swapper/222:50:012
387299241,9cyclictest0-21swapper/223:53:082
386799241,16cyclictest24956-21diskmemload00:38:011
386799241,16cyclictest0-21swapper/119:13:291
386799240,12cyclictest0-21swapper/122:39:331
387899237,2cyclictest10271-21if_enp2s000:15:213
387899231,6cyclictest0-21swapper/321:55:203
387899231,18cyclictest86550irq/126-enp1s0-21:16:063
387899230,5cyclictest0-21swapper/300:09:023
387299236,8cyclictest0-21swapper/223:25:232
387299235,6cyclictest32-21ksoftirqd/222:30:312
386799231,8cyclictest0-21swapper/122:09:091
386799231,14cyclictest0-21swapper/120:09:251
386799231,11cyclictest0-21swapper/120:36:451
386799231,11cyclictest0-21swapper/120:36:451
386799231,10cyclictest0-21swapper/100:32:131
385999231,5cyclictest11775-21kworker/u8:221:10:590
385999231,11cyclictest4894-21perf00:05:020
387899223,9cyclictest24956-21diskmemload23:22:433
387899222,11cyclictest779-21ntpd22:50:173
387899221,11cyclictest10129-21swap23:00:283
387299229,3cyclictest2031-21awk22:45:012
387299226,13cyclictest876-21systemd-logind19:40:012
387299222,2cyclictest88050irq/133-enp2s0-19:15:212
387299221,8cyclictest29805-21latency20:05:192
387299221,10cyclictest0-21swapper/220:50:232
387299221,10cyclictest0-21swapper/220:40:172
386799225,3cyclictest0-21swapper/120:18:291
386799221,9cyclictest0-21swapper/123:40:451
3867992217,2cyclictest31430-21cstates20:10:131
3867992215,2cyclictest0-21swapper/121:10:471
386799221,16cyclictest0-21swapper/122:33:391
386799221,14cyclictest0-21swapper/121:52:291
386799221,10cyclictest0-21swapper/122:55:071
386799220,12cyclictest0-21swapper/120:54:291
3859992215,4cyclictest15889-21munin-run00:30:020
147982220,4sleep232-21ksoftirqd/222:00:012
387899214,9cyclictest15167-21sendmail_mailst00:25:283
387899214,4cyclictest9406-21sshd20:33:113
387299215,4cyclictest25392-21latency19:55:192
387299212,9cyclictest11366-21kworker/2:023:55:292
387299212,9cyclictest11366-21kworker/2:023:55:292
3872992116,3cyclictest32-21ksoftirqd/223:20:012
387299211,17cyclictest20270-21tr19:45:112
386799212,10cyclictest14080-21irqstats23:10:201
3867992117,3cyclictest0-21swapper/119:58:291
3867992114,2cyclictest0-21swapper/123:45:381
386799211,3cyclictest0-21swapper/121:25:181
386799211,15cyclictest0-21swapper/100:21:011
386799211,12cyclictest0-21swapper/121:00:211
386799211,11cyclictest28727-21munin-run20:05:011
386799211,11cyclictest0-21swapper/120:49:411
386799211,10cyclictest581-21systemd-journal20:35:001
386799210,13cyclictest0-21swapper/119:16:531
385999213,15cyclictest22984-21sed19:50:180
385999211,7cyclictest21112-21seq23:25:270
385999211,12cyclictest0-21swapper/021:39:160
385999210,11cyclictest6016-21sed22:50:310
387899209,3cyclictest820-21irqstats22:40:193
387899202,10cyclictest9160-21/usr/sbin/munin19:20:273
387899201,7cyclictest0-21swapper/319:55:183
387899201,5cyclictest496-21jbd2/dm-0-821:24:183
3878992015,3cyclictest3200-21grep21:30:303
3878992014,3cyclictest15340-21pmu-power20:45:263
387899201,13cyclictest0-21swapper/322:37:563
387899201,12cyclictest15894-21ntpq22:00:253
387899201,12cyclictest0-21swapper/321:50:413
387899201,10cyclictest14634-21snmpd19:15:093
387899201,10cyclictest0-21swapper/300:35:163
387299207,3cyclictest9722-21latency23:00:222
387299206,4cyclictest0-21swapper/221:15:202
387299201,7cyclictest0-21swapper/220:55:252
3872992015,3cyclictest32-21ksoftirqd/200:10:272
387299201,10cyclictest0-21swapper/222:55:052
386799201,9cyclictest0-21swapper/119:32:371
3867992016,3cyclictest0-21swapper/121:33:211
3867992014,3cyclictest0-21swapper/119:40:261
386799201,11cyclictest0-21swapper/119:35:231
386799201,10cyclictest23588-21grep21:05:211
385999208,3cyclictest0-21swapper/021:02:110
385999202,5cyclictest18993-21munin-run22:10:010
385999202,10cyclictest24541-21/usr/sbin/munin19:55:280
385999201,4cyclictest0-21swapper/021:58:380
3859992014,3cyclictest24768-21pmu-power22:20:230
385999201,11cyclictest31178-21users21:20:310
385999201,10cyclictest15930-21cat19:35:130
3859992010,3cyclictest86750irq/128-enp1s0-23:55:130
3859992010,3cyclictest86750irq/128-enp1s0-23:55:130
387899198,4cyclictest0-21swapper/320:05:173
387899198,3cyclictest0-21swapper/323:53:083
387899197,5cyclictest0-21swapper/300:34:093
387899197,3cyclictest0-21swapper/322:34:093
387899192,9cyclictest3538-21diskstats20:20:153
387899192,9cyclictest0-21swapper/320:18:293
387899192,3cyclictest17380-21dnf22:05:193
387899191,7cyclictest0-21swapper/321:26:233
3878991912,4cyclictest27375-21cut20:00:153
387899191,15cyclictest25307-21tr21:10:143
387899191,10cyclictest23959-21df_inode22:20:153
387899191,10cyclictest1950-21smart_sda23:55:273
387899191,10cyclictest1950-21smart_sda23:55:273
387899191,10cyclictest11288-21expr23:05:133
387899190,8cyclictest0-21swapper/321:41:013
387899190,8cyclictest0-21swapper/321:41:013
387299199,2cyclictest12050irq/123-ahci[0000:10:012
387299194,8cyclictest32-21ksoftirqd/222:15:262
387299194,2cyclictest88250irq/135-enp2s0-20:45:212
387299193,8cyclictest12050irq/123-ahci[0021:35:002
387299193,4cyclictest9806-21users21:45:272
3872991913,3cyclictest5764-21grep20:25:182
387299191,11cyclictest0-21swapper/221:27:222
386799191,16cyclictest0-21swapper/120:40:221
385999192,13cyclictest16447-21needreboot23:15:200
385999191,9cyclictest22412-21needreboot22:15:220
385999191,8cyclictest24956-21diskmemload23:09:100
385999191,5cyclictest0-21swapper/022:30:370
3859991914,3cyclictest4692-21grep20:20:280
3859991913,3cyclictest9659-21latency_hist20:35:020
385999191,10cyclictest1953-21sshd22:44:180
385999191,10cyclictest14187-21/usr/sbin/munin20:45:190
387899184,2cyclictest86550irq/126-enp1s0-22:15:223
3878991812,3cyclictest26705-21expr23:40:123
3878991812,3cyclictest2247-21/usr/sbin/munin22:45:113
3878991812,3cyclictest2247-21/usr/sbin/munin22:45:113
3878991812,3cyclictest11761-21munin-plugin-st20:40:023
3878991812,3cyclictest11761-21munin-plugin-st20:40:013
387899181,11cyclictest0-21swapper/300:10:223
387899181,10cyclictest0-21swapper/321:35:173
387899181,10cyclictest0-21swapper/300:01:363
387899180,15cyclictest41-21ksoftirqd/319:40:133
387899180,10cyclictest22570-21fschecks_time23:30:163
387299185,3cyclictest30-21rcuc/222:40:012
387299185,2cyclictest32-21ksoftirqd/223:36:412
387299183,9cyclictest86550irq/126-enp1s0-23:45:122
387299183,2cyclictest12960-21df_inode21:55:182
387299182,15cyclictest0-21swapper/220:00:382
387299182,10cyclictest0-21swapper/200:01:222
387299181,8cyclictest0-21swapper/220:38:592
387299181,8cyclictest0-21swapper/220:38:592
387299181,7cyclictest0-21swapper/223:40:152
387299181,7cyclictest0-21swapper/222:29:202
387299181,6cyclictest0-21swapper/223:10:232
387299181,6cyclictest0-21swapper/200:36:482
387299181,5cyclictest17029-21/usr/sbin/munin22:05:172
3872991814,3cyclictest0-21swapper/220:10:592
3872991812,3cyclictest31183-21vmstat21:20:312
386799181,11cyclictest0-21swapper/119:50:051
385999188,3cyclictest13266-21/usr/sbin/munin23:10:230
385999183,6cyclictest1517750kworker/u8:200:38:140
385999182,12cyclictest1500-21fschecks_time20:15:160
385999181,9cyclictest0-21swapper/019:33:120
385999181,8cyclictest2247-21/usr/sbin/munin22:45:230
385999181,8cyclictest2247-21/usr/sbin/munin22:45:230
385999181,6cyclictest0-21swapper/021:42:220
385999181,6cyclictest0-21swapper/021:42:220
3859991813,3cyclictest8218-21/usr/sbin/munin21:45:160
3859991813,3cyclictest13262-21grep00:20:290
3859991812,3cyclictest8873-21unixbench_singl00:10:310
3859991811,4cyclictest4596-21grep19:10:160
385999181,10cyclictest19224-21/usr/sbin/munin22:10:240
385999180,15cyclictest81ktimersoftd/000:30:190
274322181,2sleep00-21swapper/022:25:380
194302180,3sleep00-21swapper/023:22:390
387899175,3cyclictest11837-21cpuspeed_turbos00:20:153
387899172,12cyclictest25817-21/usr/sbin/munin22:25:213
387899171,8cyclictest5715-21expr20:25:163
387899171,3cyclictest15749-21/usr/sbin/munin19:35:113
387899171,3cyclictest11370-21cpu19:25:123
387899171,3cyclictest0-21swapper/320:40:123
3878991713,3cyclictest0-21swapper/323:10:163
3878991712,3cyclictest4142-21cat19:10:143
3878991712,3cyclictest31154-21/usr/sbin/munin20:10:153
3878991712,3cyclictest21356-21processes19:45:223
3878991712,3cyclictest15717-21cut19:35:013
387899171,14cyclictest0-21swapper/322:56:383
3878991711,3cyclictest23446-21open_inodes19:50:233
387899171,13cyclictest20779-21/usr/sbin/munin21:00:253
387899170,7cyclictest0-21swapper/323:36:413
387299178,3cyclictest86550irq/126-enp1s0-21:43:252
387299178,3cyclictest86550irq/126-enp1s0-21:43:252
387299173,7cyclictest0-21swapper/200:25:242
387299172,6cyclictest0-21swapper/200:34:182
387299171,9cyclictest0-21swapper/223:23:372
387299171,6cyclictest5158-21latency_hist20:25:012
387299171,6cyclictest0-21swapper/221:35:332
387299171,6cyclictest0-21swapper/221:00:262
387299171,5cyclictest4068-21/usr/sbin/munin19:10:172
387299171,5cyclictest0-21swapper/223:30:422
3872991713,3cyclictest0-21swapper/221:05:072
3872991713,3cyclictest0-21swapper/220:16:012
3872991712,3cyclictest8179-21grep20:30:162
387299171,15cyclictest0-21swapper/221:52:262
3872991711,3cyclictest13553-21/usr/sbin/munin19:30:122
387299171,10cyclictest0-21swapper/200:18:022
3872991710,4cyclictest24311-21grep19:55:012
385999177,2cyclictest8463-21unixbench_multi19:15:260
385999176,3cyclictest86750irq/128-enp1s0-20:05:220
385999175,3cyclictest14634-21snmpd19:45:150
385999174,3cyclictest11899-21unixbench-2d21:50:290
385999173,5cyclictest0-21swapper/023:49:420
385999171,7cyclictest32388-21cat22:40:020
385999171,6cyclictest0-21swapper/023:33:580
385999171,6cyclictest0-21swapper/021:34:570
385999171,6cyclictest0-21swapper/021:16:430
385999171,4cyclictest18356-21perf20:55:010
385999171,15cyclictest0-21swapper/000:16:370
385999171,13cyclictest25812-21grep23:35:280
3859991711,3cyclictest10847-21idleruntime-cro23:05:020
385999171,11cyclictest0-21swapper/022:55:210
385999171,10cyclictest0-21swapper/021:26:030
387899162,4cyclictest0-21swapper/320:55:123
387899161,13cyclictest0-21swapper/323:25:133
387299166,3cyclictest30-21rcuc/219:29:202
387299161,7cyclictest0-21swapper/200:20:052
3872991611,3cyclictest30-21rcuc/222:50:162
385999163,4cyclictest26451-21/usr/sbin/munin23:40:120
385999162,6cyclictest10482-21switchtime19:20:260
3859991612,2cyclictest27077-21sh20:00:130
385999161,14cyclictest0-21swapper/000:07:260
385999161,12cyclictest18577-21/usr/sbin/munin20:55:180
385999161,12cyclictest10715-21missed_timers20:35:230
385999161,12cyclictest10715-21missed_timers20:35:230
3859991510,3cyclictest18243-21cat19:40:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional