You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-07 - 15:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sun Jun 07, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
48972238163,24sleep30-21swapper/307:09:163
48272232162,57sleep00-21swapper/007:08:220
49572202165,24sleep20-21swapper/207:10:002
47192201164,24sleep10-21swapper/107:06:571
1630121610,3sleep0503799cyclictest10:00:140
569321340,4sleep3505599cyclictest10:50:123
50499910197,3cyclictest0-21swapper/212:16:332
5049999691,2cyclictest24139-21users10:15:262
5049998380,2cyclictest0-21swapper/210:05:532
5049997470,3cyclictest0-21swapper/209:33:342
5049997470,3cyclictest0-21swapper/209:33:342
5049996865,2cyclictest0-21swapper/210:32:512
63822580,4sleep123-21ksoftirqd/107:10:231
234882530,1sleep20-21swapper/211:30:122
5049994845,2cyclictest0-21swapper/211:28:302
179102480,2sleep30-21swapper/311:15:253
22242430,3sleep20-21swapper/211:50:152
22242430,3sleep20-21swapper/211:50:152
505599377,3cyclictest23488-21awk12:40:003
505599361,8cyclictest0-21swapper/310:24:493
504999363,24cyclictest7255-21grep12:00:222
504999361,4cyclictest910-21systemd-logind11:35:332
504999357,16cyclictest87550irq/128-enp1s0-12:34:172
504999351,20cyclictest9362-21/usr/sbin/munin09:45:182
503799357,13cyclictest3711-21idleruntime-cro11:55:020
503799357,13cyclictest3711-21idleruntime-cro11:55:020
504999344,4cyclictest29405-21/usr/sbin/munin09:15:182
504999343,28cyclictest18453-21grep07:35:262
504999343,25cyclictest20470-40dnf11:21:392
504999341,4cyclictest2483-21expr08:15:152
504999341,18cyclictest3379-21/usr/sbin/munin10:45:202
504999333,19cyclictest32230-21diskstats11:45:142
503799337,12cyclictest12297-21idleruntime-cro07:25:010
505599325,9cyclictest0-21swapper/308:37:373
504999327,22cyclictest16578-21/usr/sbin/munin11:15:252
504999325,3cyclictest87350irq/126-enp1s0-09:25:142
504999324,3cyclictest87550irq/128-enp1s0-12:25:192
504999323,4cyclictest12804-21sendmail_mailst09:50:232
504999323,2cyclictest87350irq/126-enp1s0-10:40:242
504999323,2cyclictest87350irq/126-enp1s0-10:40:242
504999322,5cyclictest28252-21diskstats08:00:132
504999322,21cyclictest0-21swapper/209:55:242
5049993210,19cyclictest17033-21awk10:00:202
505599311,8cyclictest0-21swapper/311:05:393
505599311,14cyclictest0-21swapper/311:21:223
505599311,13cyclictest0-21swapper/307:23:303
504999315,4cyclictest24787-21switchtime07:50:232
504999312,26cyclictest13559-21grep11:05:272
504999311,3cyclictest30837-21/usr/sbin/munin11:40:212
5049993111,17cyclictest9370-21grep08:30:142
5049993111,17cyclictest28182-21sshd09:13:362
5049993110,18cyclictest27561-21expr10:25:152
504999310,2cyclictest87550irq/128-enp1s0-10:35:142
505599301,21cyclictest0-21swapper/311:27:403
504999308,19cyclictest5667-21cpuspeed_turbos10:50:132
504999307,20cyclictest6481-21unixbench_multi09:35:262
504999306,21cyclictest26812-21pluginstate07:55:222
504999306,12cyclictest87350irq/126-enp1s0-10:10:162
504999304,3cyclictest87650irq/129-enp1s0-11:14:372
504999304,3cyclictest87350irq/126-enp1s0-10:55:212
504999302,4cyclictest15075-21diskstats07:30:132
504999302,4cyclictest11401-21/usr/sbin/munin07:20:202
504999301,5cyclictest12121-21/usr/sbin/munin12:10:262
504999301,4cyclictest7844-21/usr/sbin/munin09:40:152
504999301,4cyclictest4513-21grep11:55:162
504999301,4cyclictest32307-21/usr/sbin/munin08:10:242
504999301,4cyclictest25083-21/usr/sbin/munin10:20:132
5049993010,17cyclictest7896-21cat08:25:242
5049993010,17cyclictest21458-21grep07:45:122
503799306,11cyclictest5355-21munin-run10:50:010
5037993010,7cyclictest4345-21/usr/sbin/munin08:20:240
17762300,12sleep339-21rcuc/311:50:123
17762300,12sleep339-21rcuc/311:50:123
505599291,13cyclictest0-21swapper/308:57:503
505599291,12cyclictest0-21swapper/307:38:193
504999299,17cyclictest22435-21cut09:00:142
504999295,21cyclictest31391-21gltestperf09:20:142
504999292,3cyclictest9198-21ntp_states07:15:232
504999292,3cyclictest12409-21/usr/sbin/munin08:35:252
504999291,6cyclictest0-21swapper/212:35:192
504999291,4cyclictest6308-21ntp_states07:10:222
504999291,4cyclictest24487-21/usr/sbin/munin09:05:142
504999291,4cyclictest17705-21/usr/sbin/munin08:50:132
504999291,3cyclictest30853-21if_enp2s008:05:182
504999291,3cyclictest15966-21/usr/sbin/munin08:45:162
504999291,3cyclictest10325-21/usr/sbin/munin11:00:142
5049992912,15cyclictest14924-21/usr/sbin/munin12:20:132
504299299,11cyclictest0-21swapper/109:30:201
504299299,11cyclictest0-21swapper/109:30:201
155622290,7sleep081ktimersoftd/008:45:120
505599281,3cyclictest6332-21expr12:00:123
504999288,17cyclictest8891-21fschecks_time12:05:162
504999282,3cyclictest19788-21hddtemp_smartct07:40:162
504999281,3cyclictest4693-21diskstats08:20:152
504999281,3cyclictest13229-21/usr/sbin/munin08:40:122
504999281,3cyclictest12879-21/usr/sbin/munin07:25:132
504999281,2cyclictest0-21swapper/208:55:162
503799289,16cyclictest4135-21basename08:20:010
503799283,12cyclictest31118-21munin-run10:35:000
505599271,6cyclictest31864-21tr09:20:213
505599271,13cyclictest0-21swapper/308:41:373
505599271,13cyclictest0-21swapper/307:15:503
505599270,12cyclictest0-21swapper/308:10:213
505599270,10cyclictest0-21swapper/309:55:373
503799271,13cyclictest24537-21basename10:20:000
503799271,11cyclictest28413-21unixbench_multi11:35:250
505599261,9cyclictest12875-21diskstats07:25:133
505599261,13cyclictest22999-21ntp_states09:00:223
505599261,13cyclictest0-21swapper/308:04:013
505599261,13cyclictest0-21swapper/307:46:533
505599261,12cyclictest0-21swapper/309:49:293
505599261,12cyclictest0-21swapper/308:45:143
505599261,12cyclictest0-21swapper/307:10:213
505599261,10cyclictest0-21swapper/309:53:573
503799261,12cyclictest7552-21munin-run10:55:010
505599251,14cyclictest0-21swapper/312:22:053
505599250,13cyclictest0-21swapper/311:02:143
505599250,11cyclictest0-21swapper/310:29:253
504299252,13cyclictest5447-40dnf08:20:241
503799251,12cyclictest21293-21munin-run12:35:010
505599247,10cyclictest0-21swapper/310:03:173
505599241,11cyclictest0-21swapper/309:26:133
505599240,11cyclictest0-21swapper/310:36:013
504299247,10cyclictest0-21swapper/112:04:041
5042992417,4cyclictest26270-21systemd11:35:011
504299241,17cyclictest0-21swapper/112:10:411
504299241,17cyclictest0-21swapper/111:23:321
504299241,16cyclictest0-21swapper/111:58:041
504299241,16cyclictest0-21swapper/110:52:091
504299241,16cyclictest0-21swapper/110:14:011
5042992410,12cyclictest19824-21kworker/u8:110:31:431
505599239,3cyclictest0-21swapper/312:17:493
505599238,3cyclictest19111-21taskset07:40:073
505599238,3cyclictest0-21swapper/311:13:023
505599237,3cyclictest0-21swapper/308:33:293
505599231,17cyclictest0-21swapper/308:50:263
505599231,11cyclictest0-21swapper/312:08:373
505599231,11cyclictest0-21swapper/309:38:103
504299238,7cyclictest0-21swapper/109:45:231
504299231,4cyclictest14480-21sed08:40:241
504299231,4cyclictest13597-21ntp_states07:25:201
5042992313,3cyclictest0-21swapper/111:35:101
504299231,16cyclictest0-21swapper/109:27:151
504299231,16cyclictest0-21swapper/109:15:161
504299231,13cyclictest15005-21/usr/sbin/munin09:55:231
504299231,11cyclictest0-21swapper/109:36:311
503799235,15cyclictest28933-21idleruntime-cro10:30:000
503799230,11cyclictest21104-21perf07:45:020
505599223,10cyclictest28731-21sshd11:35:353
505599221,11cyclictest23169-21/usr/sbin/munin11:30:183
505599221,10cyclictest30931-21sendmail_mailst11:40:233
505599221,10cyclictest19824-21kworker/u8:110:15:153
505599221,10cyclictest0-21swapper/312:27:513
504299228,4cyclictest0-21swapper/110:58:211
5042992216,3cyclictest26004-21chrt07:55:141
5042992214,3cyclictest0-21swapper/109:20:161
504299221,16cyclictest0-21swapper/111:16:191
504299221,16cyclictest0-21swapper/109:53:111
504299221,15cyclictest0-21swapper/110:49:271
504299221,11cyclictest0-21swapper/112:38:451
504299221,11cyclictest0-21swapper/110:23:231
504299221,10cyclictest0-21swapper/110:36:391
503799221,3cyclictest3827-21ntp_states09:30:210
503799221,3cyclictest3827-21ntp_states09:30:210
503799221,18cyclictest1477-21munin-run11:50:020
505599216,4cyclictest0-21swapper/309:44:403
505599211,9cyclictest2151-21/usr/sbin/munin08:15:193
505599211,7cyclictest21242-21taskset12:34:343
505599211,4cyclictest0-21swapper/310:44:123
505599211,4cyclictest0-21swapper/310:44:123
505599210,9cyclictest0-21swapper/309:12:493
504299216,9cyclictest0-21swapper/111:45:101
504299211,9cyclictest0-21swapper/109:14:261
504299211,3cyclictest9406-21ntp_states12:05:221
504299211,18cyclictest0-21swapper/107:50:221
504299211,15cyclictest0-21swapper/109:08:341
504299211,11cyclictest0-21swapper/110:07:451
504299211,10cyclictest0-21swapper/111:26:031
5042992110,6cyclictest0-21swapper/110:15:231
503799212,11cyclictest9362-21/usr/sbin/munin09:45:210
505599209,2cyclictest27322-21sshd07:55:363
505599204,3cyclictest0-21swapper/310:07:493
505599203,9cyclictest0-21swapper/312:14:383
505599201,11cyclictest87350irq/126-enp1s0-07:30:363
505599201,10cyclictest0-21swapper/308:20:193
504299209,3cyclictest18348-21irqstats08:50:181
504299208,3cyclictest14732-21diskstats11:10:151
504299207,4cyclictest0-21swapper/109:42:431
504299207,3cyclictest10722-21munin-run08:35:011
504299206,12cyclictest2509-21if_err_enp2s011:50:171
504299206,12cyclictest2509-21if_err_enp2s011:50:171
5042992013,4cyclictest11406-21ntpq07:20:211
504299201,18cyclictest0-21swapper/111:41:061
504299201,18cyclictest0-21swapper/108:06:261
504299201,17cyclictest0-21swapper/110:40:181
504299201,17cyclictest0-21swapper/110:40:171
504299201,17cyclictest0-21swapper/107:48:171
504299201,17cyclictest0-21swapper/107:37:031
504299201,14cyclictest0-21swapper/108:47:511
504299201,11cyclictest8581-21cat08:30:011
504299201,11cyclictest1502-21sed08:10:261
504299201,11cyclictest11624-21hddtemp_smartct08:35:151
504299201,11cyclictest10115-21tr11:00:111
5042992010,7cyclictest23-21ksoftirqd/108:55:121
504299200,18cyclictest0-21swapper/112:25:281
503799209,3cyclictest7780-21/usr/sbin/munin10:55:180
503799202,10cyclictest20973-21/usr/sbin/munin11:25:120
5037992013,4cyclictest7282-21grep09:40:120
503799201,13cyclictest0-21swapper/009:11:190
503799201,11cyclictest20355-21/usr/sbin/munin10:10:120
503799201,11cyclictest17538-21/usr/sbin/munin08:50:150
503799201,11cyclictest12187-21/usr/sbin/munin11:05:210
505599197,3cyclictest0-21swapper/310:31:133
5055991911,5cyclictest22325-21idleruntime-cro10:15:003
505599191,15cyclictest0-21swapper/310:49:173
505599191,13cyclictest8844-21ntp_states10:55:213
505599191,13cyclictest4458-21sshd09:31:373
505599191,13cyclictest4458-21sshd09:31:373
505599191,12cyclictest25200-21ntp_states09:05:223
505599191,10cyclictest0-21swapper/311:55:313
504299198,3cyclictest28116-21ps10:25:231
504299195,8cyclictest0-21swapper/112:30:061
5042991916,2cyclictest0-21swapper/107:16:591
504299191,4cyclictest29166-21smart_sda08:00:241
504299191,17cyclictest0-21swapper/112:20:221
504299191,10cyclictest3225-21ntpq08:15:231
504299191,10cyclictest19650-21fschecks_time07:40:161
504299191,10cyclictest17931-21latency_hist10:05:021
504299191,10cyclictest15306-21perl07:30:141
504299191,10cyclictest14149-21latency_hist11:10:011
504299190,14cyclictest0-21swapper/112:17:061
504299190,14cyclictest0-21swapper/109:00:101
503799195,7cyclictest25806-21sshd11:34:480
503799191,5cyclictest0-21swapper/012:10:360
503799191,3cyclictest31488-21tr10:35:120
503799191,10cyclictest3923-21/usr/sbin/munin11:55:240
503799191,10cyclictest29681-21/usr/sbin/munin11:40:140
503799191,10cyclictest25707-21/usr/sbin/munin07:55:220
503799191,10cyclictest24750-21/usr/sbin/munin10:20:190
503799191,10cyclictest21315-21/usr/sbin/munin07:45:190
503799191,10cyclictest0-21swapper/010:06:210
131862191,2sleep00-21swapper/009:50:440
505599182,11cyclictest24581-21ntp_states07:50:213
505599181,6cyclictest0-21swapper/309:15:143
505599181,14cyclictest7681-21ls08:25:213
505599181,14cyclictest0-21swapper/311:45:463
505599181,13cyclictest0-21swapper/308:06:453
503799183,12cyclictest0-21swapper/007:25:160
503799181,8cyclictest30112-21/usr/sbin/munin08:05:250
5037991811,6cyclictest20396-21kworker/u8:009:23:080
503799181,14cyclictest5440-21grep09:35:160
5037991811,4cyclictest2564-21sort09:30:020
503799181,11cyclictest0-21swapper/012:17:180
503799181,10cyclictest0-21swapper/009:15:190
503799178,4cyclictest23918-21munin-run09:05:010
503799178,3cyclictest0-21swapper/012:00:180
503799177,3cyclictest13757-21/usr/sbin/munin09:55:260
503799176,2cyclictest495-21jbd2/dm-0-808:10:230
503799172,7cyclictest8135-21/usr/sbin/munin07:15:270
503799172,13cyclictest20371-21grep08:55:150
503799172,12cyclictest9975-21/usr/sbin/munin11:00:210
503799171,7cyclictest17121-21/usr/sbin/munin12:25:160
503799171,6cyclictest0-21swapper/012:35:210
503799171,4cyclictest0-21swapper/012:05:450
5037991712,3cyclictest2395-21processes10:40:220
5037991712,3cyclictest2395-21processes10:40:220
5037991712,3cyclictest15184-21interrupts11:10:180
503799171,15cyclictest0-21swapper/008:37:040
503799170,6cyclictest0-21swapper/011:20:130
503799161,6cyclictest0-21swapper/012:21:420
5037991612,3cyclictest0-21swapper/008:41:470
503799161,13cyclictest8073-21grep08:25:240
5037991611,3cyclictest23606-21grep07:50:120
5037991611,3cyclictest18456-21sed07:35:270
503799161,12cyclictest8750-21/usr/sbin/munin08:30:120
503799161,12cyclictest27906-21/usr/sbin/munin08:00:230
5037991610,4cyclictest0-21swapper/011:15:220
5037991610,3cyclictest827-21runrttasks07:10:180
5037991511,3cyclictest0-21swapper/007:30:050
5037991510,3cyclictest24346-21cat09:05:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional