You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-21 - 12:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Thu May 21, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
173772236160,25sleep00-21swapper/019:06:000
175152229157,21sleep20-21swapper/219:07:472
176082218189,19sleep30-21swapper/319:09:003
176682216144,23sleep10-21swapper/119:09:471
1778299120100,12cyclictest0-21swapper/223:46:162
1778299109105,3cyclictest0-21swapper/200:15:102
17782999996,2cyclictest0-21swapper/222:49:232
227072590,4sleep11777799cyclictest00:10:231
1777799444,19cyclictest19440-21kworker/u8:222:55:211
17777994311,12cyclictest2058-21latency_hist21:00:011
1777799418,21cyclictest1064-21switchtime00:35:241
17777993915,11cyclictest2494-21/usr/sbin/munin19:45:171
1777799387,17cyclictest1364-21gltestperf22:10:151
1777799377,20cyclictest16229-21chrt23:56:521
1777799377,18cyclictest21814-21df_inode20:30:131
1777799369,12cyclictest14645-21latency_hist20:15:011
17777993613,11cyclictest32624-21/usr/sbin/munin23:20:211
17777993612,12cyclictest1452-21awk19:45:011
1777099367,13cyclictest16490-21munin-run00:00:010
1778299357,9cyclictest6480-21perf21:10:012
17782993512,10cyclictest15609-21/usr/sbin/munin23:55:232
1777799357,14cyclictest28893-21expr22:00:121
1777799357,12cyclictest18795-21cat00:05:021
1777799355,24cyclictest19440-21kworker/u8:221:42:451
1777799353,23cyclictest18294-21kworker/u8:023:07:031
17777993523,6cyclictest24462-40dnf23:02:261
17777993516,8cyclictest30290-21/usr/sbin/munin00:30:141
17777993516,6cyclictest808-21snmpd22:54:421
17777993511,11cyclictest18121-21cat22:50:011
17777993511,10cyclictest26258-21munin-run21:55:001
17777993510,11cyclictest31448-21grep23:20:011
1778299348,9cyclictest16013-21ls22:45:022
1778299347,16cyclictest29304-21/usr/sbin/munin23:15:162
17782993410,6cyclictest87350irq/126-enp1s0-22:25:222
17782993410,6cyclictest87350irq/126-enp1s0-22:25:222
17777993412,11cyclictest4568-21cpuspeed_turbos21:05:121
17777993410,12cyclictest1-21systemd20:00:011
17777993410,11cyclictest26749-21latency20:40:241
1777099346,13cyclictest19054-21munin-run20:25:010
1778299338,13cyclictest27741-21/usr/sbin/munin00:25:142
1778299338,11cyclictest6468-21diskmemload00:31:022
1778299337,16cyclictest6076-21/usr/sbin/munin19:55:202
1778299337,16cyclictest28348-21sendmail_mailst23:10:242
17782993314,7cyclictest27149-21hddtemp_smartct21:55:152
17782993310,12cyclictest839-21lldpd00:20:132
1777799339,13cyclictest22082-21/usr/sbin/munin21:45:211
1777799339,11cyclictest0-21swapper/100:20:181
1777799338,13cyclictest29647-21users20:45:251
1777799337,15cyclictest15027-21pmu-power22:40:221
1777799332,22cyclictest3989-21ntpq22:15:211
17777993312,17cyclictest8562-21tail23:40:151
17777993311,11cyclictest27541-21ntp_states21:55:201
17777993310,12cyclictest6704-21/usr/sbin/munin21:10:251
1778299326,15cyclictest30883-21/usr/sbin/munin22:05:162
1778299325,16cyclictest19469-21cut00:05:152
1777799329,10cyclictest808-21snmpd00:09:501
1777799328,11cyclictest15486-21/usr/sbin/munin21:30:231
1777799327,17cyclictest2672-21kworker/u8:022:24:211
1777799326,15cyclictest827-21runrttasks21:03:111
1777799323,20cyclictest31066-21kworker/u8:223:46:191
1777799323,20cyclictest28770-21kworker/u8:123:28:111
17777993213,8cyclictest9698-21interrupts21:15:181
17777993213,8cyclictest9698-21interrupts21:15:181
1777799321,24cyclictest27307-21kworker/u8:000:29:361
17777993210,11cyclictest30961-21cpuspeed_turbos22:05:121
17777993210,10cyclictest5792-21awk23:35:121
17777993210,10cyclictest5792-21awk23:35:121
1777099329,11cyclictest0-21swapper/022:55:010
1778999311,25cyclictest0-21swapper/323:38:553
1778999311,25cyclictest0-21swapper/323:38:553
1778299319,8cyclictest4447-21memory23:30:182
1778299319,13cyclictest25095-21irqstats21:50:172
1778299319,12cyclictest25260-21/usr/sbin/munin19:25:242
17782993110,10cyclictest6468-21diskmemload21:21:362
17782993110,10cyclictest30261-21users22:00:262
1777799318,12cyclictest0-21swapper/123:12:081
1777799317,13cyclictest9507-21/usr/sbin/munin22:30:111
1777099318,11cyclictest2081-21idleruntime-cro21:00:010
1777099314,13cyclictest29427-21munin-run19:35:020
1778299308,13cyclictest7517-21irqstats21:10:192
1778299308,12cyclictest87550irq/128-enp1s0-20:39:082
1778299305,9cyclictest0-21swapper/223:40:082
1777799309,12cyclictest808-21snmpd20:54:111
1777799308,19cyclictest13474-21awk23:50:221
1777799307,10cyclictest0-21swapper/121:25:411
1777799306,12cyclictest0-21swapper/121:35:171
17777993011,15cyclictest24950-21chrt00:15:251
1777099308,10cyclictest6468-21diskmemload23:45:010
1777099302,13cyclictest30654-21munin-run22:05:010
1778299297,13cyclictest21268-21ls21:40:272
1778299297,11cyclictest11434-21missed_timers20:05:212
1778299295,9cyclictest533-21unixbench_multi23:20:252
1778299294,13cyclictest0-21swapper/221:40:022
17782992912,5cyclictest20346-21ntp_states20:25:202
17782992910,7cyclictest21150-21/usr/sbin/munin00:10:102
17782992910,6cyclictest3823-21/usr/sbin/munin22:15:182
17782992910,6cyclictest20836-21df22:55:122
1777799298,9cyclictest21016-21runrttasks20:27:111
1777799297,19cyclictest28108-21sed19:30:141
1777799291,21cyclictest0-21swapper/120:35:261
1778299287,8cyclictest15307-21cat21:30:012
1778299286,10cyclictest3681-21timerandwakeup21:00:262
1778299285,14cyclictest23360-21switchtime21:45:242
1778299285,14cyclictest0-21swapper/223:02:012
1778299284,9cyclictest0-21swapper/223:10:022
17782992811,14cyclictest4541-21gltestperf19:50:142
17782992810,8cyclictest6468-21diskmemload21:30:252
1777799288,11cyclictest17969-21/usr/sbin/munin19:10:161
1777799283,12cyclictest0-21swapper/119:30:011
1777099281,13cyclictest26892-21perf23:10:020
17789992714,5cyclictest31201-21smart_sda00:30:233
1778299279,6cyclictest1938-21awk22:10:232
1778299276,11cyclictest87550irq/128-enp1s0-19:20:222
1778299274,11cyclictest18394-21cpuspeed_turbos22:50:112
17782992710,5cyclictest6468-21diskmemload00:36:542
1777799278,9cyclictest0-21swapper/120:01:121
1777799270,18cyclictest28770-21kworker/u8:123:31:551
1778999267,2cyclictest87350irq/126-enp1s0-21:50:013
1778999260,18cyclictest0-21swapper/300:36:523
1778299267,9cyclictest21027-21kworker/u8:021:15:172
1778299267,9cyclictest21027-21kworker/u8:021:15:172
1778299267,11cyclictest87650irq/129-enp1s0-20:00:202
1778299265,9cyclictest17848-21cut00:00:222
1778299265,11cyclictest0-21swapper/223:39:042
1778299265,11cyclictest0-21swapper/223:39:042
1778299263,6cyclictest0-21swapper/220:50:132
1777799266,18cyclictest0-21swapper/120:15:111
1777799262,13cyclictest0-21swapper/121:20:131
17777992618,5cyclictest4227-21diskstats19:50:121
1777099261,12cyclictest7703-21munin-run23:40:010
1777099261,12cyclictest7703-21munin-run23:40:010
1778999254,6cyclictest27106-21/usr/sbin/munin23:10:153
1778999253,7cyclictest87550irq/128-enp1s0-22:40:183
1778999251,7cyclictest9-21rcu_preempt21:39:143
1778999251,18cyclictest0-21swapper/321:31:043
1778299259,10cyclictest87650irq/129-enp1s0-23:50:232
1778299258,9cyclictest28061-21/usr/sbin/munin20:45:162
1778299257,11cyclictest0-21swapper/222:20:142
1778299256,7cyclictest13797-21cat22:40:022
17782992517,5cyclictest1683-21diskstats23:25:152
17782992513,9cyclictest9854-21diskstats22:30:132
1777799257,8cyclictest0-21swapper/122:35:201
17777992517,5cyclictest20801-21irqstats19:15:051
17777992516,6cyclictest30009-21/usr/sbin/munin19:35:141
17777992515,7cyclictest10490-21/usr/sbin/munin20:05:281
1777799251,12cyclictest0-21swapper/119:20:121
1777799251,10cyclictest0-21swapper/122:27:441
1777799251,10cyclictest0-21swapper/122:27:441
1777099251,12cyclictest10869-21munin-run21:20:000
1777099251,12cyclictest10869-21munin-run21:20:000
1778999248,8cyclictest401ktimersoftd/321:55:013
1778999241,4cyclictest0-21swapper/321:18:353
1778999241,4cyclictest0-21swapper/321:18:353
1778999241,3cyclictest0-21swapper/321:26:583
1778999241,18cyclictest0-21swapper/321:58:173
1778999241,17cyclictest0-21swapper/322:17:183
1778999241,13cyclictest27749-21/usr/sbin/munin00:25:113
1778299247,7cyclictest0-21swapper/219:15:042
1778299244,12cyclictest0-21swapper/220:30:142
1778299242,7cyclictest16849-21munin-run20:20:012
17782992415,6cyclictest337-21diskstats20:55:142
17782992413,7cyclictest909-21sssd_nss19:10:152
17777992415,6cyclictest17077-21/usr/sbin/munin20:20:151
1777099241,6cyclictest0-21swapper/023:51:100
1777099241,5cyclictest0-21swapper/023:24:050
1777099241,15cyclictest23627-21kworker/0:200:15:570
1778999237,11cyclictest401ktimersoftd/323:58:243
1778999235,5cyclictest13341-21head20:10:153
1778999231,16cyclictest39-21rcuc/323:50:253
1778999231,16cyclictest0-21swapper/323:10:003
1778999231,14cyclictest0-21swapper/322:10:293
17789992311,3cyclictest87350irq/126-enp1s0-23:20:233
17789992310,5cyclictest18470-21cstates22:50:113
1778999230,16cyclictest0-21swapper/321:21:383
1778999230,10cyclictest0-21swapper/323:50:023
17782992316,4cyclictest28523-21ntp_states19:30:192
17782992315,5cyclictest26211-21df_inode20:40:132
17782992315,5cyclictest17895-21/usr/sbin/munin20:20:182
17782992314,6cyclictest32198-21diskstats19:40:132
1778299231,14cyclictest0-21swapper/219:45:242
1777099231,11cyclictest3655-21munin-run19:50:010
1777099230,9cyclictest29394-21kworker/0:000:30:370
1778999226,3cyclictest579-21systemd-journal00:15:013
1778999222,9cyclictest41-21ksoftirqd/321:40:263
17789992212,2cyclictest401ktimersoftd/322:55:203
1778999221,17cyclictest0-21swapper/323:19:263
1778999221,16cyclictest0-21swapper/322:29:303
1778999221,16cyclictest0-21swapper/322:29:303
1778999221,15cyclictest0-21swapper/300:16:213
1778999221,11cyclictest0-21swapper/323:25:043
17789992210,7cyclictest0-21swapper/322:01:323
17789992210,5cyclictest5782-21hddtemp_smartct22:20:153
1778299225,3cyclictest87350irq/126-enp1s0-20:10:142
17782992215,4cyclictest29988-21/usr/sbin/munin19:35:122
1777099222,9cyclictest13637-21kworker/0:122:41:270
1777099221,9cyclictest5107-21kworker/0:222:29:150
1777099221,9cyclictest5107-21kworker/0:222:29:150
1777099221,9cyclictest1415-21kworker/0:119:50:170
1777099221,7cyclictest18002-21kworker/0:022:55:370
1777099221,7cyclictest1415-21kworker/0:119:55:170
1777099221,6cyclictest0-21swapper/021:36:390
1777099220,6cyclictest29024-21kworker/0:223:25:050
1778999214,8cyclictest41-21ksoftirqd/320:25:113
1778999214,7cyclictest87650irq/129-enp1s0-19:15:213
1778999211,6cyclictest0-21swapper/321:12:153
1778999211,17cyclictest17398-21hddtemp_smartct00:00:163
1778999210,13cyclictest0-21swapper/322:36:163
1777099218,3cyclictest87350irq/126-enp1s0-20:01:130
1777099215,4cyclictest23446-21munin-run20:35:010
1777099213,10cyclictest15486-21/usr/sbin/munin21:30:300
1777099212,9cyclictest31624-21kworker/0:200:36:190
1777099212,11cyclictest29024-21kworker/0:223:30:410
1777099212,11cyclictest12685-21/usr/sbin/munin20:10:110
1777099211,4cyclictest0-21swapper/022:30:220
1777099211,4cyclictest0-21swapper/021:24:090
1777099211,4cyclictest0-21swapper/000:04:480
1777099211,14cyclictest18572-21kworker/0:000:11:530
118402210,3sleep011-21rcuc/022:35:120
1778999209,3cyclictest18944-21/usr/sbin/munin00:05:213
1778999208,5cyclictest87650irq/129-enp1s0-20:32:103
1778999202,7cyclictest31851-21/usr/sbin/munin19:40:243
1778999202,2cyclictest87350irq/126-enp1s0-23:42:433
1778999201,6cyclictest39-21rcuc/322:32:593
1778999201,6cyclictest0-21swapper/323:02:023
1778999200,9cyclictest0-21swapper/322:08:593
1778999200,9cyclictest0-21swapper/300:20:113
1778999200,11cyclictest0-21swapper/323:31:563
1777099206,3cyclictest10073-21kworker/0:120:06:370
1777099202,10cyclictest29024-21kworker/0:122:17:430
1777099201,9cyclictest29394-21kworker/0:000:29:550
1777099201,9cyclictest18572-21kworker/0:000:06:350
1777099201,8cyclictest25862-21kworker/0:220:54:490
1777099201,7cyclictest19563-21kworker/0:021:43:270
1777099201,6cyclictest0-21swapper/022:46:490
1777099201,5cyclictest0-21swapper/021:59:500
1777099201,11cyclictest6704-21/usr/sbin/munin21:10:180
1777099201,11cyclictest19277-21/usr/sbin/munin20:25:240
309952190,2sleep30-21swapper/319:35:243
1778999199,3cyclictest0-21swapper/319:22:053
1778999194,3cyclictest18816-21cut19:10:143
1778999193,2cyclictest25260-21/usr/sbin/munin19:25:253
1778999191,9cyclictest41-21ksoftirqd/319:55:243
1778999191,9cyclictest0-21swapper/320:00:213
1778999191,8cyclictest6481-21basename21:10:013
1778999191,10cyclictest0-21swapper/320:09:243
17789991910,2cyclictest87350irq/126-enp1s0-20:50:133
1777099191,6cyclictest23972-21kworker/0:221:54:230
1777099191,6cyclictest0-21swapper/022:12:590
1777099191,10cyclictest30883-21/usr/sbin/munin22:05:270
1777099191,10cyclictest14878-21/usr/sbin/munin20:15:170
1777099190,9cyclictest27160-21kworker/0:100:24:430
1778999184,4cyclictest4103-21expr19:50:103
17789991814,2cyclictest4280-21awk21:05:013
1778999181,13cyclictest0-21swapper/322:45:463
1778999181,10cyclictest0-21swapper/319:45:253
1777099186,2cyclictest26972-21kworker/0:319:36:170
1777099183,9cyclictest808-21snmpd23:02:260
1777099182,3cyclictest17416-21kworker/0:219:10:450
1777099181,5cyclictest0-21swapper/021:46:520
1777099181,11cyclictest0-21swapper/021:00:250
1778999174,2cyclictest16841-21crond20:20:003
1778999171,4cyclictest27093-21ps20:40:293
1778999171,4cyclictest25642-21munin-run20:40:013
17789991712,3cyclictest28586-21sed19:30:203
1778999171,10cyclictest0-21swapper/320:21:133
1778999170,15cyclictest0-21swapper/320:56:243
1777099171,8cyclictest18769-21kworker/0:120:40:230
1777099171,7cyclictest0-21swapper/023:19:230
1777099171,7cyclictest0-21swapper/021:25:110
1777099171,4cyclictest10934-21irqstats23:45:170
1777099171,4cyclictest0-21swapper/023:11:020
1777099171,2cyclictest22424-21unixbench_singl19:15:230
17770991712,3cyclictest23726-21cut19:20:160
1778999161,12cyclictest28061-21/usr/sbin/munin20:45:133
1777099167,2cyclictest0-21swapper/022:20:210
1777099162,5cyclictest763-21dbus-daemon20:35:080
1777099161,9cyclictest0-21swapper/020:45:080
1777099161,5cyclictest0-21swapper/019:40:200
17770991611,3cyclictest25260-21/usr/sbin/munin19:25:230
1777099151,3cyclictest4503-21/usr/sbin/munin21:05:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional