You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-19 - 03:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue May 19, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2618423270,15sleep10-21swapper/119:05:111
275562242166,25sleep20-21swapper/219:06:062
278572235162,24sleep00-21swapper/019:09:580
276022202164,25sleep30-21swapper/319:06:433
27959997068,1cyclictest0-21swapper/223:41:012
251302630,3sleep10-21swapper/123:55:121
27959996158,2cyclictest0-21swapper/222:46:492
35252590,4sleep32796599cyclictest20:40:023
27959995956,2cyclictest0-21swapper/220:08:472
27959994541,3cyclictest0-21swapper/222:44:262
27959994541,3cyclictest0-21swapper/222:32:432
71632360,1sleep00-21swapper/020:45:240
2794899349,12cyclictest30132-21awk19:15:010
2795399331,26cyclictest0-21swapper/123:42:211
2796599321,6cyclictest0-21swapper/323:31:233
2794899327,12cyclictest32668-21munin-run23:00:010
108862320,4sleep30-21swapper/300:35:193
108862320,4sleep30-21swapper/300:35:183
2796599311,10cyclictest11727-21cstates23:25:123
2796599311,10cyclictest11727-21cstates23:25:123
2794899302,20cyclictest6885-21kworker/0:223:16:500
2794899302,20cyclictest32660-21kworker/0:219:19:100
2794899302,20cyclictest11858-21kworker/0:020:00:020
2794899302,13cyclictest7247-21munin-run19:35:010
2794899301,21cyclictest10250-21kworker/0:023:43:540
2795399291,6cyclictest25809-40dnf22:41:591
2794899292,20cyclictest8078-21kworker/0:221:00:260
2794899292,20cyclictest32550-21kworker/0:123:09:250
2794899292,20cyclictest16754-21kworker/0:122:39:100
2794899292,19cyclictest5997-21kworker/0:022:22:140
2794899292,19cyclictest3183-21kworker/0:120:50:110
2794899292,19cyclictest22274-21kworker/0:123:51:500
2794899292,16cyclictest28498-21kworker/0:020:29:280
2794899291,21cyclictest4780-21kworker/0:319:46:460
2794899291,21cyclictest4780-21kworker/0:319:27:060
2794899291,20cyclictest31078-21kworker/0:200:10:340
2796599287,3cyclictest0-21swapper/319:46:003
2795999281,16cyclictest0-21swapper/223:26:582
2795999281,16cyclictest0-21swapper/223:26:582
27953992812,8cyclictest5340-21crond00:25:011
2795399281,17cyclictest0-21swapper/121:13:161
2794899282,19cyclictest7338-21kworker/0:100:28:180
2794899282,19cyclictest28755-21kworker/0:000:05:340
2794899282,18cyclictest2787-21kworker/0:000:21:500
2794899281,20cyclictest32550-21kworker/0:123:11:210
2794899281,20cyclictest16754-21kworker/0:122:41:140
2794899281,20cyclictest1533-21kworker/0:121:55:220
2794899281,19cyclictest21522-21kworker/0:222:34:260
86122270,3sleep00-21swapper/019:35:260
2796599276,3cyclictest0-21swapper/319:38:083
2795399271,19cyclictest87550irq/128-enp1s0-23:45:181
2795399271,18cyclictest7175-21hddtemp_smartct22:00:171
2794899272,19cyclictest16439-21kworker/0:221:11:540
2794899272,18cyclictest28055-21kworker/0:222:52:220
2794899272,18cyclictest25339-21kworker/0:321:43:020
2794899272,18cyclictest22274-21kworker/0:100:02:580
2794899272,18cyclictest17776-21kworker/0:223:45:140
2794899272,17cyclictest10250-21kworker/0:023:30:580
2794899271,19cyclictest28498-21kworker/0:020:22:100
2794899271,19cyclictest2787-21kworker/0:000:17:420
2794899271,11cyclictest31065-21kworker/0:220:38:310
2794899270,17cyclictest11888-21kworker/0:021:08:060
2794899270,17cyclictest11888-21kworker/0:021:08:060
2796599262,21cyclictest22517-21/usr/sbin/munin23:50:203
2796599262,12cyclictest9335-21/usr/sbin/munin23:20:153
2796599261,7cyclictest0-21swapper/322:50:553
2796599260,20cyclictest0-21swapper/300:10:483
2795399262,3cyclictest87450irq/127-enp1s0-20:10:381
27953992621,2cyclictest10305-21df00:35:141
27953992621,2cyclictest10305-21df00:35:141
2795399261,6cyclictest0-21swapper/123:28:361
2795399261,6cyclictest0-21swapper/123:28:361
2795399261,18cyclictest0-21swapper/121:20:291
2795399261,17cyclictest29713-21kworker/u8:122:25:451
2795399261,17cyclictest0-21swapper/122:12:191
27953992610,8cyclictest10731-21ntpq19:40:221
2794899267,6cyclictest5997-21kworker/0:022:05:280
2794899262,15cyclictest5997-21kworker/0:022:12:400
2794899262,14cyclictest7454-21kworker/0:300:38:480
2794899262,14cyclictest7454-21kworker/0:300:38:480
27948992612,6cyclictest8078-21kworker/0:220:55:230
2794899261,19cyclictest12803-21kworker/0:222:16:420
2794899261,19cyclictest12803-21kworker/0:222:16:420
2794899261,18cyclictest29808-21kworker/0:021:50:260
2794899261,18cyclictest25830-21kworker/0:023:02:090
2794899261,18cyclictest22274-21kworker/0:123:56:180
2794899261,17cyclictest16754-21kworker/0:122:28:500
2794899260,11cyclictest24747-21kworker/0:120:18:100
2796599257,3cyclictest0-21swapper/320:01:123
2796599251,9cyclictest0-21swapper/323:39:403
27959992518,5cyclictest13919-21/usr/sbin/munin22:15:192
27959992518,5cyclictest13919-21/usr/sbin/munin22:15:192
2795399258,9cyclictest0-21swapper/121:27:221
2795399251,9cyclictest0-21swapper/123:00:401
27953992519,3cyclictest30165-21munin-node19:15:021
2795399251,17cyclictest0-21swapper/121:02:261
2795399251,16cyclictest15056-21cat22:19:561
2795399251,16cyclictest15056-21cat22:19:561
2795399251,16cyclictest0-21swapper/123:10:501
2794899252,9cyclictest2787-21kworker/0:000:31:540
2794899252,7cyclictest32550-21kworker/0:123:20:080
2794899252,12cyclictest27573-21kworker/0:121:37:340
2794899251,9cyclictest22322-21kworker/0:320:07:580
2794899251,18cyclictest5997-21kworker/0:022:03:420
2796599245,3cyclictest0-21swapper/320:56:323
2796599241,12cyclictest0-21swapper/322:10:153
2796599241,11cyclictest31759-21ntp_states22:55:233
2796599241,10cyclictest0-21swapper/319:24:483
2796599240,7cyclictest0-21swapper/300:26:193
2796599240,6cyclictest0-21swapper/321:22:363
2796599240,19cyclictest0-21swapper/323:10:003
2795999244,12cyclictest10739-21ntp_states19:40:232
2795999241,14cyclictest0-21swapper/200:10:592
2795399242,16cyclictest0-21swapper/122:50:161
27953992420,3cyclictest0-21swapper/120:39:441
27953992414,9cyclictest2386-21kworker/u8:023:35:591
2795399241,17cyclictest0-21swapper/123:30:361
2795399241,17cyclictest0-21swapper/120:26:101
2795399241,16cyclictest0-21swapper/121:50:541
2795399241,15cyclictest21-21rcuc/121:48:111
2795399241,15cyclictest0-21swapper/100:25:171
27953992410,3cyclictest24481-21sh23:55:011
2795399240,16cyclictest0-21swapper/121:34:351
2794899248,9cyclictest15475-21kworker/0:123:35:420
2794899242,8cyclictest18792-21kworker/0:121:21:240
2794899242,6cyclictest11858-21kworker/0:019:50:120
2794899242,16cyclictest25159-21kworker/0:221:27:100
2796599237,4cyclictest0-21swapper/322:44:173
2796599231,8cyclictest0-21swapper/322:23:473
2796599231,14cyclictest0-21swapper/321:45:423
2796599231,12cyclictest0-21swapper/320:19:073
2796599231,11cyclictest25667-21/usr/sbin/munin21:30:243
2796599231,11cyclictest21894-21/usr/sbin/munin22:35:233
2796599231,11cyclictest0-21swapper/321:15:293
2796599231,10cyclictest9670-21/usr/sbin/munin19:40:163
2796599230,6cyclictest0-21swapper/322:15:413
2796599230,6cyclictest0-21swapper/322:15:413
2796599230,6cyclictest0-21swapper/321:42:423
2796599230,12cyclictest0-21swapper/322:05:483
2795999235,6cyclictest32718-21ntp_states20:30:222
2795399235,11cyclictest0-21swapper/122:21:071
27953992319,3cyclictest0-21swapper/121:39:391
27953992319,3cyclictest0-21swapper/120:31:011
27953992317,3cyclictest6280-21fschecks_count20:45:151
2795399231,17cyclictest87550irq/128-enp1s0-22:30:171
2795399231,16cyclictest0-21swapper/119:20:151
2795399231,16cyclictest0-21swapper/100:10:041
2795399231,10cyclictest9479-21tr23:20:131
2795399231,10cyclictest0-21swapper/121:16:411
2795399230,6cyclictest0-21swapper/100:16:101
2794899232,10cyclictest25339-21kworker/0:321:31:260
2794899231,9cyclictest22322-21kworker/0:320:11:460
2794899231,6cyclictest4780-21kworker/0:319:43:320
2794899231,15cyclictest10250-21kworker/0:023:28:460
2794899231,15cyclictest10250-21kworker/0:023:28:460
303592220,5sleep121-21rcuc/121:40:121
2796599228,3cyclictest19696-21/usr/sbin/munin22:30:123
2796599226,3cyclictest0-21swapper/321:00:403
2796599221,7cyclictest0-21swapper/323:10:243
2796599221,6cyclictest0-21swapper/321:52:173
2796599221,5cyclictest0-21swapper/321:14:183
2796599221,3cyclictest0-21swapper/321:56:463
2796599221,13cyclictest0-21swapper/320:05:153
2796599221,11cyclictest0-21swapper/321:26:073
2796599221,10cyclictest0-21swapper/320:32:043
2796599220,8cyclictest0-21swapper/323:03:043
2796599220,12cyclictest0-21swapper/320:25:283
2795999224,7cyclictest4328-21missed_timers00:20:192
2795399229,8cyclictest0-21swapper/120:19:441
2795399225,9cyclictest0-21swapper/100:03:451
2795399221,9cyclictest10282-21/usr/sbin/munin20:55:221
27953992218,3cyclictest0-21swapper/119:55:271
27953992218,2cyclictest5744-21sed21:55:251
27953992214,3cyclictest0-21swapper/120:44:511
2795399221,16cyclictest0-21swapper/122:07:491
2795399221,15cyclictest0-21swapper/123:08:031
2795399221,15cyclictest0-21swapper/119:35:081
2795399221,15cyclictest0-21swapper/100:30:031
2795399221,15cyclictest0-21swapper/100:09:231
2795399221,10cyclictest31764-21ntp_states22:55:231
2795399220,15cyclictest0-21swapper/122:49:421
2794899222,8cyclictest5522-21kworker/0:020:44:310
2794899222,12cyclictest18792-21kworker/0:121:18:100
74482210,6sleep3401ktimersoftd/322:00:203
61572210,4sleep339-21rcuc/319:30:193
2796599217,3cyclictest808-21snmpd23:40:313
2796599211,4cyclictest0-21swapper/300:16:173
2796599211,12cyclictest0-21swapper/322:27:213
2796599211,12cyclictest0-21swapper/321:07:303
2796599211,12cyclictest0-21swapper/321:07:303
2796599211,10cyclictest16652-21diskmemload00:33:033
2796599211,10cyclictest0-21swapper/322:45:103
2796599211,10cyclictest0-21swapper/319:16:483
2796599210,4cyclictest0-21swapper/323:16:033
2795999212,7cyclictest4941-21/usr/sbin/munin23:10:242
2795999212,6cyclictest13607-21pmu-power21:00:252
2795999212,15cyclictest13410-21grep19:45:272
2795999211,8cyclictest0-21swapper/200:08:032
2795999211,6cyclictest0-21swapper/222:27:462
2795999211,4cyclictest0-21swapper/223:45:292
2795999211,3cyclictest32-21ksoftirqd/200:05:012
2795399216,3cyclictest22252-21users20:05:261
2795399212,10cyclictest0-21swapper/119:30:251
2795399211,3cyclictest19484-21/usr/sbin/munin20:00:191
2795399211,3cyclictest15263-21fschecks_time21:05:151
2795399211,3cyclictest15263-21fschecks_time21:05:151
2795399211,18cyclictest0-21swapper/122:35:031
2795399211,11cyclictest9350-21smart_sda20:50:241
2795399211,11cyclictest0-21swapper/119:28:031
2795399211,10cyclictest32759-21taskset19:16:571
2794899218,7cyclictest1533-21kworker/0:121:47:180
2794899212,6cyclictest31065-21kworker/0:220:33:160
2794899211,9cyclictest15927-21kworker/0:219:56:240
27948992110,2cyclictest28055-21kworker/0:222:49:420
2796599209,3cyclictest3377-21/usr/sbin/munin00:20:143
2796599206,3cyclictest16652-21diskmemload00:09:043
2796599205,3cyclictest0-21swapper/320:21:203
2796599203,3cyclictest0-21swapper/320:46:323
2796599201,5cyclictest0-21swapper/323:58:193
2795999206,5cyclictest0-21swapper/223:39:132
2795999206,3cyclictest28529-21gltestperf21:35:142
2795999205,4cyclictest3892-21ls20:40:132
2795999205,4cyclictest21904-21acpi22:35:122
2795999202,7cyclictest25116-21cpu20:15:112
2795999202,5cyclictest6933-21/usr/sbin/munin20:45:212
2795999201,7cyclictest1159-21/usr/sbin/munin00:15:142
2795999201,6cyclictest9447-21/usr/sbin/munin19:40:012
2795999201,6cyclictest3984-21memory19:25:182
2795999201,4cyclictest2490-21fschecks_count21:50:122
2795999200,6cyclictest31750-21ntpq22:55:222
2795399209,3cyclictest909-21sssd_nss19:45:171
2795399201,17cyclictest0-21swapper/120:20:291
2795399201,17cyclictest0-21swapper/119:50:161
2795399201,10cyclictest0-21swapper/123:15:171
2794899200,3cyclictest32660-21kworker/0:219:24:470
2796599195,3cyclictest0-21swapper/320:50:363
2796599195,3cyclictest0-21swapper/320:40:163
2796599195,3cyclictest0-21swapper/320:11:513
2796599192,2cyclictest0-21swapper/319:25:143
2796599191,4cyclictest27858-40updatedb00:00:213
2796599191,11cyclictest0-21swapper/321:35:293
2796599191,10cyclictest0-21swapper/323:45:323
2796599190,9cyclictest0-21swapper/319:53:553
2795999195,3cyclictest19748-21cat20:00:222
2795999192,6cyclictest32-21ksoftirqd/221:55:132
2795999192,15cyclictest0-21swapper/223:01:152
2795999192,15cyclictest0-21swapper/221:17:262
2795999191,7cyclictest0-21swapper/223:55:082
2795999191,7cyclictest0-21swapper/222:06:202
2795999191,6cyclictest14896-21ps23:30:232
2795999191,6cyclictest14686-21/usr/sbin/munin21:05:142
2795999191,6cyclictest14686-21/usr/sbin/munin21:05:142
2795999191,6cyclictest0-21swapper/223:19:192
27959991912,4cyclictest29677-21grep19:10:252
2795999191,15cyclictest15127-21ntpq19:50:202
2796599181,5cyclictest28482-21df19:10:123
2795999185,3cyclictest87350irq/126-enp1s0-20:10:252
2795999184,3cyclictest87650irq/129-enp1s0-19:34:442
2795999184,3cyclictest87350irq/126-enp1s0-19:55:132
2795999184,3cyclictest12050irq/123-ahci[0021:15:022
2795999184,3cyclictest0-21swapper/223:20:172
2795999182,4cyclictest29607-21pmu-power22:50:222
2795999182,15cyclictest0-21swapper/200:36:302
2795999182,15cyclictest0-21swapper/200:36:302
2795999182,14cyclictest0-21swapper/223:05:102
2795999181,6cyclictest5570-21/usr/sbin/munin00:25:212
2795999181,6cyclictest0-21swapper/222:03:222
2795999181,5cyclictest8908-21kernelversion20:50:182
2795999181,5cyclictest1874-21grep19:20:192
2795999181,5cyclictest12169-21smart_sda22:10:232
2795999181,5cyclictest11092-21irqstats20:55:182
2795999181,5cyclictest0-21swapper/221:20:132
27959991813,3cyclictest0-21swapper/200:30:262
27959991812,3cyclictest27905-21head20:20:132
2795999181,14cyclictest30698-21sed21:40:142
2796599174,3cyclictest16423-21cpuspeed_turbos19:55:133
2795999172,4cyclictest808-21snmpd21:45:082
2795999172,3cyclictest26015-21diskstats21:30:132
2795999171,4cyclictest2752-21smartctl20:35:252
2795999171,4cyclictest23197-21hddtemp_smartct23:50:172
2795999171,4cyclictest15683-21fschecks_count22:20:142
2795999171,4cyclictest0-21swapper/221:25:452
2795999171,3cyclictest0-21swapper/219:17:122
2795999153,3cyclictest87350irq/126-enp1s0-20:29:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional