You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-01 - 12:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Jun 01, 2026 00:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
243032240163,25sleep10-21swapper/119:09:551
241602237162,24sleep20-21swapper/219:08:032
241862203167,23sleep30-21swapper/319:08:243
240742202165,25sleep00-21swapper/019:06:560
1973821660,4sleep12439799cyclictest23:50:181
2653021480,3sleep32441099cyclictest19:14:373
2227421130,6sleep32441099cyclictest20:15:173
24403998581,3cyclictest0-21swapper/223:17:342
50332780,4sleep20-21swapper/220:50:172
7812520,4sleep011-21rcuc/019:25:260
259002510,1sleep10-21swapper/119:10:221
43532500,4sleep121-21rcuc/119:35:141
36122480,1sleep20-21swapper/222:00:192
53842460,1sleep30-21swapper/322:05:133
2441099441,30cyclictest0-21swapper/322:13:453
2440399411,31cyclictest0-21swapper/223:59:352
119312400,1sleep10-21swapper/122:20:131
2441099391,24cyclictest0-21swapper/321:33:423
2440399370,26cyclictest0-21swapper/221:41:412
24403993524,3cyclictest0-21swapper/222:19:212
24403993522,3cyclictest0-21swapper/221:57:292
2440399341,15cyclictest16410-21sshd23:43:352
2440399341,15cyclictest16410-21sshd23:43:352
2441099331,9cyclictest0-21swapper/322:26:413
2441099330,13cyclictest0-21swapper/321:48:373
2441099330,12cyclictest0-21swapper/322:37:013
2440399331,3cyclictest0-21swapper/223:48:512
2439799338,9cyclictest29458-21dump-pmu-power19:20:021
2438999332,18cyclictest1085-21kworker/0:023:15:210
24403993220,3cyclictest0-21swapper/221:17:572
24403993212,10cyclictest0-21swapper/200:21:212
2438999326,12cyclictest3709-21perf19:35:010
317062310,4sleep339-21rcuc/320:35:243
2441099311,9cyclictest0-21swapper/321:07:213
2441099310,18cyclictest0-21swapper/323:55:413
2441099310,11cyclictest0-21swapper/322:22:253
2440399311,3cyclictest0-21swapper/221:31:192
2440399310,24cyclictest0-21swapper/222:41:322
2440399310,21cyclictest0-21swapper/219:54:372
2439799313,18cyclictest14127-21chrt23:37:301
2438999314,13cyclictest19700-21munin-run21:25:010
2441099301,11cyclictest0-21swapper/322:04:053
2441099300,18cyclictest0-21swapper/323:06:143
2440399308,9cyclictest0-21swapper/220:37:052
2440399301,28cyclictest0-21swapper/223:05:032
2440399301,21cyclictest0-21swapper/200:36:372
2440399301,21cyclictest0-21swapper/200:27:252
2440399301,17cyclictest0-21swapper/221:06:542
2440399301,13cyclictest0-21swapper/220:30:032
2438999302,21cyclictest24078-21kworker/0:121:53:100
2438999302,13cyclictest1134-21munin-run23:10:010
2438999301,22cyclictest7499-21kworker/0:223:28:570
205622300,4sleep230-21rcuc/221:25:152
151972300,5sleep315199-21gltestperf23:40:163
151972300,5sleep315199-21gltestperf23:40:163
2441099297,9cyclictest0-21swapper/321:51:373
2441099291,12cyclictest0-21swapper/323:18:063
2441099291,12cyclictest0-21swapper/319:56:233
2441099291,10cyclictest0-21swapper/323:26:243
2441099290,15cyclictest0-21swapper/322:32:213
2441099290,15cyclictest0-21swapper/322:32:213
2440399299,3cyclictest0-21swapper/219:33:252
2440399298,8cyclictest0-21swapper/222:06:012
2440399298,10cyclictest0-21swapper/200:00:472
24403992914,9cyclictest0-21swapper/219:23:332
2440399291,12cyclictest0-21swapper/221:36:492
2440399291,10cyclictest0-21swapper/220:05:132
2440399290,27cyclictest0-21swapper/200:09:092
2439799292,13cyclictest21316-21grep23:55:131
2438999291,16cyclictest19434-21kworker/0:021:36:010
24389992910,16cyclictest4317-21awk20:50:010
2441099288,3cyclictest0-21swapper/321:38:013
2441099281,19cyclictest0-21swapper/319:24:063
2441099281,17cyclictest0-21swapper/300:39:573
2441099281,10cyclictest0-21swapper/320:08:033
2441099280,12cyclictest0-21swapper/300:02:253
2441099280,10cyclictest0-21swapper/321:29:543
2440399283,17cyclictest0-21swapper/220:25:572
2440399282,2cyclictest87450irq/127-enp1s0-22:37:422
2440399281,3cyclictest0-21swapper/200:32:072
2440399281,15cyclictest0-21swapper/222:31:052
2440399281,15cyclictest0-21swapper/222:31:052
2440399280,19cyclictest0-21swapper/220:18:292
2439799286,13cyclictest27501-21cat00:10:201
2438999281,21cyclictest30798-21kworker/0:022:03:140
2438999281,15cyclictest5889-21munin-run19:40:010
2438999281,10cyclictest18014-21kworker/0:322:52:250
24389992810,16cyclictest32572-21munin-run00:25:020
2441099279,9cyclictest0-21swapper/300:33:133
2441099279,3cyclictest0-21swapper/323:21:143
2441099276,10cyclictest0-21swapper/322:52:493
2441099271,12cyclictest0-21swapper/321:41:453
2441099271,11cyclictest0-21swapper/320:31:333
2441099271,10cyclictest0-21swapper/323:01:543
2441099271,10cyclictest0-21swapper/320:12:173
2441099270,15cyclictest0-21swapper/321:19:503
2441099270,13cyclictest0-21swapper/323:49:573
2441099270,11cyclictest0-21swapper/323:51:093
2440399279,9cyclictest0-21swapper/220:22:492
2440399271,2cyclictest0-21swapper/222:54:342
2440399271,10cyclictest0-21swapper/220:41:332
2440399270,17cyclictest0-21swapper/223:39:572
2439799272,9cyclictest20455-21sort22:40:011
2438999271,21cyclictest7499-21kworker/0:223:23:090
2438999270,22cyclictest24078-21kworker/0:121:49:120
25082260,4sleep121-21rcuc/120:45:131
2441099261,9cyclictest0-21swapper/300:26:533
2441099260,16cyclictest0-21swapper/323:38:093
2440399269,3cyclictest0-21swapper/219:29:372
2440399268,7cyclictest0-21swapper/223:32:352
2440399268,3cyclictest0-21swapper/220:01:052
24403992614,3cyclictest0-21swapper/223:10:332
2440399261,13cyclictest0-21swapper/223:04:012
2440399261,12cyclictest0-21swapper/222:13:432
2440399261,11cyclictest0-21swapper/220:12:112
2440399261,11cyclictest0-21swapper/200:13:292
2440399261,10cyclictest0-21swapper/221:21:252
2440399260,14cyclictest0-21swapper/221:54:532
2439799262,12cyclictest5916-21grep23:20:121
24397992614,5cyclictest7136-21awk22:10:011
24397992612,2cyclictest15459-21sed21:15:011
2441099259,3cyclictest0-21swapper/320:58:133
2441099257,8cyclictest0-21swapper/320:28:533
2441099256,3cyclictest0-21swapper/323:30:323
2441099256,3cyclictest0-21swapper/323:12:243
2441099251,22cyclictest0-21swapper/319:32:383
2441099251,14cyclictest0-21swapper/321:23:223
2441099251,11cyclictest0-21swapper/319:50:113
2441099251,10cyclictest666-21/usr/sbin/munin21:55:193
2441099251,10cyclictest0-21swapper/320:00:033
2441099250,19cyclictest0-21swapper/322:57:103
2441099250,10cyclictest0-21swapper/322:40:533
2440399251,3cyclictest0-21swapper/223:20:272
24403992512,3cyclictest0-21swapper/221:04:062
24403992512,3cyclictest0-21swapper/219:55:412
2440399251,10cyclictest0-21swapper/220:47:262
2440399250,13cyclictest0-21swapper/221:47:412
2440399250,11cyclictest0-21swapper/221:11:022
2440399250,10cyclictest0-21swapper/222:46:582
2439799251,7cyclictest827-21runrttasks00:18:351
2439799251,4cyclictest0-21swapper/123:25:181
2439799251,15cyclictest1127-21fschecks_count21:55:151
2439799251,12cyclictest0-21swapper/122:58:391
2438999251,6cyclictest0-21swapper/021:30:190
2438999251,16cyclictest18014-21kworker/0:322:43:570
2438999250,12cyclictest2418-21awk00:30:000
2438999250,10cyclictest9123-21kworker/0:122:23:570
298302241,3sleep229987-21users00:15:282
2441099246,3cyclictest0-21swapper/321:11:533
2441099245,8cyclictest13103-21diskmemload22:15:373
2441099241,10cyclictest31886-21/usr/sbin/munin19:25:253
2441099240,9cyclictest0-21swapper/320:24:493
2441099240,10cyclictest0-21swapper/319:45:133
2440399241,9cyclictest0-21swapper/219:47:012
2440399241,8cyclictest0-21swapper/223:51:452
2440399241,11cyclictest0-21swapper/219:40:312
24403992410,3cyclictest0-21swapper/223:25:152
2440399240,11cyclictest0-21swapper/219:38:092
2439799248,9cyclictest25342-21latency_hist00:05:011
2439799246,6cyclictest808-21snmpd21:39:581
2439799244,6cyclictest26905-21latency_hist22:55:011
2439799243,14cyclictest87650irq/129-enp1s0-00:25:261
2439799241,6cyclictest31905-21forks23:05:141
24397992414,3cyclictest0-21swapper/100:30:261
2439799241,20cyclictest28823-21cpuspeed_turbos21:45:131
2439799241,17cyclictest0-21swapper/120:16:281
2438999242,8cyclictest7499-21kworker/0:223:33:430
2438999242,10cyclictest13030-21kworker/0:121:15:030
2438999241,9cyclictest30798-21kworker/0:021:56:180
2438999241,11cyclictest25350-21munin-run00:05:020
2441099236,3cyclictest0-21swapper/322:46:433
2441099231,9cyclictest13103-21diskmemload00:05:033
2441099231,11cyclictest0-21swapper/320:54:093
2441099231,10cyclictest27495-21/usr/sbin/munin19:15:173
2440399238,3cyclictest0-21swapper/220:56:462
2440399237,3cyclictest0-21swapper/222:55:192
2440399231,11cyclictest0-21swapper/219:19:322
2439799238,8cyclictest773-21NetworkManager23:12:211
2439799236,4cyclictest10512-21/usr/sbin/munin19:50:291
2439799234,10cyclictest6498-21sh20:55:011
2439799232,8cyclictest32293-21timerandwakeup21:50:241
2439799232,6cyclictest22733-21/usr/sbin/munin22:45:181
2439799232,11cyclictest18113-21latency_hist22:35:021
2439799232,11cyclictest18113-21latency_hist22:35:011
24397992318,3cyclictest29103-21needreboot20:30:201
2439799231,16cyclictest0-21swapper/121:15:461
2438999232,9cyclictest9123-21kworker/0:122:25:570
2438999232,8cyclictest30798-21kworker/0:022:13:370
2438999232,14cyclictest9123-21kworker/0:122:33:530
2438999232,14cyclictest9123-21kworker/0:122:33:530
2438999231,8cyclictest29390-21kworker/0:019:24:490
2438999231,8cyclictest18653-21kworker/0:223:51:500
2438999231,7cyclictest24078-21kworker/0:121:43:340
24389992310,3cyclictest13435-21ntp_states23:35:220
2441099221,11cyclictest0-21swapper/319:40:273
2441099221,11cyclictest0-21swapper/300:15:313
2441099220,10cyclictest0-21swapper/300:21:433
2440399223,8cyclictest0-21swapper/219:10:112
2439799229,9cyclictest12395-21smart_sda21:05:251
2439799225,14cyclictest6729-21/usr/sbin/munin20:55:221
2439799225,14cyclictest20397-21grep21:25:141
2439799225,10cyclictest0-21swapper/123:45:231
2439799224,14cyclictest11190-21netstat23:30:191
2439799222,17cyclictest30699-21tr00:20:151
2439799221,6cyclictest14670-21latency_hist20:00:011
2439799221,4cyclictest0-21swapper/121:31:131
24397992214,3cyclictest0-21swapper/119:24:571
2439799221,18cyclictest15330-21timerandwakeup22:25:261
2439799221,16cyclictest0-21swapper/120:28:021
2439799221,16cyclictest0-21swapper/119:33:001
2439799220,16cyclictest0-21swapper/120:11:371
2439799220,14cyclictest32149-21expr19:25:141
2439799220,12cyclictest9226-21memory19:45:191
2438999222,8cyclictest7774-21kworker/0:219:49:280
2438999222,8cyclictest2251-21kworker/0:000:35:430
2438999222,8cyclictest15241-21kworker/0:220:00:560
2438999222,11cyclictest23413-21kworker/0:120:36:490
2438999221,7cyclictest16318-21kworker/0:023:56:020
2438999221,10cyclictest9123-21kworker/0:122:38:330
2441099211,7cyclictest0-21swapper/321:00:353
2441099211,11cyclictest0-21swapper/319:35:543
2441099211,10cyclictest0-21swapper/320:47:373
2439799217,7cyclictest0-21swapper/121:00:161
2439799214,2cyclictest4395-21kernelversion23:15:171
2439799212,17cyclictest0-21swapper/119:40:021
2439799211,4cyclictest18621-21kworker/u8:200:07:051
2439799211,3cyclictest18245-1kworker/u9:221:44:391
2439799211,15cyclictest0-21swapper/120:42:521
2439799211,14cyclictest24771-21ntpq20:20:231
2439799211,13cyclictest0-21swapper/123:41:491
2439799211,13cyclictest0-21swapper/123:41:491
2439799211,12cyclictest4831-21/usr/sbin/munin00:35:131
2439799210,7cyclictest30300-21/usr/sbin/munin20:35:251
2439799210,16cyclictest0-21swapper/123:01:541
2439799210,15cyclictest0-21swapper/122:41:381
2439799210,11cyclictest0-21swapper/122:01:361
2438999212,8cyclictest19434-21kworker/0:021:27:390
2438999212,11cyclictest21506-21/usr/sbin/munin20:15:170
2438999211,5cyclictest0-21swapper/000:07:050
2438999211,4cyclictest0-21swapper/023:10:320
2438999211,4cyclictest0-21swapper/022:16:240
2438999211,4cyclictest0-21swapper/000:15:100
2438999211,11cyclictest12362-21kworker/0:123:46:460
125632211,17sleep212665-21memory22:20:182
2441099200,9cyclictest0-21swapper/320:40:053
2441099200,9cyclictest0-21swapper/300:10:273
2440399206,3cyclictest13953-21/usr/sbin/munin22:25:232
2439799202,5cyclictest14906-21/usr/sbin/munin20:00:161
2439799202,3cyclictest0-21swapper/120:06:071
2439799201,11cyclictest8429-21ntp_states22:10:231
2439799201,11cyclictest18643-21memory21:20:191
2439799200,3cyclictest4830-21kworker/u8:022:18:311
2438999207,2cyclictest8403-21kworker/0:020:59:330
2438999206,4cyclictest0-21swapper/023:00:290
2438999202,9cyclictest18014-21kworker/0:322:55:230
2438999202,8cyclictest26462-21kworker/0:119:14:420
2438999202,7cyclictest12362-21kworker/0:123:40:380
2438999202,7cyclictest12362-21kworker/0:123:40:380
2438999202,4cyclictest0-21swapper/021:12:590
2438999201,9cyclictest6168-21kworker/0:220:54:230
2438999201,9cyclictest25008-21kworker/0:200:11:520
2438999201,8cyclictest8403-21kworker/0:021:01:270
2438999201,8cyclictest10025-21kworker/0:119:53:200
2438999201,7cyclictest30798-21kworker/0:022:09:250
2438999201,6cyclictest23413-21kworker/0:120:30:370
2438999201,11cyclictest0-21swapper/000:31:500
278312191,2sleep00-21swapper/020:29:360
2438999197,2cyclictest18014-21kworker/0:322:46:110
2438999192,8cyclictest8403-21kworker/0:021:09:370
2438999191,10cyclictest23704-21/usr/sbin/munin20:20:200
2438999190,6cyclictest5527-21kworker/0:019:40:530
2438999187,2cyclictest12403-21kworker/0:020:06:160
2438999183,6cyclictest0-21swapper/020:12:360
2438999181,9cyclictest0-21swapper/020:43:350
2438999162,5cyclictest27495-21/usr/sbin/munin19:15:250
2438999162,11cyclictest13305-21head19:55:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional