You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 02:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Mon Nov 24, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4019581212156,60sleep30-21swapper/319:08:323
401939128653,9sleep20-21swapper/219:06:082
401941328553,10sleep10-21swapper/119:06:231
401965127656,14sleep00-21swapper/019:09:260
4019813992824,2cyclictest0-21swapper/323:40:133
4019813992622,3cyclictest0-21swapper/322:15:193
4019813992521,2cyclictest0-21swapper/322:49:423
4019813992521,2cyclictest0-21swapper/300:15:213
401981399251,18cyclictest25311-21taskset21:50:213
4019813992421,1cyclictest0-21swapper/300:05:193
4019813992418,3cyclictest4037653-21turbostat19:25:003
758962231,9sleep24149465-21diskmemload22:21:122
401981399232,14cyclictest12299-21taskset21:44:403
4019813992321,1cyclictest0-21swapper/321:30:013
4019813992320,1cyclictest0-21swapper/320:20:013
4019813992319,2cyclictest0-21swapper/322:05:153
401981399231,13cyclictest0-21swapper/300:35:593
1665662231,2sleep00-21swapper/023:17:310
401981399222,15cyclictest259312-21sleep00:14:593
4019813992219,1cyclictest0-21swapper/320:10:233
4019813992218,2cyclictest0-21swapper/323:19:533
401981399221,16cyclictest4173397-21chrt21:24:313
4019813992210,8cyclictest1567-21chrt21:35:573
401981399220,20cyclictest0-21swapper/319:59:443
940372211,8sleep24149465-21diskmemload22:33:592
875152218,3sleep086902-21/usr/sbin/munin22:30:150
401981399212,13cyclictest84224-21taskset22:26:213
4019813992119,1cyclictest0-21swapper/322:10:153
4019813992118,1cyclictest0-21swapper/320:20:233
4019813992117,2cyclictest0-21swapper/323:10:173
4019813992117,2cyclictest0-21swapper/322:38:543
182762211,2sleep10-21swapper/121:46:191
1331672211,3sleep00-21swapper/022:56:380
1013402211,2sleep10-21swapper/122:37:351
401981399203,10cyclictest4027788-21taskset19:15:123
401981399202,13cyclictest4157026-21taskset21:14:253
401981399202,13cyclictest133918-21taskset22:57:483
4019813992017,1cyclictest0-21swapper/322:40:113
4019813992016,2cyclictest0-21swapper/319:35:253
4019813992015,2cyclictest0-21swapper/322:20:173
4019813992014,3cyclictest0-21swapper/300:00:173
4019813992013,4cyclictest217979-21turbostat23:50:003
401981399201,15cyclictest225624-21chrt23:54:123
401981399201,12cyclictest4165027-21taskset21:19:143
401981399201,11cyclictest4138056-21taskset20:55:253
401981399200,13cyclictest226180-21sleep23:59:553
2983272200,2sleep20-21swapper/200:39:152
2673802201,2sleep00-21swapper/000:19:530
178852201,2sleep20-21swapper/221:45:442
401981399192,12cyclictest290231-21taskset00:33:303
401981399192,12cyclictest190577-21taskset23:30:513
401981399192,12cyclictest150207-21taskset23:07:233
4019813991916,2cyclictest0-21swapper/323:30:013
4019813991916,2cyclictest0-21swapper/322:00:003
4019813991916,1cyclictest0-21swapper/320:10:003
4019813991912,4cyclictest66550irq/131-enp1s0-TxRx-300:28:523
401981399191,17cyclictest0-21swapper/323:04:523
401981399191,14cyclictest4187304-21chrt21:30:533
401981399191,14cyclictest176041-21chrt23:24:213
401981399191,14cyclictest125205-21taskset22:51:463
401981399191,14cyclictest0-21swapper/319:10:143
315822191,13sleep233194-21/usr/sbin/munin21:55:182
787692181,4sleep24149465-21diskmemload22:25:132
684362181,3sleep10-21swapper/122:17:261
614212181,3sleep10-21swapper/122:14:411
41895352181,2sleep00-21swapper/021:34:190
4019813991815,1cyclictest0-21swapper/323:35:163
4019813991814,1cyclictest0-21swapper/321:05:233
401981399181,14cyclictest37455-21chrt22:00:113
401981399181,13cyclictest4069609-21turbostat19:54:593
401981399181,13cyclictest12843-21chrt21:45:073
401981399181,13cyclictest0-21swapper/322:30:123
401981399181,13cyclictest0-21swapper/300:20:073
1497902181,3sleep10-21swapper/123:06:451
840182175,1sleep0171rcu_preempt22:26:030
603052171,2sleep20-21swapper/222:12:582
504102171,4sleep0171rcu_preempt22:05:270
427342171,2sleep00-21swapper/022:01:090
41794442171,2sleep10-21swapper/121:26:191
401981399179,5cyclictest4064013-21turbostat19:50:013
401981399177,1cyclictest0-21swapper/319:30:263
4019813991714,2cyclictest0-21swapper/320:30:153
4019813991714,1cyclictest0-21swapper/321:00:153
401981399171,12cyclictest0-21swapper/320:50:213
2899892171,4sleep24149465-21diskmemload00:33:052
2831872171,3sleep00-21swapper/000:29:510
2315542171,4sleep14149465-21diskmemload23:55:481
2104712171,10sleep1210686-21hddtemp_smartct23:45:141
41553892160,3sleep00-21swapper/021:11:390
41552752161,2sleep10-21swapper/121:11:271
40751742161,10sleep04075203-21grep20:00:120
401981399161,9cyclictest4112407-21taskset20:35:093
4019813991612,2cyclictest0-21swapper/320:28:543
401981399161,10cyclictest4122550-21taskset20:43:143
401981399161,10cyclictest4058399-21taskset19:41:233
2955802162,6sleep0295634-21ntpq00:35:200
1997412161,9sleep04149465-21diskmemload23:37:190
1912222161,4sleep24149465-21diskmemload23:31:512
1739312161,2sleep10-21swapper/123:21:131
1664062161,3sleep10-21swapper/123:17:161
1655042161,2sleep20-21swapper/223:15:522
1573422161,3sleep14149465-21diskmemload23:10:501
1426742161,11sleep10-21swapper/123:03:201
1260672161,3sleep20-21swapper/222:53:102
1177892161,4sleep24149465-21diskmemload22:47:502
1176472160,3sleep10-21swapper/122:47:371
770212151,2sleep10-21swapper/122:22:551
507802151,8sleep24149465-21diskmemload22:05:572
4362151,7sleep1497-21ntpq21:35:211
41902792151,3sleep24149465-21diskmemload21:35:062
41733262150,3sleep00-21swapper/021:24:240
415832721510,2sleep10-21swapper/121:15:161
41465542151,10sleep14148260-21latency21:05:181
40970402151,9sleep14097751-21cut20:20:161
4019813991512,1cyclictest0-21swapper/319:28:543
4019813991511,2cyclictest0-21swapper/320:00:163
401981399150,12cyclictest0-21swapper/320:45:213
297732151,9sleep030000-21kthreadcore21:55:160
2744322151,7sleep10-21swapper/100:23:211
2731122151,4sleep0273123-21ssh00:21:090
1810172151,9sleep1181069-21ntpq23:25:221
1588502151,9sleep0158862-21ssh23:13:100
811032140,3sleep10-21swapper/122:25:191
759542141,9sleep075964-21ssh22:21:170
681112141,5sleep087-21kswapd022:16:570
41746542141,8sleep04175192-21cut21:25:180
41718372140,2sleep10-21swapper/121:22:051
2821892141,9sleep1282198-21ssh00:28:121
262262141,8sleep226237-21ssh21:50:572
2436792141,8sleep2244610-21grep00:05:192
2078792140,8sleep24149465-21diskmemload23:42:052
2064202141,9sleep0206429-21ssh23:40:280
1992902140,3sleep20-21swapper/223:36:362
1746642141,4sleep0174667-21ssh23:22:200
1244042141,6sleep14149465-21diskmemload22:50:341
1083522141,7sleep04149465-21diskmemload22:40:510
577272131,2sleep057910-21strings22:10:190
41494512131,8sleep24149503-21systemd-run21:10:002
40500882131,7sleep14051837-21kthreadcore19:35:161
357772131,3sleep135785-21ssh21:58:021
30912131,6sleep04149465-21diskmemload21:38:170
2657852131,6sleep24149465-21diskmemload00:17:262
2513912133,5sleep1251417-21apt-get00:10:151
2509682131,4sleep1250980-21rm00:09:591
2404122133,4sleep2431ktimers/200:01:342
2353182131,8sleep0236288-21kthreadcore00:00:160
2336612137,3sleep0584-21gpm23:59:040
2239152131,8sleep24149465-21diskmemload23:51:352
1995812132,2sleep1171rcu_preempt23:37:041
1841072131,9sleep00-21swapper/023:29:010
1337292131,3sleep1133738-21ssh22:57:311
1176362131,7sleep0117646-21ssh22:47:360
1082472131,5sleep20-21swapper/222:40:402
41894482121,5sleep14149465-21diskmemload21:34:121
41722272121,4sleep24149465-21diskmemload21:22:432
41570172121,4sleep24149465-21diskmemload21:14:242
41444282121,7sleep04144597-21cstates21:05:130
40703592121,7sleep14070496-21kthreadcore19:55:161
40332132121,7sleep14034685-21kthreadcore19:20:161
2885072120,8sleep0288519-21ssh00:30:350
2876342120,8sleep1287737-21sh00:30:191
279862121,5sleep14149465-21diskmemload21:53:401
2650742120,2sleep10-21swapper/100:16:211
2572522121,4sleep24149465-21diskmemload00:11:562
2499612121,5sleep00-21swapper/000:08:260
2253362120,1sleep00-21swapper/023:53:470
2090292121,3sleep10-21swapper/123:43:501
1820562121,3sleep2182067-21ssh23:25:592
1423682121,5sleep00-21swapper/023:02:510
1331912121,4sleep24149465-21diskmemload22:56:402
104022121,5sleep24149465-21diskmemload21:41:512
914192111,3sleep14149465-21diskmemload22:30:241
88012111,4sleep04149465-21diskmemload21:40:200
522442111,4sleep14149465-21diskmemload22:08:121
452142111,3sleep10-21swapper/122:04:591
424962111,4sleep24149465-21diskmemload22:00:482
41822412111,5sleep24182467-21df_inode21:30:132
41794572111,3sleep20-21swapper/221:26:202
41636972110,1sleep20-21swapper/221:17:142
41633422111,4sleep04149465-21diskmemload21:16:400
40956262110,1sleep10-21swapper/120:16:171
40431532112,4sleep24043172-21apt-get19:30:112
40274542111,5sleep14027577-21diskstats19:15:121
40200162113,3sleep14020274-21df_inode19:10:151
2969832111,3sleep10-21swapper/100:36:331
2814472111,4sleep24149465-21diskmemload00:26:562
2729322111,4sleep24149465-21diskmemload00:20:502
260892110,5sleep04149465-21diskmemload21:50:420
2571352111,3sleep00-21swapper/000:11:470
2415372111,4sleep14149465-21diskmemload00:03:141
2319912111,6sleep2231993-21ssh23:56:292
2228312110,7sleep1218322-21/usr/sbin/munin23:50:271
2177552111,2sleep20-21swapper/223:49:442
1717472111,4sleep24149465-21diskmemload23:20:202
1593922110,1sleep20-21swapper/223:13:592
1490682110,4sleep04149465-21diskmemload23:05:360
1411522111,4sleep24149465-21diskmemload23:00:562
1263052111,4sleep04149465-21diskmemload22:53:330
122292111,3sleep10-21swapper/121:44:341
1094132111,3sleep10-21swapper/122:42:271
996222101,5sleep299642-21grep22:35:272
676552101,3sleep24149465-21diskmemload22:16:192
41428702101,4sleep24142919-21ntpq21:00:212
41375352101,4sleep04137583-21ntpq20:55:190
41217252101,4sleep14121779-21ntpq20:40:201
41111322101,5sleep14111183-21ld20:30:181
41085302104,3sleep24110265-21kthreadcore20:30:152
41022222101,4sleep24102338-21irqstats20:25:152
40959222101,4sleep04095949-21munin-run20:19:590
40766682101,5sleep14078412-21cut20:00:171
40748822101,5sleep04075045-21cat20:00:000
40575592101,4sleep14057680-21ntpq19:40:201
40347612101,5sleep24036427-21grep19:20:182
2154432101,3sleep00-21swapper/023:46:090
1920512101,5sleep0192063-21ssh23:33:060
1443952100,2sleep20-21swapper/223:05:142
131812101,3sleep04149465-21diskmemload21:45:140
1007922101,3sleep04149465-21diskmemload22:36:460
4143216291,5sleep04143491-21head21:00:280
4132367290,1sleep10-21swapper/120:50:221
4132309291,4sleep04132366-21gcc20:50:220
4117086290,1sleep20-21swapper/220:35:292
4112692291,4sleep04112795-21perl20:35:150
4086283291,4sleep04087749-21cut20:10:180
4069240290,2sleep10-21swapper/119:50:351
4061905291,3sleep24062799-21memory19:45:192
4053982291,4sleep04054083-21fschecks_time19:40:160
4047475291,3sleep04047578-21/usr/sbin/munin19:30:260
4041914291,3sleep14041964-21ntpq19:25:231
185282291,3sleep1184778-21apt-get23:30:031
4138834281,3sleep14138853-21apt-get21:00:121
4138197280,1sleep10-21swapper/120:55:321
4128733281,3sleep24128910-21grep20:50:152
4111683280,1sleep00-21swapper/020:30:260
4103483281,3sleep04105291-21kthreadcore20:25:170
4096465280,2sleep20-21swapper/220:20:012
4085348280,1sleep20-21swapper/220:09:042
4081707281,3sleep04083406-21kthreadcore20:05:170
4079269280,1sleep20-21swapper/220:00:212
4058968280,5sleep10-21swapper/119:45:141
4027578281,3sleep24027787-21hddtemp_smartct19:15:132
4138367270,1sleep20-21swapper/220:57:422
4127617270,1sleep10-21swapper/120:45:381
4126942270,2sleep20-21swapper/220:45:192
4117725270,2sleep00-21swapper/020:40:120
4106671270,4sleep10-21swapper/120:28:031
4080430270,1sleep10-21swapper/120:05:061
4069538270,2sleep20-21swapper/219:54:202
4059500271,2sleep00-21swapper/019:45:170
4048441271,2sleep00-21swapper/019:35:110
4042821270,1sleep00-21swapper/019:29:100
4042778270,1sleep20-21swapper/219:28:362
4024785270,1sleep00-21swapper/019:11:340
4023074271,3sleep24023902-21missed_timers19:10:212
4122587260,1sleep20-21swapper/220:43:422
4117395260,1sleep10-21swapper/120:39:251
4101330260,1sleep00-21swapper/020:22:450
4090431260,3sleep20-21swapper/220:12:162
4090332260,1sleep10-21swapper/120:11:001
4069873260,2sleep20-21swapper/219:55:132
4067706261,2sleep00-21swapper/019:50:180
4053706260,2sleep20-21swapper/219:40:132
4048085260,1sleep10-21swapper/119:33:591
4037490260,1sleep00-21swapper/019:23:040
4027311261,2sleep00-21swapper/019:15:110
4127015250,1sleep00-21swapper/020:45:210
4090698250,1sleep20-21swapper/220:15:012
4051839250,2sleep20-21swapper/219:35:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional