You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-27 - 05:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot7s.osadl.org (updated Wed Aug 27, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23163210576,10sleep10-21swapper/119:08:111
2299729985,9sleep30-21swapper/319:06:043
2326427457,6sleep20-21swapper/219:09:282
2307327460,9sleep00-21swapper/019:07:020
2339799280,28cyclictest0-21swapper/221:12:102
2339799250,0cyclictest0-21swapper/222:45:532
2339799230,23cyclictest0-21swapper/221:45:122
2339599221,2cyclictest0-21swapper/123:00:151
2340199210,17cyclictest0-21swapper/319:30:163
2339599200,0cyclictest0-21swapper/121:51:371
2339199200,18cyclictest0-21swapper/019:10:130
2339799199,7cyclictest30136-21apache223:43:562
2339799190,19cyclictest0-21swapper/223:25:462
23391991918,1cyclictest0-21swapper/023:38:090
2339199190,3cyclictest0-21swapper/021:37:280
2339199190,2cyclictest0-21swapper/023:28:390
23401991817,1cyclictest0-21swapper/321:55:153
2340199180,1cyclictest0-21swapper/321:16:263
2339199183,5cyclictest20987-21apt-get23:10:190
2339199182,1cyclictest0-21swapper/000:10:150
2339199180,1cyclictest0-21swapper/000:22:420
2340199170,17cyclictest0-21swapper/322:53:543
2339799173,13cyclictest13623-21kernelversion19:55:202
23395991716,1cyclictest0-21swapper/122:27:381
2339599171,1cyclictest141rcu_preempt22:41:161
2339599171,12cyclictest0-21swapper/100:20:131
2339599170,16cyclictest0-21swapper/121:55:201
23391991717,0cyclictest0-21swapper/022:08:540
2339199170,3cyclictest0-21swapper/000:04:520
2339199170,3cyclictest0-21swapper/000:04:510
2339199170,1cyclictest0-21swapper/021:00:150
2340199160,1cyclictest0-21swapper/323:57:083
2339599163,12cyclictest23626-21apt-get19:10:141
2339199165,4cyclictest0-21swapper/021:45:130
2339199161,8cyclictest25186-21idleruntime-cro22:49:590
23391991610,5cyclictest30123-21apache223:52:430
2339799150,15cyclictest0-21swapper/200:38:492
2339799150,0cyclictest0-21swapper/221:30:242
2339599152,11cyclictest0-21swapper/119:32:571
2339199150,1cyclictest0-21swapper/021:33:420
21942150,0chrt0-21swapper/021:23:080
210982150,0sleep20-21swapper/200:22:262
2340199142,2cyclictest141rcu_preempt00:29:113
2339799142,12cyclictest6380-21ssh21:50:182
2339799141,1cyclictest29346-21/usr/sbin/munin20:30:212
2339599144,9cyclictest0-21swapper/120:15:161
23395991412,2cyclictest30136-21apache200:03:321
23395991412,2cyclictest30136-21apache200:03:311
23395991412,1cyclictest241rcuc/123:50:011
23391991413,1cyclictest0-21swapper/023:58:560
23391991410,2cyclictest2064-21ssh22:10:140
2339199140,13cyclictest8275-21head21:50:310
60732131,1sleep316014-21diskmemload00:36:323
2340199139,3cyclictest5970-21diskstats23:45:223
23401991310,2cyclictest6074-21head22:35:233
23401991310,2cyclictest5804-21grep19:40:163
23401991310,2cyclictest30943-21cat20:30:253
23401991310,2cyclictest1940-21grep22:10:143
23401991310,2cyclictest1897-21grep22:55:273
23401991310,2cyclictest12747-21grep23:50:183
2339799139,3cyclictest27824-21df_abs22:05:172
2339799139,3cyclictest20851-21grep22:00:202
2339799137,2cyclictest24496-21munin-run21:40:002
2339799133,4cyclictest224-21systemd-journal22:21:292
2339799131,11cyclictest9375-21grep20:55:152
23397991310,2cyclictest4815-21grep21:25:172
23397991310,2cyclictest32329-21sed19:25:262
23397991310,2cyclictest31077-21uname21:20:192
23397991310,2cyclictest26809-21ssh00:00:202
23397991310,2cyclictest26809-21ssh00:00:192
23397991310,2cyclictest25611-21grep23:35:282
2339799130,3cyclictest0-21swapper/222:31:012
2339799130,1cyclictest0-21swapper/221:42:152
2339799130,13cyclictest0-21swapper/222:54:262
2339799130,12cyclictest32298-21grep00:05:192
2339799130,12cyclictest24373-21ssh22:25:212
2339799130,12cyclictest0-21swapper/222:16:412
23395991311,1cyclictest25-21ksoftirqd/122:05:551
23395991310,2cyclictest9768-21grep22:15:181
23395991310,2cyclictest9314-21wc00:40:001
23395991310,2cyclictest4278-21uname19:35:211
23395991310,2cyclictest32292-21tune2fs00:05:171
23395991310,2cyclictest2236-21tr20:40:151
23395991310,2cyclictest19514-21grep22:45:171
2339599130,1cyclictest0-21swapper/122:55:221
2339599130,12cyclictest25540-21ssh21:40:191
2339599130,12cyclictest0-21swapper/123:23:261
2339199139,3cyclictest30025-21grep19:20:270
2339199139,3cyclictest28641-21grep20:25:260
2339199132,6cyclictest5290-21munin-run19:40:000
23391991312,1cyclictest0-21swapper/023:42:460
23391991310,2cyclictest4575-21cron00:10:000
23391991310,2cyclictest23195-21/usr/sbin/munin21:15:310
23391991310,2cyclictest21466-21wc22:00:240
23391991310,2cyclictest17482-21tune2fs23:30:180
23391991310,2cyclictest12519-21cut22:40:230
34372121,4sleep03683-21grep20:40:270
2340199129,2cyclictest884-21cat00:05:283
2340199129,2cyclictest3541-21cat20:40:253
2340199129,2cyclictest28506-21sshd20:25:273
2340199129,2cyclictest26520-21cron22:05:013
2340199129,2cyclictest22479-21grep20:15:173
2340199129,2cyclictest17294-21grep21:10:193
2340199129,2cyclictest13391-21ssh22:40:313
2340199120,2cyclictest0-21swapper/323:05:173
2339799129,2cyclictest6302-21head19:40:182
2339799129,2cyclictest30051-21sed19:20:282
2339799129,2cyclictest24720-21grep19:10:202
2339799129,2cyclictest22811-21sed23:10:322
2339799129,2cyclictest19594-21basename20:10:002
2339799129,2cyclictest19025-21cut23:55:162
2339799129,2cyclictest13683-21ssh23:50:272
2339799129,2cyclictest13217-21sed00:15:272
2339799128,4cyclictest8192-21ssh00:11:442
2339799128,3cyclictest1814-21taskset22:10:112
23397991210,1cyclictest27293-21grep20:25:152
23397991210,1cyclictest19856-21basename20:10:182
2339799120,1cyclictest0-21swapper/223:24:142
2339599129,2cyclictest6555-21grep22:35:271
2339599129,2cyclictest29141-21sh00:30:011
2339599129,2cyclictest25-21ksoftirqd/100:10:121
2339599129,2cyclictest20126-21aten2_r7power_e22:00:171
2339599129,2cyclictest17554-21tail23:30:181
2339599129,2cyclictest16003-21cron21:10:001
2339599129,2cyclictest14811-21sed23:05:201
2339599129,2cyclictest14114-21chrt23:27:511
2339599129,2cyclictest12108-21chrt23:50:151
2339599128,3cyclictest29639-21head19:20:231
2339599128,3cyclictest241rcuc/120:40:001
2339599128,3cyclictest20810-21sed23:55:281
2339599128,3cyclictest17825-21wc22:20:301
23395991210,1cyclictest14794-21cron19:59:591
2339199129,2cyclictest6558-21cat23:45:260
2339199129,2cyclictest4215-21grep23:20:280
2339199129,2cyclictest31730-21wc22:30:270
2339199129,2cyclictest26809-21wc20:25:010
2339199129,2cyclictest26360-21sed21:40:260
2339199129,2cyclictest1979-21head22:55:270
2339199129,2cyclictest13117-21wc00:15:260
2339199128,3cyclictest4922-21wc00:35:270
2339199123,8cyclictest18246-21gltestperf20:05:200
23391991211,1cyclictest13-21ksoftirqd/000:25:230
2339199120,4cyclictest0-21swapper/021:57:190
2340199119,1cyclictest4791-21grep21:25:173
2340199119,1cyclictest15819-21head20:00:183
2340199119,1cyclictest10888-21grep22:15:283
2340199118,2cyclictest9406-21tr20:55:143
2340199118,2cyclictest5733-21df00:10:163
2340199118,2cyclictest566-21ssh21:45:193
2340199118,2cyclictest27856-21ssh23:15:103
2340199118,2cyclictest27528-21sort00:00:283
2340199118,2cyclictest27528-21sort00:00:273
2340199118,2cyclictest19319-21ssh22:45:173
2340199118,2cyclictest18375-21grep00:20:173
2340199118,2cyclictest18123-21chrt23:30:223
2340199118,2cyclictest1291-21runrttasks22:20:163
2340199118,2cyclictest12590-21ssh21:30:203
2340199118,2cyclictest10524-21cat19:50:163
2340199113,4cyclictest0-21swapper/322:25:153
2340199111,9cyclictest16014-21diskmemload21:23:383
2339799118,2cyclictest7633-21ssh23:00:172
2339799118,2cyclictest5642-21if_enp2s022:35:192
2339799118,2cyclictest5111-21ssh23:45:072
2339799118,2cyclictest415-21cat20:35:262
2339799118,2cyclictest3942-21head19:35:182
2339799118,2cyclictest32384-21ssh22:55:062
2339799118,2cyclictest26955-21ssh21:17:202
2339799118,2cyclictest24824-21tr20:20:162
2339799118,2cyclictest24014-21df00:25:172
2339799118,2cyclictest2202-21cat20:40:142
2339799118,2cyclictest18103-21cut23:30:222
2339799118,2cyclictest17604-21ssh23:07:112
2339799118,2cyclictest14514-21ntpq21:55:232
2339799118,2cyclictest12591-21head22:40:232
2339799118,2cyclictest12257-21head21:00:182
2339799117,3cyclictest29715-21tr00:30:152
2339799117,2cyclictest29646-21sh23:15:302
2339799117,1cyclictest2703-21seq19:32:342
2339799110,10cyclictest8880-21basename20:55:002
2339599119,1cyclictest11590-21grep21:00:151
2339599118,2cyclictest9389-21tr20:55:151
2339599118,2cyclictest8853-21cron20:55:001
2339599118,2cyclictest5611-21head20:45:271
2339599118,2cyclictest5592-21wc21:25:211
2339599118,2cyclictest32477-21grep19:25:261
2339599118,2cyclictest31090-21grep22:30:221
2339599118,2cyclictest30954-21ssh23:40:201
2339599118,2cyclictest30600-21cut00:30:211
2339599118,2cyclictest29321-21ssh23:15:251
2339599118,2cyclictest25347-21tail20:20:191
2339599118,2cyclictest25184-21wc23:35:241
2339599118,2cyclictest24377-21ssh21:15:241
2339599118,2cyclictest22933-21wc21:14:591
2339599118,2cyclictest21128-21cut23:10:201
2339599118,2cyclictest19623-21wc21:35:271
2339599118,2cyclictest18908-21sed20:05:271
2339599118,2cyclictest1720-21ssh22:10:051
2339599118,2cyclictest12716-21tr00:15:231
2339599117,3cyclictest1335-21grep21:45:271
2339599110,10cyclictest15449-21ssh21:32:401
2339199118,2cyclictest9388-21tr20:55:150
2339199118,2cyclictest8827-21grep23:00:270
2339199118,2cyclictest7239-21df20:50:170
2339199118,2cyclictest5896-21sed21:25:240
2339199118,2cyclictest32307-21rm00:31:290
2339199118,2cyclictest31813-21bc20:35:170
2339199118,2cyclictest31092-21tr19:25:150
2339199118,2cyclictest29607-21tr20:30:150
2339199118,2cyclictest26441-21tail22:50:190
2339199118,2cyclictest23138-21apache_processe22:25:150
2339199118,2cyclictest2253-21ls23:19:590
2339199118,2cyclictest18953-21ssh22:21:250
2339199118,2cyclictest16348-21ssh21:10:060
2339199118,2cyclictest14827-21awk20:00:000
2339199118,2cyclictest13934-21sh23:05:140
2339199118,2cyclictest12807-21ssh22:17:190
2339199118,2cyclictest11974-21sed19:50:270
2339199117,3cyclictest17349-21ls20:05:010
2339199117,2cyclictest26778-21cut19:15:190
2339199110,10cyclictest5621-21ssh22:35:190
2340199108,2cyclictest24194-21expr23:35:173
2340199108,1cyclictest79012sleep319:45:063
2340199108,1cyclictest6802-21ssh23:00:023
2340199108,1cyclictest6763-21basename20:50:153
2340199108,1cyclictest6235-21sh21:50:113
2340199108,1cyclictest3602-21expr19:35:163
2340199108,1cyclictest32006-21egrep20:35:213
2340199108,1cyclictest31308-21cut22:30:243
2340199108,1cyclictest31074-21ssh22:07:133
2340199108,1cyclictest2835-21sh23:20:183
2340199108,1cyclictest2571-21sh23:42:473
2340199108,1cyclictest18194-21grep20:05:193
2340199108,1cyclictest11382-21ssh00:15:133
2340199107,2cyclictest9583-21ssh23:25:083
2340199107,2cyclictest523-21awk19:29:593
2340199107,2cyclictest32133-21ssh00:31:173
2340199107,2cyclictest24841-21ssh21:40:083
2340199107,2cyclictest21456-21grep20:10:303
2340199107,2cyclictest20663-21sh23:10:003
2340199107,2cyclictest18103-21ssh21:35:103
2340199107,2cyclictest11799-21grep21:00:173
2339799108,1cyclictest15372-21expr20:00:162
2339799108,1cyclictest14760-21if_enp2s021:05:202
2339799108,1cyclictest10687-21expr19:50:162
2339799107,2cyclictest9100-21ntpq19:45:232
2339799107,2cyclictest27658-21grep19:15:272
2339799107,2cyclictest23827-21date20:15:282
2339799104,2cyclictest224-21systemd-journal20:50:012
2339599109,1cyclictest0-21swapper/120:00:141
2339599108,1cyclictest31619-21sort20:35:011
2339599108,1cyclictest28481-21wc20:25:261
2339599108,1cyclictest123862sleep119:54:391
2339599107,2cyclictest27718-21ssh22:50:391
2339599107,2cyclictest27230-21tr19:15:231
2339599106,3cyclictest30895-21cut21:20:171
2339599100,3cyclictest0-21swapper/119:45:161
2339199108,1cyclictest7613-21chrt19:44:240
2339199108,1cyclictest5377-21cat20:45:230
2339199108,1cyclictest27542sleep019:33:140
2339199108,1cyclictest19908-21cut20:10:180
2339199108,1cyclictest15605-21sed21:05:290
2339199107,2cyclictest9908-21taskset19:48:220
234019998,1cyclictest28482-21aten2_r7power_p19:20:153
234019998,1cyclictest15061-21ntpq21:05:223
234019997,1cyclictest6678-21sed20:50:013
234019997,1cyclictest27162-21ntpq19:15:223
234019997,1cyclictest12714-21taskset19:55:153
234019996,2cyclictest25381-21grep20:20:193
234019992,5cyclictest25244-21grep19:10:263
233959996,2cyclictest21699-21taskset20:11:551
233959995,3cyclictest6694-21tr19:40:221
233919997,1cyclictest24185-21chrt20:18:090
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional