You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-02 - 03:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Wed Jul 02, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2146021140,0sleep10-21swapper/123:37:271
9354210461,39sleep30-21swapper/319:09:553
9354210461,39sleep30-21swapper/319:09:553
9150210288,9sleep10-21swapper/119:07:161
9150210288,9sleep10-21swapper/119:07:151
908028370,9sleep00-21swapper/019:06:220
908028370,9sleep00-21swapper/019:06:210
905727868,6sleep20-21swapper/219:06:042
905727868,6sleep20-21swapper/219:06:042
944999280,28cyclictest0-21swapper/222:39:212
9445992621,4cyclictest29839-21cat23:20:271
944999250,25cyclictest0-21swapper/222:40:012
944999250,25cyclictest0-21swapper/200:28:502
9445992518,6cyclictest2711-21diskmemload21:42:371
9453992421,2cyclictest4194-21hddtemp_smartct23:25:233
9453992421,2cyclictest1998-21apache_volume00:10:173
9453992421,2cyclictest1998-21apache_volume00:10:163
9449992421,2cyclictest3961-21df_inode23:25:212
944599245,15cyclictest0-21swapper/122:25:181
9440992421,2cyclictest15500-21munin-node23:15:010
9440992417,3cyclictest224-21systemd-journal21:20:000
9453992320,2cyclictest29242-21diskstats22:35:203
9453992320,2cyclictest10641-21apache_accesses22:45:143
9449992321,1cyclictest14045-21apache_processe21:40:122
9440992321,1cyclictest1291-21runrttasks00:22:480
9449992219,2cyclictest1291-21runrttasks23:22:302
944599229,5cyclictest2711-21diskmemload23:15:111
944599229,1cyclictest2711-21diskmemload22:30:531
9445992219,2cyclictest3530-21/usr/sbin/munin22:40:191
9445992217,2cyclictest2711-21diskmemload22:49:391
9445992213,8cyclictest2711-21diskmemload00:00:451
944099224,9cyclictest0-21swapper/000:25:140
9440992219,2cyclictest32293-21apache_volume22:15:190
9440992219,2cyclictest25469-21df22:10:180
9440992219,2cyclictest25239-21df_inode22:55:180
9440992219,2cyclictest15500-21munin-node21:37:250
9440992219,2cyclictest14674-21diskstats21:40:170
9453992119,1cyclictest1291-21runrttasks22:16:523
9449992119,1cyclictest21173-21apache_processe21:45:122
944999210,21cyclictest0-21swapper/221:38:072
944599217,13cyclictest0-21swapper/121:35:121
944599216,13cyclictest0-21swapper/123:06:331
9445992118,2cyclictest15500-21munin-node22:57:211
9445992116,3cyclictest2711-21diskmemload22:17:431
9445992112,8cyclictest2711-21diskmemload00:28:101
9440992118,2cyclictest32341-21df_inode23:00:190
9440992118,2cyclictest1291-21runrttasks22:40:270
9453992017,2cyclictest32351-21df_inode23:45:203
9453992017,2cyclictest10981-21df23:30:193
9449992017,2cyclictest0-21swapper/200:36:282
944999200,5cyclictest0-21swapper/200:15:122
944599208,10cyclictest0-21swapper/121:34:571
9445992018,2cyclictest0-21swapper/119:35:141
9445992016,1cyclictest2711-21diskmemload23:57:151
9445992010,6cyclictest0-21swapper/100:20:001
944099203,7cyclictest2588-21munin-run23:25:000
9453991916,2cyclictest25605-21ssh21:25:213
9449991917,1cyclictest447-21df_inode22:15:242
9449991916,2cyclictest1215-21/usr/sbin/munin23:00:292
944999190,19cyclictest0-21swapper/222:55:182
944599198,9cyclictest2711-21diskmemload21:55:161
944599198,8cyclictest2711-21diskmemload00:20:441
9445991916,2cyclictest0-21swapper/122:51:261
9445991911,7cyclictest2711-21diskmemload21:22:481
944599190,7cyclictest0-21swapper/119:15:141
944599190,19cyclictest0-21swapper/121:16:461
9440991917,1cyclictest1291-21runrttasks22:28:060
9440991916,2cyclictest25830-21apache_volume00:30:140
9453991816,1cyclictest9774-21latency_hist21:15:003
9453991816,1cyclictest15882-21ntp_states00:20:223
9453991815,2cyclictest26677-21hddtemp_smartct00:30:183
9453991815,2cyclictest13159-21munin-run23:10:003
945399180,2cyclictest0-21swapper/323:50:453
9449991816,1cyclictest7838-21grep22:20:192
9449991816,1cyclictest1291-21runrttasks23:40:592
944999180,18cyclictest0-21swapper/222:01:572
944599189,8cyclictest0-21swapper/123:32:171
944599188,9cyclictest0-21swapper/122:00:151
9445991816,1cyclictest25424-21latency_hist00:30:011
9445991815,2cyclictest1291-21runrttasks00:08:541
9445991811,6cyclictest11545-21fschecks_time20:20:171
9445991811,6cyclictest0-21swapper/121:25:211
9445991810,6cyclictest0-21swapper/119:25:171
9440991815,2cyclictest25903-21/usr/sbin/munin21:25:230
9453991714,2cyclictest21567-21/usr/sbin/munin23:15:183
945399170,14cyclictest0-21swapper/321:43:283
944999170,17cyclictest0-21swapper/223:32:462
944599179,4cyclictest0-21swapper/100:35:221
944599178,6cyclictest0-21swapper/122:20:071
944599177,9cyclictest28048-21aten2_r7power_v20:55:151
9445991714,2cyclictest0-21swapper/122:05:141
9445991713,3cyclictest16569-21sh19:25:011
9440991717,0cyclictest0-21swapper/000:11:360
9440991717,0cyclictest0-21swapper/000:11:360
9453991614,1cyclictest32396-21cstates00:35:173
9453991614,1cyclictest20325-21idleruntime-cro23:14:593
9453991614,1cyclictest1291-21runrttasks21:47:283
944599167,7cyclictest0-21swapper/123:00:171
944599165,10cyclictest0-21swapper/123:41:061
944599163,6cyclictest0-21swapper/119:15:011
944599163,4cyclictest5853-21diskmemload23:26:041
9445991613,2cyclictest3010-21irqstats00:10:241
9445991613,2cyclictest3010-21irqstats00:10:231
9445991613,2cyclictest1291-21runrttasks21:52:351
9440991613,2cyclictest9118-21df_abs00:15:150
9440991613,2cyclictest15500-21munin-node21:22:560
944099160,1cyclictest0-21swapper/021:58:190
9453991512,2cyclictest22001-21diskstats22:30:183
9453991512,2cyclictest1291-21runrttasks23:03:223
9453991512,2cyclictest1291-21runrttasks21:30:073
944999152,3cyclictest32-21ksoftirqd/222:30:482
944999150,15cyclictest0-21swapper/200:10:182
944999150,15cyclictest0-21swapper/200:10:172
944999150,15cyclictest0-21swapper/200:09:392
944599156,3cyclictest7197-21sh23:50:171
9445991512,2cyclictest0-21swapper/122:11:031
944599150,13cyclictest30642-21sed19:50:271
944099153,6cyclictest0-21swapper/021:10:000
9440991512,2cyclictest31713-21apache_accesses23:45:160
944099150,15cyclictest0-21swapper/023:35:350
9453991414,0cyclictest0-21swapper/300:19:343
9453991411,2cyclictest31224-21kworker/u8:222:10:223
9453991411,2cyclictest27855-21cat20:55:013
9453991411,2cyclictest17324-21ssh23:35:063
9453991411,2cyclictest1291-21runrttasks23:57:423
9453991410,2cyclictest25246-21forks23:40:203
945399140,12cyclictest30337-21cat21:00:013
945399140,11cyclictest0-21swapper/320:30:153
944999149,4cyclictest26054-21wc21:25:252
9449991413,1cyclictest0-21swapper/220:30:152
944999140,1cyclictest0-21swapper/200:31:222
944999140,14cyclictest0-21swapper/223:45:172
944999140,14cyclictest0-21swapper/222:47:512
944999140,12cyclictest0-21swapper/222:50:282
944599149,4cyclictest0-21swapper/121:48:391
944599148,4cyclictest1119-21grep23:45:301
944599147,6cyclictest0-21swapper/119:30:151
9445991412,1cyclictest2711-21diskmemload21:10:581
9445991411,2cyclictest20331-21munin-run20:40:001
9445991410,3cyclictest5821-21turbostat.cron20:10:001
944599140,10cyclictest0-21swapper/123:10:101
9440991410,3cyclictest7125-21grep22:20:170
944099140,1cyclictest0-21swapper/023:41:360
9453991312,1cyclictest0-21swapper/300:25:233
9453991311,1cyclictest207872sleep322:06:333
9453991310,2cyclictest30224-21cat21:50:263
9453991310,2cyclictest18956-21tr20:35:213
9453991310,2cyclictest15500-21munin-node22:02:013
945399130,9cyclictest0-21swapper/323:20:143
945399130,2cyclictest0-21swapper/321:55:453
944999139,3cyclictest8955-21sed20:15:162
944999139,3cyclictest23903-21taskset22:09:312
944999139,2cyclictest26156-21grep22:10:222
944999137,5cyclictest29663-21apache220:10:002
944999133,3cyclictest0-21swapper/221:20:152
944999133,1cyclictest2711-21diskmemload23:15:392
944999131,1cyclictest141rcu_preempt21:13:422
9449991310,2cyclictest7092-21df_abs23:05:212
9449991310,2cyclictest24867-21chrt00:02:512
944999130,13cyclictest0-21swapper/221:16:232
944999130,12cyclictest11118-21grep20:20:142
944599139,3cyclictest29740-21wc22:35:231
944599138,4cyclictest26158-21date19:45:001
944599138,3cyclictest26205-21grep20:50:151
944099139,3cyclictest20834-21grep19:30:280
9440991310,2cyclictest24695-21cat20:45:270
9440991310,2cyclictest22175-21grep22:30:200
944099130,1cyclictest0-21swapper/023:58:030
944099130,12cyclictest32723-21grep00:35:180
944099130,12cyclictest30862-21grep21:00:160
944099130,12cyclictest30862-21grep21:00:150
945399129,2cyclictest9168-21tr20:15:173
945399129,2cyclictest22874-21cat20:45:013
945399129,2cyclictest21804-21grep00:00:233
945399129,2cyclictest12136-21tr19:15:153
945399128,3cyclictest7949-21cat21:35:203
945399126,6cyclictest15536-21open_files22:25:263
945399126,4cyclictest16905-21turbostat.cron21:20:013
945399125,5cyclictest7744-21snmpd20:02:003
945399124,5cyclictest0-21swapper/319:35:013
945399122,6cyclictest11799-21latency_hist19:15:013
945399120,9cyclictest0-21swapper/322:20:283
944999129,2cyclictest6767-21grep20:10:182
944999129,2cyclictest29170-21grep19:50:162
944999129,2cyclictest22255-21grep20:40:272
944999129,2cyclictest15395-21head00:20:182
944999129,2cyclictest13852-21sort20:25:192
944999128,3cyclictest6515-21tr23:50:132
9449991212,0cyclictest0-21swapper/223:55:252
9449991210,1cyclictest4185-21cat21:55:212
944599129,2cyclictest14667-21wc20:25:281
944599128,3cyclictest7027-21taskset20:10:201
944599127,3cyclictest8748-21tr20:15:151
944599123,2cyclictest224-21systemd-journal19:56:591
9445991210,1cyclictest0-21swapper/121:05:141
944099129,2cyclictest7463-21sed23:50:190
944099129,2cyclictest22329-21grep19:35:180
944099129,2cyclictest20482-21cat20:40:010
944099129,2cyclictest12342-21grep19:15:150
944099128,3cyclictest7164-21sort23:05:210
944099128,3cyclictest27349-21grep19:45:190
944099128,3cyclictest13130-21cat20:25:000
944099128,2cyclictest648-21grep19:55:280
944099124,4cyclictest224-21systemd-journal21:00:010
944099123,8cyclictest15500-21munin-node23:29:200
944099122,7cyclictest2711-21diskmemload22:39:050
944099121,8cyclictest2711-21diskmemload22:45:240
944099120,3cyclictest0-21swapper/022:04:120
945399118,2cyclictest857-21cat21:05:163
945399118,2cyclictest4265-21ssh22:40:233
945399118,2cyclictest32719-21turbostat21:05:013
945399118,2cyclictest32719-21turbostat21:05:003
945399118,2cyclictest32565-21ls19:55:243
945399118,2cyclictest29417-21df19:50:153
945399118,2cyclictest28322-21grep00:05:203
945399118,2cyclictest25005-21cat19:40:223
945399118,2cyclictest23948-21expr20:45:193
945399118,2cyclictest21790-21cat19:35:163
945399118,2cyclictest19480-21grep22:50:283
945399118,2cyclictest17537-21tr21:20:153
945399118,2cyclictest15071-21taskset20:26:313
945399118,2cyclictest14535-21tr19:20:153
945399117,3cyclictest25378-21grep22:55:203
944999119,1cyclictest136452sleep223:10:142
944999118,2cyclictest3586-21ssh21:32:352
944999118,2cyclictest31659-21grep19:55:172
944999118,2cyclictest28645-21head20:55:182
944999118,2cyclictest27353-21head19:45:202
944999118,2cyclictest19053-21ssh22:28:132
944999113,4cyclictest224-21systemd-journal19:16:352
9449991111,0cyclictest0-21swapper/220:50:142
944599118,2cyclictest32300-21head21:00:281
944599118,2cyclictest32300-21head21:00:271
944599118,2cyclictest25344-21cat20:50:011
944599117,2cyclictest21150-21grep20:40:161
944099118,2cyclictest3876-21sort21:10:190
944099118,2cyclictest3786-21grep20:05:150
944099118,2cyclictest29559-21grep19:50:180
944099118,2cyclictest26087-21apt-get19:45:000
944099118,2cyclictest25814-21grep20:50:140
944099118,2cyclictest2527-21sh21:31:320
944099118,2cyclictest22289-21sed00:00:290
944099118,2cyclictest21493-21grep21:45:140
944099118,2cyclictest2122-21tail20:00:200
944099118,2cyclictest20957-21ssh22:51:580
944099118,2cyclictest19576-21grep22:05:260
944099118,2cyclictest13486-21ssh23:31:350
944099118,2cyclictest11016-21run-parts19:10:240
944099118,2cyclictest10281-21chrt20:16:180
944099117,3cyclictest32708-21ssh21:52:400
944099115,5cyclictest15520-21latency_hist20:30:010
945399108,1cyclictest54952sleep320:05:563
945399108,1cyclictest18413-21cat19:25:293
945399107,2cyclictest8034-21taskset20:13:013
945399107,2cyclictest26838-21tr19:45:183
945399107,2cyclictest11844-21cat20:20:203
944999108,1cyclictest23335-21egrep20:45:162
944999108,1cyclictest233292sleep219:35:322
944999108,1cyclictest20066-21chrt19:30:202
944999108,1cyclictest19007-21which20:35:212
944999108,1cyclictest16917-21aten2_r7power_p19:25:152
944999108,1cyclictest1510-21grep21:05:212
944999107,2cyclictest9972-21tr19:10:162
944999107,2cyclictest28289-21chrt21:50:082
944999107,2cyclictest24248-21grep19:40:182
944999107,2cyclictest17319-21ssh23:35:062
944999107,2cyclictest15194-21grep19:20:182
944599108,1cyclictest32172sleep120:02:091
944599101,5cyclictest0-21swapper/119:45:151
944599100,6cyclictest224-21systemd-journal20:32:351
944099108,1cyclictest205632sleep020:40:050
944099108,1cyclictest16727-21chrt20:30:200
944099108,1cyclictest14825-21ssh23:10:210
944099107,2cyclictest29002-21expr00:05:250
944099100,9cyclictest18759-21taskset19:27:460
94499997,1cyclictest305602sleep221:00:042
94499997,1cyclictest305602sleep221:00:032
94499993,5cyclictest2980-21grep20:00:292
94409997,1cyclictest6160-21wget20:10:160
94409994,2cyclictest14466-21apache_volume19:20:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional