You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-19 - 10:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sat Jul 19, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1729829884,9sleep30-21swapper/319:09:543
1696429676,9sleep10-21swapper/119:05:341
1718327762,10sleep00-21swapper/019:08:220
1719027464,6sleep20-21swapper/219:08:282
17405992725,1cyclictest0-21swapper/300:12:083
17401992624,1cyclictest12631-21sed21:30:262
1740199260,22cyclictest0-21swapper/220:20:132
1740199242,1cyclictest0-21swapper/222:05:122
1739999231,21cyclictest0-21swapper/123:35:431
17396992221,1cyclictest0-21swapper/000:20:530
316142210,0chrt0-21swapper/022:22:430
1739999217,2cyclictest9986-21diskmemload23:51:131
1740599200,19cyclictest0-21swapper/323:40:283
28372190,1chrt2838-21/usr/sbin/munin00:05:271
17401991918,1cyclictest21238-21wget20:25:162
17401991917,1cyclictest11910-21grep21:50:222
1740199190,1cyclictest0-21swapper/223:28:062
1739699191,18cyclictest0-21swapper/022:51:350
1740199187,9cyclictest7744-21snmpd00:26:392
17401991813,3cyclictest9986-21diskmemload22:59:272
1740199180,1cyclictest141rcu_preempt00:12:422
1740199180,1cyclictest0-21swapper/200:00:542
1740199180,17cyclictest0-21swapper/219:20:182
17396991818,0cyclictest0-21swapper/000:05:460
17396991813,3cyclictest13-21ksoftirqd/021:47:310
17396991813,2cyclictest13-21ksoftirqd/021:44:590
216252170,0sleep20-21swapper/200:20:022
17399991716,1cyclictest25-21ksoftirqd/121:16:541
17399991710,5cyclictest26221-21perf22:40:011
1739699178,7cyclictest9897-21munin-run23:09:590
17396991715,1cyclictest13-21ksoftirqd/021:26:180
17396991714,2cyclictest13-21ksoftirqd/023:35:040
17396991713,2cyclictest13-21ksoftirqd/022:07:520
1740599164,11cyclictest0-21swapper/321:00:143
1740199160,3cyclictest0-21swapper/221:18:082
1740199160,2cyclictest0-21swapper/223:40:462
1740199160,1cyclictest0-21swapper/222:22:322
1740199160,1cyclictest0-21swapper/222:03:012
1740199160,1cyclictest0-21swapper/221:40:592
1740199160,16cyclictest0-21swapper/223:47:162
1740199160,14cyclictest0-21swapper/223:18:032
1739999160,3cyclictest0-21swapper/122:22:511
1739699169,2cyclictest13-21ksoftirqd/023:18:060
1739699163,2cyclictest141rcu_preempt19:25:150
1739699162,13cyclictest18641-21apache_volume22:55:160
17396991614,2cyclictest13-21ksoftirqd/021:35:000
1739699160,1cyclictest141rcu_preempt23:28:010
1740599150,15cyclictest0-21swapper/321:30:103
1740199150,15cyclictest0-21swapper/219:40:182
1740199150,14cyclictest0-21swapper/222:37:192
1740199150,0cyclictest0-21swapper/222:44:482
1739999151,1cyclictest0-21swapper/123:40:091
1739699153,1cyclictest141rcu_preempt21:10:140
1740599149,3cyclictest1939-21dump-pmu-power22:05:003
1740599149,3cyclictest19191-21ssh23:55:173
1740199148,5cyclictest24664-21grep20:30:212
1740199145,9cyclictest7744-21snmpd00:16:582
1740199145,7cyclictest11728-21gdbus00:06:182
17401991411,3cyclictest3693-21apache222:18:502
17401991411,2cyclictest6505-21chrt22:27:062
17401991411,2cyclictest21453-21cat19:15:262
1740199140,1cyclictest0-21swapper/200:30:532
1740199140,0cyclictest0-21swapper/223:55:122
1739999148,2cyclictest224-21systemd-journal00:20:001
17399991413,1cyclictest0-21swapper/123:34:371
17399991411,2cyclictest10002-21turbostat21:50:001
17399991410,2cyclictest18177-21sed19:10:161
1739999140,14cyclictest0-21swapper/100:00:431
1739699149,1cyclictest13-21ksoftirqd/020:50:160
1739699146,7cyclictest17534-21munin-node23:53:110
1739699142,6cyclictest9979-21munin-run22:30:000
1739699140,2cyclictest0-21swapper/023:44:060
1739699140,0cyclictest141rcu_preempt22:37:000
1740599139,3cyclictest23681-21tr20:30:153
1740599139,3cyclictest16972-21grep00:15:283
1740599139,2cyclictest32350-21sed20:45:273
1740599132,9cyclictest26892-21sh22:20:143
17405991312,1cyclictest0-21swapper/323:53:443
17405991312,1cyclictest0-21swapper/323:28:153
1740599131,11cyclictest21229-21sh23:35:573
1740599131,11cyclictest0-21swapper/321:43:133
1740599131,10cyclictest0-21swapper/323:21:143
17405991310,2cyclictest951-21cron00:05:003
17405991310,2cyclictest7885-21wc19:55:193
17405991310,2cyclictest18904-21tr22:55:183
17405991310,2cyclictest17767-21grep19:10:153
17405991310,2cyclictest10807-21cut22:30:183
1740599130,2cyclictest0-21swapper/321:20:003
1740199139,3cyclictest9986-21diskmemload21:10:212
1740199139,3cyclictest10577-21grep00:35:172
17401991310,2cyclictest9009-21munin-plugin-st20:00:002
17401991310,2cyclictest4142-21sed23:05:272
17401991310,2cyclictest3290-21grep21:25:162
17401991310,2cyclictest30066-21grep20:40:282
17401991310,2cyclictest13789-21awk20:10:002
17401991310,2cyclictest10099-21wc21:50:002
1740199130,13cyclictest0-21swapper/223:36:312
1740199130,12cyclictest17871-21cat20:15:242
1740199130,0cyclictest0-21swapper/222:13:462
1739999139,3cyclictest12100-21wc21:30:211
1739999139,2cyclictest2776-21sh20:55:001
1739999135,7cyclictest17534-21munin-node23:06:361
1739999133,6cyclictest14056-21aten2_r7power_c20:10:131
1739999131,7cyclictest2457-21ssh22:05:131
1739999131,11cyclictest3900-21grep00:30:171
17399991310,2cyclictest2864-21ls00:30:011
17399991310,2cyclictest20637-21uniq19:15:181
17399991310,2cyclictest19500-21grep20:20:161
17399991310,2cyclictest18071-21cron22:55:011
1739999130,1cyclictest141rcu_preempt23:15:081
1739699139,3cyclictest9979-21cron21:10:000
1739699139,3cyclictest17188-21grep20:15:160
1739699136,3cyclictest13-21ksoftirqd/000:00:050
1739699132,6cyclictest9986-21diskmemload21:15:120
1739699132,6cyclictest26607-21/usr/sbin/munin23:00:120
17396991311,1cyclictest18711-21ssh22:15:130
17396991310,2cyclictest6283-21grep21:00:180
17396991310,2cyclictest29089-21tail20:40:180
17396991310,2cyclictest21711-21grep19:15:270
17396991310,2cyclictest21452-21tr20:25:170
17396991310,2cyclictest11447-21sort22:30:210
17396991310,2cyclictest10071-21cat23:50:000
1739699130,0cyclictest141rcu_preempt00:12:330
268652120,0sleep326881-21ssh00:23:373
1740599129,2cyclictest9261-21taskset22:09:213
1740599129,2cyclictest3453-21ssh23:05:213
1740599129,2cyclictest31105-21wc19:35:283
1740599129,2cyclictest29084-21ssh23:00:483
1740599129,2cyclictest28014-21wc21:20:203
1740599129,2cyclictest2322-21tail20:50:293
1740599129,2cyclictest19102-21grep20:20:143
1740599129,2cyclictest1796-21cat19:45:003
1740599129,2cyclictest13376-21grep20:05:293
1740599129,2cyclictest11715-21wc00:35:233
1740599128,3cyclictest5842-21wc19:50:233
1740599128,3cyclictest4524-21wc20:55:273
1740599128,3cyclictest4188-21expr21:45:273
1740599128,3cyclictest29412-21cut00:25:193
1740599128,3cyclictest19416-21cut21:55:173
17405991211,1cyclictest0-21swapper/300:08:483
1740599120,2cyclictest0-21swapper/323:30:333
1740199129,2cyclictest27082-21tr23:20:152
1740199129,2cyclictest19265-21df21:35:182
1740199129,2cyclictest18751-21tr19:10:212
1740199129,2cyclictest1812-21cat23:05:002
1740199129,2cyclictest11685-21grep22:30:212
17401991211,1cyclictest14862-21kworker/2:123:14:382
1740199120,1cyclictest0-21swapper/221:23:562
1740199120,12cyclictest0-21swapper/222:50:152
1739999129,2cyclictest4360-21ssh22:25:241
1739999129,2cyclictest31680-21basename19:40:001
1739999129,2cyclictest28011-21cut23:20:211
1739999129,2cyclictest27356-21cut21:40:141
1739999129,2cyclictest18112-21basename21:55:001
1739999129,2cyclictest10995-21cut00:35:191
1739999123,2cyclictest11684-21cat22:30:211
1739999122,8cyclictest26175-21turbostat.cron23:00:001
1739699129,2cyclictest29599-21cut19:35:160
1739699129,2cyclictest23645-21grep20:30:150
1739699129,2cyclictest19541-21tail21:35:200
1739699129,2cyclictest13359-21head20:05:290
1739699128,3cyclictest9570-21tr20:00:150
1739699122,1cyclictest141rcu_preempt00:31:580
1739699121,3cyclictest224-21systemd-journal20:50:000
1739699121,1cyclictest9986-21diskmemload21:22:470
17396991210,1cyclictest18088-21cron23:35:000
1740599118,2cyclictest5059-21ssh23:45:583
1740599118,2cyclictest3642-21grep00:30:163
1740599118,2cyclictest3551-21sed22:25:173
1740599118,2cyclictest29235-21sh22:40:533
1740599118,2cyclictest2882-21tr21:25:143
1740599118,2cyclictest26004-21sed19:25:233
1740599118,2cyclictest24096-21ssh22:18:133
1740599118,2cyclictest23160-21cut19:20:213
1740599118,2cyclictest19533-21grep22:35:193
1740599118,2cyclictest16499-21cut20:15:133
1740599118,2cyclictest15054-21ssh22:12:273
1740599118,2cyclictest15037-21sh21:12:293
1740599118,2cyclictest14184-21cut20:10:143
1740599118,2cyclictest11741-21tr23:10:193
1740599118,2cyclictest11484-21sh22:50:183
1740599118,2cyclictest10369-21sh21:50:073
1740599118,2cyclictest10043-21sed20:00:183
1740599117,3cyclictest19674-21ssh23:15:213
1740599117,3cyclictest19588-21cut21:35:203
1740599117,2cyclictest28849-21grep19:30:263
1740599117,2cyclictest2415-21apt-get22:45:143
1740599115,2cyclictest224-21systemd-journal20:30:003
1740199118,2cyclictest7979-21wget21:05:142
1740199118,2cyclictest5734-21grep21:00:152
1740199118,2cyclictest4405-21grep22:45:282
1740199118,2cyclictest30323-21head19:35:192
1740199118,2cyclictest27660-21sed20:35:302
1740199118,2cyclictest27660-21sed20:35:302
1740199118,2cyclictest25054-21tr19:25:162
1740199118,2cyclictest21899-21sh21:56:302
1740199118,2cyclictest11812-21ntpq23:30:222
1740199110,1cyclictest0-21swapper/220:45:142
1740199110,10cyclictest16275-21sh23:53:322
1739999118,2cyclictest8223-21ssh21:28:251
1739999118,2cyclictest5966-21cat19:50:261
1739999118,2cyclictest3402-21ssh23:25:181
1739999118,2cyclictest32441-21head20:45:271
1739999118,2cyclictest3164-21sed23:45:171
1739999118,2cyclictest29278-21basename19:34:591
1739999118,2cyclictest27085-21expr23:00:151
1739999118,2cyclictest26394-21grep19:25:271
1739999118,2cyclictest26247-21tr20:35:181
1739999118,2cyclictest26247-21tr20:35:181
1739999118,2cyclictest24328-21rm00:21:021
1739999118,2cyclictest21637-21cat20:25:171
1739999118,2cyclictest19875-21ntpq21:55:201
1739999118,2cyclictest18763-21cut21:35:161
1739999118,2cyclictest18550-21apt20:20:001
1739999118,2cyclictest14410-21rm21:12:011
1739999118,2cyclictest12903-21ssh23:10:531
1739999118,2cyclictest11244-21ssh00:11:221
1739999118,2cyclictest10053-21head20:00:181
1739999116,4cyclictest3693-21apache222:48:401
1739999111,5cyclictest5229-21perf21:00:011
1739999111,1cyclictest27054-21cpu21:20:161
1739999110,10cyclictest30358-21ssh22:01:451
1739999110,10cyclictest20153-21which23:55:241
1739699119,2cyclictest161rcuc/022:10:180
1739699118,2cyclictest8912-21chrt19:58:580
1739699118,2cyclictest5123-21ssh22:46:020
1739699118,2cyclictest31975-21ssh00:26:500
1739699118,2cyclictest30120-21ssh23:21:350
1739699118,2cyclictest27163-21apt-get19:30:130
1739699118,2cyclictest26798-21ssh22:00:140
1739699118,2cyclictest25807-21ssh21:59:390
1739699118,2cyclictest19279-21tail19:10:260
1739699118,2cyclictest18521-21ssh23:55:100
1739699118,2cyclictest17987-21ssh00:16:270
1739699118,2cyclictest15715-21grep20:10:250
1739699118,2cyclictest14239-21ssh21:51:510
1739699118,2cyclictest10160-21ssh00:35:070
1739699114,6cyclictest9986-21diskmemload22:40:330
1739699112,2cyclictest141rcu_preempt23:10:200
1740599108,1cyclictest277842sleep320:36:363
1740599108,1cyclictest277842sleep320:36:353
1740599107,2cyclictest8285-21expr21:05:153
1740599107,2cyclictest30301-21chrt20:43:083
1740599107,2cyclictest21772-21seq19:15:433
1740199108,1cyclictest3924-21basename20:55:192
1740199108,1cyclictest3797-21grep19:45:262
1740199107,2cyclictest1903-21tr20:50:262
1740199107,2cyclictest10060-21head20:00:182
1740199100,9cyclictest28547-21chrt19:30:232
1739999108,1cyclictest7950-21grep21:05:141
1739999108,1cyclictest31928-21taskset19:40:131
1739999108,1cyclictest223642sleep119:20:051
1739999108,1cyclictest18512-21ssh22:15:051
1739999108,1cyclictest10434-21ssh22:10:071
1739999107,2cyclictest29021-21sh22:40:431
1739999107,2cyclictest28803-21expr20:40:161
1739999107,2cyclictest23995-21expr20:30:151
1739999107,2cyclictest13557-21taskset20:07:161
1739999106,3cyclictest7120-21tr19:55:141
1739699108,1cyclictest3016-21taskset20:55:110
1739699108,1cyclictest24160-21chrt19:20:350
1739699107,2cyclictest3804-21tail19:45:260
1739699107,2cyclictest32194-21grep19:40:170
1739699107,2cyclictest19600-21grep20:20:160
1739699104,2cyclictest224-21systemd-journal20:39:110
1739699104,2cyclictest224-21systemd-journal20:39:100
174059997,1cyclictest3612-21sed19:45:253
174019997,1cyclictest14015-21acpi20:10:112
174019990,9cyclictest0-21swapper/219:50:142
173999997,1cyclictest5732-21expr21:00:151
173999997,1cyclictest1919-21dpkg19:45:001
173969997,1cyclictest6564-21taskset19:54:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional