You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-11 - 10:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue Nov 11, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3443210575,10sleep30-21swapper/319:06:083
3585210289,9sleep10-21swapper/119:07:551
343528564,6sleep00-21swapper/019:06:010
368328265,13sleep20-21swapper/219:09:142
381899465,21cyclictest847-21sh23:05:320
381899402,20cyclictest7775-21users23:10:280
3818993718,18cyclictest0-21swapper/022:55:520
382299310,2cyclictest0-21swapper/123:44:361
381899300,30cyclictest0-21swapper/023:17:120
381899300,30cyclictest0-21swapper/023:17:120
51422270,0sleep30-21swapper/322:00:243
382299273,21cyclictest0-21swapper/123:00:001
382299270,26cyclictest0-21swapper/121:05:221
382299261,1cyclictest29113-21diskmemload23:17:441
382299261,1cyclictest29113-21diskmemload23:17:441
3827992423,1cyclictest0-21swapper/223:59:552
382299245,18cyclictest29113-21diskmemload21:31:281
382299243,20cyclictest29113-21diskmemload23:25:061
382299240,2cyclictest0-21swapper/122:39:341
382299240,2cyclictest0-21swapper/120:10:141
382299231,1cyclictest0-21swapper/122:10:121
383299220,18cyclictest0-21swapper/323:45:153
382299224,15cyclictest29113-21diskmemload23:57:281
382299221,3cyclictest0-21swapper/100:35:251
382299221,2cyclictest29113-21diskmemload22:32:171
382299221,1cyclictest29113-21diskmemload23:21:131
382299220,2cyclictest0-21swapper/122:00:141
381899220,20cyclictest0-21swapper/022:52:200
382299218,5cyclictest30907-21rm22:18:581
382299214,15cyclictest29113-21diskmemload23:32:191
382299212,16cyclictest387-21cstates21:35:161
382299210,3cyclictest0-21swapper/119:10:151
382299210,2cyclictest29180-21tr21:55:151
382299210,2cyclictest29113-21diskmemload00:29:401
382299210,1cyclictest0-21swapper/122:43:431
382299210,19cyclictest0-21swapper/121:12:461
381899210,19cyclictest0-21swapper/023:02:160
3822992020,0cyclictest0-21swapper/100:05:501
382299201,16cyclictest24307-21sendmail-msp21:00:001
382299200,3cyclictest28088-21cat20:00:011
382299200,2cyclictest0-21swapper/100:17:151
382299200,1cyclictest0-21swapper/120:35:131
383299190,19cyclictest0-21swapper/322:54:543
382299193,14cyclictest13584-21runrttasks22:29:391
382299192,3cyclictest17498-21latency_hist22:10:001
382299192,2cyclictest0-21swapper/122:20:231
382299192,1cyclictest19392-21nfs4_client21:25:231
382299191,1cyclictest29113-21diskmemload00:11:011
382299190,1cyclictest0-21swapper/123:46:321
381899197,6cyclictest18308-21munin-run19:40:000
381899197,6cyclictest18308-21munin-run19:40:000
382299182,14cyclictest7529-21ssh23:10:281
382299181,3cyclictest29113-21diskmemload23:36:351
382299181,16cyclictest0-21swapper/100:00:321
382299180,1cyclictest0-21swapper/123:03:011
381899186,6cyclictest13998-21munin-run21:45:000
382799174,5cyclictest0-21swapper/223:00:152
382799170,17cyclictest0-21swapper/223:14:122
382299170,3cyclictest0-21swapper/121:17:531
382299170,2cyclictest0-21swapper/122:49:561
382299170,1cyclictest0-21swapper/123:53:401
382299170,1cyclictest0-21swapper/123:06:271
381899171,8cyclictest3185-21munin-run22:00:010
381899170,2cyclictest0-21swapper/022:39:450
383299164,5cyclictest224-21systemd-journal20:41:173
383299161,13cyclictest23078-21sensors22:35:273
383299160,2cyclictest0-21swapper/300:03:153
383299160,1cyclictest0-21swapper/323:42:363
382799160,15cyclictest0-21swapper/222:16:142
382799160,15cyclictest0-21swapper/200:36:002
382299163,1cyclictest29113-21diskmemload21:53:401
382299161,1cyclictest29113-21diskmemload00:22:191
382299160,3cyclictest29113-21diskmemload21:44:201
382299160,3cyclictest16431-21grep21:45:311
382299160,3cyclictest0-21swapper/100:30:521
383299151,1cyclictest0-21swapper/300:10:083
382799159,5cyclictest1444-21head20:10:182
382799152,13cyclictest29113-21diskmemload23:18:462
382799152,13cyclictest29113-21diskmemload23:18:462
3827991510,3cyclictest10984-21tail20:30:182
382799150,1cyclictest0-21swapper/223:54:112
382799150,0cyclictest0-21swapper/222:39:012
3822991511,3cyclictest150452sleep121:22:421
382299150,2cyclictest16144-21needreboot20:40:221
382299150,14cyclictest252852sleep119:53:021
3832991411,2cyclictest9776-21head19:20:183
3832991411,2cyclictest21002-21uniq23:20:173
3832991410,3cyclictest19183-21grep21:25:213
3832991410,3cyclictest14680-21ssh21:45:193
3832991410,3cyclictest12397-21cat19:25:193
382799148,4cyclictest10550irq/122-ahci[0022:15:002
382799144,3cyclictest224-21systemd-journal22:56:452
382799143,7cyclictest0-21swapper/200:25:002
3827991411,2cyclictest13020-21taskset22:50:502
3827991410,3cyclictest24259-21cron21:00:002
3827991410,3cyclictest20557-21grep20:50:182
382799140,13cyclictest905-21cron00:15:002
382799140,13cyclictest0-21swapper/200:31:312
382799140,12cyclictest0-21swapper/223:32:192
382299149,4cyclictest7130-21sed20:20:281
382299149,4cyclictest5209-21cron20:19:591
382299148,5cyclictest19695-21chrt19:40:211
382299140,1cyclictest0-21swapper/119:30:181
381899140,5cyclictest0-21swapper/020:55:150
381899140,13cyclictest30141-21sed21:10:140
383299139,3cyclictest941-21grep20:10:153
383299139,3cyclictest28708-21sh23:25:243
383299139,3cyclictest17290-21grep19:35:203
383299139,3cyclictest17290-21grep19:35:203
383299139,2cyclictest5205-21cron20:19:593
383299131,9cyclictest4635-21smartctl21:15:173
3832991312,1cyclictest3816-21cyclictest21:40:193
3832991310,2cyclictest8272-21grep19:15:303
3832991310,2cyclictest30927-21grep21:10:213
3832991310,2cyclictest19528-21sleep20:50:003
3832991310,2cyclictest19324-21grep19:40:173
3832991310,2cyclictest16121-21cat22:30:283
3832991310,2cyclictest11695-21smartctl21:20:193
3832991310,2cyclictest10588-21hddtemp_smartct23:35:183
382799139,3cyclictest17164-21ssh22:09:412
382799139,3cyclictest13990-21dump-pmu-power21:45:002
382799138,4cyclictest0-21swapper/200:19:352
382799134,7cyclictest7744-21snmpd00:00:322
382799133,3cyclictest224-21systemd-journal21:55:002
382799131,5cyclictest29620-21fschecks_time21:55:172
3827991310,2cyclictest6190-21awk19:15:002
3827991310,2cyclictest15309-21head22:30:202
382799130,12cyclictest10346-21idleruntime-cro22:05:002
382299137,5cyclictest19929-21wc20:50:151
382299130,4cyclictest224-21systemd-journal19:30:001
382299130,3cyclictest0-21swapper/122:50:361
381899139,3cyclictest32388-21sed20:05:260
381899139,3cyclictest1435-21grep21:35:230
381899134,5cyclictest224-21systemd-journal19:15:000
381899133,1cyclictest141rcu_preempt00:29:210
381899131,6cyclictest7259-21munin-run00:20:000
3818991310,2cyclictest29257-21cat21:10:000
3818991310,2cyclictest28817-21grep20:00:160
3818991310,2cyclictest27867-21dpkg20:00:000
3818991310,2cyclictest24833-21tr21:00:160
3818991310,2cyclictest13898-21cat00:25:000
3818991310,2cyclictest10304-21cron21:20:000
95342120,0sleep00-21swapper/022:49:240
383299129,2cyclictest31039-21tr20:05:143
383299129,2cyclictest31002-21wc23:05:003
383299129,2cyclictest27024-21grep22:15:293
383299129,2cyclictest24784-21grep21:00:163
383299129,2cyclictest22583-21grep19:45:263
383299129,2cyclictest2226-21head00:15:223
383299129,2cyclictest1912-21grep21:35:273
383299129,2cyclictest14902-21grep23:15:283
383299129,2cyclictest14902-21grep23:15:283
383299128,3cyclictest26814-21grep00:35:213
383299126,3cyclictest7884-21apt-get20:25:133
383299123,5cyclictest224-21systemd-journal22:13:383
3832991210,1cyclictest13412-21sed20:35:173
383299120,7cyclictest1291-21runrttasks23:55:003
383299120,12cyclictest0-21swapper/321:59:493
382799129,2cyclictest8463-21wc22:25:242
382799129,2cyclictest29825-21grep22:40:212
382799129,2cyclictest29580-21seq21:32:392
382799129,2cyclictest27925-21wc21:05:212
382799129,2cyclictest20277-21grep19:40:282
382799129,2cyclictest15257-21tr20:40:162
382799129,2cyclictest14953-21wc20:40:002
382799129,2cyclictest10741-21grep19:20:272
382799128,3cyclictest25971-21grep19:55:172
382799128,3cyclictest11084-21ssh23:35:222
382799128,2cyclictest7441-21cut19:15:222
382799122,6cyclictest224-21systemd-journal22:46:382
3827991210,1cyclictest0-21swapper/200:08:042
382799120,2cyclictest0-21swapper/223:21:012
382299129,2cyclictest7915-21cut20:25:131
382299129,2cyclictest10669-21grep19:20:261
382299128,3cyclictest303772sleep120:03:361
382299126,5cyclictest6911-21tr19:15:181
382299125,6cyclictest21009-21cut19:45:131
382299121,2cyclictest224-21systemd-journal20:50:001
382299120,4cyclictest12171-21sshd20:32:381
382299120,1cyclictest0-21swapper/121:00:171
382299120,11cyclictest32453-21head20:05:261
381899129,2cyclictest8634-21sed20:25:170
381899129,2cyclictest6223-21grep20:20:190
381899129,2cyclictest4813-21sed20:15:270
381899129,2cyclictest26929-21head00:35:220
381899129,2cyclictest26868-21cat21:30:260
381899129,2cyclictest1792-21wc23:30:010
381899129,2cyclictest17100-21grep23:40:160
381899129,2cyclictest15519-21grep19:30:300
381899129,2cyclictest14620-21cut21:45:190
381899129,2cyclictest12924-21grep20:35:150
381899129,1cyclictest4945-21ntp_states22:00:210
381899128,3cyclictest5174-21munin-run23:54:590
381899128,3cyclictest26943-21latency_hist23:25:000
381899125,6cyclictest17150-21munin-run20:45:010
381899124,4cyclictest278-21systemd-udevd19:18:210
381899123,3cyclictest224-21systemd-journal23:50:000
381899122,7cyclictest17859-21ssh22:31:510
383299118,2cyclictest9725-21ssh00:20:333
383299118,2cyclictest878-21ssh23:05:343
383299118,2cyclictest7688-21ssh22:47:393
383299118,2cyclictest5086-21rm22:23:243
383299118,2cyclictest4649-21ssh23:30:413
383299118,2cyclictest4291-21expr19:10:163
383299118,2cyclictest31943-21ssh21:34:573
383299118,2cyclictest24832-21ssh00:08:353
383299118,2cyclictest24079-21expr19:50:213
383299118,2cyclictest23763-21munin-plugin-st23:00:003
383299118,2cyclictest21444-21ssh21:50:073
383299118,2cyclictest21197-21cat20:50:263
383299118,2cyclictest2038-21ssh23:51:583
383299118,2cyclictest16908-21ssh00:26:233
383299118,2cyclictest12130-21ssh23:14:353
383299118,2cyclictest11434-21ssh22:05:163
383299118,2cyclictest10445-21ssh22:26:403
383299113,4cyclictest24281-21munin-run21:00:013
383299110,10cyclictest29362-21grep22:40:183
382799119,1cyclictest21344-21ssh23:43:062
382799118,2cyclictest6922-21wc20:20:282
382799118,2cyclictest652-21grep23:05:272
382799118,2cyclictest4975-21grep21:15:202
382799118,2cyclictest3933-21ssh21:37:172
382799118,2cyclictest32537-21ssh21:11:342
382799118,2cyclictest31991-21grep20:05:212
382799118,2cyclictest28571-21cut23:25:232
382799118,2cyclictest26229-21tail21:00:282
382799118,2cyclictest23822-21sh23:45:072
382799118,2cyclictest22357-21taskset19:45:242
382799118,2cyclictest19941-21taskset21:48:522
382799118,2cyclictest18396-21wc20:45:202
382799118,2cyclictest17794-21cat21:25:022
382799117,3cyclictest23482-21grep19:50:202
382799110,1cyclictest0-21swapper/200:26:472
382299112,2cyclictest17321-21grep19:35:211
382299112,2cyclictest17321-21grep19:35:211
381899118,2cyclictest801-21grep22:20:190
381899118,2cyclictest32569-21ssh22:42:230
381899118,2cyclictest3124-21cut23:30:170
381899118,2cyclictest28771-21ssh22:17:010
381899118,2cyclictest24700-21sendmail_mailtr19:50:270
381899118,2cyclictest22607-21sh00:06:270
381899118,2cyclictest20984-21cut20:50:230
381899118,2cyclictest20543-21taskset19:43:110
381899118,2cyclictest18015-21cut20:45:170
381899118,2cyclictest17723-21ssh22:10:000
381899118,2cyclictest16078-21ssh23:39:490
381899118,2cyclictest13911-21cat00:00:200
381899118,2cyclictest11872-21grep20:30:270
381899110,10cyclictest12871-21ssh21:20:340
383299108,1cyclictest5653-21wc20:20:163
383299108,1cyclictest25669-21seq19:55:023
383299108,1cyclictest20935-21cat00:30:173
383299107,2cyclictest14852-21ls19:30:233
383299107,2cyclictest11223-21cat20:30:203
383299102,4cyclictest224-21systemd-journal20:05:003
382799108,1cyclictest8099-21sort20:25:142
382799108,1cyclictest13784-21taskset19:30:122
382799108,1cyclictest13689-21cat19:30:012
382799107,2cyclictest3318-21grep20:15:152
382799107,2cyclictest32575-21grep22:20:172
382799107,2cyclictest10993-21grep21:20:162
382799102,6cyclictest28373-21aten2_r7power_e20:00:162
381899108,1cyclictest703-21apache_accesses20:10:120
381899108,1cyclictest6817-21cat23:55:190
381899108,1cyclictest24243-21ssh21:51:090
381899108,1cyclictest20944-21basename19:45:110
381899108,1cyclictest20721-21ssh21:26:050
381899108,1cyclictest13246-21chrt19:26:170
381899108,1cyclictest10702-21rm22:05:080
381899108,1cyclictest10370-21gzip19:20:230
381899108,1cyclictest10071-21ssh22:26:170
381899107,2cyclictest27744-21expr00:10:170
381899107,2cyclictest20581-21bc00:30:130
38329997,1cyclictest27276-21expr21:05:173
38279997,1cyclictest16770-21taskset19:35:172
38279997,1cyclictest16770-21taskset19:35:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional