You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-13 - 12:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot7s.osadl.org (updated Tue May 13, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3429210087,9sleep30-21swapper/319:09:543
3215210087,9sleep10-21swapper/119:07:071
310729566,10sleep00-21swapper/019:05:420
309428867,15sleep20-21swapper/219:05:312
3527993025,1cyclictest25-21ksoftirqd/122:45:211
353599271,23cyclictest0-21swapper/323:35:233
3527992523,2cyclictest25-21ksoftirqd/100:40:001
3527992117,3cyclictest25-21ksoftirqd/121:46:081
3525992121,0cyclictest0-21swapper/022:58:180
3535992019,1cyclictest0-21swapper/322:53:113
353099201,16cyclictest7376-21switchtime21:40:272
3525992020,0cyclictest0-21swapper/023:25:510
353599190,1cyclictest0-21swapper/321:31:353
3530991918,1cyclictest0-21swapper/220:00:152
3530991917,1cyclictest0-21swapper/223:21:192
353099190,1cyclictest0-21swapper/223:50:002
3527991915,1cyclictest25-21ksoftirqd/121:44:371
3525991915,2cyclictest0-21swapper/021:35:330
352599190,18cyclictest0-21swapper/021:31:360
3527991816,1cyclictest0-21swapper/121:19:261
353599175,4cyclictest0-21swapper/322:30:153
3535991712,2cyclictest0-21swapper/321:38:133
353599170,8cyclictest141rcu_preempt19:10:153
3530991713,3cyclictest0-21swapper/222:05:152
352799170,1cyclictest0-21swapper/100:33:551
353599166,8cyclictest31004-21/usr/sbin/munin21:10:273
353599161,4cyclictest0-21swapper/300:35:183
3530991612,3cyclictest9631-21grep00:30:202
353099160,8cyclictest0-21swapper/222:33:012
353099160,16cyclictest0-21swapper/223:55:422
352799168,3cyclictest25-21ksoftirqd/123:35:181
3527991611,3cyclictest25-21ksoftirqd/122:10:001
3527991610,2cyclictest25-21ksoftirqd/100:04:551
352799160,16cyclictest0-21swapper/123:47:371
352799160,15cyclictest0-21swapper/123:12:401
352599165,6cyclictest3767-21munin-run23:35:000
3525991615,1cyclictest0-21swapper/023:45:150
3525991614,2cyclictest0-21swapper/021:29:440
352599160,1cyclictest0-21swapper/000:25:140
352599160,16cyclictest0-21swapper/021:19:370
353599150,2cyclictest0-21swapper/321:59:123
353599150,1cyclictest0-21swapper/323:05:483
353599150,14cyclictest0-21swapper/322:57:113
353099158,6cyclictest7709-21wc19:15:292
3530991513,1cyclictest0-21swapper/222:50:002
353099150,1cyclictest0-21swapper/220:10:172
352799151,1cyclictest141rcu_preempt00:05:231
3527991510,5cyclictest0-21swapper/123:29:511
3527991510,1cyclictest141rcu_preempt20:15:181
352799150,1cyclictest0-21swapper/123:04:371
352599154,7cyclictest7744-21snmpd20:00:130
3525991514,1cyclictest0-21swapper/023:53:160
3525991512,2cyclictest0-21swapper/022:26:300
352599150,1cyclictest141rcu_preempt21:05:000
353599146,8cyclictest0-21swapper/323:16:143
353599141,4cyclictest4737-21sh20:20:003
3535991410,3cyclictest39-21ksoftirqd/322:20:003
3530991414,0cyclictest0-21swapper/220:41:322
3530991412,2cyclictest32-21ksoftirqd/223:32:332
3530991412,1cyclictest13813-21irqstats00:05:212
3530991412,1cyclictest0-21swapper/222:55:182
3530991410,3cyclictest29741-21sendmail-msp00:20:012
353099140,14cyclictest0-21swapper/221:58:472
353099140,13cyclictest0-21swapper/222:42:142
352799144,3cyclictest141rcu_preempt22:35:231
352799144,3cyclictest141rcu_preempt21:55:141
3527991413,1cyclictest25-21ksoftirqd/122:17:381
3527991412,2cyclictest25-21ksoftirqd/123:39:591
3527991411,2cyclictest25-21ksoftirqd/121:20:131
3525991410,3cyclictest10704-21dump-pmu-power19:25:000
353599139,3cyclictest39-21ksoftirqd/321:44:153
353599135,7cyclictest0-21swapper/322:47:083
353599134,8cyclictest7744-21snmpd22:05:113
353599134,3cyclictest32293-21cron20:10:003
3535991310,2cyclictest5439-21grep00:25:303
3535991310,2cyclictest21112-21head23:00:223
3535991310,2cyclictest12068-21cut23:40:183
3535991310,2cyclictest10244-21chrt22:30:003
353599130,2cyclictest0-21swapper/323:26:563
353599130,12cyclictest0-21swapper/300:31:263
353599130,0cyclictest0-21swapper/322:10:243
353099139,3cyclictest3986-21awk20:15:212
353099139,3cyclictest30266-21grep20:05:152
353099136,6cyclictest3871-21apache223:00:222
353099133,9cyclictest3884-21cups-browsed20:45:252
3530991310,2cyclictest8389-21cut20:25:202
3530991310,2cyclictest4036-21grep21:15:182
3530991310,2cyclictest29770-21grep22:20:172
3530991310,2cyclictest22564-21grep20:55:152
3530991310,2cyclictest22300-21sed19:45:272
3530991310,2cyclictest10901-21grep21:20:162
353099130,1cyclictest0-21swapper/200:10:192
352799131,2cyclictest141rcu_preempt20:00:151
3527991310,2cyclictest5258-21cron21:40:001
3527991310,2cyclictest26355-21cron21:05:001
3527991310,2cyclictest1358-21cut20:10:251
352799130,1cyclictest0-21swapper/123:20:441
352799130,13cyclictest0-21swapper/122:33:261
352599139,3cyclictest17261-21tr20:45:170
352599131,1cyclictest141rcu_preempt22:05:290
3525991310,2cyclictest32647-21grep22:45:180
3525991310,2cyclictest28743-21diskmemload23:15:270
3525991310,2cyclictest21584-21cat22:15:000
353599129,2cyclictest7520-21find19:15:253
353599129,2cyclictest29315-21rm22:20:153
353599129,2cyclictest27741-21uname21:05:213
353599129,2cyclictest21589-21sh20:55:003
353599129,2cyclictest18175-21tr22:35:183
353599129,2cyclictest16593-21grep23:20:203
353599129,2cyclictest12826-21grep19:25:263
353599128,3cyclictest25605-21expr23:50:163
353599126,4cyclictest28743-21diskmemload23:48:543
353599121,8cyclictest49-21kcompactd000:10:213
353599121,1cyclictest0-21swapper/323:34:123
353599120,12cyclictest0-21swapper/321:25:203
353599120,11cyclictest5719-21dump-pmu-power00:00:003
353599120,11cyclictest0-21swapper/322:43:253
353099129,2cyclictest7501-21ssh22:50:202
353099129,2cyclictest5117-21head23:35:212
353099129,2cyclictest24327-21cron22:40:002
353099129,2cyclictest2242-21df23:10:202
353099129,2cyclictest20275-21grep20:50:202
353099129,2cyclictest18395-21sort23:45:162
353099129,2cyclictest17284-21cat19:35:232
353099129,2cyclictest15777-21/usr/sbin/munin22:10:182
353099129,2cyclictest15661-21awk19:35:012
353099129,2cyclictest11411-21cut19:25:142
353099128,3cyclictest9311-21tail19:20:212
353099128,3cyclictest5527-21grep19:10:272
353099128,3cyclictest13422-21cut21:45:192
353099128,2cyclictest30166-21cut00:20:152
353099127,4cyclictest7037-21ssh22:04:222
353099127,4cyclictest24372-21cut21:00:142
353099127,2cyclictest3958-21ssh21:38:412
353099125,6cyclictest28743-21diskmemload21:33:422
353099121,3cyclictest224-21systemd-journal20:35:002
352799129,2cyclictest31501-21cron23:55:001
352799129,2cyclictest30533-21head22:20:231
352799129,2cyclictest20400-21head00:10:241
352799129,2cyclictest17891-21sed20:45:201
352799129,2cyclictest17753-21tr21:25:151
352799129,1cyclictest141rcu_preempt19:15:191
352799128,3cyclictest31923-21grep00:20:341
352799127,4cyclictest25-21ksoftirqd/120:25:171
352799127,3cyclictest26636-21apt-get21:05:151
3527991210,1cyclictest25230-21sed21:30:181
3527991210,1cyclictest20583-21cut20:50:221
3527991210,1cyclictest13969-21grep20:35:291
352799120,1cyclictest141rcu_preempt20:10:001
352799120,12cyclictest0-21swapper/123:31:461
352799120,11cyclictest7939-21kworker/u8:422:12:371
352599129,2cyclictest9823-21wc20:30:000
352599129,2cyclictest348-21grep23:55:190
352599129,2cyclictest31444-21ssh00:20:280
352599129,2cyclictest29442-21tr22:20:160
352599129,2cyclictest22457-21grep20:55:160
352599129,2cyclictest22319-21cut22:15:170
352599129,2cyclictest13653-21head00:05:200
352599129,2cyclictest13516-21grep21:45:200
352599129,2cyclictest11018-21grep22:30:160
352599128,3cyclictest10530-21rm22:52:400
352599122,6cyclictest17810-21/usr/sbin/munin22:35:180
3525991210,1cyclictest31890-21ssh21:11:140
3525991210,1cyclictest13-21ksoftirqd/021:40:000
352599120,11cyclictest4905-21tail19:10:200
353599119,1cyclictest4590-21grep21:15:213
353599119,1cyclictest10389-21taskset21:20:023
353599118,2cyclictest6901-21grep20:20:283
353599118,2cyclictest27552-21run-parts20:00:003
353599118,2cyclictest13525-21cut21:45:203
353599116,1cyclictest224-21systemd-journal00:02:383
353599115,6cyclictest4439-21munin-node20:14:193
353599115,5cyclictest5803-21kworker/u8:323:12:403
353599112,1cyclictest496-21dbus-daemon00:08:583
353599110,1cyclictest141rcu_preempt21:50:143
353599110,1cyclictest0-21swapper/300:16:223
353099119,1cyclictest21510-21uptime21:50:252
353099118,2cyclictest6887-21grep00:00:192
353099118,2cyclictest4993-21grep00:25:262
353099118,2cyclictest30845-21wc21:10:262
353099118,2cyclictest23568-21grep23:25:202
353099118,2cyclictest17589-21cut21:25:152
353099118,2cyclictest12457-21tr23:40:212
353099118,2cyclictest11734-21sh23:16:332
353099117,3cyclictest24848-21taskset19:51:062
353099117,1cyclictest12361-21apt20:35:162
353099111,7cyclictest25395-21/usr/sbin/munin19:55:142
353099110,1cyclictest0-21swapper/222:15:072
353099110,10cyclictest29175-21sh23:05:342
352799119,1cyclictest56742sleep123:59:581
352799119,1cyclictest16164-21tr19:35:151
352799118,2cyclictest7814-21head22:50:231
352799118,2cyclictest7440-21sed20:25:001
352799118,2cyclictest5916-21cron19:15:001
352799118,2cyclictest4739-21sh00:25:231
352799118,2cyclictest4148-21cat22:25:181
352799118,2cyclictest3235-21sh22:00:371
352799118,2cyclictest31021-21grep21:10:271
352799118,2cyclictest26130-21sh00:15:221
352799118,2cyclictest25513-21cut19:55:141
352799118,2cyclictest24712-21ssh22:40:091
352799118,2cyclictest21150-21cat19:45:161
352799118,2cyclictest19708-21ssh21:50:131
352799118,2cyclictest19039-21tail19:40:181
352799118,2cyclictest15201-21head22:55:271
352799118,2cyclictest12342-21sh23:17:091
352799118,2cyclictest11499-21cat19:25:151
352799118,2cyclictest10971-21wc20:30:211
352799117,3cyclictest15980-21head20:40:231
352799110,5cyclictest28434-21ntp_states23:05:241
352599119,1cyclictest15172-21grep19:30:270
352599119,1cyclictest13-21ksoftirqd/022:40:220
352599118,2cyclictest6663-21ssh23:35:550
352599118,2cyclictest30857-21ssh23:07:190
352599118,2cyclictest27711-21wc21:55:230
352599118,2cyclictest26939-21cat19:55:260
352599118,2cyclictest23336-21tr19:50:180
352599118,2cyclictest22499-21ssh21:51:220
352599118,2cyclictest19452-21sort00:10:180
352599118,2cyclictest18503-21ssh00:37:440
352599118,2cyclictest18413-21cut19:40:150
352599118,2cyclictest17611-21grep23:20:290
352599118,2cyclictest1723-21aten2_r7power_e23:10:170
352599118,2cyclictest14853-21ssh21:22:460
352599118,2cyclictest1429-21tr22:00:140
352599118,2cyclictest11153-21ssh23:40:120
352599118,2cyclictest11078-21sh00:03:390
352599118,2cyclictest10676-21cat00:30:280
352599118,2cyclictest10634-21grep20:30:190
352599116,4cyclictest6193-21apache_accesses19:15:150
353599108,2cyclictest21689-21sed19:45:203
353599108,1cyclictest922-21ssh22:00:023
353599108,1cyclictest27886-21basename20:00:153
353599108,1cyclictest21970-21aten2_r7power_v20:55:133
353599108,1cyclictest177272sleep319:36:183
353599108,1cyclictest14712-21seq20:40:103
353599107,2cyclictest8669-21cat20:25:223
353599107,2cyclictest19034-21head19:40:183
353599107,2cyclictest12633-21grep20:35:183
353599107,2cyclictest11192-21cut20:30:243
353599106,3cyclictest25268-21cat19:55:003
353599100,1cyclictest0-21swapper/300:23:063
353099108,1cyclictest6466-21ssh22:26:162
353099108,1cyclictest283522sleep221:05:272
353099108,1cyclictest20325-21taskset19:43:572
353099107,2cyclictest15598-21head00:35:202
352799107,2cyclictest22114-21cut20:55:141
352799106,1cyclictest141rcu_preempt19:30:181
352599108,1cyclictest30549-21ls20:05:150
352599108,1cyclictest2789-21cut20:15:130
352599108,1cyclictest26629-21apache_processe21:05:140
352599108,1cyclictest25570-21cut00:15:180
352599108,1cyclictest21514-21basename19:45:180
352599108,1cyclictest13905-21cat20:35:290
352599108,1cyclictest12859-21grep19:25:260
352599107,2cyclictest5070-21touch20:20:000
352599107,2cyclictest20807-21ssh23:00:200
352599107,2cyclictest20247-21grep20:50:200
352599100,6cyclictest0-21swapper/020:10:170
35359998,1cyclictest10364-21chrt19:20:513
35359997,2cyclictest14010-21expr19:30:163
35359997,1cyclictest19088-21taskset20:48:233
35359996,2cyclictest26044-21seq21:01:013
35309996,2cyclictest5519-21grep20:20:162
35279997,1cyclictest86032sleep119:20:161
35279997,1cyclictest24125-21sed19:50:231
35259996,2cyclictest17771-21chrt19:36:510
35259996,2cyclictest15520-21tail20:40:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional