You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-27 - 07:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sat Jul 27, 2024 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
228732238164,25sleep20-21swapper/219:06:592
230482237163,24sleep30-21swapper/319:09:133
227982237165,58sleep00-21swapper/019:06:000
230982205167,25sleep10-21swapper/119:09:521
711421630,4sleep12320599cyclictest23:25:131
895821030,4sleep38964-21gltestperf22:15:143
232109910391,3cyclictest0-21swapper/200:08:422
225942610,4sleep12320599cyclictest22:45:231
167972610,2sleep20-21swapper/221:20:102
167972610,2sleep20-21swapper/221:20:102
23210995955,3cyclictest0-21swapper/221:03:272
17952580,4sleep22321099cyclictest00:25:202
161282540,2sleep339-21rcuc/320:05:123
256992530,3sleep339-21rcuc/321:40:143
2320599447,24cyclictest26721-21cut22:55:201
2320599441,10cyclictest6504-21perf23:25:011
2320599421,32cyclictest29078-21users21:45:281
2321599411,29cyclictest0-21swapper/323:40:213
23215994025,10cyclictest0-21swapper/323:53:263
2319999401,15cyclictest18682-21date21:25:010
2321599391,28cyclictest0-21swapper/322:48:133
2321599381,20cyclictest0-21swapper/322:38:033
2321599381,20cyclictest0-21swapper/320:02:453
2321599371,27cyclictest0-21swapper/322:41:413
2321599371,27cyclictest0-21swapper/322:22:533
2321599370,13cyclictest0-21swapper/300:02:143
23215993617,3cyclictest0-21swapper/323:09:493
2321599361,24cyclictest0-21swapper/322:13:493
2321599361,13cyclictest0-21swapper/321:13:013
2320599367,2cyclictest17862-21latency_hist20:10:011
2320599361,7cyclictest23398-21/usr/sbin/munin19:10:261
23205993615,18cyclictest18768-21ntp_states23:50:211
23205993612,21cyclictest19064-21latency_hist22:40:011
23205993610,3cyclictest0-21swapper/120:56:591
2320599360,8cyclictest16395-21kworker/u8:119:15:021
2319999365,23cyclictest2074-21kworker/u8:123:36:460
2319999361,10cyclictest0-21swapper/023:40:210
2321599351,25cyclictest0-21swapper/323:26:293
2321599351,25cyclictest0-21swapper/320:34:533
2321599351,16cyclictest17172-21hddtemp_smartct21:20:133
2321599351,16cyclictest17172-21hddtemp_smartct21:20:133
2320599353,6cyclictest8785-40dnf19:46:321
2320599352,25cyclictest0-21swapper/123:30:421
2320599352,25cyclictest0-21swapper/123:30:421
2320599351,3cyclictest14885-21perl21:15:151
2319999351,12cyclictest0-21swapper/023:55:500
2319999351,10cyclictest0-21swapper/022:46:530
2321599341,26cyclictest0-21swapper/322:07:413
2320599348,3cyclictest1960-21cat19:30:261
2320599346,3cyclictest30568-21sshd00:18:331
23205993410,3cyclictest0-21swapper/120:48:481
2321599331,23cyclictest0-21swapper/323:17:573
2321599331,20cyclictest0-21swapper/322:56:373
2321599331,20cyclictest0-21swapper/322:52:533
2321599331,18cyclictest0-21swapper/320:37:173
2321599331,18cyclictest0-21swapper/319:44:213
2321599331,18cyclictest0-21swapper/319:39:253
2321599331,14cyclictest0-21swapper/320:15:133
2321599331,14cyclictest0-21swapper/320:15:133
2320599339,4cyclictest1027-21grep21:55:311
2320599337,23cyclictest10559-21cpu22:20:101
2320599335,5cyclictest24871-21kworker/u8:122:04:051
2320599335,3cyclictest86750irq/128-enp1s0-23:15:191
2320599332,8cyclictest22534-21users21:31:071
23205993313,17cyclictest15545-21gltestperf22:30:141
2320599331,29cyclictest13080-21latency_hist23:40:021
23205993312,18cyclictest9910-21/usr/sbin/munin21:05:161
2320599331,12cyclictest581-21systemd-journal22:55:011
23205993310,2cyclictest12652-21/usr/sbin/munin19:55:261
23205993310,20cyclictest1411-21/usr/sbin/munin00:25:141
2319999331,8cyclictest0-21swapper/022:31:010
2319999331,11cyclictest0-21swapper/022:54:450
2319999331,10cyclictest0-21swapper/022:41:410
259112320,3sleep32321599cyclictest20:25:233
2321599321,22cyclictest0-21swapper/319:52:053
2321599321,18cyclictest0-21swapper/321:26:133
2321599321,14cyclictest16317-21irqstats23:45:173
2321599321,14cyclictest0-21swapper/319:57:093
2321599321,13cyclictest0-21swapper/319:11:253
2321599321,12cyclictest0-21swapper/321:38:213
2321599321,10cyclictest0-21swapper/300:05:263
2320599327,2cyclictest24412-21sshd20:24:321
2320599326,3cyclictest31035-21cat20:39:561
2320599324,9cyclictest0-21swapper/100:21:341
2320599322,27cyclictest22390-21cat20:20:011
2320599322,27cyclictest22390-21cat20:20:011
2320599321,28cyclictest827-21cat20:44:561
2320599321,28cyclictest13687-21grep21:10:251
2320599321,24cyclictest30454-21latency_hist19:25:011
23205993212,17cyclictest5145-21/usr/sbin/munin00:35:161
23205993212,17cyclictest4911-21/usr/sbin/munin19:40:151
23205993212,17cyclictest15504-21/usr/sbin/munin23:45:191
23205993210,2cyclictest15389-21sshd20:01:311
2319999325,12cyclictest26320-21perf00:10:020
2319999321,13cyclictest0-21swapper/020:45:410
23215993119,3cyclictest0-21swapper/321:19:013
2321599311,20cyclictest0-21swapper/320:44:133
2321599311,19cyclictest0-21swapper/323:31:253
2321599311,19cyclictest0-21swapper/323:31:253
2321599311,19cyclictest0-21swapper/323:01:563
2321599311,19cyclictest0-21swapper/320:21:413
2321599311,19cyclictest0-21swapper/320:10:133
2321599311,18cyclictest0-21swapper/300:23:183
2321599311,15cyclictest0-21swapper/320:48:053
2321599311,14cyclictest0-21swapper/322:25:173
2321599311,14cyclictest0-21swapper/322:25:173
2321599311,14cyclictest0-21swapper/321:53:573
2321599311,14cyclictest0-21swapper/321:46:133
2321599311,13cyclictest0-21swapper/320:54:053
2321599311,13cyclictest0-21swapper/319:31:253
2320599319,3cyclictest24686-21/usr/sbin/munin20:25:181
2320599316,3cyclictest86750irq/128-enp1s0-20:10:251
2320599315,2cyclictest8305-21kworker/u8:222:17:371
2320599311,27cyclictest4025-21cpuspeed_turbos22:05:141
2320599311,27cyclictest30641-21df_inode23:05:131
2320599311,24cyclictest0-21swapper/121:44:581
23205993112,16cyclictest9694-21idleruntime-cro21:05:011
2319999311,9cyclictest0-21swapper/023:26:290
2319999311,13cyclictest0-21swapper/023:46:530
2321599301,19cyclictest0-21swapper/321:33:333
2321599301,19cyclictest0-21swapper/300:18:063
2321599301,18cyclictest0-21swapper/300:10:463
2321599301,14cyclictest0-21swapper/321:56:053
2321599301,14cyclictest0-21swapper/320:59:173
2321599301,14cyclictest0-21swapper/300:29:263
2321599301,12cyclictest0-21swapper/319:21:333
2320599307,3cyclictest26535-21/usr/sbin/munin00:10:191
2320599307,3cyclictest0-21swapper/100:04:331
2320599305,4cyclictest0-21swapper/119:50:241
2320599302,4cyclictest4540-21users00:30:261
2320599302,2cyclictest12718-21kworker/u8:200:08:361
2320599302,25cyclictest29159-21ntp_states23:00:241
2320599301,25cyclictest19433-21cpuspeed_turbos22:40:111
23205993010,17cyclictest30681-21/usr/sbin/munin19:25:121
2319999305,12cyclictest31667-21munin-run21:55:010
2319999301,12cyclictest0-21swapper/019:46:370
2319999301,11cyclictest0-21swapper/021:57:410
2319999301,11cyclictest0-21swapper/019:17:570
2319999300,6cyclictest0-21swapper/021:29:240
2321599291,3cyclictest0-21swapper/323:38:343
2321599291,14cyclictest0-21swapper/323:58:463
2321599291,12cyclictest0-21swapper/319:17:333
2321599291,12cyclictest0-21swapper/300:34:383
2321599291,11cyclictest0-21swapper/323:11:013
2320599299,2cyclictest16975-21fschecks_time21:20:121
2320599299,2cyclictest16975-21fschecks_time21:20:121
2320599298,4cyclictest0-21swapper/120:30:171
2320599297,4cyclictest20546-21grep23:55:171
2320599293,6cyclictest4388-21ntp_states20:50:231
2320599291,4cyclictest29688-21/usr/sbin/munin21:50:281
2320599291,3cyclictest86550irq/126-enp1s0-21:25:481
2319999291,6cyclictest0-21swapper/023:53:260
2319999291,15cyclictest3451-21kworker/0:022:10:100
2321599281,16cyclictest0-21swapper/300:36:143
2321599281,15cyclictest0-21swapper/322:31:093
2321599281,13cyclictest0-21swapper/323:21:573
2320599287,3cyclictest32495-21/usr/sbin/munin23:10:251
2320599286,3cyclictest7418-21switchtime22:10:241
2320599281,3cyclictest23086-21/usr/sbin/munin21:35:141
2320599280,27cyclictest0-21swapper/123:40:261
2319999281,14cyclictest0-21swapper/023:10:290
2319999281,13cyclictest0-21swapper/023:24:210
2319999281,12cyclictest0-21swapper/000:25:340
2321599271,13cyclictest0-21swapper/321:05:173
2320599275,2cyclictest0-21swapper/119:35:221
2320599271,7cyclictest0-21swapper/122:26:131
2320599271,7cyclictest0-21swapper/122:26:131
2319999271,11cyclictest0-21swapper/023:18:290
2321599261,15cyclictest0-21swapper/321:01:493
2321599261,10cyclictest8973-21taskset19:48:513
2321099261,9cyclictest12776-21kworker/2:120:25:152
2319999261,14cyclictest0-21swapper/021:08:450
2319999261,13cyclictest0-21swapper/000:30:060
2319999261,12cyclictest0-21swapper/023:05:090
2319999261,12cyclictest0-21swapper/022:55:490
2319999261,10cyclictest0-21swapper/021:31:330
2319999257,15cyclictest7116-21munin-run00:40:010
2319999251,21cyclictest26892-21/usr/sbin/munin20:30:150
2319999251,21cyclictest14634-21snmpd22:00:130
2319999251,13cyclictest0-21swapper/021:01:410
2319999251,13cyclictest0-21swapper/020:04:450
2319999251,12cyclictest0-21swapper/020:05:330
2321599241,10cyclictest0-21swapper/322:01:573
2321099241,9cyclictest0-21swapper/220:24:592
23210992411,3cyclictest24027-21df22:50:112
2319999241,20cyclictest22103-21/usr/sbin/munin00:00:280
2319999241,13cyclictest0-21swapper/000:13:500
2319999241,10cyclictest0-21swapper/021:37:410
2321599231,12cyclictest0-21swapper/319:26:533
23210992317,3cyclictest29393-21/usr/sbin/munin19:20:192
2321099231,19cyclictest27382-21grep22:55:302
2319999231,6cyclictest0-21swapper/021:19:010
2319999231,5cyclictest18089-21users21:20:260
2319999231,19cyclictest0-21swapper/023:34:300
2319999231,19cyclictest0-21swapper/023:34:300
2319999231,15cyclictest0-21swapper/019:20:050
2319999231,14cyclictest0-21swapper/020:11:090
2319999231,14cyclictest0-21swapper/000:15:420
2319999231,13cyclictest0-21swapper/020:38:210
2319999231,11cyclictest0-21swapper/022:17:330
2319999231,11cyclictest0-21swapper/022:08:530
2319999231,11cyclictest0-21swapper/020:57:330
2319999231,11cyclictest0-21swapper/020:29:580
2319999231,10cyclictest17657-21perl22:35:170
2319999231,10cyclictest0-21swapper/019:31:090
2321099221,5cyclictest12026-21memory23:35:182
2321099220,5cyclictest18041-1kworker/u9:223:00:192
2319999228,3cyclictest11880-21diskmemload23:01:480
2319999227,3cyclictest13181-21ntp_states21:10:210
2319999222,6cyclictest25373-21munin-run19:15:000
2319999221,12cyclictest0-21swapper/022:20:450
2319999221,12cyclictest0-21swapper/019:54:210
2319999221,11cyclictest0-21swapper/021:41:080
2319999221,10cyclictest27488-21/usr/sbin/munin21:45:110
2321099216,5cyclictest0-21swapper/222:10:202
2319999211,14cyclictest0-21swapper/019:39:090
2319999211,12cyclictest0-21swapper/020:41:410
2319999211,11cyclictest0-21swapper/020:51:330
2319999211,10cyclictest24103-21sshd20:20:320
2321099207,2cyclictest12776-21kworker/2:121:17:492
23210992015,3cyclictest969-21grep21:55:312
23210992015,3cyclictest16738-21find23:45:232
2321099201,13cyclictest0-21swapper/219:12:562
23199992013,4cyclictest14228-21unixbench_multi22:25:310
23199992013,4cyclictest14228-21unixbench_multi22:25:310
2319999201,11cyclictest0-21swapper/019:55:090
2319999201,11cyclictest0-21swapper/019:40:530
269312191,2sleep20-21swapper/221:40:542
2321099198,3cyclictest21878-21awk00:00:012
2321099198,3cyclictest12050irq/123-ahci[0021:10:012
2321099195,8cyclictest23335-40dnf22:48:332
2321099191,8cyclictest657-21tail23:10:142
2321099191,8cyclictest11880-21diskmemload21:45:272
2321099191,7cyclictest4429-21grep00:30:252
2321099191,7cyclictest0-21swapper/221:30:552
2321099191,6cyclictest0-21swapper/219:15:262
2319999191,9cyclictest0-21swapper/020:17:490
2319999191,9cyclictest0-21swapper/020:17:490
2319999190,12cyclictest0-21swapper/000:20:180
41712181,2sleep20-21swapper/223:18:332
325712181,3sleep00-21swapper/019:29:090
2321099184,8cyclictest8785-40dnf19:46:342
2321099182,7cyclictest6719-21/usr/sbin/munin23:25:232
2321099181,8cyclictest0-21swapper/222:00:222
2321099181,7cyclictest30315-21users00:15:272
2321099181,7cyclictest0-21swapper/223:08:352
2321099181,6cyclictest0-21swapper/223:52:292
2321099181,6cyclictest0-21swapper/219:51:022
2321099181,6cyclictest0-21swapper/200:20:582
2321099181,5cyclictest0-21swapper/222:38:092
2321099181,5cyclictest0-21swapper/200:35:112
23210991813,3cyclictest13677-21grep23:40:122
2321099181,10cyclictest0-21swapper/222:20:362
2321099180,15cyclictest3186-21grep19:35:152
2321099175,3cyclictest0-21swapper/222:07:232
2321099173,6cyclictest32-21ksoftirqd/223:30:182
2321099173,6cyclictest32-21ksoftirqd/223:30:182
2321099173,5cyclictest30-21rcuc/221:25:442
2321099173,11cyclictest8934-21grep22:15:142
2321099171,7cyclictest26892-21/usr/sbin/munin20:30:152
2321099171,7cyclictest0-21swapper/220:50:202
2321099171,6cyclictest5445-21memory23:20:222
2321099171,6cyclictest15976-21ntp_states22:30:212
2321099171,6cyclictest0-21swapper/220:11:012
23210991713,3cyclictest26871-21df00:10:142
23210991713,3cyclictest0-21swapper/222:40:192
23210991713,3cyclictest0-21swapper/220:02:312
23210991712,3cyclictest5511-21/usr/sbin/munin20:55:272
23210991712,3cyclictest22439-21df00:00:142
2321099171,15cyclictest0-21swapper/220:47:022
2321099171,15cyclictest0-21swapper/220:16:342
2321099171,15cyclictest0-21swapper/220:16:342
23210991711,3cyclictest1663-21grep19:30:222
23210991711,3cyclictest12003-21cut19:55:152
2321099165,3cyclictest14505-21taskset22:27:582
2321099165,3cyclictest14505-21taskset22:27:582
2321099163,4cyclictest31054-21latency_hist20:40:022
2321099161,6cyclictest31070-21ls21:50:282
23210991612,3cyclictest0-21swapper/221:10:332
23210991612,2cyclictest31305-21gltestperf19:25:152
2321099161,14cyclictest0-21swapper/220:40:492
23210991611,3cyclictest23804-21cat21:35:162
23210991611,3cyclictest15899-21/usr/sbin/munin20:05:302
23210991511,2cyclictest6484-21head19:40:302
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional