You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-26 - 09:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sat Apr 26, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
197932241161,26sleep00-21swapper/019:06:570
198112237161,23sleep30-21swapper/319:07:123
197442235163,24sleep20-21swapper/219:06:192
200042227153,22sleep10-21swapper/119:09:401
20136999793,3cyclictest0-21swapper/219:37:532
20136998481,2cyclictest0-21swapper/219:53:002
245472790,5sleep12013199cyclictest20:35:141
198672570,2sleep00-21swapper/023:15:210
209822500,2sleep20-21swapper/221:50:132
209822500,2sleep20-21swapper/221:50:132
20136994945,3cyclictest0-21swapper/223:03:482
2012599421,13cyclictest16528-21munin-run23:10:010
2012599381,12cyclictest19649-21kworker/0:120:31:310
20125993712,12cyclictest26393-21munin-run20:40:000
20125993610,12cyclictest5180-21munin-run00:05:010
20125993610,12cyclictest5180-21munin-run00:05:010
2012599359,2cyclictest8425-21kworker/0:122:43:130
2012599357,13cyclictest15922-21idleruntime-cro23:05:010
2012599355,14cyclictest5639-21munin-run21:10:010
2014299341,25cyclictest0-21swapper/321:28:313
20125993414,3cyclictest23722-21idleruntime-cro23:30:010
2014299331,24cyclictest0-21swapper/320:34:073
2012599337,12cyclictest23120-21munin-run23:25:010
2012599332,9cyclictest19304-21chrt21:43:320
2012599331,25cyclictest12345-21kworker/0:000:31:170
2012599331,21cyclictest10034-21kworker/0:100:21:070
2014299321,24cyclictest0-21swapper/323:08:313
2014299321,23cyclictest0-21swapper/300:05:193
2014299321,20cyclictest0-21swapper/300:11:353
2012599327,12cyclictest25181-21munin-run19:20:010
2012599325,13cyclictest11520-21perf22:55:010
2012599322,14cyclictest27269-21munin-run22:10:010
2014299311,24cyclictest0-21swapper/321:08:473
2014299311,23cyclictest0-21swapper/322:07:513
2014299311,13cyclictest0-21swapper/319:19:123
2012599312,22cyclictest23658-21kworker/0:023:46:450
2012599312,10cyclictest19776-21kworker/0:219:10:210
120242310,20sleep10-21swapper/100:20:301
2014299302,22cyclictest0-21swapper/319:49:213
2014299301,18cyclictest0-21swapper/323:59:213
2014299301,14cyclictest0-21swapper/323:12:233
2014299301,14cyclictest0-21swapper/322:51:593
2014299301,12cyclictest0-21swapper/300:37:593
20142992916,6cyclictest0-21swapper/322:47:293
2014299291,17cyclictest0-21swapper/300:28:473
2014299290,22cyclictest0-21swapper/320:53:433
2012599291,11cyclictest5968-21kworker/0:219:50:270
2012599291,10cyclictest1360-21kworker/0:122:28:190
2014299281,9cyclictest0-21swapper/320:48:553
2014299281,9cyclictest0-21swapper/320:48:553
2014299281,20cyclictest0-21swapper/322:30:013
2014299281,20cyclictest0-21swapper/321:59:273
2014299281,12cyclictest0-21swapper/320:37:273
2014299281,12cyclictest0-21swapper/320:09:053
2014299281,12cyclictest0-21swapper/320:04:313
2014299281,12cyclictest0-21swapper/319:22:393
20125992813,3cyclictest29234-21kworker/0:122:15:010
2012599281,22cyclictest19649-21kworker/0:120:28:450
2014299278,4cyclictest0-21swapper/321:54:153
2014299278,4cyclictest0-21swapper/321:54:153
2014299271,9cyclictest0-21swapper/319:50:313
2014299271,9cyclictest0-21swapper/300:31:593
2014299271,12cyclictest0-21swapper/323:30:173
2014299271,12cyclictest0-21swapper/320:25:033
2014299271,11cyclictest0-21swapper/319:31:353
2014299271,11cyclictest0-21swapper/319:29:033
2014299270,12cyclictest0-21swapper/320:12:313
2014299270,12cyclictest0-21swapper/319:10:153
2013199271,19cyclictest0-21swapper/123:43:571
2012599271,10cyclictest7952-21kworker/0:319:55:270
2012599271,10cyclictest28243-21kworker/0:320:55:490
2012599271,10cyclictest1441-21kworker/0:319:39:170
2014299268,9cyclictest0-21swapper/322:43:593
2014299261,9cyclictest0-21swapper/320:16:073
2014299261,12cyclictest0-21swapper/320:20:033
2014299261,11cyclictest0-21swapper/321:32:073
2014299260,21cyclictest0-21swapper/319:43:433
2014299260,21cyclictest0-21swapper/319:43:433
2014299260,18cyclictest0-21swapper/322:24:393
2013199269,4cyclictest17227-21sa121:40:011
2013199268,10cyclictest0-21swapper/123:55:141
2013199262,20cyclictest20045-21munin-run21:50:011
2013199261,8cyclictest25959-21sed23:35:021
20125992612,3cyclictest2043-21kworker/0:022:30:190
2012599261,18cyclictest9117-21kworker/0:023:13:330
2012599261,17cyclictest10223-21kworker/0:020:01:580
2012599261,13cyclictest15420-21awk20:15:020
257012250,2sleep230-21rcuc/222:00:202
2014299252,11cyclictest0-21swapper/323:25:533
2014299251,11cyclictest0-21swapper/323:48:473
2014299251,11cyclictest0-21swapper/321:48:313
2014299251,10cyclictest0-21swapper/300:21:113
2014299250,17cyclictest0-21swapper/319:57:513
2014299250,12cyclictest0-21swapper/321:39:593
2013199256,10cyclictest0-21swapper/123:54:301
2013199252,16cyclictest7707-21kworker/u8:100:15:361
20131992520,3cyclictest22401-21cat19:15:011
2013199251,4cyclictest581-21systemd-journal22:25:021
20125992512,2cyclictest9978-21kworker/0:021:27:170
2012599251,14cyclictest17439-21kworker/0:320:20:150
2012599251,12cyclictest1463-21idleruntime-cro22:25:010
20125992510,3cyclictest24100-21kworker/0:022:00:140
2014299245,9cyclictest0-21swapper/322:35:163
2014299241,12cyclictest0-21swapper/320:42:553
2014299241,11cyclictest0-21swapper/319:37:513
2014299240,13cyclictest0-21swapper/321:22:153
2014299240,11cyclictest0-21swapper/323:54:313
2014299240,10cyclictest0-21swapper/323:20:393
2013699245,5cyclictest1544-21tr21:00:142
2013199249,9cyclictest0-21swapper/100:06:271
2013199241,20cyclictest23061-21memory19:15:041
2013199241,17cyclictest0-21swapper/123:12:061
2013199241,17cyclictest0-21swapper/122:00:261
2013199241,16cyclictest0-21swapper/123:06:221
2013199240,16cyclictest0-21swapper/122:11:221
2012599248,3cyclictest19648-21kworker/0:121:50:120
2012599248,3cyclictest19648-21kworker/0:121:50:120
2012599242,13cyclictest21891-21kworker/0:222:19:250
2012599241,9cyclictest27125-21kworker/0:019:28:130
2012599241,9cyclictest21891-21kworker/0:221:55:410
2012599241,8cyclictest9117-21kworker/0:022:48:050
20125992411,2cyclictest29347-21kworker/0:219:31:330
20125992411,2cyclictest12819-21kworker/0:220:16:530
2012599240,10cyclictest7736-21kworker/0:221:18:460
309152231,18sleep230929-21munin-run23:50:002
2014299238,8cyclictest0-21swapper/323:43:433
2014299231,6cyclictest0-21swapper/323:00:173
2014299231,12cyclictest0-21swapper/322:32:473
2014299231,11cyclictest0-21swapper/323:37:473
2014299231,10cyclictest0-21swapper/321:14:153
20142992310,3cyclictest0-21swapper/300:15:213
2013699233,2cyclictest14698-21kworker/u8:221:39:262
2013699230,21cyclictest0-21swapper/221:34:242
2013199238,3cyclictest2969-21unixbench_multi21:00:281
2013199237,9cyclictest0-21swapper/100:11:521
2013199236,9cyclictest0-21swapper/121:29:471
2013199234,10cyclictest0-21swapper/122:48:331
2013199233,7cyclictest14405-21wc00:30:191
2013199233,14cyclictest18607-21kworker/u8:023:48:331
2013199231,3cyclictest7688-21taskset21:13:181
2013199231,10cyclictest0-21swapper/123:19:101
2013199230,17cyclictest0-21swapper/121:44:251
2012599239,4cyclictest20781-21kworker/0:223:30:170
2012599235,2cyclictest30680-21kworker/0:223:53:370
2012599231,9cyclictest17085-21kworker/0:100:37:330
2012599231,8cyclictest23658-21kworker/0:023:38:330
2012599231,13cyclictest9978-21kworker/0:021:30:130
20125992311,3cyclictest4201-21kworker/0:322:38:130
2012599231,12cyclictest7736-21kworker/0:221:14:130
20125992311,2cyclictest5594-21kworker/0:300:12:530
2012599231,12cyclictest28243-21kworker/0:320:44:130
2012599231,11cyclictest3718-21kworker/0:019:48:050
143752230,3sleep123-21ksoftirqd/120:10:161
2014299227,7cyclictest0-21swapper/323:18:393
2014299226,7cyclictest780-21rngd22:18:393
2014299225,7cyclictest88250irq/135-enp2s0-22:55:433
2014299225,7cyclictest88250irq/135-enp2s0-22:55:433
2014299221,12cyclictest0-21swapper/300:04:053
2014299221,12cyclictest0-21swapper/300:04:053
2014299221,11cyclictest0-21swapper/321:02:253
2014299221,10cyclictest0-21swapper/321:41:473
20142992210,3cyclictest0-21swapper/322:00:553
2013699222,16cyclictest4349-21grep00:00:242
2013699222,16cyclictest4349-21grep00:00:242
2013699221,3cyclictest0-21swapper/223:43:512
2013699220,21cyclictest0-21swapper/220:22:232
2013699220,20cyclictest0-21swapper/220:47:512
2013699220,20cyclictest0-21swapper/220:47:512
2013699220,20cyclictest0-21swapper/220:38:172
2013699220,20cyclictest0-21swapper/220:18:452
2013199225,9cyclictest30027-21diskstats22:15:121
2013199225,9cyclictest0-21swapper/122:30:201
2013199223,2cyclictest0-21swapper/122:55:181
2013199223,2cyclictest0-21swapper/122:55:171
2013199222,15cyclictest9962-21kworker/u8:221:20:341
2013199222,11cyclictest28435-40dnf20:43:151
20131992217,2cyclictest26990-21uptime19:20:271
20131992214,2cyclictest7707-21kworker/u8:100:25:151
2013199221,19cyclictest0-21swapper/120:33:101
2013199221,18cyclictest27710-21tr19:25:151
2013199221,16cyclictest0-21swapper/100:00:111
2013199221,16cyclictest0-21swapper/100:00:111
2013199221,15cyclictest374-21kworker/u8:121:16:371
2013199221,15cyclictest0-21swapper/123:04:011
2013199221,15cyclictest0-21swapper/122:40:021
2013199221,15cyclictest0-21swapper/100:37:311
2013199221,14cyclictest0-21swapper/121:52:111
2013199221,14cyclictest0-21swapper/121:52:111
2013199221,13cyclictest0-21swapper/122:25:381
2013199221,10cyclictest0-21swapper/120:20:251
2012599229,3cyclictest26363-21kworker/0:220:45:410
2012599229,3cyclictest26363-21kworker/0:220:45:410
2012599229,2cyclictest9978-21kworker/0:021:23:090
2012599229,2cyclictest5126-21kworker/0:200:05:090
2012599227,3cyclictest444-21kworker/0:123:55:230
2012599221,10cyclictest10223-21kworker/0:020:06:220
20125992210,3cyclictest9978-21kworker/0:021:39:090
20125992210,3cyclictest5594-21kworker/0:300:19:490
85802211,3sleep214634-21snmpd22:43:412
242432211,3sleep20-21swapper/221:57:402
2014299219,3cyclictest0-21swapper/320:56:393
2014299212,11cyclictest9130-21ntp_states21:15:213
2013199215,4cyclictest9719-21cstates22:50:151
20131992119,1cyclictest0-21swapper/119:50:151
20131992117,3cyclictest0-21swapper/121:07:571
20131992114,2cyclictest0-21swapper/123:36:541
2013199211,15cyclictest0-21swapper/123:21:011
2013199211,14cyclictest0-21swapper/122:40:181
2013199211,13cyclictest0-21swapper/122:05:291
2013199211,11cyclictest17594-21sh20:20:011
2013199210,6cyclictest0-21swapper/121:31:211
2012599216,3cyclictest28243-21kworker/0:320:50:530
2012599211,12cyclictest4201-21kworker/0:322:55:300
2012599211,12cyclictest4201-21kworker/0:322:55:290
2012599211,11cyclictest29977-21kworker/0:123:42:050
2012599211,10cyclictest5594-21kworker/0:300:27:010
20125992110,2cyclictest9978-21kworker/0:021:45:420
152282210,2sleep20-21swapper/220:12:552
2014299200,10cyclictest0-21swapper/322:12:073
2013699206,3cyclictest86550irq/126-enp1s0-00:35:182
2013699203,14cyclictest8899-21cut00:15:162
2013699201,6cyclictest10149-21tail22:50:182
20136992014,4cyclictest21992-21ls20:30:002
2013699201,11cyclictest0-21swapper/223:24:012
2013699201,11cyclictest0-21swapper/222:14:302
2013699201,10cyclictest0-21swapper/223:32:102
2013699201,10cyclictest0-21swapper/219:17:362
2013199208,3cyclictest2898-21ntp_states19:40:251
2013199208,3cyclictest2898-21ntp_states19:40:251
2013199203,4cyclictest1607-21munin-run19:40:021
2013199201,6cyclictest22476-21/usr/sbin/munin21:55:281
20131992016,3cyclictest0-21swapper/120:50:141
20131992016,3cyclictest0-21swapper/120:28:561
20131992013,4cyclictest11857-21find20:00:261
2013199201,18cyclictest0-21swapper/120:48:491
2013199201,18cyclictest0-21swapper/120:48:491
2012599202,8cyclictest31577-21kworker/0:119:41:530
2012599202,8cyclictest31577-21kworker/0:119:41:530
319672190,5sleep22013699cyclictest20:55:172
2013699195,3cyclictest5632-21diskmemload22:21:232
2013699192,4cyclictest2105-21munin-run22:30:012
2013699192,4cyclictest10962-21df_inode20:00:162
2013699191,8cyclictest28587-21awk20:45:012
2013699191,6cyclictest5632-21diskmemload23:11:042
2013699191,6cyclictest31157-21/usr/sbin/munin23:50:172
2013699191,6cyclictest2323-21/usr/sbin/munin22:30:102
2013699191,5cyclictest27494-21cat19:25:012
2013699191,5cyclictest0-21swapper/222:05:402
20136991914,3cyclictest14562-21switchtime00:30:202
20136991913,4cyclictest30472-21tail19:30:182
20136991912,4cyclictest20958-21cut19:10:192
20136991911,4cyclictest5180-21grep21:05:272
20131991916,2cyclictest0-21swapper/120:09:141
2013199191,2cyclictest0-21swapper/119:30:251
2013199191,17cyclictest0-21swapper/119:57:131
2013199191,16cyclictest0-21swapper/120:56:401
2013199191,13cyclictest0-21swapper/123:27:251
2012599195,2cyclictest3248-21kworker/0:021:04:210
2012599192,5cyclictest385-21kworker/0:119:23:160
311512181,2sleep20-21swapper/220:52:052
26542181,3sleep20-21swapper/223:56:152
2013699185,3cyclictest86750irq/128-enp1s0-23:06:312
2013699184,3cyclictest86550irq/126-enp1s0-21:25:282
2013699182,4cyclictest5804-21latency_hist00:10:012
2013699182,3cyclictest5701-21wc22:35:202
2013699181,6cyclictest9206-21if_err_enp1s019:55:192
2013699181,5cyclictest7697-21sshd00:11:142
2013699181,5cyclictest4840-21irqstats19:45:182
2013699181,5cyclictest17459-21/usr/sbin/munin21:40:162
2013699181,5cyclictest10612-21diskstats21:20:192
2013699181,3cyclictest0-21swapper/223:38:182
20136991813,3cyclictest6940-21ntp_states21:10:262
2013699181,15cyclictest2961-21pmu-power19:40:262
2013699181,15cyclictest2961-21pmu-power19:40:262
2013699181,15cyclictest0-21swapper/222:45:262
2013699181,10cyclictest0-21swapper/222:56:302
2013699181,10cyclictest0-21swapper/222:56:302
20131991813,2cyclictest4422-21fschecks_count19:45:171
131222181,2sleep20-21swapper/220:09:062
2013699174,3cyclictest86550irq/126-enp1s0-20:30:242
2013699171,4cyclictest12952-21chrt00:29:262
2013699171,2cyclictest86550irq/126-enp1s0-22:15:152
2013699171,2cyclictest0-21swapper/223:19:432
2013699171,15cyclictest0-21swapper/219:29:112
20136991711,3cyclictest9213-21proc_pri21:15:222
196722170,2sleep20-21swapper/221:45:242
120462171,2sleep20-21swapper/200:20:392
2013699161,10cyclictest0-21swapper/223:28:342
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional