You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-17 - 11:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Wed Sep 17, 2025 00:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1074995924,18cyclictest25366irq/58-eth023:00:133
1066995614,8cyclictest131rcu_preempt21:50:132
105899554,11cyclictest131rcu_preempt00:15:141
104999541,8cyclictest0-21swapper/020:00:140
104999521,2cyclictest18927-21diskmemload23:15:120
104999494,38cyclictest28599-21ssh21:55:120
104999492,41cyclictest0-21swapper/020:50:150
1074994734,10cyclictest25366irq/58-eth022:30:143
107499473,29cyclictest25366irq/58-eth022:45:173
105899471,40cyclictest8767-21ssh22:25:161
104999472,7cyclictest0-21swapper/022:50:130
104999471,4cyclictest0-21swapper/022:45:160
104999463,2cyclictest31314-21apt-get00:20:140
104999453,5cyclictest0-21swapper/019:35:170
104999452,2cyclictest24958-21apt-get22:55:140
104999451,5cyclictest0-21swapper/000:05:140
104999451,2cyclictest0-21swapper/021:15:140
104999441,7cyclictest0-21swapper/019:55:170
104999433,3cyclictest9280-21apt-get23:45:110
104999431,3cyclictest0-21swapper/019:45:130
104999421,10cyclictest26261-21apt-get20:10:120
105899403,8cyclictest131rcu_preempt20:25:151
105899401,32cyclictest4407-21ssh22:00:141
104999402,31cyclictest16421-21ssh00:10:140
104999401,7cyclictest0-21swapper/019:40:140
104999401,2cyclictest0-21swapper/021:50:130
1074993913,14cyclictest25366irq/58-eth023:10:173
104999391,4cyclictest0-21swapper/020:45:130
104999390,4cyclictest27951-21apt-get21:35:140
104999381,30cyclictest4397-21ssh22:00:130
104999380,3cyclictest25180-21apt-get22:35:160
106699372,18cyclictest17827-21ssh00:31:472
1058993710,23cyclictest21-21ksoftirqd/123:15:121
104999372,27cyclictest16270-21ssh22:10:100
107499364,21cyclictest25366irq/58-eth022:10:103
105899362,26cyclictest11582-21ssh21:25:181
104999362,3cyclictest17938-21apt-get20:35:120
104999361,3cyclictest14353-21apt-get21:05:130
104999361,3cyclictest0-21swapper/021:40:150
104999361,3cyclictest0-21swapper/021:40:150
104999361,2cyclictest23933-21apt-get00:15:140
106699352,28cyclictest25525-21ssh00:38:462
1058993515,10cyclictest21-21ksoftirqd/100:25:141
104999353,3cyclictest0-21swapper/021:45:140
104999352,4cyclictest0-21swapper/020:55:140
1066993421,10cyclictest8773-21sh22:25:172
105899342,28cyclictest19713-21ssh21:30:151
104999345,25cyclictest20280-21tr23:10:260
104999342,2cyclictest21453-21apt-get20:05:160
104999341,9cyclictest26309-21cut23:35:200
104999341,8cyclictest2415-21cut00:00:200
104999341,8cyclictest19312-21cut23:30:230
104999341,4cyclictest0-21swapper/022:15:150
104999341,3cyclictest17346-21apt-get23:50:160
104999341,3cyclictest0-21swapper/019:30:160
104999341,10cyclictest786-21/usr/sbin/munin23:20:240
104999340,4cyclictest931-21apt-get22:40:140
104999340,4cyclictest8300-21apt-get20:25:140
104999335,25cyclictest8741-21cut19:50:190
104999335,25cyclictest8741-21cut19:50:190
104999331,9cyclictest32697-21ssh22:20:060
104999331,9cyclictest13290-21cut21:25:250
104999331,8cyclictest9878-21kthreadcore23:05:180
104999331,8cyclictest3389-21df21:20:150
104999331,8cyclictest23056-21irqcore20:40:180
104999331,8cyclictest19909-21fschecks_time21:30:180
104999331,7cyclictest9314-21grep19:15:260
104999331,4cyclictest0-21swapper/000:25:150
1049993311,20cyclictest4742-21unixbench_multi23:00:290
104999326,23cyclictest16729-21mailstats20:30:260
104999322,6cyclictest19598-21apt-get00:35:140
104999322,2cyclictest3521-21apt-get20:20:140
104999321,9cyclictest26223-21kthreadcore23:55:190
104999321,9cyclictest13191-21cat23:25:300
104999321,8cyclictest16381-21tr19:25:170
104999321,8cyclictest16381-21tr19:25:170
104999321,7cyclictest22090-21kthreadcore21:10:230
104999321,7cyclictest16669-21latency_hist22:30:010
104999321,7cyclictest12927-21sshd22:05:460
104999321,7cyclictest12927-21sshd22:05:460
104999321,7cyclictest10774-21sort19:20:130
104999321,4cyclictest16937-21ssh22:30:120
104999320,4cyclictest1171-21apt-get19:10:140
106699317,6cyclictest131rcu_preempt19:40:142
106699315,19cyclictest21846irq/40-dwc2_hso19:50:132
106699315,19cyclictest21846irq/40-dwc2_hso19:50:132
106699315,18cyclictest21846irq/40-dwc2_hso19:10:142
1058993115,10cyclictest0-21swapper/122:30:141
104999311,7cyclictest1518-21ssh23:40:200
104999311,7cyclictest12826-21fschecks_time00:30:170
104999311,4cyclictest9472-21apt-get21:00:160
105899302,22cyclictest27963-21ssh21:35:141
104999301,6cyclictest2595-21head20:15:310
1074992916,7cyclictest25366irq/58-eth023:05:133
1074992910,4cyclictest0-21swapper/323:23:083
106699295,14cyclictest21846irq/40-dwc2_hso21:00:162
106699292,7cyclictest131rcu_preempt23:34:462
1066992917,9cyclictest872-21sendmail23:40:012
105899293,7cyclictest131rcu_preempt00:09:461
105899293,15cyclictest21-21ksoftirqd/120:20:131
107499289,4cyclictest0-21swapper/322:15:373
1074992811,14cyclictest0-21swapper/322:40:213
106699281,20cyclictest948-21sh22:40:142
1066992811,13cyclictest27-21ksoftirqd/200:20:162
105899283,9cyclictest0-21swapper/100:13:471
105899283,21cyclictest17352-21ssh23:50:141
107499273,16cyclictest0-21swapper/300:33:023
107499273,16cyclictest0-21swapper/300:26:463
107499272,8cyclictest0-21swapper/322:55:153
1074992714,8cyclictest0-21swapper/321:50:143
106699276,12cyclictest21846irq/40-dwc2_hso20:35:112
106699273,11cyclictest25188-21ssh22:35:152
106699272,20cyclictest0-21swapper/200:08:462
105899279,10cyclictest21-21ksoftirqd/121:45:151
105899271,21cyclictest25533-21ssh23:35:151
104999270,8cyclictest1077-21latency_hist19:10:010
107499263,17cyclictest0-21swapper/323:58:463
107499262,17cyclictest0-21swapper/300:03:463
107499261,10cyclictest0-21swapper/322:20:273
107499253,16cyclictest0-21swapper/321:40:163
107499253,16cyclictest0-21swapper/321:40:163
106699257,15cyclictest3302-21diskmemload21:20:152
106699256,13cyclictest21846irq/40-dwc2_hso21:35:132
106699254,13cyclictest21846irq/40-dwc2_hso20:10:132
106699251,19cyclictest11581-21sh21:25:182
105899254,16cyclictest950-21ssh22:40:141
107499248,3cyclictest0-21swapper/300:24:203
1074992411,8cyclictest25366irq/58-eth000:05:263
1074992410,3cyclictest0-21swapper/300:16:163
106699244,13cyclictest21846irq/40-dwc2_hso00:15:142
106699242,14cyclictest21846irq/40-dwc2_hso21:05:142
106699242,10cyclictest0-21swapper/200:00:142
106699241,13cyclictest26569-21cut22:15:222
105899242,16cyclictest201rcuc/121:40:151
105899242,16cyclictest201rcuc/121:40:151
105899241,19cyclictest9301-21ssh23:45:121
107499239,8cyclictest25366irq/58-eth023:40:183
107499233,9cyclictest0-21swapper/322:27:463
107499232,9cyclictest0-21swapper/323:31:463
106699232,6cyclictest0-21swapper/222:13:462
106699232,16cyclictest0-21swapper/200:13:472
1066992311,5cyclictest23913-21ssh22:54:002
1066992310,10cyclictest1868-21ssh21:56:502
105899231,9cyclictest12528-21awk23:05:261
105899231,17cyclictest25394-21ssh23:55:141
105899231,10cyclictest21-21ksoftirqd/123:20:211
107499223,9cyclictest0-21swapper/323:37:503
107499222,6cyclictest0-21swapper/300:12:583
106699228,11cyclictest9853-21grep19:15:312
106699227,12cyclictest25357-21sh22:55:192
106699224,9cyclictest21846irq/40-dwc2_hso20:25:152
106699223,16cyclictest3826-21ssh21:40:162
106699223,16cyclictest3826-21ssh21:40:162
106699222,16cyclictest0-21swapper/200:29:462
1066992215,4cyclictest11023-21ssh22:45:242
1066992211,9cyclictest0-21swapper/220:45:142
106699221,11cyclictest3231-21tr23:20:222
106699221,10cyclictest1020-21if_eth123:00:172
1066992210,10cyclictest11013-21ssh22:03:142
106699220,19cyclictest28882-21timerwakeupswit19:35:322
106699220,17cyclictest14100-21missed_timers19:20:242
106699220,11cyclictest26042-21idleruntime-cro20:10:012
105899227,12cyclictest0-21swapper/100:30:141
105899223,13cyclictest0-21swapper/121:05:131
105899221,2cyclictest131rcu_preempt20:50:151
105899221,16cyclictest19311-21ssh21:10:151
105899221,14cyclictest28753-21ssh22:55:291
107499214,12cyclictest0-21swapper/323:15:113
107499213,12cyclictest0-21swapper/322:39:453
1074992110,7cyclictest0-21swapper/321:25:243
106699219,10cyclictest29074-21proc_pri23:55:262
106699218,10cyclictest15459-21ssh23:27:572
106699212,10cyclictest21260-21kthreadcore21:10:222
1066992111,8cyclictest0-21swapper/221:45:052
106699211,16cyclictest9554-21grep23:45:152
106699211,16cyclictest23522-21kthreadcore20:40:202
106699211,11cyclictest9545-21ssh23:05:182
106699211,11cyclictest17367-21ssh23:50:152
106699211,10cyclictest21395-21cut21:30:222
1066992110,9cyclictest7990-21ssh23:43:282
1066992110,9cyclictest24548-21ssh23:14:452
1066992110,8cyclictest0-21swapper/219:45:282
1058992113,5cyclictest0-21swapper/123:40:281
105899211,16cyclictest13165-21apt-get20:30:121
107499208,9cyclictest0-21swapper/322:03:123
107499203,13cyclictest0-21swapper/321:10:153
107499203,10cyclictest0-21swapper/321:20:153
107499201,12cyclictest25366irq/58-eth021:30:153
106699208,5cyclictest18966-21kthreadcore22:30:202
106699208,5cyclictest14998-21ssh22:08:442
106699208,5cyclictest14998-21ssh22:08:442
106699207,4cyclictest27457-21sh21:15:142
106699202,9cyclictest1112-21kthreadcore20:15:242
106699202,10cyclictest16916-21kthreadcore19:25:182
106699202,10cyclictest16916-21kthreadcore19:25:182
106699201,9cyclictest3741-21grep20:20:172
106699201,9cyclictest25829-21kthreadcore23:15:192
106699201,9cyclictest22070-21kthreadcore19:30:242
106699201,16cyclictest2841-21ssh22:20:212
106699201,10cyclictest5010-21irqcore20:55:192
106699201,10cyclictest2436-21kthreadcore20:50:222
105899201,5cyclictest25979-21apt-get00:40:021
1058992013,5cyclictest0-21swapper/123:10:351
1058992013,5cyclictest0-21swapper/120:10:181
1058992012,6cyclictest0-21swapper/120:05:201
105899201,15cyclictest15549-21apt-get19:25:101
105899201,15cyclictest15549-21apt-get19:25:101
1058992010,7cyclictest17394-21cut22:10:161
107499198,9cyclictest0-21swapper/323:26:103
107499193,11cyclictest0-21swapper/323:52:463
107499192,13cyclictest0-21swapper/321:45:143
107499192,10cyclictest0-21swapper/320:15:043
1074991910,3cyclictest0-21swapper/321:55:253
106699197,5cyclictest12009-21fschecks_time19:55:202
106699192,8cyclictest14560-21tr20:30:192
1066991913,4cyclictest0-21swapper/220:00:132
105899193,11cyclictest0-21swapper/122:15:461
105899191,9cyclictest6888-21kthreadcore20:55:221
1058991911,6cyclictest1-21systemd21:17:201
105899191,15cyclictest331-21ssh22:20:141
105899191,12cyclictest31321-21ssh00:20:151
107499182,11cyclictest0-21swapper/321:15:153
107499182,10cyclictest0-21swapper/320:05:183
107499182,10cyclictest0-21swapper/319:40:143
107499182,10cyclictest0-21swapper/319:20:153
107499181,12cyclictest0-21swapper/320:20:233
1074991810,6cyclictest0-21swapper/322:50:123
106699181,9cyclictest1120-21latency_hist19:10:012
105899189,7cyclictest28692-21sed22:35:251
105899189,6cyclictest1-21systemd22:07:301
105899189,6cyclictest1-21systemd22:07:301
105899189,3cyclictest0-21swapper/120:35:121
105899186,7cyclictest0-21swapper/121:00:151
105899185,10cyclictest4333-21kthreadcore19:10:231
105899183,9cyclictest17236-21ssh23:30:151
105899181,5cyclictest52846usb-storage20:15:101
1058991810,5cyclictest7151-21switchtime21:20:291
107499179,6cyclictest0-21swapper/322:06:313
107499179,6cyclictest0-21swapper/322:06:313
107499178,6cyclictest0-21swapper/321:37:573
107499177,6cyclictest0-21swapper/323:48:293
107499172,10cyclictest0-21swapper/320:55:153
107499172,10cyclictest0-21swapper/320:50:153
107499172,10cyclictest0-21swapper/320:45:143
107499172,10cyclictest0-21swapper/320:30:133
107499172,10cyclictest0-21swapper/300:35:463
107499171,11cyclictest0-21swapper/320:00:143
105899179,5cyclictest31223-21timerandwakeup20:45:281
105899176,8cyclictest22880-21fschecks_time20:40:171
105899173,10cyclictest0-21swapper/119:15:171
1058991712,3cyclictest17302-21ls22:50:171
1058991711,4cyclictest8488-21sh00:05:011
105899171,13cyclictest9101-21apt-get22:45:171
105899171,13cyclictest640-21find23:00:121
107499163,7cyclictest0-21swapper/320:25:243
107499162,10cyclictest0-21swapper/319:35:183
107499162,10cyclictest0-21swapper/319:15:173
107499161,12cyclictest0-21swapper/320:35:223
105899169,5cyclictest20411-21kworker/u8:221:55:331
105899169,4cyclictest0-21swapper/119:50:281
105899169,4cyclictest0-21swapper/119:50:281
105899165,8cyclictest0-21swapper/119:40:141
105899163,10cyclictest10250-21tr23:25:211
105899161,12cyclictest21020-21cut19:30:231
105899161,12cyclictest12190-21grep19:55:221
105899161,10cyclictest6987-21wc19:50:021
105899161,10cyclictest21538-21kthreadcore21:50:201
1058991610,4cyclictest0-21swapper/119:35:261
107499151,12cyclictest0-21swapper/319:10:243
105899157,5cyclictest17206-21cut20:00:191
105899151,11cyclictest11007-21grep19:20:171
107499149,3cyclictest0-21swapper/320:40:243
107499149,3cyclictest0-21swapper/319:45:133
107499142,8cyclictest0-21swapper/321:05:043
107499138,3cyclictest0-21swapper/319:28:563
107499138,3cyclictest0-21swapper/319:28:563
107499131,7cyclictest0-21swapper/319:55:323
107499131,7cyclictest0-21swapper/319:50:273
107499131,7cyclictest0-21swapper/319:50:273
107499131,10cyclictest0-21swapper/320:10:143
105899131,9cyclictest1079-21latency_hist19:10:011
107499122,8cyclictest0-21swapper/321:05:013
107499112,5cyclictest0-21swapper/319:30:183
107499101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional