You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-13 - 10:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun Jul 13, 2025 00:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7681996422,29cyclictest131rcu_preempt21:20:140
768899635,47cyclictest32693-21ssh00:15:151
768899605,40cyclictest15176-21ssh00:25:141
768899584,40cyclictest22524-21ssh00:30:171
768899584,40cyclictest22524-21ssh00:30:161
769899571,3cyclictest0-21swapper/220:35:132
769899564,46cyclictest729-21apt-get22:15:172
7706995523,18cyclictest32466irq/58-eth022:15:163
768899556,38cyclictest17205-21ssh22:05:161
768199532,7cyclictest131rcu_preempt19:10:160
769899521,3cyclictest0-21swapper/223:55:152
768899523,39cyclictest29678-21ssh00:35:151
7698995131,15cyclictest24697-21sh22:30:172
768199512,12cyclictest131rcu_preempt22:35:160
769899481,5cyclictest0-21swapper/222:55:162
769899475,33cyclictest17490-21ssh23:25:162
769899472,28cyclictest1592-21ssh21:35:142
769899471,5cyclictest0-21swapper/220:15:162
769899471,41cyclictest0-21swapper/200:10:142
768199472,33cyclictest16777-21ssh00:05:170
769899463,37cyclictest23346irq/40-dwc2_hso23:45:182
769899463,37cyclictest23346irq/40-dwc2_hso23:45:182
7688994616,9cyclictest5988-21grep20:45:311
770699453,26cyclictest32466irq/58-eth022:45:133
769899451,3cyclictest0-21swapper/222:10:132
769899451,3cyclictest0-21swapper/222:10:122
769899451,3cyclictest0-21swapper/220:20:162
768199451,36cyclictest25040-21ssh23:10:140
769899441,39cyclictest0-21swapper/222:25:182
769899441,2cyclictest9569-21ssh21:40:152
769899441,2cyclictest1194-21apt-get23:35:162
770699434,25cyclictest32466irq/58-eth022:40:123
769899431,5cyclictest0-21swapper/222:35:162
769899431,3cyclictest0-21swapper/221:50:172
769899420,2cyclictest0-21swapper/221:15:122
7688994228,11cyclictest17485-21ssh23:25:161
768199423,6cyclictest131rcu_preempt20:40:150
769899414,33cyclictest0-21swapper/219:55:152
769899411,36cyclictest0-21swapper/200:35:152
769899411,35cyclictest24911-21apt-get22:50:182
769899411,2cyclictest4291-21apt-get19:40:182
769899401,2cyclictest22219-21apt-get19:25:132
769899401,2cyclictest0-21swapper/221:25:152
769899401,2cyclictest0-21swapper/221:25:142
7688994030,5cyclictest479-21apt-get22:35:161
7688994023,13cyclictest0-21swapper/123:55:161
7688994023,12cyclictest0-21swapper/121:50:161
7688994013,10cyclictest24494-21ssh23:49:331
7688994013,10cyclictest24494-21ssh23:49:321
768199401,8cyclictest131rcu_preempt19:30:150
769899391,7cyclictest0-21swapper/221:20:142
769899391,34cyclictest23346irq/40-dwc2_hso00:25:122
769899391,2cyclictest16657-21apt-get21:00:152
769899391,2cyclictest0-21swapper/200:05:172
769899391,28cyclictest25438-21sed23:10:182
769899391,10cyclictest26890-21sed21:30:192
769899391,10cyclictest26890-21sed21:30:182
768899395,25cyclictest24696-21ssh00:10:141
7688993912,9cyclictest25013-21ssh22:10:121
7688993912,9cyclictest25013-21ssh22:10:121
768899391,29cyclictest18322-21ssh21:25:151
768899391,29cyclictest18322-21ssh21:25:151
7688993911,10cyclictest8113-21ssh22:19:401
769899381,7cyclictest0-21swapper/200:15:152
769899381,5cyclictest0-21swapper/200:20:162
769899381,33cyclictest23346irq/40-dwc2_hso19:10:172
769899381,2cyclictest9036-21apt-get22:00:162
7688993833,2cyclictest29765-21apt-get20:40:161
7688993824,8cyclictest25165-21apt-get23:50:151
7688993819,14cyclictest0-21swapper/121:45:131
768199384,27cyclictest8781-21ssh00:00:130
768199384,20cyclictest25187-21ssh23:50:160
768199382,30cyclictest7868-21ssh00:20:170
769899372,2cyclictest8549-21apt-get23:00:142
769899371,6cyclictest2993-21kthreadcore20:45:212
769899371,33cyclictest12928-21ps19:15:212
7688993729,4cyclictest7851-21apt-get00:20:171
7688993714,8cyclictest29933-21ssh22:31:131
7688993713,13cyclictest7159-21ssh21:36:511
7688993712,10cyclictest30486-21sh22:51:501
7688993710,9cyclictest6419-21ssh23:16:341
769899362,24cyclictest22647-21fschecks_count00:30:192
769899362,24cyclictest22647-21fschecks_count00:30:182
769899361,8cyclictest16312-21cron22:25:022
769899361,6cyclictest13089-21grep19:45:302
769899361,3cyclictest0-21swapper/223:20:132
7688993631,2cyclictest7807-21apt-get19:10:161
7688993631,2cyclictest16770-21apt-get00:05:181
7688993630,3cyclictest10292-21apt-get21:20:151
7688993620,13cyclictest16948-21ssh22:45:131
7688993611,13cyclictest31790-21ssh21:31:551
7688993611,13cyclictest31790-21ssh21:31:551
7688993610,8cyclictest0-21swapper/121:19:271
7681993611,18cyclictest12-21ksoftirqd/023:45:170
7681993611,18cyclictest12-21ksoftirqd/023:45:170
770699352,19cyclictest32466irq/58-eth020:25:143
769899351,6cyclictest735-21sort20:10:012
769899351,5cyclictest0-21swapper/220:40:152
769899351,3cyclictest0-21swapper/223:50:152
769899351,3cyclictest0-21swapper/219:50:152
769899351,2cyclictest16930-21apt-get22:45:132
769899350,4cyclictest25669-21apt-get23:30:162
769899350,3cyclictest0-21swapper/220:30:152
768899359,8cyclictest0-21swapper/122:23:081
768899359,7cyclictest0-21swapper/121:56:481
7688993512,8cyclictest14210-21ssh22:01:111
7688993511,9cyclictest7385-21sh23:37:461
7688993511,8cyclictest32712-21ssh23:14:391
768199351,31cyclictest833-21ssh21:55:150
769899342,29cyclictest7797-21kthreadcore20:50:202
769899341,7cyclictest31149-21cat21:10:312
769899341,6cyclictest31569-21idleruntime-cro19:35:022
769899341,3cyclictest0-21swapper/220:55:152
769899341,2cyclictest8722-21apt-get22:40:122
768899348,11cyclictest16639-21/usr/sbin/munin21:00:261
768899347,11cyclictest14009-21ssh23:40:391
7688993427,3cyclictest31799-21apt-get19:35:161
7688993419,12cyclictest0-21swapper/122:55:171
7688993411,8cyclictest0-21swapper/123:00:031
7688993410,11cyclictest22681-21rm22:27:361
7688993410,10cyclictest14034-21ssh22:41:161
769899334,21cyclictest9319-21fschecks_count23:40:172
769899331,4cyclictest4643-21cat20:10:312
769899331,21cyclictest0-21swapper/221:58:042
769899330,6cyclictest3139-21timerwakeupswit19:35:302
768899337,11cyclictest9459-21sed20:15:311
7688993311,6cyclictest0-21swapper/121:12:251
7688993311,12cyclictest14875-21ssh00:02:371
769899321,3cyclictest26058-21awk21:10:012
768899329,8cyclictest15152-21ssh21:41:441
7688993217,11cyclictest7855-21mailstats19:40:311
7688993213,8cyclictest14687-21ssh23:21:301
768199324,19cyclictest12-21ksoftirqd/019:15:160
768199323,21cyclictest12-21ksoftirqd/020:00:160
770699312,14cyclictest32466irq/58-eth023:25:163
769899311,5cyclictest19124-21cut23:05:232
769899311,5cyclictest19124-21cut23:05:232
769899311,27cyclictest885-21apt-get23:15:142
769899311,27cyclictest17165-21/usr/sbin/munin22:05:212
7698993110,18cyclictest21967-21sshd19:22:342
7688993117,11cyclictest16869-21diskstats23:05:181
7688993117,11cyclictest16869-21diskstats23:05:171
7688993111,10cyclictest14081-21cat19:50:161
7706993015,9cyclictest32466irq/58-eth022:50:163
769899301,2cyclictest8770-21apt-get00:00:142
769899301,2cyclictest0-21swapper/221:49:102
768899307,8cyclictest25140-21cat20:00:221
7688993018,9cyclictest15638-21grep20:55:311
7688993016,9cyclictest0-21swapper/119:55:151
7688993015,12cyclictest9149-21/usr/sbin/munin19:45:251
7688993015,12cyclictest0-21swapper/123:30:171
7681993020,7cyclictest742-21ssh22:15:160
769899293,23cyclictest24014-21if_eth120:00:202
769899291,6cyclictest18763-21ntp_kernel_pll_20:25:262
768899297,10cyclictest0-21swapper/121:05:231
7688992922,4cyclictest27016-21apt-get19:30:151
7688992920,7cyclictest0-21swapper/120:20:131
7688992920,6cyclictest0-21swapper/120:05:141
7688992918,8cyclictest22201-21/usr/sbin/munin19:25:171
7688992917,8cyclictest4378-21ntpq20:10:281
7688992916,10cyclictest29545-21sendmail-msp20:40:021
7688992915,10cyclictest7207-21if_err_eth120:50:181
770699284,13cyclictest32466irq/58-eth020:40:143
770699282,19cyclictest32466irq/58-eth023:15:133
770699281,14cyclictest32466irq/58-eth021:35:143
7688992823,2cyclictest15330-21apt-get20:25:151
7688992816,9cyclictest21020-21mailstats19:20:301
7688992816,8cyclictest20115-21/usr/sbin/munin20:30:131
7688992814,11cyclictest16668-21usage_Anybus19:15:321
770699272,19cyclictest0-21swapper/300:05:173
770699271,13cyclictest32466irq/58-eth019:30:143
7706992711,11cyclictest32466irq/58-eth023:45:183
7706992711,11cyclictest32466irq/58-eth023:45:183
770699264,14cyclictest32566irq/59-eth000:25:143
770699262,13cyclictest32466irq/58-eth022:10:133
770699262,13cyclictest32466irq/58-eth022:10:133
7688992513,8cyclictest7670-21cyclictest19:10:021
7706992415,6cyclictest321rcuc/320:10:173
770699241,15cyclictest32466irq/58-eth000:20:173
768199242,17cyclictest2036-21ssh21:15:140
768199241,19cyclictest25692-21ssh23:30:170
768199231,3cyclictest131rcu_preempt20:50:140
768199231,2cyclictest131rcu_preempt20:25:220
7681992312,5cyclictest0-21swapper/019:35:160
770699227,10cyclictest32466irq/58-eth022:35:183
770699222,4cyclictest0-21swapper/320:15:153
7706992217,3cyclictest321rcuc/300:39:473
7706992216,4cyclictest321rcuc/323:33:583
770699221,3cyclictest0-21swapper/323:50:153
770699221,15cyclictest32466irq/58-eth022:20:153
769899221,5cyclictest7716-21latency_hist19:10:022
768199224,11cyclictest628-21Async09:12:570
768199222,16cyclictest15831-21irqstats00:25:190
768199221,3cyclictest0-21swapper/019:55:040
770699213,11cyclictest32466irq/58-eth021:00:183
7706992116,3cyclictest321rcuc/300:11:513
7706992115,3cyclictest321rcuc/323:45:013
7706992111,7cyclictest32466irq/58-eth021:25:153
7706992111,7cyclictest32466irq/58-eth021:25:143
768199212,2cyclictest131rcu_preempt19:45:140
768199212,16cyclictest25041-21apt-get22:10:130
768199212,16cyclictest25041-21apt-get22:10:130
768199212,12cyclictest12-21ksoftirqd/022:40:120
768199211,15cyclictest426-21cut00:15:180
768199211,13cyclictest28519-21find20:05:140
770699209,8cyclictest32466irq/58-eth021:50:183
770699204,9cyclictest0-21swapper/322:55:153
7706992015,4cyclictest321rcuc/323:20:203
7706992015,3cyclictest321rcuc/322:30:343
7706992015,3cyclictest321rcuc/322:05:223
7706992015,3cyclictest321rcuc/322:05:013
7706992015,3cyclictest321rcuc/320:55:243
7706992014,4cyclictest321rcuc/323:00:173
7706992014,4cyclictest0-21swapper/323:38:253
7706992014,4cyclictest0-21swapper/321:20:013
7706992014,3cyclictest321rcuc/321:42:173
7706992013,5cyclictest0-21swapper/321:05:203
7706992012,5cyclictest321rcuc/319:50:163
7706992011,6cyclictest0-21swapper/300:00:193
768199208,3cyclictest131rcu_preempt23:25:240
768199204,6cyclictest11176-21tr22:00:240
7681992015,3cyclictest17053-21tr22:25:230
768199201,16cyclictest21615-21tune2fs21:05:180
768199201,13cyclictest17641-21switchtime19:50:300
768199201,10cyclictest30257-21irqrtprio00:35:200
768199200,16cyclictest16523-21ls23:05:010
7706991914,3cyclictest321rcuc/322:29:403
7706991913,4cyclictest0-21swapper/323:06:423
7706991913,4cyclictest0-21swapper/323:06:423
7706991913,4cyclictest0-21swapper/321:30:263
7706991913,4cyclictest0-21swapper/321:30:253
7706991913,4cyclictest0-21swapper/321:20:193
7706991913,4cyclictest0-21swapper/321:15:023
7706991913,4cyclictest0-21swapper/319:15:183
7706991913,4cyclictest0-21swapper/300:17:283
7706991912,4cyclictest0-21swapper/300:30:563
7706991912,4cyclictest0-21swapper/300:30:563
7706991911,6cyclictest0-21swapper/323:10:333
768199199,8cyclictest738-21apt-get23:55:140
768199197,3cyclictest131rcu_preempt23:15:250
768199196,7cyclictest12-21ksoftirqd/023:41:420
768199193,3cyclictest131rcu_preempt20:20:140
768199192,3cyclictest28989-21rm00:34:160
768199192,3cyclictest28989-21rm00:34:160
768199192,2cyclictest0-21swapper/021:41:290
768199192,15cyclictest5269-21kthreadcore19:40:230
768199191,4cyclictest0-21swapper/021:26:560
768199191,4cyclictest0-21swapper/021:26:560
768199191,10cyclictest28359-21tr21:10:220
7681991910,6cyclictest25498-21kthreadcore22:30:220
770699182,10cyclictest0-21swapper/320:45:153
770699182,10cyclictest0-21swapper/320:20:153
7706991813,3cyclictest0-21swapper/321:56:443
7706991812,4cyclictest321rcuc/320:30:133
7706991812,4cyclictest0-21swapper/323:56:053
7706991812,4cyclictest0-21swapper/321:46:123
768199185,3cyclictest131rcu_preempt22:55:150
768199183,4cyclictest12-21ksoftirqd/000:10:300
768199181,9cyclictest4842-21kthreadcore21:35:240
768199181,6cyclictest20773-21/usr/sbin/munin22:05:270
768199181,6cyclictest18095-21kthreadcore23:05:220
768199181,6cyclictest18095-21kthreadcore23:05:220
7681991811,4cyclictest9198-21sh23:20:130
7681991810,5cyclictest17107-21fschecks_count22:45:160
768199180,8cyclictest12-21ksoftirqd/022:20:150
768199180,16cyclictest5222-21timerwakeupswit23:35:330
770699177,7cyclictest321rcuc/320:05:173
7706991712,3cyclictest321rcuc/319:35:223
7706991712,3cyclictest0-21swapper/320:50:173
7706991712,3cyclictest0-21swapper/320:38:033
7706991712,3cyclictest0-21swapper/319:55:233
768199178,6cyclictest21354-21wc21:45:290
768199175,9cyclictest30036-21switchtime21:30:280
768199175,9cyclictest30036-21switchtime21:30:280
768199175,9cyclictest21984-21cron19:25:010
768199174,11cyclictest131rcu_preempt20:10:250
768199171,9cyclictest29028-21ssh21:50:300
768199171,13cyclictest20132-21apt-get20:30:130
768199170,3cyclictest131rcu_preempt21:00:240
770699162,9cyclictest0-21swapper/319:10:173
7706991611,3cyclictest321rcuc/319:45:303
7706991611,3cyclictest0-21swapper/319:25:273
7706991611,3cyclictest0-21swapper/319:20:213
768199166,8cyclictest25024-21df_inode20:35:160
768199162,11cyclictest4402-21kthreadcore20:45:220
768199161,4cyclictest0-21swapper/020:56:400
768199161,11cyclictest5744-21cpu20:15:170
7681991610,3cyclictest12-21ksoftirqd/019:25:250
768199160,6cyclictest7689-21latency_hist19:10:020
7706991510,3cyclictest0-21swapper/320:00:273
770699149,3cyclictest0-21swapper/319:40:163
77069991,5cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional