You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-17 - 14:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Nov 17, 2025 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2414899612,40cyclictest8289-21ssh00:30:170
2415799592,48cyclictest26966-21ssh21:20:161
2414899585,45cyclictest796-21ssh23:05:180
2414899573,39cyclictest10580-21ssh21:30:140
2416599552,48cyclictest2116-21apt-get21:45:162
2414899521,40cyclictest24776-21ssh00:20:170
2416599511,46cyclictest0-21swapper/223:50:142
2416599471,33cyclictest0-21swapper/200:00:152
2416599461,39cyclictest4384-21apt-get20:30:142
2416599461,39cyclictest0-21swapper/222:25:142
2414899463,33cyclictest465-21ssh00:25:140
2417399453,24cyclictest29566irq/58-eth021:20:173
2416599452,38cyclictest17631-21apt-get22:15:142
2416599450,40cyclictest20857-21apt-get19:40:182
2416599441,39cyclictest26963-21apt-get21:20:182
2416599441,39cyclictest0-21swapper/219:50:162
2416599441,34cyclictest24767-21apt-get00:20:172
2416599440,38cyclictest8286-21apt-get00:30:192
2414899443,32cyclictest10385-21diskmemload23:00:130
2416599421,33cyclictest15642-21apt-get00:35:172
2416599420,36cyclictest460-21ssh00:25:132
2416599420,35cyclictest10773-21apt-get21:10:172
2416599411,24cyclictest16641-21ssh23:35:142
2414899418,21cyclictest12-21ksoftirqd/019:45:150
2414899414,26cyclictest26970-21ssh21:20:170
24148994122,13cyclictest420-21ssh23:45:170
2417399403,31cyclictest29566irq/58-eth021:45:153
2416599402,27cyclictest0-21swapper/220:50:132
2416599401,30cyclictest10385-21diskmemload21:50:132
2416599400,36cyclictest0-21swapper/220:10:142
2414899400,31cyclictest24763-21ssh23:40:140
2416599391,33cyclictest471-21apt-get00:05:152
2416599381,34cyclictest32554-21ssh23:25:012
2416599381,34cyclictest28514-21apt-get20:55:132
2416599381,22cyclictest0-21swapper/223:32:102
2416599381,22cyclictest0-21swapper/223:32:092
24173993722,7cyclictest29566irq/58-eth023:20:133
24165993718,13cyclictest0-21swapper/222:40:122
2416599371,33cyclictest0-21swapper/221:00:142
2416599371,32cyclictest32412-21kthreadcore19:15:232
2416599371,31cyclictest0-21swapper/219:25:132
2416599371,22cyclictest14753-21ssh22:51:502
2416599370,32cyclictest0-21swapper/223:25:142
24165993619,13cyclictest0-21swapper/222:10:152
2416599361,32cyclictest4954-21cat22:45:292
2416599361,31cyclictest9267-21ntp_states21:05:292
2416599361,30cyclictest0-21swapper/222:55:162
2416599361,29cyclictest18012-21ssh21:55:172
24157993630,3cyclictest0-21swapper/121:40:161
2414899360,27cyclictest1526-21rm22:05:160
2416599352,30cyclictest13018-21find22:30:322
2416599352,29cyclictest26346irq/40-dwc2_hso23:10:152
24165993517,4cyclictest0-21swapper/219:10:162
2416599351,31cyclictest24565-21kthreadcore20:15:222
2416599351,29cyclictest0-21swapper/220:45:132
2416599351,26cyclictest16926-21ssh21:32:582
2416599350,28cyclictest2886-21memory20:25:252
2416599350,27cyclictest20148-21ntp_192.168.11400:15:272
2416599350,24cyclictest14214-21ssh00:11:552
2416599342,27cyclictest0-21swapper/223:55:152
2416599342,27cyclictest0-21swapper/223:55:142
24165993418,13cyclictest27-21ksoftirqd/221:40:142
2416599341,26cyclictest5044-21ntp_states19:20:252
2416599340,28cyclictest2576-21apt-get21:25:152
2415799342,29cyclictest26007-21ssh22:00:161
24157993415,16cyclictest21-21ksoftirqd/100:05:151
24148993412,13cyclictest12-21ksoftirqd/020:50:140
2416599330,30cyclictest10385-21diskmemload22:07:552
2417399324,24cyclictest29566irq/58-eth000:35:163
2417399321,19cyclictest29566irq/58-eth019:55:143
2416599322,27cyclictest26346irq/40-dwc2_hso23:15:152
2416599321,28cyclictest0-21swapper/221:15:172
2416599321,27cyclictest751-21sendmail-mta20:41:102
2416599321,22cyclictest5693-21ssh23:05:402
2416599321,21cyclictest24724-21/usr/sbin/munin23:40:152
2416599320,24cyclictest30536-21ssh23:01:132
2416599320,24cyclictest17233-21/usr/sbin/munin20:05:342
2416599320,23cyclictest6004-21ssh23:46:472
2416599320,23cyclictest18441-21date21:35:022
24157993210,13cyclictest21-21ksoftirqd/121:10:171
2416599311,27cyclictest4685-21cut19:55:222
2416599311,22cyclictest29124-21kthreadcore20:20:222
2416599311,22cyclictest17818-21irqrtprio22:35:192
2416599310,24cyclictest31785-21ssh22:02:082
2415799315,21cyclictest8888-21ssh23:10:161
2415799314,19cyclictest2123-21ssh21:45:151
2414899313,24cyclictest930-21ssh22:45:150
2416599306,16cyclictest13718-21cat20:35:332
2416599301,26cyclictest30244-21sshd19:46:482
2416599301,24cyclictest11184-21find19:30:142
2416599301,24cyclictest11184-21find19:30:142
2416599301,21cyclictest7834-21/usr/sbin/munin20:00:272
2416599300,28cyclictest31843-21ssh22:22:342
24173992914,10cyclictest29566irq/58-eth022:37:303
2415799292,18cyclictest2585-21ssh21:25:151
2417399281,15cyclictest29566irq/58-eth021:55:163
2416599280,23cyclictest16012-21/usr/sbin/munin19:35:182
2415799284,20cyclictest24918-21ssh23:20:151
2415799281,22cyclictest9076-21ssh22:50:161
2415799281,22cyclictest0-21swapper/120:15:151
2414899288,11cyclictest12-21ksoftirqd/020:30:130
2414899282,20cyclictest14469-21ssh00:12:180
2415799272,19cyclictest23371-21ssh23:58:211
2415799272,19cyclictest23371-21ssh23:58:201
2417399263,17cyclictest0-21swapper/322:46:503
2415799264,20cyclictest8486-21ssh23:50:141
2415799262,18cyclictest31737-21ssh22:22:281
24157992613,10cyclictest17262-21cron20:10:011
2414899262,19cyclictest7686-21ssh00:09:160
2414899261,20cyclictest0-21swapper/022:55:160
2417399254,15cyclictest0-21swapper/321:25:133
2417399251,11cyclictest29566irq/58-eth020:20:153
24173992510,10cyclictest0-21swapper/300:07:163
2415799253,7cyclictest131rcu_preempt22:35:141
24157992521,2cyclictest0-21swapper/123:00:141
2415799251,2cyclictest131rcu_preempt00:20:181
2415799251,19cyclictest31992-21apt-get20:25:151
2414899251,16cyclictest629-21Async14:00:480
24157992410,11cyclictest21-21ksoftirqd/120:10:141
2414899244,14cyclictest151rcuc/023:15:170
2414899241,20cyclictest10844-21ls21:10:180
2417399231,16cyclictest29566irq/58-eth000:25:133
2416599231,19cyclictest24306-21grep19:10:022
2416599231,19cyclictest24306-21grep19:10:022
2415799238,12cyclictest21-21ksoftirqd/119:40:151
2415799232,19cyclictest201rcuc/100:10:161
2415799232,17cyclictest8684-21ssh23:30:171
2415799232,17cyclictest8684-21ssh23:30:171
2415799232,16cyclictest25459-21ssh23:40:191
2414899239,7cyclictest0-21swapper/022:30:160
2414899231,18cyclictest22444-21cut20:15:160
2417399222,13cyclictest29566irq/58-eth023:00:143
24173992216,4cyclictest0-21swapper/323:35:133
2417399221,15cyclictest0-21swapper/300:12:183
2415799225,13cyclictest24877-21ssh00:00:141
24157992212,7cyclictest0-21swapper/120:45:191
2415799221,18cyclictest25302-21find22:40:131
24157992211,3cyclictest21-21ksoftirqd/119:35:151
2414899221,15cyclictest842-21munin-node21:50:150
24148992210,5cyclictest12-21ksoftirqd/000:35:150
2417399219,7cyclictest0-21swapper/323:27:113
24173992113,5cyclictest29566irq/58-eth023:55:163
24173992113,5cyclictest29566irq/58-eth023:55:163
2415799219,9cyclictest19561-21irqstats21:15:221
2415799217,10cyclictest1717-21cut22:05:191
2415799213,13cyclictest21-21ksoftirqd/120:40:151
24157992113,5cyclictest660-21cron21:00:011
24157992112,6cyclictest12906-21ntpq22:30:281
2415799211,17cyclictest17634-21sh22:15:121
2415799211,15cyclictest9148-21ls21:05:261
2415799211,15cyclictest13691-21sh21:50:261
2415799211,14cyclictest8962-21latency_hist20:35:011
2415799211,14cyclictest15667-21sshd00:35:171
2414899214,8cyclictest0-21swapper/020:25:140
2414899212,13cyclictest151rcuc/023:25:150
2414899211,6cyclictest131rcu_preempt22:42:300
2414899211,17cyclictest32694-21ls00:05:010
2414899211,16cyclictest9142-21ls23:10:190
2414899211,16cyclictest17635-21rm22:15:130
2414899211,16cyclictest12083-21ssh23:50:240
2414899211,14cyclictest16656-21ssh23:35:150
2417399209,3cyclictest0-21swapper/322:31:303
2417399203,13cyclictest0-21swapper/322:15:123
24173992011,6cyclictest0-21swapper/321:15:163
24173992010,7cyclictest0-21swapper/322:24:123
24173992010,6cyclictest29566irq/58-eth000:15:143
2415799205,4cyclictest131rcu_preempt20:00:011
2415799205,12cyclictest3348-21cut00:25:221
2415799203,13cyclictest1270-21ssh22:25:131
2415799202,4cyclictest131rcu_preempt23:35:341
2415799202,15cyclictest14239-21ssh21:30:271
24157992011,6cyclictest19835-21sh23:15:241
2415799201,11cyclictest1421-21wc19:50:241
2415799200,2cyclictest131rcu_preempt22:45:121
2414899208,4cyclictest131rcu_preempt21:25:320
2414899204,3cyclictest131rcu_preempt22:10:200
2414899202,14cyclictest18864-21ssh21:15:160
24148992010,8cyclictest6134-21sed21:45:300
24148992010,6cyclictest21157-21timerandwakeup22:35:300
2414899200,3cyclictest131rcu_preempt22:25:140
2417399193,11cyclictest0-21swapper/322:42:313
24173991912,4cyclictest0-21swapper/322:10:193
24173991912,4cyclictest0-21swapper/321:40:293
24173991910,6cyclictest0-21swapper/322:50:483
24173991910,3cyclictest0-21swapper/300:00:253
2415799199,8cyclictest5303-21cat23:05:351
2415799199,7cyclictest18539-21sh00:15:231
2415799191,9cyclictest3131-21kthreadcore21:00:231
24157991912,4cyclictest17325-21cpu22:55:161
24157991911,5cyclictest18950-21grep21:35:221
2414899198,8cyclictest27237-21df20:20:160
2414899198,8cyclictest25995-21apt-get22:00:160
2414899197,8cyclictest12-21ksoftirqd/021:55:180
2414899192,11cyclictest16714-21ssh22:54:480
24148991914,3cyclictest0-21swapper/020:55:120
24148991911,6cyclictest14774-21sshd23:32:410
24148991911,6cyclictest14774-21sshd23:32:400
2414899191,14cyclictest16564-21ssh00:15:140
2414899191,11cyclictest24884-21find23:20:130
2417399189,7cyclictest0-21swapper/323:46:233
2417399189,6cyclictest0-21swapper/322:06:403
2417399189,6cyclictest0-21swapper/321:35:493
2417399188,6cyclictest0-21swapper/300:24:203
2417399183,10cyclictest0-21swapper/322:27:293
2417399182,10cyclictest0-21swapper/320:30:153
2417399182,10cyclictest0-21swapper/300:30:193
2417399181,14cyclictest0-21swapper/321:10:173
2417399181,11cyclictest0-21swapper/320:05:183
24173991810,6cyclictest0-21swapper/323:05:333
24173991810,6cyclictest0-21swapper/321:34:303
24173991810,5cyclictest29566irq/58-eth023:40:203
24173991810,4cyclictest29566irq/58-eth023:50:143
2415799188,7cyclictest932-21snmpd20:50:031
2415799188,7cyclictest32087-21kthreadcore19:15:211
2415799184,11cyclictest10800-21kthreadcore00:30:251
2415799181,8cyclictest1814-21ssh23:45:241
24157991811,4cyclictest20035-21kthreadcore21:55:241
2415799181,13cyclictest696-21diskstats23:25:161
2415799181,13cyclictest6383-21apt-get19:25:121
2414899189,7cyclictest24191-21memory19:40:280
2414899187,3cyclictest131rcu_preempt19:17:500
2414899186,9cyclictest1-21systemd20:47:090
2414899183,10cyclictest17673-21ls20:40:290
2414899182,11cyclictest21672-21ssh23:55:490
2414899182,11cyclictest21672-21ssh23:55:480
24148991812,4cyclictest0-21swapper/020:10:170
2414899181,11cyclictest18737-21uniq21:35:190
2417399179,6cyclictest0-21swapper/323:32:323
2417399179,6cyclictest0-21swapper/323:32:313
2417399179,6cyclictest0-21swapper/323:15:573
2417399179,6cyclictest0-21swapper/323:13:383
2417399179,6cyclictest0-21swapper/322:55:133
2417399172,11cyclictest0-21swapper/322:00:163
2417399171,13cyclictest0-21swapper/320:20:013
2417399171,11cyclictest0-21swapper/319:30:303
2417399171,11cyclictest0-21swapper/319:30:303
2417399171,10cyclictest0-21swapper/320:35:183
2415799173,11cyclictest9413-21wc20:35:191
2415799173,11cyclictest26798-21kthreadcore19:45:211
2415799172,12cyclictest1979-21irqcore19:20:171
24157991710,3cyclictest9509-21/usr/sbin/munin22:10:171
2414899179,5cyclictest0-21swapper/019:26:500
2414899178,7cyclictest16002-21memory20:05:250
24148991711,4cyclictest7521-21sendmail20:00:000
2414899171,12cyclictest26602-21apt-get21:40:150
2414899171,10cyclictest15903-21ls19:35:020
2414899171,10cyclictest15903-21ls19:35:010
2414899170,12cyclictest27644-21ls19:10:250
2417399168,6cyclictest0-21swapper/319:27:493
2417399161,9cyclictest0-21swapper/321:50:153
2417399161,9cyclictest0-21swapper/320:10:243
2417399161,12cyclictest0-21swapper/320:55:153
2417399161,11cyclictest0-21swapper/319:20:023
2417399161,10cyclictest0-21swapper/319:45:163
24173991610,4cyclictest0-21swapper/319:35:143
2415799169,4cyclictest10690-21cut20:00:241
2415799162,11cyclictest24227-21latency_hist19:10:021
2415799162,11cyclictest24227-21latency_hist19:10:011
2415799162,10cyclictest11320-21fschecks_count19:30:181
2415799162,10cyclictest11320-21fschecks_count19:30:171
2415799161,12cyclictest24531-21fschecks_time19:10:201
2415799161,11cyclictest27379-21hwlatdetect20:20:171
2414899167,7cyclictest1-21systemd20:01:580
2414899166,8cyclictest16086-21sort19:35:150
2414899162,7cyclictest0-21swapper/019:50:160
2414899162,12cyclictest10640-21kthreadcore20:35:210
2414899162,10cyclictest0-21swapper/019:20:120
2414899161,12cyclictest5927-21cut21:05:170
2414899161,12cyclictest1640-21tr21:00:210
2417399151,11cyclictest0-21swapper/320:25:183
2417399149,3cyclictest0-21swapper/320:01:363
2417399141,9cyclictest0-21swapper/319:20:123
2417399141,10cyclictest0-21swapper/321:05:183
2417399138,3cyclictest0-21swapper/319:50:193
2417399138,3cyclictest0-21swapper/319:43:163
2417399135,5cyclictest29566irq/58-eth020:45:133
2417399131,7cyclictest0-21swapper/321:00:153
2417399131,5cyclictest0-21swapper/320:40:213
2414899131,10cyclictest24140-21cyclictest19:10:010
2414899131,10cyclictest24140-21cyclictest19:10:000
2417399127,3cyclictest0-21swapper/319:10:303
2417399121,9cyclictest0-21swapper/319:10:013
2417399121,9cyclictest0-21swapper/319:10:013
2417399111,8cyclictest0-21swapper/320:50:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional