You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 05:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue Jul 01, 2025 00:45:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1573799801,69cyclictest0-21swapper/219:30:152
15745996040,15cyclictest32466irq/58-eth021:45:183
1572899521,42cyclictest2084-21ssh23:30:131
1573799502,43cyclictest27027-21apt-get19:55:152
1573799501,42cyclictest23346irq/40-dwc2_hso23:05:172
1573799490,43cyclictest25546-21apt-get19:20:192
1573799490,43cyclictest25546-21apt-get19:20:182
1573799461,41cyclictest30356-21apt-get19:25:152
1573799460,41cyclictest0-21swapper/200:35:132
1573799451,41cyclictest15881-21apt-get19:10:132
1573799450,38cyclictest0-21swapper/220:30:172
1573799442,37cyclictest23346irq/40-dwc2_hso22:20:172
1573799441,40cyclictest0-21swapper/223:40:142
1573799430,34cyclictest0-21swapper/222:55:152
1574599422,27cyclictest32466irq/58-eth022:10:203
1573799421,36cyclictest0-21swapper/222:25:182
1573799420,39cyclictest0-21swapper/200:00:142
1573799411,34cyclictest31830-21apt-get20:00:132
1573799410,36cyclictest0-21swapper/223:45:152
1573799401,35cyclictest26723-21if_err_eth123:25:182
1573799401,34cyclictest0-21swapper/222:15:152
1573799395,31cyclictest6454-21cat23:30:322
1573799391,35cyclictest19517-21kthreadcore20:20:242
1573799391,34cyclictest15090-21ssh22:35:412
1573799391,34cyclictest0-21swapper/222:05:162
1574599384,21cyclictest32466irq/58-eth021:20:163
1574599383,18cyclictest32466irq/58-eth000:10:103
1574599381,19cyclictest32466irq/58-eth000:30:153
1573799381,33cyclictest26623-21/usr/sbin/munin21:45:342
1573799381,33cyclictest0-21swapper/219:35:182
1573799381,32cyclictest0-21swapper/200:30:142
15728993824,10cyclictest0-21swapper/100:00:121
1572099388,25cyclictest12-21ksoftirqd/022:10:180
1573799371,33cyclictest15525-21df_inode20:50:162
1573799370,33cyclictest10582-21apt-get21:35:192
1573799370,33cyclictest0-21swapper/221:30:152
1573799370,33cyclictest0-21swapper/220:55:122
1573799362,31cyclictest29903-21apt-get00:10:102
1573799361,32cyclictest9278-21ntp_states20:40:272
1573799361,32cyclictest20725-21missed_timers19:45:232
1573799361,31cyclictest17486-21ntp_states20:15:302
1573799361,30cyclictest0-21swapper/221:20:162
1573799361,27cyclictest26401-21apt-get22:45:162
1573799361,27cyclictest26401-21apt-get22:45:152
15737993611,13cyclictest18419-21sh22:00:132
1573799360,33cyclictest4252-21memory20:35:232
1573799360,29cyclictest0-21swapper/221:40:132
1572099363,22cyclictest12-21ksoftirqd/019:55:130
1573799351,31cyclictest9876-21grep23:55:012
1573799351,31cyclictest2462-21sed21:50:182
1573799351,27cyclictest15169-21users20:45:312
1573799350,32cyclictest2316-21fschecks_time22:30:182
1573799350,30cyclictest0-21swapper/223:15:122
1573799350,28cyclictest7914-21/usr/sbin/munin20:05:262
1573799350,28cyclictest12581-21ntp_192.168.11420:10:272
1572099352,28cyclictest18368-21ssh23:20:170
1572099351,31cyclictest18234-21sh22:40:170
1574599343,19cyclictest32466irq/58-eth022:25:163
1573799342,26cyclictest0-21swapper/221:10:152
1573799342,24cyclictest5540-21grep23:10:252
1573799341,30cyclictest5896-21if_eth100:15:182
1573799341,29cyclictest22947-21kthreadcore19:50:202
1573799341,28cyclictest13868-21apt-get00:20:152
1573799341,25cyclictest25831-21mailstats00:25:292
1573799341,25cyclictest15073-21users23:55:322
15737993411,21cyclictest24498-21ssh23:02:392
1573799340,31cyclictest0-21swapper/222:40:142
1573799340,25cyclictest10296-21grep23:35:152
1572099342,27cyclictest18800-21ssh21:20:160
1573799331,29cyclictest9963-21cron22:55:012
1573799331,29cyclictest22455-21unixbench_singl23:20:342
1573799331,29cyclictest10047-21ssh21:55:052
1573799330,30cyclictest0-21swapper/221:00:152
1574599323,19cyclictest32466irq/58-eth021:00:163
15745993225,4cyclictest32466irq/58-eth023:25:153
1573799321,28cyclictest16018-21ntp_states19:40:322
1573799320,29cyclictest2089-21diskmemload00:09:452
1573799320,29cyclictest14123-21ssh21:15:262
1573799320,22cyclictest0-21swapper/222:13:312
1572899322,21cyclictest662-21ntpd23:10:151
1572099321,25cyclictest10228-21ssh23:55:130
1573799311,27cyclictest24388-21/usr/sbin/munin19:15:272
1573799311,26cyclictest27108-21if_eth121:25:182
1573799310,28cyclictest27766-21usage_Anybus20:25:312
15728993118,6cyclictest0-21swapper/121:25:161
1572099313,19cyclictest10609-21ssh21:35:180
1572099311,23cyclictest26395-21ssh23:05:150
1574599303,19cyclictest0-21swapper/300:20:173
1573799301,26cyclictest863-21missed_timers21:05:252
1572899303,20cyclictest10141-21rm23:15:151
1572099305,19cyclictest12-21ksoftirqd/022:45:170
1572099305,19cyclictest12-21ksoftirqd/022:45:160
1572099301,23cyclictest26375-21ssh22:25:170
1574599298,13cyclictest32466irq/58-eth021:35:183
1574599293,15cyclictest32466irq/58-eth021:15:143
1572899293,19cyclictest18374-21ssh23:40:141
1572899292,19cyclictest26422-21ssh23:25:161
1572099294,17cyclictest10229-21ssh22:35:160
1572099292,22cyclictest5575-21ssh00:15:130
15720992917,10cyclictest12-21ksoftirqd/019:25:150
15720992916,8cyclictest15712-21cyclictest20:50:140
1572899282,21cyclictest5055-21ssh00:35:141
1572899281,3cyclictest131rcu_preempt23:20:161
1572099282,17cyclictest2204-21sh21:50:150
1574599277,8cyclictest0-21swapper/321:50:153
1574599274,13cyclictest0-21swapper/322:35:163
1574599272,17cyclictest0-21swapper/323:10:163
15745992611,10cyclictest32466irq/58-eth023:00:343
1572099263,15cyclictest29809-21ssh00:30:160
1572099261,19cyclictest0-21swapper/019:20:190
1572099261,19cyclictest0-21swapper/019:20:190
1573799251,20cyclictest15741-21cut19:10:022
1572899251,15cyclictest10217-21ssh21:55:171
1572099252,19cyclictest0-21swapper/019:35:200
1574599249,10cyclictest32466irq/58-eth021:30:163
1574599242,19cyclictest32466irq/58-eth021:25:153
15745992411,10cyclictest0-21swapper/322:10:023
1572899243,16cyclictest26358-21apt-get22:25:171
1572899241,19cyclictest29920-21ssh00:10:091
15720992412,9cyclictest10638-21ssh22:55:190
1574599238,8cyclictest32466irq/58-eth023:40:153
1574599233,10cyclictest32466irq/58-eth022:30:163
1574599231,14cyclictest0-21swapper/322:40:173
1572899232,17cyclictest10353-21ssh22:55:171
1572899231,15cyclictest10244-21ssh22:35:181
15728992310,3cyclictest131rcu_preempt20:40:011
1572099232,15cyclictest13889-21ssh00:20:170
15720992311,7cyclictest2116-21ssh22:30:160
1574599229,8cyclictest32466irq/58-eth023:15:133
1574599222,11cyclictest32466irq/58-eth023:30:143
1572899221,18cyclictest2066-21ssh22:10:181
1572899221,17cyclictest15754-21ls19:10:021
1572099223,12cyclictest0-21swapper/020:35:130
1572099222,17cyclictest15386-21ls20:50:020
15720992213,6cyclictest2559-21ssh21:30:160
1574599214,14cyclictest32466irq/58-eth020:30:153
1574599213,13cyclictest0-21swapper/300:15:133
1574599212,12cyclictest0-21swapper/323:20:173
1574599211,14cyclictest0-21swapper/322:50:133
1572899211,17cyclictest26384-21find22:45:161
1572899211,17cyclictest26384-21find22:45:151
1572899211,13cyclictest15866-21find19:10:131
1572899210,17cyclictest23451-21latency_hist20:25:011
1572899210,15cyclictest18706-21ls22:20:211
1572099212,14cyclictest26649-21ssh21:45:180
1572099212,13cyclictest10534-21ssh21:15:150
1572099211,9cyclictest21957-21memory00:00:240
1572099211,16cyclictest29706-21sh00:10:010
1572099211,15cyclictest22193-21apt-get19:50:130
1572099210,14cyclictest9871-21latency_hist23:55:020
1574599203,14cyclictest0-21swapper/323:45:223
1574599202,12cyclictest0-21swapper/322:00:143
1574599202,11cyclictest0-21swapper/319:30:033
15745992012,6cyclictest0-21swapper/320:45:193
15745992011,6cyclictest0-21swapper/323:50:283
1574599201,15cyclictest32466irq/58-eth000:35:133
1572899209,8cyclictest22163-21df00:25:151
1572899209,3cyclictest131rcu_preempt23:45:331
1572899205,5cyclictest2932-21kthreadcore22:50:201
1572899201,14cyclictest30303-21ntp_192.168.11421:45:301
1572099201,16cyclictest3029-21ps21:10:200
15720992011,6cyclictest10109-21ssh23:35:020
15720992011,6cyclictest0-21swapper/020:05:020
1572099201,14cyclictest18036-21latency_hist22:00:010
1572099201,13cyclictest29924-21sh00:10:110
1572099201,11cyclictest26820-21if_eth122:05:200
1574599197,7cyclictest0-21swapper/300:00:133
1574599192,11cyclictest0-21swapper/319:15:123
1574599191,14cyclictest0-21swapper/322:20:013
1574599191,10cyclictest0-21swapper/320:35:183
1572899191,9cyclictest0-21swapper/100:07:511
1572899191,16cyclictest25568-21missed_timers19:50:231
15728991911,5cyclictest4433-21kthreadcore21:10:231
1572899191,13cyclictest9796-21apt-get23:35:031
15728991910,7cyclictest24825-21cron21:00:011
15728991910,6cyclictest0-21swapper/123:05:341
15728991910,6cyclictest0-21swapper/120:15:221
1572899190,15cyclictest9131-21find20:10:131
1572099195,11cyclictest29663-21ssh23:45:240
1572099192,14cyclictest26749-21apt-get21:25:130
1572099192,14cyclictest14649-21cut20:15:230
1572099191,3cyclictest0-21swapper/000:27:370
15720991912,4cyclictest13672-21missed_timers23:15:240
15720991911,6cyclictest25277-21sshd19:17:420
1572099191,15cyclictest14965-21kthreadcore19:40:270
1572099191,14cyclictest3402-21kthreadcore22:50:220
1572099190,15cyclictest13817-21ls20:15:020
1572099190,13cyclictest22092-21missed_timers21:40:250
1572099190,13cyclictest18560-21cut22:00:160
1574599189,6cyclictest0-21swapper/300:06:173
1574599187,7cyclictest0-21swapper/323:39:413
1574599184,8cyclictest41-21kworker/3:120:15:193
1574599184,8cyclictest0-21swapper/321:10:193
1574599182,9cyclictest0-21swapper/323:05:183
1574599182,9cyclictest0-21swapper/319:10:143
1574599182,12cyclictest0-21swapper/322:55:183
1574599182,11cyclictest0-21swapper/321:55:173
1574599182,11cyclictest0-21swapper/319:35:203
1574599182,10cyclictest0-21swapper/320:00:143
1574599182,10cyclictest0-21swapper/319:45:133
15745991812,4cyclictest0-21swapper/300:25:193
1572899189,4cyclictest16912-21kthreadcore00:20:261
1572899187,7cyclictest0-21swapper/123:00:151
1572899186,9cyclictest1-21systemd20:47:511
1572899183,12cyclictest30330-21ps00:30:191
1572899182,8cyclictest27611-21kthreadcore22:05:231
1572899182,12cyclictest2177-21ssh23:50:151
1572899181,9cyclictest8094-21kthreadcore20:40:211
1572899181,9cyclictest19443-21ssh21:20:211
1572899181,7cyclictest21068-21irqcore19:15:161
15728991812,4cyclictest14663-21usage_Anybus21:15:321
15728991812,4cyclictest0-21swapper/121:30:191
1572899181,15cyclictest22026-21ls19:50:021
15728991810,5cyclictest5893-21mailstats22:30:301
1572099186,7cyclictest12-21ksoftirqd/021:00:150
1572099182,11cyclictest151rcuc/023:40:140
1572099181,8cyclictest9163-21wc20:40:240
1572099181,7cyclictest6090-21timerwakeupswit23:30:310
1572099181,5cyclictest18360-21apt-get23:00:150
15720991811,5cyclictest2451-21ssh23:10:200
1572099181,13cyclictest13846-21wc22:15:300
1572099181,12cyclictest0-21swapper/019:45:110
1574599179,6cyclictest0-21swapper/323:55:293
1574599179,6cyclictest0-21swapper/321:42:393
1574599178,6cyclictest0-21swapper/319:25:163
1574599172,9cyclictest0-21swapper/321:05:163
1574599172,9cyclictest0-21swapper/320:10:163
1574599172,10cyclictest0-21swapper/322:20:193
1574599171,13cyclictest0-21swapper/322:45:263
1574599171,13cyclictest0-21swapper/322:45:253
1574599171,10cyclictest0-21swapper/320:50:173
1572899175,9cyclictest612-21dbus-daemon20:52:511
1572899175,9cyclictest6083-21irqcore00:15:181
1572899175,9cyclictest27230-21kthreadcore19:20:261
1572899175,9cyclictest27230-21kthreadcore19:20:251
1572899173,8cyclictest0-21swapper/122:00:131
1572899173,11cyclictest25654-21irqrtprio21:00:201
1572899173,10cyclictest29221-21kthreadcore20:30:211
1572899172,10cyclictest0-21swapper/122:15:181
1572899171,12cyclictest26356-21cat21:45:011
1572899171,12cyclictest20193-21kthreadcore22:40:251
15728991710,4cyclictest0-21swapper/119:40:201
1572099176,8cyclictest20403-21tune2fs20:55:160
1572099176,3cyclictest89166lldpd20:32:260
1572099172,10cyclictest0-21swapper/019:10:140
15720991711,4cyclictest0-21swapper/000:38:220
15720991710,5cyclictest23619-21ssh22:21:150
15720991710,5cyclictest0-21swapper/020:20:270
1572099170,9cyclictest30047-21ssh23:25:270
1574599162,10cyclictest0-21swapper/320:00:003
1574599161,9cyclictest0-21swapper/319:40:283
1574599161,12cyclictest0-21swapper/319:20:223
1574599161,12cyclictest0-21swapper/319:20:213
1572899168,5cyclictest4568-21cut21:50:231
1572899163,10cyclictest7852-21fschecks_time19:35:231
1572899162,9cyclictest0-21swapper/120:25:161
1572899162,10cyclictest0-21swapper/120:05:141
1572899162,10cyclictest0-21swapper/119:30:161
15728991611,3cyclictest4268-21ls20:05:011
1572899161,12cyclictest201rcuc/121:35:271
1572899161,11cyclictest1191-21timerandwakeup21:05:321
1572899161,11cyclictest10331-21df23:55:161
1572099168,5cyclictest6002-21kthreadcore19:30:250
1572099161,12cyclictest15642-21latency_hist19:10:010
1574599151,10cyclictest0-21swapper/321:00:013
1572899153,9cyclictest30907-21irqrtprio19:25:201
1572899151,10cyclictest27248-21sed19:55:181
1572099157,5cyclictest30221-21/usr/sbin/munin21:05:200
1572099152,10cyclictest4004-21apt-get20:05:030
1572099151,11cyclictest23683-21apt-get20:25:160
1574599142,8cyclictest0-21swapper/320:25:013
1574599141,9cyclictest0-21swapper/320:45:003
1574599141,7cyclictest0-21swapper/319:50:233
1574599141,11cyclictest0-21swapper/320:25:183
1574599131,10cyclictest0-21swapper/320:05:283
1574599111,7cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional