You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 08:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Tue Nov 18, 2025 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3240599601,51cyclictest7394-21ssh22:40:171
32405995939,11cyclictest21-21ksoftirqd/120:30:121
3240599532,42cyclictest21965-21ssh22:10:141
3239799471,40cyclictest20899-21apt-get20:05:150
3241099461,27cyclictest18142-21ssh00:25:152
3240599465,34cyclictest26573-21ssh23:50:161
3239799462,41cyclictest0-21swapper/000:15:150
3239799461,41cyclictest14632-21apt-get19:25:140
3239799461,40cyclictest11321-21apt-get21:25:170
32397994533,8cyclictest0-21swapper/021:45:120
3239799451,38cyclictest26603-21apt-get00:10:020
3239799451,38cyclictest24048-21apt-get22:50:130
3239799451,38cyclictest17449-21apt-get20:35:160
3241799436,23cyclictest29566irq/58-eth020:55:183
32417994328,8cyclictest29566irq/58-eth022:15:173
3240599432,32cyclictest2949-21ssh21:20:131
32397994337,3cyclictest16375-21sh23:05:160
3239799433,37cyclictest0-21swapper/021:00:150
3239799433,37cyclictest0-21swapper/021:00:140
3239799431,38cyclictest0-21swapper/000:00:140
3239799431,37cyclictest30547-21apt-get20:15:140
3239799430,2cyclictest0-21swapper/021:55:150
3239799423,35cyclictest21956-21apt-get22:10:140
3239799422,34cyclictest1364-21apt-get23:35:130
3241799413,22cyclictest29566irq/58-eth000:20:123
3239799411,36cyclictest31430-21apt-get22:35:150
3239799410,35cyclictest0-21swapper/022:00:120
3239799410,2cyclictest0-21swapper/021:05:150
32417994017,11cyclictest29566irq/58-eth021:40:143
3241099401,5cyclictest131rcu_preempt21:20:132
3239799405,31cyclictest26992-21ssh21:15:130
3239799402,36cyclictest24582-21ssh23:10:140
3239799401,36cyclictest18612-21diskmemload21:10:170
3239799401,35cyclictest0-21swapper/022:45:170
3239799401,34cyclictest9858-21apt-get23:40:170
3239799390,27cyclictest26856-21sendmail-mta20:41:100
3239799390,27cyclictest26856-21sendmail-mta20:41:100
3239799390,24cyclictest8394-21ssh21:21:230
3241799389,18cyclictest29566irq/58-eth023:10:133
3239799381,34cyclictest0-21swapper/022:20:130
3239799380,5cyclictest1470-21apt-get00:35:160
3239799370,33cyclictest6421-21apt-get19:50:150
3239799370,2cyclictest2475-21ssh23:55:130
3241099362,17cyclictest2661-21ssh21:38:352
3239799361,32cyclictest2989-21apt-get20:20:140
3239799361,32cyclictest2989-21apt-get20:20:130
3239799361,31cyclictest18135-21apt-get00:25:150
3239799361,30cyclictest18161-21apt-get23:45:140
3239799361,30cyclictest12613-21apt-get20:30:120
3239799361,30cyclictest0-21swapper/023:25:150
3239799360,33cyclictest11234-21apt-get19:55:170
3239799360,24cyclictest731-21sh23:53:040
3239799351,30cyclictest11449-21ssh22:40:320
3239799351,30cyclictest0-21swapper/000:20:130
3239799351,22cyclictest556-21sh21:35:390
3241799343,22cyclictest29566irq/58-eth021:10:173
32417993419,10cyclictest29566irq/58-eth022:05:143
3241799341,22cyclictest0-21swapper/319:45:163
3240599342,22cyclictest19116-21ssh00:05:131
3239799341,31cyclictest30447-21ssh22:15:190
3239799341,30cyclictest19118-21apt-get00:05:130
3239799341,30cyclictest1156-21apt-key00:35:010
3239799341,29cyclictest19440-21apt-get19:30:140
3239799340,30cyclictest9795-21apt-get19:20:160
3239799340,30cyclictest18394-21ntp_192.168.11422:25:270
32417993317,10cyclictest0-21swapper/323:25:173
3241799330,28cyclictest0-21swapper/300:10:023
3239799331,29cyclictest5764-21cut21:40:210
3239799331,29cyclictest0-21swapper/019:40:150
3239799331,28cyclictest31980-21grep20:50:150
3239799330,30cyclictest20807-21cut21:30:210
3239799330,2cyclictest32272-21ssh22:55:150
3241799328,16cyclictest29566irq/58-eth000:30:143
3241799323,26cyclictest0-21swapper/321:15:133
3241799321,28cyclictest0-21swapper/322:45:173
3241099322,23cyclictest18612-21diskmemload23:35:122
3239799327,22cyclictest22550-21cut21:50:230
3239799327,22cyclictest22550-21cut21:50:220
32397993217,9cyclictest0-21swapper/023:00:150
3239799321,28cyclictest9183-21usage_EthcTask19:15:290
3239799321,28cyclictest4430-21df_inode20:55:180
3239799321,28cyclictest25443-21df23:30:140
3239799321,28cyclictest1243-21kthreadcore19:10:210
3239799321,27cyclictest32755-21/usr/sbin/munin23:15:280
3239799320,2cyclictest0-21swapper/023:20:150
3239799311,27cyclictest7781-21/usr/sbin/munin20:25:130
3239799311,27cyclictest32511-21latency_hist19:10:010
3239799311,27cyclictest18226-21cat22:05:340
3239799311,27cyclictest1297-21apt-get19:45:030
3239799311,26cyclictest16070-21/usr/sbin/munin20:00:230
3239799310,28cyclictest27790-21ps19:35:270
3239799310,27cyclictest26260-21cut22:30:230
32417993018,5cyclictest0-21swapper/321:05:143
3241799301,21cyclictest0-21swapper/300:00:143
3241099303,12cyclictest0-21swapper/221:48:392
3241099302,21cyclictest0-21swapper/219:45:052
3239799301,26cyclictest27949-21cut20:45:210
3239799301,25cyclictest0-21swapper/020:10:140
3241799293,18cyclictest0-21swapper/321:35:153
32417992918,7cyclictest0-21swapper/321:55:153
3241799291,24cyclictest0-21swapper/319:35:133
3241099293,21cyclictest0-21swapper/223:19:342
3241099293,21cyclictest0-21swapper/221:07:352
3241099293,20cyclictest27030-21ssh00:10:132
3241099291,10cyclictest2590966ssh00:30:142
3241799283,21cyclictest0-21swapper/319:40:153
3241799282,15cyclictest29566irq/58-eth000:05:153
3241799281,20cyclictest0-21swapper/300:15:153
3241099283,19cyclictest26346irq/40-dwc2_hso19:55:182
3241099283,19cyclictest0-21swapper/223:09:342
3241099281,9cyclictest651966ssh22:20:152
32410992812,13cyclictest0-21swapper/220:45:272
3241099281,17cyclictest26346irq/40-dwc2_hso19:50:142
3241099281,16cyclictest19140-21ssh00:05:142
32417992719,5cyclictest0-21swapper/322:55:153
32417992718,5cyclictest0-21swapper/322:20:133
3241799270,23cyclictest0-21swapper/323:45:133
3241099271,11cyclictest22458-21ssh22:29:352
3241099271,10cyclictest1774-21ssh21:55:352
3240599272,20cyclictest2693-21ssh21:38:371
3240599272,19cyclictest23895-21ssh23:09:351
3240599271,21cyclictest24054-21ssh22:50:141
3241799260,23cyclictest0-21swapper/322:40:163
3241099262,8cyclictest6108-21ssh22:38:482
3241099262,19cyclictest11159-21ssh21:43:382
3241099261,9cyclictest18612-21diskmemload21:33:362
3241099261,21cyclictest15818-21ssh23:23:522
3241099261,15cyclictest26346irq/40-dwc2_hso20:20:142
3241099261,15cyclictest26346irq/40-dwc2_hso20:20:132
32410992611,12cyclictest7953-21grep20:25:172
3241099260,18cyclictest25615-21ls23:30:172
3240599262,18cyclictest17009-21/usr/sbin/munin23:25:341
3241799253,18cyclictest0-21swapper/320:40:133
3241799253,18cyclictest0-21swapper/320:40:133
3241799252,18cyclictest0-21swapper/323:20:143
3241799250,21cyclictest0-21swapper/322:00:123
3241099251,19cyclictest19916-21ssh22:07:342
3241799245,15cyclictest0-21swapper/321:25:163
3241799241,20cyclictest0-21swapper/323:05:153
3241799241,20cyclictest0-21swapper/320:25:143
3241799241,19cyclictest0-21swapper/321:20:143
3241799241,19cyclictest0-21swapper/319:30:153
3241799241,18cyclictest0-21swapper/323:15:153
32417992411,6cyclictest0-21swapper/321:00:143
32417992411,6cyclictest0-21swapper/321:00:143
3241799240,21cyclictest0-21swapper/320:15:133
3241799240,20cyclictest0-21swapper/322:25:143
3241799240,19cyclictest0-21swapper/323:55:133
3241799240,19cyclictest0-21swapper/323:40:163
3241799240,18cyclictest0-21swapper/323:35:143
3241099242,8cyclictest18612-21diskmemload00:04:352
3241099242,6cyclictest0-21swapper/221:53:382
3241099242,6cyclictest0-21swapper/221:53:382
3241099242,17cyclictest0-21swapper/222:48:342
3241099242,16cyclictest0-21swapper/222:58:522
3241099241,7cyclictest844566ssh00:39:372
3241099241,7cyclictest1031466rm22:00:472
3241099241,6cyclictest2585066ssh21:13:422
3241099241,18cyclictest30240-21ssh22:52:342
3241099241,18cyclictest11344-21ssh21:25:172
3241099241,14cyclictest932-21snmpd20:50:142
32410992411,10cyclictest0-21swapper/221:15:242
3241099240,12cyclictest847266ssh23:57:102
3240599243,15cyclictest73-21kswapd022:45:181
3240599242,9cyclictest0-21swapper/100:25:151
32405992413,8cyclictest0-21swapper/121:25:171
3240599241,18cyclictest1387-21ssh23:35:131
3241799231,19cyclictest0-21swapper/323:50:153
3241799231,19cyclictest0-21swapper/322:34:223
3241799231,18cyclictest0-21swapper/320:35:163
3241799230,20cyclictest0-21swapper/323:30:133
3241099235,10cyclictest6365-21grep00:15:312
3241099232,13cyclictest26346irq/40-dwc2_hso23:00:152
3241099231,6cyclictest2854066ssh22:31:202
3241099231,6cyclictest2339066ssh23:46:042
32410992312,9cyclictest15390-21ssh00:21:072
3240599238,11cyclictest21-21ksoftirqd/100:34:341
3240599232,16cyclictest2481-21ssh23:55:141
3240599231,9cyclictest503-21irqstats22:55:201
3240599231,17cyclictest10419-21ssh21:42:351
32417992215,3cyclictest0-21swapper/320:10:143
3241799221,18cyclictest0-21swapper/321:30:293
3241799221,18cyclictest0-21swapper/300:35:113
3241799221,17cyclictest0-21swapper/321:45:053
3241799220,19cyclictest0-21swapper/322:50:283
3241099229,9cyclictest17232-21kthreadcore20:00:202
3241099222,17cyclictest26346irq/40-dwc2_hso22:40:162
3241099221,7cyclictest472966ssh22:18:122
3241099221,6cyclictest2722966ssh22:11:072
32410992211,8cyclictest0-21swapper/221:00:252
32410992211,8cyclictest0-21swapper/221:00:242
3241099221,12cyclictest19546-21ssh23:25:242
3240599222,3cyclictest131rcu_preempt23:10:141
3240599221,6cyclictest6059-21ntp_states00:15:271
32405992213,6cyclictest22434-21kthreadcore21:30:221
3240599221,17cyclictest30457-21ssh22:15:181
3240599220,2cyclictest131rcu_preempt00:00:141
32417992114,4cyclictest0-21swapper/319:10:153
3241799211,18cyclictest0-21swapper/300:25:143
3241799211,17cyclictest0-21swapper/323:01:433
3241799211,17cyclictest0-21swapper/322:10:133
3241799211,17cyclictest0-21swapper/321:52:033
3241799211,17cyclictest0-21swapper/321:52:023
3241799211,17cyclictest0-21swapper/320:50:173
3241799211,17cyclictest0-21swapper/320:20:163
3241799211,17cyclictest0-21swapper/320:20:163
3241799211,17cyclictest0-21swapper/320:00:173
3241799211,17cyclictest0-21swapper/319:55:233
3241799211,17cyclictest0-21swapper/319:25:133
3241799211,17cyclictest0-21swapper/319:15:123
3241799210,18cyclictest0-21swapper/320:45:173
3241099213,9cyclictest28153-21ntp_192.168.11423:10:272
3241099212,17cyclictest0-21swapper/223:50:182
3241099211,17cyclictest25850-21find20:40:282
3241099211,17cyclictest25850-21find20:40:272
3241099211,11cyclictest9842-21acpi23:40:162
3241099211,10cyclictest25508-21ls20:10:012
32410992110,8cyclictest0-21swapper/220:30:182
32410992110,4cyclictest9777-21/usr/sbin/munin19:20:192
3240599212,4cyclictest21-21ksoftirqd/121:45:251
32405992113,5cyclictest18465-21grep19:25:281
3240599211,15cyclictest3503-21ls19:10:241
3240599211,10cyclictest425066ssh23:15:311
32405992110,8cyclictest24699-21kthreadcore20:40:221
32405992110,8cyclictest24699-21kthreadcore20:40:211
3241799201,17cyclictest0-21swapper/320:05:143
3241799201,16cyclictest0-21swapper/322:35:143
3241799201,16cyclictest0-21swapper/319:20:193
3241799200,17cyclictest0-21swapper/319:50:263
3241799200,15cyclictest0-21swapper/320:30:123
3241099207,10cyclictest14762-21grep19:25:162
3241099202,10cyclictest28701-21kthreadcore20:10:232
3241099201,16cyclictest4325-21/usr/sbin/munin20:55:282
3241099201,16cyclictest2636-21kthreadcore19:10:222
3241099201,10cyclictest8567-21mailstats19:15:272
3241099201,10cyclictest31899-21kthreadcore20:15:212
3241099201,10cyclictest30220-21kthreadcore19:40:222
3241099201,10cyclictest24690-21sed19:35:192
3240599205,3cyclictest131rcu_preempt22:25:241
3240599202,12cyclictest31447-21ssh22:35:161
32405992011,6cyclictest15834-21cron20:00:011
3240599201,14cyclictest14166-21cut21:05:181
3240599200,9cyclictest8642-21/usr/sbin/munin23:20:281
3241099191,9cyclictest32465-21latency_hist19:10:022
3241099191,4cyclictest0-21swapper/220:35:172
3241099191,15cyclictest19618-21fschecks_time19:30:162
3240599193,12cyclictest32380-21cyclictest21:50:161
3240599193,12cyclictest32380-21cyclictest21:50:151
3240599192,12cyclictest0-21swapper/123:00:141
3240599191,9cyclictest20187-21tr23:45:231
32405991911,6cyclictest7541-21sshd20:20:361
32405991911,6cyclictest7541-21sshd20:20:351
3240599191,16cyclictest30749-21switchtime20:45:281
32405991911,6cyclictest30295-21sshd20:12:361
32405991911,6cyclictest17349-21ntpq22:05:281
32405991911,5cyclictest0-21swapper/123:30:161
3240599188,7cyclictest19769-21ps19:30:181
3240599187,8cyclictest670-21cat21:55:291
3240599181,9cyclictest11316-21cut00:20:201
3240599181,3cyclictest131rcu_preempt21:14:461
3240599181,14cyclictest10859-21kthreadcore23:40:231
32405991810,6cyclictest20657-21sshd20:03:361
3240599177,7cyclictest6186-21cron19:50:001
3240599172,9cyclictest0-21swapper/120:15:141
3240599171,9cyclictest11096-21usage_EthcTask22:20:331
3240599171,8cyclictest27715-21tr21:15:191
32405991712,3cyclictest8460-21proc_pri19:15:261
32405991712,3cyclictest21533-21irqstats20:05:201
3240599171,12cyclictest25450-21kthreadcore22:30:231
32405991710,5cyclictest8926-21sshd20:56:351
3240599168,6cyclictest6806-21sed19:50:201
3240599168,5cyclictest30806-21/usr/sbin/munin00:10:271
3240599168,5cyclictest28095-21grep19:35:291
3240599163,8cyclictest0-21swapper/120:35:181
3240599162,11cyclictest12112-21kthreadcore19:20:231
3240599162,10cyclictest0-21swapper/120:50:161
3240599161,9cyclictest29198-21/usr/sbin/munin19:40:191
3240599161,13cyclictest7798-21apt-get20:25:151
3240599161,12cyclictest8775-21ls00:40:021
3240599161,12cyclictest11225-21tr21:00:231
3240599161,12cyclictest11225-21tr21:00:221
3240599161,11cyclictest5965-21sed22:00:181
3241799151,12cyclictest0-21swapper/319:10:013
3240599151,11cyclictest32322-21latency_hist19:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional