You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-20 - 04:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Wed May 20, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25512995230,10cyclictest12-21ksoftirqd/019:35:140
25548995123,23cyclictest35966irq/58-eth000:00:153
2551899511,41cyclictest12846-21ssh00:10:141
2551299511,40cyclictest13866-21ssh22:05:130
2551899481,41cyclictest32649-21ssh23:45:151
2551899471,41cyclictest0-21swapper/123:40:121
2551899471,36cyclictest8593-21ssh22:20:141
2551899460,42cyclictest3860-21ssh00:05:181
2551299463,38cyclictest29782-21rm21:20:140
2551899452,39cyclictest25473-21apt-get00:35:131
2551899452,38cyclictest0-21swapper/100:30:141
2551899451,38cyclictest31868-21apt-get22:15:141
2552799440,27cyclictest930-21snmpd23:00:122
2551899441,38cyclictest18623-21apt-get20:10:141
2551899441,37cyclictest21793-21apt-get00:15:121
25518994318,13cyclictest630-21Async09:36:311
2551899422,36cyclictest22908-21apt-get22:10:121
2551899422,34cyclictest25729-21apt-get19:10:151
2551899421,37cyclictest13867-21apt-get22:05:141
2551899421,28cyclictest10860-21ssh23:15:141
2551899411,26cyclictest17964-21ssh22:41:261
2551899410,2cyclictest0-21swapper/119:30:111
2551899403,33cyclictest17346-21apt-get19:35:141
2551899403,28cyclictest11690-21ssh21:10:151
2551899401,35cyclictest0-21swapper/121:30:131
2551899401,34cyclictest0-21swapper/123:10:121
2551899400,2cyclictest0-21swapper/120:20:121
25512994011,24cyclictest12-21ksoftirqd/000:25:130
2554899395,19cyclictest35966irq/58-eth023:40:123
2551899392,32cyclictest0-21swapper/123:25:131
2551899391,31cyclictest670-21apt-get20:25:121
2551899390,2cyclictest0-21swapper/122:45:121
2554899382,22cyclictest35966irq/58-eth021:05:133
2552799381,22cyclictest3269-21ssh22:35:122
2551899381,32cyclictest14499-21apt-get23:35:141
2551899381,32cyclictest14499-21apt-get23:35:141
2551899381,32cyclictest0-21swapper/100:20:161
2551899380,3cyclictest29757-21apt-get21:20:121
2551899380,26cyclictest15084-21/usr/sbin/munin20:40:131
2551899380,26cyclictest15084-21/usr/sbin/munin20:40:131
2551899371,31cyclictest3263-21apt-get22:35:131
2551899371,31cyclictest30537-21apt-get19:15:121
2551899371,31cyclictest0-21swapper/121:50:121
2551899371,30cyclictest7373-21apt-get00:25:121
2551899370,24cyclictest23006-21ssh22:26:071
25512993714,17cyclictest17857-21sh23:18:050
2552799362,27cyclictest18141-21ssh23:55:112
2551899362,30cyclictest0-21swapper/122:30:121
2551899362,27cyclictest5409-21ssh23:30:121
2551899361,32cyclictest0-21swapper/123:05:121
2551899361,30cyclictest18220-21ls23:55:121
2551899361,30cyclictest0-21swapper/119:50:131
2551899361,29cyclictest13839-21apt-get20:05:131
2551899360,31cyclictest0-21swapper/121:25:111
2554899352,21cyclictest35966irq/58-eth000:10:133
2551899351,31cyclictest6718-21apt-get21:05:141
2551899351,29cyclictest3194-21cut21:40:211
2551899350,32cyclictest2938-21apt-get19:20:121
2551899350,24cyclictest1374-21sh00:02:381
2551899350,23cyclictest5430-21ssh22:53:351
25527993421,7cyclictest27-21ksoftirqd/200:10:132
2551899342,27cyclictest7121-21ssh22:55:121
2551899340,30cyclictest7719-21apt-get19:25:111
2551899340,30cyclictest7719-21apt-get19:25:111
2554899332,21cyclictest35966irq/58-eth023:00:133
25527993311,9cyclictest0-21swapper/221:15:162
2551899331,29cyclictest23670-21sendmail_mailst23:20:291
2551899331,28cyclictest25387-21kthreadcore20:50:191
2551899330,2cyclictest0-21swapper/120:45:261
2554899321,13cyclictest35966irq/58-eth022:55:113
2552799327,10cyclictest17856-21sh23:18:052
25527993210,19cyclictest29839-21diskmemload22:13:032
2551899321,28cyclictest0-21swapper/121:55:121
2551899321,27cyclictest27248-21/usr/sbin/munin19:45:171
2551899321,27cyclictest13634-21ssh22:05:011
2551899321,27cyclictest13634-21ssh22:05:011
2551899321,27cyclictest11107-21kthreadcore20:35:191
2551899321,25cyclictest10141-21/usr/sbin/munin21:45:141
2551899320,2cyclictest28010-21ssh21:35:231
2551899320,2cyclictest28010-21ssh21:35:231
2551899320,29cyclictest29584-21df20:55:141
2551899320,27cyclictest16194-21apt-get23:00:111
2552799311,20cyclictest28275-21ssh21:55:122
2551899311,27cyclictest5989-21sed20:30:201
2551899311,27cyclictest12411-21ntp_states20:00:251
2551899311,26cyclictest0-21swapper/120:15:121
2551899310,27cyclictest4258-21/usr/sbin/munin19:55:151
2551899310,27cyclictest25664-21sed19:10:021
2551899310,21cyclictest15993-21ssh23:52:531
2554899301,20cyclictest0-21swapper/322:45:123
2552799306,15cyclictest27346irq/40-dwc2_hso19:35:132
2552799304,17cyclictest27346irq/40-dwc2_hso19:55:122
2552799304,17cyclictest27243-21kworker/u8:100:25:132
2551899301,26cyclictest2774-21tr21:00:191
2551899301,26cyclictest22264-21df_inode19:40:141
2551299302,24cyclictest26680-21ssh22:30:120
2551299301,25cyclictest1127-21ssh21:40:120
2554899294,15cyclictest35966irq/58-eth023:35:143
2554899294,15cyclictest35966irq/58-eth023:35:143
2554899291,20cyclictest35966irq/58-eth023:30:123
2552799292,20cyclictest0-21swapper/222:45:132
2554899286,19cyclictest35966irq/58-eth000:35:133
2552799283,20cyclictest27346irq/40-dwc2_hso22:15:132
2552799283,11cyclictest13875-21ssh22:05:142
2552799282,21cyclictest0-21swapper/221:30:142
2552799282,19cyclictest27346irq/40-dwc2_hso20:25:122
2552799282,19cyclictest19000-21ssh21:50:032
2552799282,10cyclictest0-21swapper/219:20:132
25512992810,12cyclictest12-21ksoftirqd/000:35:130
2552799273,17cyclictest27346irq/40-dwc2_hso22:25:122
2552799272,10cyclictest0-21swapper/223:05:052
2552799271,18cyclictest5599-21cut20:30:162
2551299275,16cyclictest0-21swapper/022:35:140
2551299274,15cyclictest151rcuc/022:15:140
2551299272,20cyclictest30674-21ssh22:14:040
2551299271,23cyclictest0-21swapper/023:35:120
2551299271,23cyclictest0-21swapper/023:35:120
25548992613,9cyclictest0-21swapper/322:15:223
25548992613,8cyclictest35966irq/58-eth000:15:123
2552799261,22cyclictest28379-21ls20:50:282
2552799261,19cyclictest27346irq/40-dwc2_hso19:30:112
2552799261,18cyclictest27-21ksoftirqd/223:40:122
25512992611,12cyclictest16463-21diskmemload00:30:140
25548992511,8cyclictest35966irq/58-eth021:30:143
2552799253,5cyclictest4905-21ssh22:00:142
2552799253,5cyclictest4905-21ssh22:00:132
2552799253,15cyclictest27346irq/40-dwc2_hso19:40:142
2552799252,8cyclictest17113-21ssh21:48:032
2552799251,9cyclictest30287-21ssh22:50:062
2551299253,17cyclictest9205-21ssh23:50:130
2551299252,21cyclictest1826-21ssh23:10:110
2551299252,19cyclictest21822-21ssh00:15:120
2551299251,19cyclictest32224-21latency_hist23:45:010
2554899241,15cyclictest0-21swapper/322:25:123
2554899241,13cyclictest35966irq/58-eth020:15:143
2552799244,15cyclictest27346irq/40-dwc2_hso00:15:122
2552799242,16cyclictest0-21swapper/220:35:142
2552799242,16cyclictest0-21swapper/200:35:082
25527992412,5cyclictest1-21systemd21:40:572
2551299247,3cyclictest131rcu_preempt21:15:280
2551299242,17cyclictest30436-21ssh22:50:130
2554899231,15cyclictest0-21swapper/322:35:143
2554899231,12cyclictest35966irq/58-eth022:00:143
2554899231,12cyclictest35966irq/58-eth022:00:133
2552799235,10cyclictest29267-21diskmemload00:00:232
2552799232,17cyclictest27346irq/40-dwc2_hso21:20:132
2552799232,16cyclictest0-21swapper/220:40:142
2552799232,16cyclictest0-21swapper/220:40:132
2552799232,15cyclictest0-21swapper/222:40:062
2552799231,18cyclictest10407-21timerandwakeup21:05:282
2552799231,13cyclictest26919-21cut21:35:212
2552799231,13cyclictest26919-21cut21:35:212
2551299234,15cyclictest28844-21ssh23:25:130
2551299232,17cyclictest27268-21ssh00:00:150
2551299232,14cyclictest630-21Async09:36:310
25512992312,3cyclictest6322-21/usr/sbin/munin21:25:270
2551299231,19cyclictest10171-21ssh21:45:150
2551299231,18cyclictest12298-21ls20:00:220
2551299231,14cyclictest25109-21sh22:45:260
2552799228,10cyclictest0-21swapper/223:20:142
2552799221,8cyclictest1452566ssh23:51:082
2552799221,17cyclictest9141-21fschecks_time20:00:152
2552799221,17cyclictest4519-21cut00:05:242
2552799221,17cyclictest29648-21tune2fs20:55:152
2552799221,13cyclictest29171-21ssh23:25:182
2552799221,12cyclictest31059-21fschecks_time00:20:182
2552799221,10cyclictest11857-21cut22:20:232
2551299223,6cyclictest12-21ksoftirqd/023:20:310
2551299223,13cyclictest0-21swapper/023:55:110
2551299222,17cyclictest25260-21ssh23:05:130
25512992212,7cyclictest0-21swapper/020:05:170
25512992211,8cyclictest11663-21sshd00:09:070
2551299221,16cyclictest8559-21find22:20:130
2551299221,15cyclictest28291-21ssh21:55:130
2554899216,8cyclictest35966irq/58-eth021:35:143
2554899216,8cyclictest35966irq/58-eth021:35:133
25548992114,4cyclictest35966irq/58-eth000:30:143
2552799217,11cyclictest30207-21sed22:30:232
2552799215,14cyclictest14455-21ssh21:10:222
2552799213,15cyclictest28339-21cut20:20:152
2552799212,15cyclictest0-21swapper/200:30:082
2552799212,10cyclictest20464-21kthreadcore20:10:212
25527992114,4cyclictest2000-21ssh23:10:152
2552799211,17cyclictest1254-21ssh23:45:212
2552799211,17cyclictest11118-21grep22:55:272
2552799211,16cyclictest30722-21fschecks_time19:15:162
2552799211,15cyclictest14526-21ssh23:35:152
2552799211,15cyclictest14526-21ssh23:35:142
2552799211,11cyclictest29051-21missed_timers19:10:262
2552799211,10cyclictest9296-21ssh21:25:212
2552799211,10cyclictest9256-21rm23:30:272
2552799211,10cyclictest27307-21irqcore19:45:182
2552799211,10cyclictest25277-21grep20:15:212
2552799211,10cyclictest10274-21kthreadcore19:25:212
2552799211,10cyclictest10274-21kthreadcore19:25:212
2551299212,14cyclictest12434-21diskstats22:40:150
25512992113,6cyclictest0-21swapper/019:20:160
2551299210,14cyclictest4138-21find19:55:110
2554899209,7cyclictest0-21swapper/323:28:163
2554899209,7cyclictest0-21swapper/321:15:183
2554899209,6cyclictest0-21swapper/300:24:193
2554899208,9cyclictest0-21swapper/322:53:323
25548992011,6cyclictest35966irq/58-eth023:55:113
25548992011,6cyclictest35966irq/58-eth022:10:133
25548992010,6cyclictest35966irq/58-eth023:15:153
2552799201,11cyclictest1884-21cpu21:00:132
2552799201,11cyclictest1476-21kthreadcore19:50:212
2552799201,10cyclictest17325-21proc_pri20:05:272
2551299209,8cyclictest23420-21apt-get20:15:130
2551299203,3cyclictest131rcu_preempt20:25:010
25512992011,6cyclictest6479-21sshd21:04:590
25512992011,6cyclictest20266-21cron21:15:010
2551299201,16cyclictest12790-21ls19:30:160
2551299201,14cyclictest21127-21ssh22:25:220
2551299200,14cyclictest2379-21ls00:20:300
2554899199,7cyclictest35966irq/58-eth022:40:133
2554899198,6cyclictest0-21swapper/322:22:463
2554899192,10cyclictest0-21swapper/323:05:053
2554899192,10cyclictest0-21swapper/321:50:033
25548991911,5cyclictest0-21swapper/322:05:203
2554899191,14cyclictest0-21swapper/320:00:013
25548991910,7cyclictest0-21swapper/300:09:373
2552799192,14cyclictest25404-21apt-get19:10:022
2552799191,9cyclictest20715-21kthreadcore20:45:192
2551299199,7cyclictest16305-21df23:00:130
2551299196,9cyclictest22474-21grep20:10:300
2551299192,12cyclictest23200-21ssh21:34:030
25512991912,5cyclictest14104-21cron23:35:000
25512991912,4cyclictest14841-21sendmail20:40:010
2551299191,15cyclictest329-21fschecks_count23:45:170
2551299191,13cyclictest27817-21munin-run21:55:010
25512991910,6cyclictest0-21swapper/020:55:170
2554899189,6cyclictest0-21swapper/323:21:593
2554899188,8cyclictest0-21swapper/321:58:593
2554899187,9cyclictest0-21swapper/321:43:333
2554899186,7cyclictest0-21swapper/321:46:423
2554899182,10cyclictest0-21swapper/320:25:043
2554899182,10cyclictest0-21swapper/320:10:153
2554899182,10cyclictest0-21swapper/320:05:053
2554899182,10cyclictest0-21swapper/319:50:143
2554899182,10cyclictest0-21swapper/319:30:123
2554899182,10cyclictest0-21swapper/319:20:123
25548991811,5cyclictest35966irq/58-eth023:47:173
2554899181,11cyclictest0-21swapper/300:30:023
2554899181,10cyclictest35966irq/58-eth021:25:113
25548991810,6cyclictest0-21swapper/323:51:413
25548991810,6cyclictest0-21swapper/323:11:093
2551299186,9cyclictest9109-21switchtime20:30:280
2551299186,5cyclictest6976-21cut22:00:230
2551299186,5cyclictest6976-21cut22:00:230
2551299183,7cyclictest12321-21apt19:30:010
2551299183,7cyclictest12321-21apt19:30:010
2551299182,12cyclictest6654-21apt-get22:55:030
25512991812,3cyclictest0-21swapper/021:35:110
25512991812,3cyclictest0-21swapper/021:35:110
2551299181,14cyclictest7034-21irqcore21:05:180
2551299181,14cyclictest5376-21latency_hist20:30:030
25512991811,4cyclictest0-21swapper/000:10:070
2554899179,6cyclictest0-21swapper/321:21:423
2554899179,6cyclictest0-21swapper/321:12:393
2554899172,10cyclictest35966irq/58-eth019:15:143
2554899172,10cyclictest0-21swapper/320:40:133
2554899172,10cyclictest0-21swapper/320:40:133
2554899172,10cyclictest0-21swapper/320:00:133
25548991710,5cyclictest0-21swapper/322:30:213
2551299179,4cyclictest32118-21irqcore19:50:180
2551299175,9cyclictest20579-21kthreadcore20:45:190
2551299173,11cyclictest15557-21sed20:40:170
2551299173,11cyclictest15557-21sed20:40:170
2551299172,11cyclictest0-21swapper/019:10:160
2551299167,6cyclictest29376-21latency_hist20:55:020
2551299163,10cyclictest22644-21irqcore19:40:170
2551299162,11cyclictest28573-21kthreadcore19:45:200
2551299162,11cyclictest25407-21latency_hist19:10:020
2554899151,12cyclictest0-21swapper/320:35:153
2554899149,3cyclictest0-21swapper/321:04:343
2554899149,3cyclictest0-21swapper/320:48:043
2554899149,3cyclictest0-21swapper/319:40:273
2554899143,8cyclictest0-21swapper/319:25:123
2554899143,8cyclictest0-21swapper/319:25:123
2554899141,7cyclictest0-21swapper/320:55:133
2554899141,7cyclictest0-21swapper/319:10:253
2554899141,10cyclictest0-21swapper/319:10:023
2554899131,9cyclictest0-21swapper/319:40:023
2554899131,10cyclictest0-21swapper/320:20:293
2554899123,7cyclictest0-21swapper/320:50:163
2554899121,5cyclictest0-21swapper/319:50:023
2554899111,8cyclictest0-21swapper/320:30:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional