You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-25 - 06:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue Aug 25, 2026 00:45:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2303299522,30cyclictest13944-21ssh22:25:132
2302499511,2cyclictest0-21swapper/122:05:141
2302499511,2cyclictest0-21swapper/122:05:131
23040995030,11cyclictest35966irq/58-eth023:40:163
23032995033,12cyclictest930-21snmpd21:00:132
23032995023,16cyclictest0-21swapper/222:05:142
23032995023,16cyclictest0-21swapper/222:05:132
2301699501,23cyclictest31811-21ssh22:35:120
23040994834,8cyclictest35966irq/58-eth021:55:143
23032994837,8cyclictest16302-21apt-get21:50:132
2302499482,35cyclictest9239-21ssh21:10:111
2302499481,44cyclictest0-21swapper/119:15:131
23032994727,11cyclictest6175-21ssh00:35:112
23032994516,14cyclictest7416-21ssh00:15:222
23032994515,11cyclictest30521-21ssh23:48:522
23032994515,11cyclictest30521-21ssh23:48:512
2302499452,37cyclictest2968-21apt-get20:30:141
2302499451,6cyclictest0-21swapper/119:35:141
2302499450,3cyclictest3687-21apt-get21:25:121
2302499450,3cyclictest3687-21apt-get21:25:111
2304099440,39cyclictest0-21swapper/322:45:143
23032994430,9cyclictest12840-21apt-get21:30:162
2302499442,36cyclictest19592-21apt-get22:10:121
23032994318,14cyclictest0-21swapper/219:30:112
23032994314,24cyclictest0-21swapper/200:05:162
2302499432,36cyclictest0-21swapper/121:50:131
2302499431,6cyclictest0-21swapper/120:25:131
2303299423,27cyclictest25309-21ssh21:55:132
23032994222,8cyclictest0-21swapper/219:40:122
2302499421,5cyclictest0-21swapper/120:40:131
2302499421,5cyclictest0-21swapper/120:40:121
2302499421,2cyclictest21833-21apt-get00:05:151
2301699423,22cyclictest10725-21ssh22:05:160
2301699423,22cyclictest10725-21ssh22:05:150
23032994121,9cyclictest18670-21ssh23:40:142
23032994112,13cyclictest3371-21ssh22:18:322
2302499412,35cyclictest17260-21apt-get22:45:141
2302499412,35cyclictest0-21swapper/123:40:151
2302499411,35cyclictest0-21swapper/119:40:121
2302499411,30cyclictest18253-21sh21:15:151
2302499411,2cyclictest12776-21apt-get00:00:121
23032994034,3cyclictest6370-21apt-get20:00:132
23032994017,18cyclictest0-21swapper/220:40:132
23032994017,18cyclictest0-21swapper/220:40:122
23032994017,18cyclictest0-21swapper/200:00:122
23032994015,9cyclictest4673-21ssh00:12:162
23032994011,15cyclictest16864-21rm21:13:512
2302499402,35cyclictest0-21swapper/121:00:121
2302499401,34cyclictest15164-21apt-get23:20:131
2302499401,2cyclictest0-21swapper/120:00:131
23032993918,17cyclictest0-21swapper/223:05:142
23032993915,8cyclictest1513-21ssh22:53:562
23032993915,7cyclictest10335-21ssh23:57:312
23032993818,15cyclictest0-21swapper/223:10:132
23032993813,18cyclictest0-21swapper/221:45:152
2302499381,4cyclictest0-21swapper/119:55:131
23032993714,9cyclictest21521-21ssh23:22:222
23032993714,7cyclictest22873-21ssh22:46:242
2302499372,29cyclictest9828-21apt-get23:35:231
2302499371,33cyclictest20716-21apt-get23:05:151
2302499371,2cyclictest0-21swapper/121:45:161
2301699376,25cyclictest22880-21ssh22:30:140
2301699372,32cyclictest17281-21ssh22:45:120
23032993630,2cyclictest11660-21apt-get00:20:092
23032993620,12cyclictest5006-21ssh22:20:132
23032993617,15cyclictest433-21latency_hist23:30:032
23032993614,7cyclictest7556-21ssh22:39:352
2302499361,32cyclictest0-21swapper/122:50:111
2302499361,2cyclictest4987-21apt-get22:20:141
2302499361,2cyclictest28478-21apt-get22:15:131
2302499361,2cyclictest0-21swapper/121:30:151
2302499361,22cyclictest0-21swapper/121:55:331
23016993614,19cyclictest12-21ksoftirqd/021:50:110
2303299358,11cyclictest0-21swapper/221:44:552
23032993519,13cyclictest8032-21/usr/sbin/munin21:05:292
23032993519,13cyclictest8032-21/usr/sbin/munin21:05:282
23032993518,8cyclictest1794-21/usr/sbin/munin19:55:182
23032993518,14cyclictest21753-21/usr/sbin/munin21:35:222
23032993518,14cyclictest0-21swapper/222:00:132
23032993517,9cyclictest25888-21df20:20:142
23032993517,8cyclictest11212-21open_inodes20:35:272
23032993516,16cyclictest23325-21grep22:10:242
23032993510,17cyclictest0-21swapper/222:33:162
2302499351,30cyclictest30788-21/usr/sbin/munin21:40:251
2302499350,31cyclictest11680-21ssh00:20:101
2301699352,29cyclictest20698-21ssh00:25:140
2301699351,30cyclictest20719-21ssh23:05:130
2301699351,30cyclictest12786-21ssh00:00:100
2303299349,9cyclictest0-21swapper/200:29:382
23032993420,11cyclictest0-21swapper/220:05:152
23032993418,8cyclictest12031-21ntp_states22:40:262
23032993418,13cyclictest543-21ssh23:51:402
23032993417,14cyclictest8587-21kthreadcore23:15:222
23032993417,14cyclictest28296-21/usr/sbin/munin19:45:302
23032993416,14cyclictest21056-21timerandwakeup20:45:282
23032993416,14cyclictest1544-21ssh21:22:512
23032993416,14cyclictest1544-21ssh21:22:502
23032993413,8cyclictest10583-21ssh21:27:532
23032993413,8cyclictest10583-21ssh21:27:522
2302499343,21cyclictest21-21ksoftirqd/119:50:141
2302499343,21cyclictest21-21ksoftirqd/119:50:131
2302499341,2cyclictest0-21swapper/123:10:121
2302499341,29cyclictest20533-21ssh00:25:031
2302499340,31cyclictest0-21swapper/120:05:141
23032993317,13cyclictest8655-21ntp_states19:25:312
23032993317,13cyclictest8655-21ntp_states19:25:302
23032993316,14cyclictest3196-21/usr/sbin/munin20:30:172
23032993315,16cyclictest27346irq/40-dwc2_hso19:10:142
23032993315,14cyclictest30393-21awk20:55:242
23032993313,7cyclictest9133-21ssh23:34:552
23032993313,7cyclictest2505-21sh00:31:032
2302499331,3cyclictest0-21swapper/123:15:151
2302499331,2cyclictest0-21swapper/119:25:141
2302499331,2cyclictest0-21swapper/119:25:131
2302499331,29cyclictest7378-21sh22:55:331
2302499331,29cyclictest4957-21sshd19:22:101
2302499331,28cyclictest24464-21tr19:10:191
2302499331,27cyclictest11791-21apt-get23:00:121
2301699332,27cyclictest11801-21rm23:00:130
2301699331,30cyclictest1797-21ssh22:00:130
2301699331,29cyclictest3692-21ssh23:55:090
2304099321,27cyclictest0-21swapper/319:40:133
23032993218,11cyclictest31810-21sed19:15:282
23032993218,10cyclictest0-21swapper/220:25:132
23032993216,8cyclictest12094-21cat23:00:182
23032993213,7cyclictest14976-21ssh23:36:022
23032993212,17cyclictest18940-21kthreadcore20:10:222
23032993211,13cyclictest4311-21kthreadcore22:55:202
2302499321,4cyclictest0-21swapper/100:10:151
2302499321,3cyclictest24419-21apt-get19:45:141
2302499321,3cyclictest0-21swapper/119:30:131
2302499321,2cyclictest0-21swapper/120:15:131
2302499321,29cyclictest13151-21ssh00:38:061
2302499321,28cyclictest12458-21unixbench_singl22:40:281
2302499321,27cyclictest19084-21kthreadcore20:10:231
2301699321,26cyclictest4997-21ssh22:20:130
2304099311,27cyclictest0-21swapper/300:10:133
2304099311,25cyclictest0-21swapper/320:40:123
2304099311,25cyclictest0-21swapper/320:40:123
2304099311,22cyclictest0-21swapper/322:05:143
2304099311,22cyclictest0-21swapper/322:05:133
23032993117,11cyclictest18457-21grep19:35:292
23032993116,12cyclictest514-21fschecks_time19:20:172
23032993116,12cyclictest22148-21/usr/sbin/munin20:50:132
23032993114,14cyclictest23456-21ssh21:15:572
2302499311,2cyclictest0-21swapper/123:30:301
2302499311,27cyclictest6362-21cut23:57:101
2302499311,27cyclictest11399-21ls00:20:011
2302499311,26cyclictest24541-21cut20:50:211
2302499311,26cyclictest20722-21missed_timers20:45:241
2302499311,26cyclictest12334-21latency_hist20:40:011
2302499310,10cyclictest2249-21cut22:35:211
2301699311,24cyclictest12-21ksoftirqd/021:25:110
2301699311,24cyclictest12-21ksoftirqd/021:25:100
2304099301,22cyclictest0-21swapper/319:20:133
2304099301,21cyclictest0-21swapper/321:30:163
2304099301,19cyclictest0-21swapper/321:05:133
2304099301,19cyclictest0-21swapper/321:05:123
2304099300,27cyclictest0-21swapper/322:10:113
2304099300,27cyclictest0-21swapper/300:00:093
23032993019,7cyclictest27346irq/40-dwc2_hso20:15:142
23032993015,12cyclictest854-21sshd19:50:102
23032993015,12cyclictest854-21sshd19:50:092
2302499301,27cyclictest8259-21ssh22:02:311
2302499301,26cyclictest30840-21ssh23:27:461
2302499301,26cyclictest29668-21cpu00:30:141
2302499301,26cyclictest21961-21fschecks_count21:35:161
2302499301,25cyclictest4294-21uniq21:05:161
2302499301,25cyclictest4294-21uniq21:05:161
2302499301,25cyclictest18036-21unixbench_multi22:25:291
2301699302,25cyclictest28504-21ssh22:15:140
2301699302,21cyclictest23164-21ssh21:15:350
2304099292,22cyclictest0-21swapper/320:50:133
2304099291,25cyclictest0-21swapper/323:10:123
2304099291,25cyclictest0-21swapper/322:55:133
2304099291,23cyclictest0-21swapper/321:00:123
2304099290,25cyclictest0-21swapper/323:00:123
2302499290,5cyclictest25483-21apt-get20:20:031
2302499290,26cyclictest3349-21latency_hist23:55:011
2301699293,22cyclictest15178-21ssh23:20:130
2301699293,16cyclictest26268-21ssh22:50:130
2301699292,21cyclictest29667-21ssh00:30:150
2301699292,19cyclictest151rcuc/021:35:120
2301699291,24cyclictest27249-21sh21:20:140
2301699291,24cyclictest27249-21sh21:20:130
2301699291,22cyclictest24113-21ssh23:25:140
2304099284,15cyclictest35966irq/58-eth000:35:113
2304099281,24cyclictest0-21swapper/320:15:143
2304099281,24cyclictest0-21swapper/319:25:143
2304099281,24cyclictest0-21swapper/319:25:133
2304099281,24cyclictest0-21swapper/300:30:133
2304099281,22cyclictest0-21swapper/320:45:143
2304099281,22cyclictest0-21swapper/300:05:153
2304099281,21cyclictest0-21swapper/320:00:123
2304099281,20cyclictest0-21swapper/323:15:153
2302499281,2cyclictest0-21swapper/120:55:121
2302499281,24cyclictest22838-21/usr/sbin/munin22:30:181
2302499280,7cyclictest27534-21sed21:20:181
2302499280,7cyclictest27534-21sed21:20:171
23016992815,10cyclictest23005-21cyclictest22:10:110
2304099273,19cyclictest0-21swapper/300:20:093
2304099271,23cyclictest0-21swapper/321:45:153
2304099271,22cyclictest0-21swapper/322:20:133
2304099271,20cyclictest0-21swapper/321:40:123
2304099270,22cyclictest0-21swapper/322:15:133
2304099270,22cyclictest0-21swapper/319:50:123
2304099270,22cyclictest0-21swapper/319:50:113
23032992716,8cyclictest23052-21latency_hist19:10:012
2302499271,24cyclictest27871-21ssh23:45:391
2302499271,24cyclictest27871-21ssh23:45:381
2301699272,19cyclictest32278-21ssh21:58:110
2304099262,20cyclictest0-21swapper/320:25:123
2304099261,22cyclictest0-21swapper/322:35:103
2304099261,22cyclictest0-21swapper/322:00:123
2304099261,22cyclictest0-21swapper/319:15:133
2304099261,21cyclictest0-21swapper/321:10:123
2304099261,20cyclictest0-21swapper/321:50:123
2304099260,22cyclictest0-21swapper/319:30:123
2302499260,5cyclictest23005-21cyclictest19:10:011
2301699264,15cyclictest23019-21kworker/u8:122:40:120
23016992615,5cyclictest0-21swapper/019:45:130
2304099252,18cyclictest0-21swapper/300:17:013
2304099251,21cyclictest0-21swapper/323:55:093
2304099251,21cyclictest0-21swapper/323:35:053
2304099251,21cyclictest0-21swapper/321:29:563
2304099251,21cyclictest0-21swapper/321:29:553
2304099251,20cyclictest0-21swapper/321:20:133
2304099251,20cyclictest0-21swapper/321:20:123
2304099251,19cyclictest0-21swapper/323:30:113
2304099251,19cyclictest0-21swapper/323:20:113
2304099251,19cyclictest0-21swapper/322:50:113
2301699258,10cyclictest0-21swapper/023:30:110
2301699252,18cyclictest844-21munin-node19:10:140
23016992510,5cyclictest131rcu_preempt21:00:180
2304099241,20cyclictest0-21swapper/322:40:143
2304099241,20cyclictest0-21swapper/321:35:123
2304099241,20cyclictest0-21swapper/321:15:333
2304099241,20cyclictest0-21swapper/320:30:153
2304099241,20cyclictest0-21swapper/319:35:123
2304099241,20cyclictest0-21swapper/319:10:273
2304099241,19cyclictest0-21swapper/323:25:123
2304099241,19cyclictest0-21swapper/320:10:123
2304099241,19cyclictest0-21swapper/319:45:123
2304099231,19cyclictest0-21swapper/323:05:223
2304099231,19cyclictest0-21swapper/322:25:153
2304099231,19cyclictest0-21swapper/320:55:123
2304099231,19cyclictest0-21swapper/300:25:263
2304099231,18cyclictest0-21swapper/322:30:133
2304099221,18cyclictest0-21swapper/323:50:463
2304099221,18cyclictest0-21swapper/323:47:243
2304099221,18cyclictest0-21swapper/323:47:243
2304099221,17cyclictest0-21swapper/320:20:123
2304099221,17cyclictest0-21swapper/320:05:153
2301699222,11cyclictest12852-21ssh21:30:150
2301699221,18cyclictest27265-21irqcore20:55:170
2301699221,18cyclictest11955-21ls00:20:130
2301699221,16cyclictest7162-21ls21:45:020
2301699221,13cyclictest878-21ntp_192.168.11423:10:240
2304099211,17cyclictest0-21swapper/319:55:143
2304099210,18cyclictest0-21swapper/320:35:163
2301699216,7cyclictest131rcu_preempt20:30:130
2301699213,13cyclictest0-21swapper/000:05:160
2301699212,14cyclictest6530-21ssh22:55:280
23016992114,5cyclictest18075-21ssh23:39:500
2301699211,16cyclictest7755-21apt-get20:35:130
2301699211,15cyclictest12673-21latency21:10:230
2301699201,10cyclictest20589-21cron20:15:000
2301699193,9cyclictest0-21swapper/000:20:000
23016991911,6cyclictest12613-21ssh00:37:250
2301699191,15cyclictest28100-21cut19:15:160
23016991911,5cyclictest15811-21sh22:25:190
2301699191,14cyclictest14800-21apt-get19:35:120
23016991910,6cyclictest18892-21fschecks_time23:40:170
2301699185,10cyclictest518-21cut00:10:220
2301699184,11cyclictest12691-21fschecks_count20:40:160
2301699184,11cyclictest12691-21fschecks_count20:40:150
2301699182,11cyclictest10945-21munin-run20:05:010
2301699181,9cyclictest4265-21df21:05:140
2301699181,9cyclictest4265-21df21:05:130
2301699181,8cyclictest30718-21sort20:25:170
2301699181,15cyclictest5442-21ls19:25:210
2301699181,15cyclictest5442-21ls19:25:210
2301699181,14cyclictest7434-21apt-get21:45:150
2301699181,14cyclictest22033-21latency_hist20:50:000
2301699181,14cyclictest13811-21tail19:30:280
2301699179,5cyclictest25832-21wc20:50:290
2301699173,11cyclictest3518-21kthreadcore19:55:220
2301699172,11cyclictest8582-21ssh23:15:230
2301699171,14cyclictest25574-21apt-config20:20:020
2301699171,13cyclictest3813-21ntp_192.168.11419:20:250
2301699171,13cyclictest24185-21latency_hist19:45:020
2301699171,12cyclictest31524-21ssh23:50:020
2301699171,12cyclictest31524-21ssh23:50:010
2301699171,12cyclictest29297-21diskstats19:50:160
2301699171,12cyclictest29297-21diskstats19:50:150
2301699171,12cyclictest25897-21diskstats20:20:150
2304099161,13cyclictest0-21swapper/319:10:013
2301699156,6cyclictest1-21systemd23:52:570
2301699152,10cyclictest11483-21grep20:05:180
2301699151,11cyclictest23035-21latency_hist19:10:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional