You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-04 - 04:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Tue Aug 04, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29133996423,9cyclictest12-21ksoftirqd/000:30:130
29152996114,6cyclictest131rcu_preempt22:35:122
29133995035,6cyclictest14756-21diskmemload21:20:130
2913399472,39cyclictest15740-21ssh22:40:130
2915299454,21cyclictest30145-21ssh22:30:142
29152994518,19cyclictest27-21ksoftirqd/222:15:132
2915299451,24cyclictest8470-21ssh23:30:142
2913399420,36cyclictest29973-21ssh00:00:120
2914299413,36cyclictest15545-21ssh00:10:121
29142994033,3cyclictest0-21swapper/123:45:141
29133994025,7cyclictest12-21ksoftirqd/019:45:270
2914299392,34cyclictest28293-21ssh23:05:131
2914299392,33cyclictest3021-21ssh22:15:141
2913399392,29cyclictest26384-21diskmemload23:40:110
2913399391,29cyclictest1001-21ssh00:20:140
29152993810,6cyclictest15829-21kworker/2:119:30:132
2914299381,32cyclictest27783-21ssh00:35:121
2916599372,20cyclictest35966irq/58-eth019:55:143
2916599361,22cyclictest35966irq/58-eth022:55:123
29152993617,10cyclictest6563-21sh00:05:142
2915299351,8cyclictest131rcu_preempt20:35:122
2914299358,23cyclictest17470-21ssh22:05:161
2914299358,23cyclictest17470-21ssh22:05:151
2914299343,26cyclictest11937-21ssh23:50:141
2913399341,21cyclictest22885-21ssh23:20:130
2915299333,23cyclictest22906-21ssh23:20:142
2915299333,20cyclictest0-21swapper/223:05:142
29152993215,10cyclictest27-21ksoftirqd/223:50:132
2915299321,19cyclictest27346irq/40-dwc2_hso23:15:142
2914299323,23cyclictest30130-21ssh22:30:121
2914299321,28cyclictest10315-21ssh22:55:131
2913399322,25cyclictest17464-21ssh23:35:150
29152993117,7cyclictest0-21swapper/200:10:132
29165993015,9cyclictest35966irq/58-eth023:40:113
2916599300,16cyclictest35966irq/58-eth020:30:133
29152993015,11cyclictest27-21ksoftirqd/221:20:152
2915299301,16cyclictest26375-21ssh23:40:122
2914299302,8cyclictest131rcu_preempt20:35:131
2914299302,22cyclictest22617-21sed23:20:021
2913399302,21cyclictest8038-21ssh22:00:010
29165992912,11cyclictest35966irq/58-eth021:55:013
29165992912,11cyclictest35966irq/58-eth021:55:003
2915299293,8cyclictest0-21swapper/223:57:002
2914299292,16cyclictest21-21ksoftirqd/119:50:121
2913399293,20cyclictest24770-21ssh22:45:170
2913399292,21cyclictest27514-21idleruntime-cro21:35:000
2915299285,15cyclictest0-21swapper/222:50:162
2915299285,15cyclictest0-21swapper/222:50:152
2913399282,20cyclictest4661-21ssh22:33:010
2913399281,19cyclictest19108-21latency_hist23:00:010
2913399281,16cyclictest4599-21ssh21:40:130
2916599279,12cyclictest0-21swapper/322:19:003
29152992710,7cyclictest876-21ps20:55:192
2914299272,19cyclictest31285-21ssh21:55:011
2914299272,19cyclictest31285-21ssh21:55:011
2914299272,19cyclictest27504-21apt-get21:35:011
2914299272,19cyclictest27038-21rm23:57:001
2914299272,16cyclictest6565-21sh00:05:151
2914299271,21cyclictest6728-21ssh22:35:131
29142992711,11cyclictest19361-21ssh23:00:141
2913399272,19cyclictest2632-21latency_hist22:15:010
2913399272,19cyclictest20543-21latency_hist23:55:000
29133992712,5cyclictest131rcu_preempt21:40:010
2916599269,11cyclictest0-21swapper/321:47:193
2916599264,12cyclictest36066irq/59-eth023:45:153
2916599263,17cyclictest0-21swapper/322:33:013
2916599262,17cyclictest0-21swapper/323:33:003
2915299261,21cyclictest0-21swapper/200:30:142
2914299261,19cyclictest12357-21ssh21:44:021
2913399262,19cyclictest20931-21ssh23:55:140
2913399262,19cyclictest14567-21sh23:32:010
2913399262,18cyclictest4871-21diskmemload23:10:140
2913399262,18cyclictest14756-21diskmemload21:47:200
29165992516,6cyclictest35966irq/58-eth023:05:133
2915299259,9cyclictest261rcuc/220:00:132
2915299259,9cyclictest261rcuc/220:00:122
2915299252,18cyclictest0-21swapper/222:03:022
2915299252,16cyclictest27346irq/40-dwc2_hso20:10:122
2915299252,16cyclictest0-21swapper/222:05:162
2915299252,16cyclictest0-21swapper/222:05:162
2915299251,21cyclictest5297-21ls19:15:302
2915299251,16cyclictest0-21swapper/219:50:122
2915299250,2cyclictest131rcu_preempt21:35:262
29142992510,10cyclictest14756-21diskmemload23:32:001
2913399252,19cyclictest22179-21apt-get20:10:110
2916599241,9cyclictest35966irq/58-eth020:50:153
29165992413,7cyclictest35966irq/58-eth000:20:143
29165992411,8cyclictest35966irq/58-eth021:55:153
2915299246,4cyclictest131rcu_preempt20:20:142
2915299244,12cyclictest6156-21munin-plugin-st00:05:022
29152992418,4cyclictest31694-21ssh21:55:152
29152992413,9cyclictest18047-21irqrtprio23:35:192
2915299241,20cyclictest618-21ls19:10:282
2915299240,19cyclictest11943-21acpi22:20:112
2914299242,19cyclictest29993-21ssh00:00:151
2914299241,16cyclictest11987-21ssh22:20:111
2913399241,15cyclictest10001-21rm00:25:130
2916599239,10cyclictest35966irq/58-eth021:20:143
29165992315,5cyclictest35966irq/58-eth023:15:143
2915299238,13cyclictest27-21ksoftirqd/221:45:222
2915299232,16cyclictest0-21swapper/223:01:022
2915299232,16cyclictest0-21swapper/221:42:182
2915299232,13cyclictest4872-21ssh23:10:142
2915299232,12cyclictest31930-21ssh23:25:142
2915299231,7cyclictest9676-21latency_hist00:25:022
2915299231,3cyclictest131rcu_preempt23:45:282
29152992313,8cyclictest0-21swapper/220:50:242
29152992311,9cyclictest31398-21sh00:18:022
29152992311,10cyclictest29021-21sshd21:52:212
29152992311,10cyclictest29021-21sshd21:52:202
29152992310,6cyclictest0-21swapper/220:30:252
2914299231,7cyclictest25166-21irqrtprio00:15:201
2914299231,17cyclictest1334-21ssh22:50:161
2914299231,17cyclictest1334-21ssh22:50:151
2914299231,16cyclictest24184-21sh21:15:161
2913399234,13cyclictest8327-21ssh22:00:140
2913399232,17cyclictest22670-21rm21:50:150
2913399232,17cyclictest22670-21rm21:50:140
29133992311,10cyclictest12339-21cron20:00:010
2916599223,16cyclictest35966irq/58-eth022:35:113
2916599223,14cyclictest36066irq/59-eth021:15:153
2916599222,11cyclictest35966irq/58-eth022:25:163
2916599222,11cyclictest35966irq/58-eth022:00:123
29165992210,9cyclictest0-21swapper/323:54:183
2915299227,5cyclictest24368-21ssh21:15:172
2915299227,12cyclictest21101-21grep22:25:152
2915299223,16cyclictest31671-21cat20:20:012
2915299222,16cyclictest904-21awk22:50:002
2915299222,11cyclictest19573-21grep22:40:292
2915299221,17cyclictest27689-21rm21:35:022
2915299221,14cyclictest27346irq/40-dwc2_hso00:35:132
2914299221,8cyclictest13573-21/usr/sbin/munin21:25:271
2914299221,18cyclictest23195-21ls20:45:021
29142992210,9cyclictest10052-21cat19:20:281
2913399221,16cyclictest24190-21ssh21:15:160
2913399221,14cyclictest11541-21latency_hist23:50:020
2916599219,9cyclictest0-21swapper/322:46:203
2916599214,15cyclictest35966irq/58-eth023:00:153
29165992112,7cyclictest0-21swapper/300:05:173
29165992112,6cyclictest35966irq/58-eth000:10:123
29165992111,7cyclictest35966irq/58-eth000:35:123
29165992110,8cyclictest35966irq/58-eth022:40:143
2915299219,9cyclictest0-21swapper/200:25:162
2915299216,8cyclictest26376-21sort22:10:122
2915299212,9cyclictest18760-21ntp_192.168.11421:10:252
2915299212,17cyclictest22505-21kthreadcore19:35:212
2915299212,12cyclictest4518-21irqcore20:25:172
2915299212,12cyclictest4518-21irqcore20:25:162
2915299211,11cyclictest9904-21apt-get21:25:132
2915299211,11cyclictest8720-21missed_timers21:00:222
2915299211,11cyclictest10460-21fschecks_count22:55:152
2915299211,10cyclictest10363-21unixbench_singl19:20:302
2914299211,9cyclictest22522-21ntpq00:30:271
2914299211,6cyclictest31584-21awk20:20:021
29133992113,5cyclictest17602-21ssh22:05:180
29133992113,5cyclictest17602-21ssh22:05:170
2913399211,15cyclictest4021-21ls20:25:020
2913399211,15cyclictest24551-21rm00:15:160
2916599202,11cyclictest0-21swapper/322:10:143
2916599201,16cyclictest35966irq/58-eth022:05:163
2916599201,16cyclictest35966irq/58-eth022:05:153
29165992010,7cyclictest35966irq/58-eth021:10:133
29165992010,4cyclictest0-21swapper/321:30:243
2915299204,13cyclictest19715-21kthreadcore20:40:192
2915299201,9cyclictest8842-21kthreadcore19:55:202
2915299201,9cyclictest18317-21kthreadcore20:05:182
2915299201,10cyclictest32123-21kthreadcore19:45:342
2915299201,10cyclictest13205-21kthreadcore21:05:212
2914299209,8cyclictest25525-21cat22:25:331
2914299208,9cyclictest15166-21grep19:25:271
2914299202,13cyclictest31677-21kworker/u8:122:03:021
29142992013,5cyclictest26612-21fschecks_time22:10:161
29142992012,5cyclictest21-21ksoftirqd/123:20:161
29142992012,5cyclictest0-21swapper/120:55:301
29142992011,6cyclictest0-21swapper/120:20:281
2914299201,15cyclictest10208-21apt-get21:05:131
29142992010,7cyclictest15000-21cat21:45:211
2913399208,10cyclictest20650-21cron22:25:020
2913399202,5cyclictest0-21swapper/020:05:130
29133992013,5cyclictest10541-21sendmail_mailtr22:35:260
29133992013,4cyclictest6084-21cut22:15:210
29133992012,6cyclictest27614-21ssh00:35:030
29133992012,6cyclictest12-21ksoftirqd/020:30:120
29133992011,6cyclictest0-21swapper/021:25:130
2913399201,14cyclictest0-21swapper/023:25:140
2916599199,6cyclictest35966irq/58-eth020:10:113
29165991913,4cyclictest0-21swapper/321:40:303
2915299195,12cyclictest29156-21latency_hist19:10:012
2915299192,8cyclictest26461-21kthreadcore19:40:212
2915299192,14cyclictest25069-21kthreadcore20:45:222
2915299191,10cyclictest12587-21kthreadcore19:25:192
2914299199,7cyclictest5507-21cat21:20:321
2914299199,7cyclictest20518-21cut23:35:241
2914299191,6cyclictest1478-21kthreadcore23:25:221
29142991913,4cyclictest0-21swapper/120:00:241
29142991913,4cyclictest0-21swapper/120:00:231
2914299191,15cyclictest4419-21cut19:15:221
29142991910,6cyclictest27136-21cat20:45:291
29142991910,6cyclictest0-21swapper/120:25:241
29142991910,6cyclictest0-21swapper/120:25:231
2914299190,2cyclictest131rcu_preempt23:10:131
2913399198,8cyclictest5458-21df_inode21:00:140
2913399192,11cyclictest10339-21grep21:05:150
29133991913,4cyclictest0-21swapper/020:40:160
29133991913,4cyclictest0-21swapper/020:15:140
29133991912,5cyclictest27553-21ssh23:04:370
29133991911,6cyclictest12793-21ssh00:07:160
29133991910,7cyclictest21221-21ssh00:11:300
2916599185,6cyclictest0-21swapper/323:35:143
2916599182,10cyclictest0-21swapper/319:10:133
29165991812,4cyclictest0-21swapper/323:10:303
2916599181,13cyclictest35966irq/58-eth022:50:163
2916599181,13cyclictest35966irq/58-eth022:50:153
2916599181,10cyclictest0-21swapper/319:35:123
29165991810,6cyclictest35966irq/58-eth021:35:133
29165991810,6cyclictest0-21swapper/323:26:133
29165991810,6cyclictest0-21swapper/323:23:313
29165991810,6cyclictest0-21swapper/319:40:293
29165991810,5cyclictest35966irq/58-eth000:00:133
29165991810,5cyclictest0-21swapper/323:55:163
2914299189,3cyclictest21934-21sshd20:08:011
2914299188,7cyclictest7784-21apt-get19:55:111
2914299182,3cyclictest6227-21ssh21:58:021
2914299182,12cyclictest15706-21/usr/sbin/munin22:40:301
2914299182,11cyclictest4164-21ssh21:40:001
29142991812,3cyclictest21-21ksoftirqd/100:25:141
2914299181,12cyclictest26495-21sh23:40:141
29142991810,4cyclictest21-21ksoftirqd/122:45:171
2914299180,13cyclictest957-21/usr/sbin/munin00:20:161
2913399182,8cyclictest25035-21kthreadcore20:45:200
2913399181,5cyclictest31890-21ntp_192.168.11423:05:230
29133991811,5cyclictest15377-21grep21:10:160
29133991811,5cyclictest0-21swapper/019:50:220
2913399181,14cyclictest29574-21kthreadcore20:50:210
2913399181,14cyclictest21080-21ssh22:25:140
29133991810,6cyclictest6938-21ssh22:51:290
29133991810,6cyclictest6938-21ssh22:51:280
29133991810,6cyclictest19545-21ssh23:16:310
2916599179,6cyclictest0-21swapper/322:20:273
2916599172,10cyclictest0-21swapper/320:40:143
2916599172,10cyclictest0-21swapper/300:15:163
29165991711,4cyclictest0-21swapper/320:05:273
29165991711,4cyclictest0-21swapper/300:30:253
2916599171,10cyclictest0-21swapper/320:35:113
2916599171,10cyclictest0-21swapper/320:00:153
2916599171,10cyclictest0-21swapper/320:00:153
2914299179,6cyclictest13539-21sshd20:35:011
2914299179,5cyclictest18947-21sendmail_mailst21:10:271
2914299175,9cyclictest29787-21grep19:10:181
2914299171,14cyclictest5423-21cpuspeed21:00:121
29142991711,4cyclictest0-21swapper/119:30:181
2914299171,12cyclictest2679-21awk19:50:011
2913399172,10cyclictest0-21swapper/019:15:150
2913399171,8cyclictest17161-21latency_hist20:05:020
2913399171,8cyclictest17161-21latency_hist20:05:010
29133991711,4cyclictest0-21swapper/020:35:180
29133991711,4cyclictest0-21swapper/019:40:190
2913399171,13cyclictest19168-21kthreadcore19:30:200
2913399171,12cyclictest22105-21kthreadcore19:35:200
2916599168,6cyclictest0-21swapper/300:25:123
2916599162,10cyclictest0-21swapper/320:15:153
2916599161,9cyclictest0-21swapper/321:25:133
2916599161,10cyclictest0-21swapper/321:00:273
2916599161,10cyclictest0-21swapper/319:15:193
2914299163,10cyclictest29475-21tr20:50:211
2914299162,10cyclictest0-21swapper/119:40:161
2914299160,6cyclictest23140-21kthreadcore19:35:201
2913399162,10cyclictest0-21swapper/019:10:130
2913399161,8cyclictest1611-21kthreadcore20:55:200
2913399161,6cyclictest4874-21kthreadcore20:25:180
2913399161,6cyclictest4874-21kthreadcore20:25:170
2913399161,12cyclictest6581-21cut19:20:160
2916599157,6cyclictest0-21swapper/320:49:013
2914299151,11cyclictest22554-21irqcore20:10:161
2913399151,10cyclictest29159-21grep19:10:020
2913399151,10cyclictest11501-21tune2fs19:25:160
2916599149,3cyclictest0-21swapper/320:55:163
2916599149,3cyclictest0-21swapper/320:28:313
2916599149,3cyclictest0-21swapper/320:28:303
2916599143,8cyclictest35966irq/58-eth019:30:113
2916599141,8cyclictest0-21swapper/319:50:273
2916599131,7cyclictest0-21swapper/319:45:423
2916599131,10cyclictest0-21swapper/320:20:323
2916599131,10cyclictest0-21swapper/319:20:163
2914299131,9cyclictest29122-21cyclictest19:10:021
2916599127,3cyclictest0-21swapper/321:05:183
2916599112,5cyclictest0-21swapper/319:25:143
2916599101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional