You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-15 - 08:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Mon Jun 15, 2026 00:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29868995731,20cyclictest35966irq/58-eth000:35:133
2983699573,45cyclictest30204-21ssh23:05:140
2984799522,40cyclictest5445-21ssh23:45:141
2984799522,40cyclictest5445-21ssh23:45:141
29868995024,15cyclictest35966irq/58-eth022:35:123
2984799503,7cyclictest131rcu_preempt00:10:151
2984799483,9cyclictest131rcu_preempt20:20:131
2985899472,32cyclictest24896-21ssh23:20:122
2983699474,5cyclictest131rcu_preempt20:10:120
2983699461,39cyclictest28860-21ssh21:35:140
2984799434,32cyclictest1610-21ssh21:20:121
29847994328,6cyclictest21-21ksoftirqd/123:20:121
2983699433,5cyclictest131rcu_preempt22:30:140
2986899422,24cyclictest35966irq/58-eth023:20:123
2985899421,29cyclictest6785-21ssh23:10:112
2985899421,29cyclictest6785-21ssh23:10:112
2984799411,25cyclictest23701-21ssh21:50:131
2983699412,4cyclictest131rcu_preempt22:25:120
2986899403,24cyclictest35966irq/58-eth022:30:143
2984799407,10cyclictest131rcu_preempt20:30:121
2983699405,23cyclictest14577-21ssh23:50:130
2983699404,25cyclictest4219-21sh22:15:130
2986899393,23cyclictest35966irq/58-eth021:55:133
2983699392,33cyclictest28841-21ssh23:40:130
2983699392,29cyclictest31116-21ssh00:35:120
2986899381,20cyclictest35966irq/58-eth019:15:133
2985899382,17cyclictest19773-21ssh23:35:122
2985899382,17cyclictest19773-21ssh23:35:112
2986899372,23cyclictest35966irq/58-eth021:25:133
2986899372,23cyclictest35966irq/58-eth021:25:133
29868993511,12cyclictest35966irq/58-eth022:05:143
29868993511,12cyclictest35966irq/58-eth022:05:143
2985899353,16cyclictest30195-21ssh23:05:132
2983699352,24cyclictest23675-21ssh23:55:130
2986899343,16cyclictest35966irq/58-eth000:10:143
2985899349,20cyclictest27-21ksoftirqd/219:50:132
2985899343,19cyclictest27-21ksoftirqd/223:00:122
2984799342,2cyclictest131rcu_preempt22:15:131
2983699348,17cyclictest18496-21ssh22:05:130
2983699348,17cyclictest18496-21ssh22:05:120
2984799334,17cyclictest844-21munin-node21:00:141
2984799333,19cyclictest21-21ksoftirqd/122:00:121
29847993318,10cyclictest0-21swapper/123:00:131
29858993213,11cyclictest31350-21ssh22:30:142
2983699321,27cyclictest32732-21ssh00:00:130
2985899314,18cyclictest27346irq/40-dwc2_hso21:10:122
2985899304,12cyclictest261rcuc/200:10:142
2985899303,21cyclictest0-21swapper/219:40:132
2985899303,16cyclictest10714-21ssh21:25:132
2985899303,16cyclictest10714-21ssh21:25:132
2985899302,21cyclictest0-21swapper/220:15:152
2985899302,18cyclictest27346irq/40-dwc2_hso19:55:132
29847993023,4cyclictest12042-21sh22:55:121
2983699302,22cyclictest15802-21ssh23:15:130
2983699302,22cyclictest15802-21ssh23:15:130
2986899296,15cyclictest35966irq/58-eth022:15:133
2986899282,15cyclictest35966irq/58-eth022:20:123
2985899283,17cyclictest27346irq/40-dwc2_hso21:50:132
2985899282,20cyclictest4014-21ssh00:20:142
2985899282,19cyclictest0-21swapper/220:10:132
2985899282,16cyclictest27346irq/40-dwc2_hso21:05:142
29858992810,13cyclictest27-21ksoftirqd/223:15:132
29858992810,13cyclictest27-21ksoftirqd/223:15:132
2984799283,6cyclictest131rcu_preempt19:35:141
2984799282,21cyclictest14607-21ssh21:45:151
2984799282,19cyclictest15859-21ssh21:10:141
29847992817,4cyclictest21-21ksoftirqd/121:15:151
2983699283,16cyclictest8072-21ssh22:35:130
2983699281,21cyclictest2988-21ssh22:50:140
2986899275,14cyclictest0-21swapper/323:25:133
2985899278,9cyclictest14256-21cron20:35:022
2985899273,14cyclictest27346irq/40-dwc2_hso19:45:142
2985899272,20cyclictest0-21swapper/222:35:132
2985899271,19cyclictest5435-21ssh23:45:142
2985899271,19cyclictest5435-21ssh23:45:142
2985899271,10cyclictest279766ssh00:35:302
2984799277,10cyclictest27562-21ssh22:10:121
2984799271,14cyclictest5532-21ssh21:40:131
2986899262,15cyclictest35966irq/58-eth000:20:143
29868992616,6cyclictest0-21swapper/322:00:123
29868992616,4cyclictest0-21swapper/323:35:123
29868992616,4cyclictest0-21swapper/323:35:123
2985899263,16cyclictest27346irq/40-dwc2_hso20:35:142
2985899261,18cyclictest11945-21find19:25:092
2985899261,17cyclictest27346irq/40-dwc2_hso22:00:122
29847992612,9cyclictest211ksoftirqd/100:00:131
29847992610,11cyclictest21-21ksoftirqd/100:30:111
2983699265,18cyclictest12-21ksoftirqd/021:15:140
2983699262,19cyclictest19796-21ssh21:30:140
2983699261,19cyclictest1597-21ssh23:25:130
2986899252,11cyclictest35966irq/58-eth020:55:133
29868992514,7cyclictest0-21swapper/321:15:123
2986899251,15cyclictest35966irq/58-eth021:45:143
29868992510,10cyclictest35966irq/58-eth022:40:143
2985899258,13cyclictest261rcuc/223:50:122
2985899256,16cyclictest13331-21ssh22:20:142
2985899251,16cyclictest1056-21latency_hist20:55:012
2985899251,16cyclictest1056-21latency_hist20:55:012
2985899251,14cyclictest4809-21kthreadcore23:25:222
2985899251,10cyclictest0-21swapper/220:05:152
2984799253,17cyclictest4024-21ssh00:20:151
2984799252,15cyclictest0-21swapper/100:25:141
29868992415,7cyclictest0-21swapper/323:45:143
29868992415,7cyclictest0-21swapper/323:45:143
2986899241,2cyclictest0-21swapper/322:10:133
2986899241,15cyclictest35966irq/58-eth019:20:123
2985899249,7cyclictest261rcuc/222:45:112
2985899243,13cyclictest1601-21ssh21:20:112
2985899241,19cyclictest930-21snmpd20:40:142
2985899241,15cyclictest27346irq/40-dwc2_hso19:15:122
29847992413,7cyclictest0-21swapper/121:55:121
2983699242,18cyclictest10728-21ssh21:25:140
2983699242,18cyclictest10728-21ssh21:25:140
2985899232,9cyclictest0-21swapper/221:45:152
2985899231,9cyclictest22074-21ssh00:30:132
29858992312,4cyclictest0-21swapper/200:00:312
2985899231,19cyclictest26125-21latency_hist22:45:022
2985899231,18cyclictest27433-21ssh00:15:142
2985899231,18cyclictest21663-21grep19:35:162
2985899231,15cyclictest21832-21ssh00:30:022
2985899231,13cyclictest18887-21irqcore22:05:192
2985899231,13cyclictest18887-21irqcore22:05:192
2984799232,19cyclictest17153-21ssh22:40:121
29847992316,5cyclictest0-21swapper/121:35:131
29847992313,7cyclictest15442-21cron21:10:011
2983699233,16cyclictest12063-21ssh22:55:140
29836992318,3cyclictest0-21swapper/020:45:120
2986899227,10cyclictest35966irq/58-eth000:25:143
2986899225,11cyclictest35966irq/58-eth022:50:143
2986899225,10cyclictest35966irq/58-eth023:00:113
2986899223,13cyclictest0-21swapper/323:10:123
2986899223,13cyclictest0-21swapper/323:10:123
2985899229,3cyclictest131rcu_preempt23:40:262
2985899223,15cyclictest21501-21latency_hist19:35:022
2985899222,16cyclictest12155-21df22:55:142
2985899221,5cyclictest26278-21timerandwakeup22:25:292
29858992211,9cyclictest27-21ksoftirqd/222:53:102
29858992211,9cyclictest0-21swapper/222:10:422
2985899221,17cyclictest25599-21irqrtprio21:15:192
2984799224,14cyclictest10719-21ssh21:25:131
2984799224,14cyclictest10719-21ssh21:25:131
2984799222,16cyclictest22359-21ssh22:25:141
2984799222,16cyclictest15444-21diskmemload00:05:151
2983699221,16cyclictest32539-21apt-get20:20:130
2983699221,13cyclictest4009-21ssh00:20:140
2983699221,11cyclictest0-21swapper/023:30:290
2986899215,9cyclictest0-21swapper/321:20:123
2986899213,13cyclictest0-21swapper/320:20:143
29868992114,4cyclictest0-21swapper/323:15:123
29868992114,4cyclictest0-21swapper/323:15:113
29868992113,5cyclictest0-21swapper/320:45:123
29868992112,6cyclictest0-21swapper/321:10:283
29868992112,5cyclictest0-21swapper/322:25:133
29868992111,7cyclictest0-21swapper/300:00:143
29868992110,9cyclictest0-21swapper/323:09:523
2985899219,9cyclictest0-21swapper/220:00:152
2985899217,7cyclictest261rcuc/221:55:122
2985899217,12cyclictest27-21ksoftirqd/221:35:232
2985899211,15cyclictest27346irq/40-dwc2_hso19:20:132
2985899211,12cyclictest4977-21wc20:55:282
2985899211,11cyclictest4956-21grep22:15:182
2985899211,11cyclictest12108-21tr00:05:242
2985899211,10cyclictest31589-21kthreadcore19:10:182
2985899210,10cyclictest19319-21latency_hist23:35:002
2984799211,9cyclictest18245-21ntp_192.168.11423:50:251
29847992111,7cyclictest2082-21cron19:15:011
2984799211,14cyclictest18184-21cut20:05:171
2984799210,13cyclictest7492-21ls00:40:021
2983699217,4cyclictest131rcu_preempt22:10:180
2983699211,16cyclictest14493-21apt-get20:35:130
2983699210,3cyclictest131rcu_preempt19:45:130
2986899209,8cyclictest0-21swapper/323:55:213
2986899203,9cyclictest0-21swapper/323:40:133
2986899203,12cyclictest0-21swapper/300:05:133
2986899202,13cyclictest0-21swapper/300:15:133
29868992013,4cyclictest0-21swapper/319:30:143
2985899207,5cyclictest23420-21sh21:30:252
2985899203,9cyclictest6098-21kthreadcore20:25:202
2985899203,14cyclictest25131-21kthreadcore23:55:212
2985899202,9cyclictest25446-21kthreadcore20:45:182
2985899201,9cyclictest3467-21grep20:20:232
2985899201,15cyclictest8817-21tr21:00:212
29858992010,8cyclictest12445-21ssh21:42:572
2984799207,3cyclictest131rcu_preempt19:25:011
2984799203,12cyclictest0-21swapper/123:10:121
2984799203,12cyclictest0-21swapper/123:10:121
2984799202,15cyclictest23663-21ssh23:55:121
2984799201,8cyclictest9458-21munin-run20:30:011
2984799201,16cyclictest24262-21cut20:45:151
2984799201,14cyclictest2689-21ssh22:30:291
29847992010,8cyclictest21-21ksoftirqd/123:40:131
29847992010,7cyclictest13307-21apt-get22:20:131
2983699201,12cyclictest6896-21latency_hist19:20:010
29836992010,5cyclictest0-21swapper/020:25:120
2983699200,17cyclictest29102-21fschecks_time20:50:160
2983699200,17cyclictest29102-21fschecks_time20:50:160
2986899198,7cyclictest0-21swapper/322:45:173
2986899198,7cyclictest0-21swapper/300:30:313
2986899192,10cyclictest0-21swapper/319:55:143
2986899191,14cyclictest0-21swapper/319:35:283
2986899191,11cyclictest0-21swapper/321:00:173
29868991910,6cyclictest0-21swapper/321:50:033
29868991910,6cyclictest0-21swapper/321:39:383
2984799199,7cyclictest19892-21df_inode21:30:151
2984799192,12cyclictest21-21ksoftirqd/119:50:131
2984799192,11cyclictest23397-21sh23:35:241
2984799192,11cyclictest23397-21sh23:35:241
29847991911,4cyclictest536-21usb-storage22:45:041
29847991910,6cyclictest10280-21cron23:30:011
2984799190,15cyclictest31325-21switchtime20:15:291
2983699199,7cyclictest22651-21cron20:10:020
2983699194,4cyclictest536-21usb-storage23:35:050
2983699194,4cyclictest536-21usb-storage23:35:050
2983699193,9cyclictest0-21swapper/021:05:140
2983699193,12cyclictest14389-21date23:50:030
2983699193,12cyclictest14389-21date23:50:020
2983699193,10cyclictest0-21swapper/000:25:150
2983699192,15cyclictest24932-21mailstats23:00:270
2983699192,14cyclictest30653-21kthreadcore19:10:180
2983699191,9cyclictest6197-21fschecks_count21:00:160
29836991913,3cyclictest0-21swapper/022:40:120
29836991911,5cyclictest0-21swapper/021:50:010
2983699191,13cyclictest9410-21sh22:00:120
2983699191,13cyclictest26329-21ssh22:45:120
2986899188,6cyclictest35966irq/58-eth023:30:123
2986899188,4cyclictest0-21swapper/322:55:183
2986899188,3cyclictest0-21swapper/321:43:523
2986899188,3cyclictest0-21swapper/321:32:053
2986899186,4cyclictest0-21swapper/320:30:013
2986899182,10cyclictest0-21swapper/320:35:153
2986899182,10cyclictest0-21swapper/320:10:133
2986899182,10cyclictest0-21swapper/319:50:143
2986899182,10cyclictest0-21swapper/319:10:133
2985899181,13cyclictest29904-21latency_hist19:10:022
2984799182,14cyclictest13558-21kthreadcore23:30:211
2984799182,13cyclictest8437-21apt-get19:55:121
2984799181,4cyclictest131rcu_preempt20:50:121
2984799181,4cyclictest131rcu_preempt20:50:121
2984799181,2cyclictest131rcu_preempt20:10:111
29847991812,4cyclictest1870-21grep23:05:291
29847991812,3cyclictest0-21swapper/122:35:041
2984799181,14cyclictest24701-21latency_hist23:20:011
2984799181,14cyclictest24701-21latency_hist23:20:011
2984799181,14cyclictest12276-21grep19:25:151
2984799181,12cyclictest6695-21ssh22:50:271
29847991810,5cyclictest27347-21/usr/sbin/munin00:15:291
2984799180,15cyclictest14484-21find20:35:131
2983699187,3cyclictest131rcu_preempt00:30:190
2983699183,12cyclictest21306-21munin-run19:35:010
2983699182,14cyclictest1583-21apt-get21:20:100
2983699182,13cyclictest7476-21irqrtprio23:10:170
2983699182,13cyclictest7476-21irqrtprio23:10:170
2983699181,9cyclictest0-21swapper/023:20:110
2983699181,8cyclictest19706-21/usr/sbin/munin21:10:280
2983699181,7cyclictest9006-21munin-run22:00:010
2983699181,14cyclictest5541-21ssh21:40:130
29836991810,5cyclictest10120-21kthreadcore00:05:210
2986899179,5cyclictest0-21swapper/321:05:263
2986899179,5cyclictest0-21swapper/320:00:243
2986899177,7cyclictest0-21swapper/323:51:043
2986899172,6cyclictest0-21swapper/320:50:143
2986899172,6cyclictest0-21swapper/320:50:143
2986899171,13cyclictest0-21swapper/320:30:113
2984799178,7cyclictest13440-21tune2fs20:00:171
2984799171,13cyclictest1483-21tune2fs20:55:151
2984799171,10cyclictest21-21ksoftirqd/122:05:081
2984799171,10cyclictest21-21ksoftirqd/122:05:081
2983699178,6cyclictest30220-21grep19:40:280
2983699178,6cyclictest12153-21tune2fs19:25:130
2983699175,5cyclictest24862-21cat19:35:230
2983699174,9cyclictest30542-21kthreadcore00:15:230
2983699174,10cyclictest9549-21kthreadcore19:20:210
2983699173,10cyclictest0-21swapper/020:15:150
2983699172,3cyclictest131rcu_preempt00:10:140
2983699171,13cyclictest8591-21fschecks_time19:55:160
2983699171,13cyclictest21652-21cut20:40:210
2983699171,13cyclictest13151-21ntp_states20:30:250
2983699171,11cyclictest23573-21ssh21:50:050
29836991710,4cyclictest13284-21/usr/sbin/munin22:20:160
2986899162,9cyclictest0-21swapper/320:40:143
2986899162,9cyclictest0-21swapper/319:45:043
2986899162,8cyclictest0-21swapper/319:25:113
29868991610,3cyclictest0-21swapper/320:15:143
29868991610,3cyclictest0-21swapper/319:40:113
2984799167,6cyclictest2824-21irqrtprio19:15:181
2984799164,9cyclictest20548-21grep19:30:301
2984799162,11cyclictest31223-21/usr/sbin/munin19:45:271
2984799161,12cyclictest19414-21diskstats20:40:151
2983699163,10cyclictest4656-21kthreadcore19:50:200
2983699163,10cyclictest0-21swapper/020:55:140
2983699162,11cyclictest15180-21kthreadcore20:00:210
29868991510,3cyclictest0-21swapper/320:05:153
2984799151,11cyclictest26451-21fschecks_count19:40:151
2984799141,9cyclictest29915-21cat19:10:021
2986899138,3cyclictest0-21swapper/319:10:023
2983699131,8cyclictest29872-21latency_hist19:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional