You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-16 - 04:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Thu Jul 16, 2026 00:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2819199702,49cyclictest20093-21ssh22:25:131
2818399700,8cyclictest131rcu_preempt00:15:130
28191996017,11cyclictest131rcu_preempt23:15:121
2819999584,32cyclictest7322-21ssh23:30:122
2818399563,40cyclictest16532-21ssh22:05:110
2818399522,13cyclictest131rcu_preempt22:35:120
2819999513,36cyclictest25528-21ssh22:10:142
28191995016,24cyclictest21-21ksoftirqd/119:35:141
2819999473,24cyclictest27269-21ssh21:35:162
2818399473,34cyclictest30894-21ssh21:55:130
2819199462,39cyclictest27219-21ssh23:05:131
2819199462,33cyclictest21876-21ssh21:50:141
2819199461,34cyclictest23564-21ssh21:15:111
28207994530,8cyclictest35966irq/58-eth023:20:123
2819199453,36cyclictest3837-21ssh21:40:141
2819999446,27cyclictest27-21ksoftirqd/223:25:132
2819999441,25cyclictest28914-21ssh00:00:132
2819999441,25cyclictest28914-21ssh00:00:122
2818399442,35cyclictest14593-21ssh22:40:130
2819199422,34cyclictest32488-21ssh00:20:151
2820799403,21cyclictest35966irq/58-eth021:20:143
2819999402,20cyclictest2063-21ssh22:15:132
2819999402,20cyclictest2063-21ssh22:15:122
28191993912,19cyclictest21-21ksoftirqd/121:45:121
28183993828,6cyclictest536-21usb-storage23:55:100
2818399372,20cyclictest12-21ksoftirqd/020:20:130
2819999366,20cyclictest27346irq/40-dwc2_hso21:20:132
2819999364,23cyclictest30875-21ssh21:55:122
2819199367,19cyclictest21-21ksoftirqd/120:00:131
2818399354,24cyclictest12-21ksoftirqd/023:05:130
28183993515,14cyclictest12-21ksoftirqd/021:25:130
2820799343,20cyclictest35966irq/58-eth021:30:153
2820799343,20cyclictest35966irq/58-eth021:30:143
28199993420,10cyclictest536-21usb-storage20:10:112
2819199345,20cyclictest14146-21diskmemload23:00:121
28199993318,11cyclictest536-21usb-storage20:30:122
2819199330,23cyclictest9060-21ssh00:25:121
2819199322,23cyclictest21-21ksoftirqd/120:10:101
2819199322,10cyclictest131rcu_preempt20:55:121
28207993122,5cyclictest35966irq/58-eth000:15:123
2819999314,15cyclictest27346irq/40-dwc2_hso23:05:122
2819999313,21cyclictest0-21swapper/200:38:002
2819999312,22cyclictest20085-21diskmemload22:25:112
2819999311,24cyclictest23580-21ssh21:15:132
2819199315,18cyclictest11128-21sh22:20:121
2819199312,23cyclictest18245-21ssh21:30:151
2819199312,23cyclictest18245-21ssh21:30:141
2820799303,21cyclictest35966irq/58-eth022:55:133
2820799303,17cyclictest0-21swapper/321:55:123
2819999307,16cyclictest27-21ksoftirqd/221:52:192
2819999304,16cyclictest27346irq/40-dwc2_hso19:55:132
2819999304,16cyclictest27346irq/40-dwc2_hso00:10:142
2819999303,10cyclictest27-21ksoftirqd/220:20:122
2819999302,19cyclictest27346irq/40-dwc2_hso23:55:122
28191993019,7cyclictest32623-21ssh21:20:141
28191993014,9cyclictest21-21ksoftirqd/121:35:141
2819199301,20cyclictest32552-21ssh22:50:131
2818399304,14cyclictest18198-21ssh23:00:110
2818399302,19cyclictest2050-21ssh22:15:110
2818399302,19cyclictest2050-21ssh22:15:110
2820799291,18cyclictest35966irq/58-eth022:05:113
2819999299,7cyclictest27-21ksoftirqd/200:23:082
2819999298,15cyclictest0-21swapper/219:50:142
2819999297,10cyclictest27-21ksoftirqd/223:50:452
2819999293,16cyclictest27346irq/40-dwc2_hso20:15:122
2819999292,20cyclictest5512-21ssh00:05:142
28199992911,12cyclictest27-21ksoftirqd/223:15:122
2819199292,21cyclictest29071-21ssh22:30:131
2818399293,18cyclictest7540-21ssh22:00:140
2820799282,18cyclictest35966irq/58-eth019:20:123
28207992816,8cyclictest35966irq/58-eth022:15:133
28207992816,8cyclictest35966irq/58-eth022:15:133
28207992812,9cyclictest0-21swapper/323:15:123
2819999283,16cyclictest27346irq/40-dwc2_hso22:45:122
2819999281,15cyclictest27346irq/40-dwc2_hso21:25:122
28199992810,9cyclictest27-21ksoftirqd/222:50:402
2819999280,7cyclictest118431kernelversion22:20:182
2818399283,16cyclictest11114-21ssh22:20:130
28183992815,10cyclictest12-21ksoftirqd/023:35:140
28183992815,10cyclictest12-21ksoftirqd/023:35:130
2818399281,3cyclictest131rcu_preempt23:10:130
2818399281,22cyclictest21768-21ssh23:20:130
2820799273,17cyclictest35966irq/58-eth023:30:123
2820799272,20cyclictest35966irq/58-eth023:55:103
2820799272,16cyclictest35966irq/58-eth020:40:123
2819999279,11cyclictest27-21ksoftirqd/221:30:372
2819999279,11cyclictest27-21ksoftirqd/221:30:372
2819999276,10cyclictest27346irq/40-dwc2_hso20:05:142
2819999275,13cyclictest27346irq/40-dwc2_hso20:35:132
2819999271,17cyclictest21788-21ssh23:20:142
28199992710,4cyclictest27-21ksoftirqd/222:58:042
2819199272,18cyclictest5592-21ssh22:35:131
2819199271,19cyclictest25340-21ssh23:40:151
2818399271,20cyclictest26996-21ssh00:35:140
2819999265,11cyclictest27346irq/40-dwc2_hso20:40:112
2819999262,15cyclictest27-21ksoftirqd/219:25:232
2819999261,12cyclictest3738-21apt-get23:10:132
2818399268,3cyclictest131rcu_preempt00:25:190
28199992513,9cyclictest29159-21sh22:30:142
2819999251,19cyclictest3844-21sh21:40:142
2819999251,11cyclictest2090-21fschecks_time23:45:152
2819999250,18cyclictest27-21ksoftirqd/200:27:122
2818399259,10cyclictest12-21ksoftirqd/019:30:140
2818399257,15cyclictest12-21ksoftirqd/022:55:120
2818399251,18cyclictest30805-21ssh23:25:130
28183992510,12cyclictest14105-21cron00:10:010
2820799244,15cyclictest35966irq/58-eth000:05:133
2820799242,13cyclictest35966irq/58-eth022:10:133
28207992413,8cyclictest35966irq/58-eth022:00:143
2820799241,15cyclictest0-21swapper/323:05:133
2819999249,7cyclictest27-21ksoftirqd/220:25:172
2819999248,9cyclictest27-21ksoftirqd/223:00:042
2819999248,8cyclictest27-21ksoftirqd/200:33:422
2819999247,14cyclictest27-21ksoftirqd/220:45:222
2819999242,18cyclictest5848-21tr19:20:182
2819999242,13cyclictest27-21ksoftirqd/221:48:042
2819999242,11cyclictest14560-21ssh21:10:142
28199992416,5cyclictest25872-21ssh00:15:202
28199992413,9cyclictest27-21ksoftirqd/223:40:012
28199992413,9cyclictest27-21ksoftirqd/223:40:012
2819999241,21cyclictest26005-21ssh23:40:182
2819999241,19cyclictest26025-21kthreadcore19:40:192
2819999241,15cyclictest27-21ksoftirqd/222:00:132
28191992415,6cyclictest14395-21ssh23:50:241
2819199241,17cyclictest9699-21grep21:05:181
2818399241,19cyclictest23696-21cut22:45:150
2820799232,12cyclictest35966irq/58-eth022:40:123
2820799232,12cyclictest35966irq/58-eth021:40:143
2819999236,13cyclictest27-21ksoftirqd/222:40:182
2819999231,11cyclictest8694-21ssh22:35:222
2819199232,17cyclictest32439-1kworker/u9:023:45:111
2820799229,9cyclictest0-21swapper/321:15:123
2820799226,10cyclictest35966irq/58-eth000:35:133
28207992210,8cyclictest35966irq/58-eth000:00:253
28207992210,8cyclictest35966irq/58-eth000:00:243
2819999222,12cyclictest30803-21kthreadcore19:45:192
2819999222,10cyclictest16171-21latency_hist20:05:012
2819999221,17cyclictest32502-21fschecks_time20:55:162
2819999221,15cyclictest27-21ksoftirqd/222:06:572
2819999221,11cyclictest31009-21cat20:50:282
2819199228,9cyclictest28174-21cyclictest00:38:591
2819199223,3cyclictest131rcu_preempt23:30:131
2819199223,3cyclictest131rcu_preempt23:10:121
2819199222,16cyclictest7775-21cut22:00:161
28191992214,4cyclictest131rcu_preempt19:25:121
28191992211,8cyclictest803-21ssh00:00:301
28191992211,8cyclictest803-21ssh00:00:301
2819199221,17cyclictest27595-21cut20:50:171
28183992212,7cyclictest0-21swapper/021:05:160
2818399221,15cyclictest20094-21ssh22:25:130
2818399221,15cyclictest1901-21ssh23:45:130
2820799215,9cyclictest321rcuc/319:25:113
2820799211,16cyclictest0-21swapper/323:35:133
2820799211,16cyclictest0-21swapper/323:35:133
2820799211,14cyclictest0-21swapper/323:40:143
2819999212,9cyclictest3812-21kthreadcore19:15:192
2819999212,10cyclictest29316-21kthreadcore19:10:202
2819999211,16cyclictest4936-21fschecks_time21:00:142
2819999211,15cyclictest20133-21fschecks_time19:35:162
2819999211,15cyclictest17348-21kthreadcore19:30:212
2819999211,11cyclictest0-21swapper/221:05:202
2819199214,3cyclictest131rcu_preempt21:55:121
2819199214,3cyclictest131rcu_preempt00:06:521
2819199212,5cyclictest131rcu_preempt22:55:131
28191992113,5cyclictest27680-21unixbench_singl22:45:291
2818399213,11cyclictest0-21swapper/020:35:140
2818399212,15cyclictest705-21apt-get19:15:100
28183992112,6cyclictest11005-21awk23:50:140
2818399211,17cyclictest28415-21cut19:10:160
2820799203,10cyclictest0-21swapper/320:30:123
28207992012,6cyclictest0-21swapper/300:20:213
28207992011,5cyclictest35966irq/58-eth021:49:243
2820799201,12cyclictest35966irq/58-eth023:10:123
28207992010,7cyclictest35966irq/58-eth021:35:143
2819199207,4cyclictest131rcu_preempt23:55:271
2819199202,14cyclictest2052-21apt-get22:15:121
2819199202,14cyclictest2052-21apt-get22:15:111
2819199201,13cyclictest21152-21latency20:40:221
2819199200,17cyclictest20575-21timerwakeupswit22:05:291
2818399201,8cyclictest967-21tr22:50:190
28183992014,4cyclictest21834-21/usr/sbin/munin21:50:240
28183992013,5cyclictest12-21ksoftirqd/000:00:110
28183992013,5cyclictest12-21ksoftirqd/000:00:110
2818399201,2cyclictest0-21swapper/023:44:500
2818399201,11cyclictest3699-21ntp_192.168.11400:20:250
2818399201,10cyclictest4262-21kthreadcore19:50:210
2818399200,14cyclictest14831-21ls21:10:170
2820799199,8cyclictest0-21swapper/322:37:563
2820799199,8cyclictest0-21swapper/322:31:273
2820799198,6cyclictest0-21swapper/323:54:493
2820799191,9cyclictest0-21swapper/322:45:113
28207991912,5cyclictest0-21swapper/322:20:183
28207991910,7cyclictest0-21swapper/323:45:133
28207991910,6cyclictest0-21swapper/300:27:453
2819199198,9cyclictest20935-21cron20:10:011
2819199198,8cyclictest26059-21apt-get20:15:131
2819199193,12cyclictest25440-21ssh23:20:251
2819199191,9cyclictest10403-21cat19:55:281
2818399194,12cyclictest27967-21cat21:15:290
2818399193,12cyclictest9478-21latency_hist21:05:020
2818399192,13cyclictest29339-21kworker/u8:020:10:120
2818399192,12cyclictest0-21swapper/019:55:140
2818399191,9cyclictest3631-21wc20:55:270
2818399191,9cyclictest29080-21kthreadcore21:35:220
2818399191,8cyclictest15240-21irqstats00:10:190
2818399191,15cyclictest12576-21latency_hist21:45:020
28183991910,5cyclictest8079-21kthreadcore23:30:200
2820799189,6cyclictest35966irq/58-eth022:25:123
2820799189,6cyclictest35966irq/58-eth021:25:133
2820799189,6cyclictest0-21swapper/322:51:303
2820799182,10cyclictest0-21swapper/320:55:053
2820799182,10cyclictest0-21swapper/320:25:053
2820799182,10cyclictest0-21swapper/320:20:053
2820799182,10cyclictest0-21swapper/320:15:143
2820799182,10cyclictest0-21swapper/319:35:143
28207991812,4cyclictest0-21swapper/323:27:293
28207991812,4cyclictest0-21swapper/321:50:383
2820799181,12cyclictest35966irq/58-eth000:30:113
2820799181,12cyclictest0-21swapper/320:45:153
28207991810,5cyclictest35966irq/58-eth023:04:433
2819199188,7cyclictest19102-21cat00:10:321
2819199181,3cyclictest131rcu_preempt21:00:231
2819199181,15cyclictest31389-21kthreadcore19:10:221
2819199181,13cyclictest2071-21ntp_states23:25:271
2819199181,13cyclictest17982-21ssh00:30:121
2819199181,11cyclictest13271-21cut20:35:161
28191991810,5cyclictest30084-21grep19:45:181
2818399182,14cyclictest11155-21kthreadcore19:25:180
2818399182,12cyclictest18219-21apt-get21:30:140
2818399182,12cyclictest18219-21apt-get21:30:130
2818399181,9cyclictest16477-21/usr/sbin/munin23:15:270
28183991811,4cyclictest0-21swapper/019:20:270
2818399181,13cyclictest18103-21diskstats00:30:130
2820799179,6cyclictest0-21swapper/321:11:303
2820799172,10cyclictest0-21swapper/320:35:143
2820799172,10cyclictest0-21swapper/320:10:123
2820799172,10cyclictest0-21swapper/320:00:133
2820799172,10cyclictest0-21swapper/319:30:143
2820799171,13cyclictest0-21swapper/319:40:273
28207991710,5cyclictest35966irq/58-eth000:10:203
2820799170,10cyclictest35966irq/58-eth021:00:103
2819199179,6cyclictest29344-21sshd19:42:441
2819199173,3cyclictest131rcu_preempt20:25:171
2819199172,12cyclictest1977-21cpu19:50:141
2819199172,11cyclictest0-21swapper/121:10:141
2819199172,10cyclictest15086-21kworker/u8:120:30:131
2819199171,13cyclictest9333-21rm21:25:151
28191991711,3cyclictest24450-21cut00:15:181
2819199171,11cyclictest15494-21sed19:30:171
28191991710,5cyclictest29127-21ssh22:10:251
28191991710,4cyclictest17098-21kthreadcore23:35:211
28191991710,4cyclictest17098-21kthreadcore23:35:201
2819199170,9cyclictest3810-21kthreadcore19:15:191
2818399179,6cyclictest30741-21sed20:20:030
2818399178,7cyclictest29156-21df22:30:150
2818399175,9cyclictest775-21wc19:45:270
2818399173,12cyclictest32575-21/usr/sbin/munin21:20:190
2818399172,10cyclictest0-21swapper/020:45:150
2818399171,9cyclictest28081-21ssh22:10:220
28183991711,4cyclictest5288-21kthreadcore20:25:200
2818399171,12cyclictest14850-21ssh21:45:200
2820799162,10cyclictest0-21swapper/319:45:143
2820799161,11cyclictest0-21swapper/319:15:173
2819999161,12cyclictest28211-21latency_hist19:10:012
2819999161,12cyclictest28211-21latency_hist19:10:012
2819199163,11cyclictest378-21kthreadcore20:20:211
2819199162,2cyclictest131rcu_preempt22:40:071
2819199161,11cyclictest22773-21diskstats20:45:161
2818399168,5cyclictest9409-21kthreadcore20:30:190
2818399162,8cyclictest0-21swapper/020:50:140
2818399161,13cyclictest18074-21cut20:40:150
2818399161,11cyclictest21172-21kthreadcore19:35:200
2819199151,11cyclictest5525-21apt-get19:20:131
2819199151,10cyclictest28216-21latency_hist19:10:021
2819199151,10cyclictest28216-21latency_hist19:10:011
2818399156,5cyclictest0-21swapper/019:40:110
2818399152,9cyclictest0-21swapper/020:05:150
2818399152,10cyclictest11740-21fschecks_time20:00:160
2820799149,3cyclictest0-21swapper/321:05:263
2820799132,8cyclictest0-21swapper/319:55:143
2820799132,7cyclictest35966irq/58-eth019:10:123
2820799131,8cyclictest35966irq/58-eth020:50:143
2820799131,7cyclictest0-21swapper/320:05:173
2818399131,9cyclictest28189-21latency_hist19:10:020
2818399131,9cyclictest28189-21latency_hist19:10:010
2820799121,7cyclictest0-21swapper/319:10:013
2820799121,7cyclictest0-21swapper/319:10:013
2820799121,5cyclictest0-21swapper/319:50:193
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional