You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-27 - 23:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Apr 27, 2026 12:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1656199541,43cyclictest6066-21ssh11:00:150
1657899520,49cyclictest3994-21apt-get11:35:152
1657899511,42cyclictest2565-21apt-get10:40:142
1657899501,43cyclictest0-21swapper/208:00:112
1657899491,42cyclictest25490-21apt-get12:05:132
1657899491,41cyclictest28046-21apt-get10:00:132
1657899491,41cyclictest11540-21apt-get10:45:122
1658999485,27cyclictest35966irq/58-eth009:30:133
1657899481,40cyclictest24067-21apt-get08:25:152
1657899470,40cyclictest0-21swapper/208:40:142
1657899461,40cyclictest0-21swapper/208:50:152
1657099461,4cyclictest131rcu_preempt09:25:121
1656199462,39cyclictest29202-21ssh12:25:100
1657899440,29cyclictest7200-21ls11:55:012
1657099443,33cyclictest8156-21ssh10:25:141
1657099443,29cyclictest2595-21ssh10:40:141
16570994416,18cyclictest21-21ksoftirqd/108:40:141
1656199447,31cyclictest131rcu_preempt10:00:140
1657899431,37cyclictest26005-21ssh10:35:142
1657899431,37cyclictest26005-21ssh10:35:132
1657899431,25cyclictest4576-21acpi10:05:142
1657899430,38cyclictest27541-21apt-get07:55:132
1657099433,32cyclictest10024-21ssh09:50:121
1657899422,36cyclictest0-21swapper/211:30:132
1657899421,37cyclictest0-21swapper/207:15:132
1657899421,36cyclictest22579-21apt-get10:15:132
1657899420,34cyclictest20128-21apt-get12:20:132
1657899420,29cyclictest2382-21diskmemload11:23:162
1657099421,31cyclictest22607-21ssh10:15:131
1656199423,36cyclictest23977-21ssh11:10:130
1658999417,26cyclictest35966irq/58-eth009:45:133
1656199414,35cyclictest20840-21ssh09:20:130
1657899401,33cyclictest18990-21apt-get09:55:112
1657099403,3cyclictest131rcu_preempt12:30:121
1657099403,3cyclictest131rcu_preempt12:30:111
1658999395,19cyclictest35966irq/58-eth011:05:123
1657899391,33cyclictest6340-21apt-get09:30:142
1657899391,32cyclictest20499-21ssh10:50:132
1657899391,22cyclictest0-21swapper/209:49:412
1657899390,34cyclictest0-21swapper/210:30:112
1657899390,34cyclictest0-21swapper/210:30:112
1657899390,24cyclictest29761-21apt-get09:25:132
16561993925,8cyclictest131rcu_preempt08:30:130
1656199391,34cyclictest14980-21ssh11:05:130
1657899381,33cyclictest0-21swapper/210:20:112
1657899381,32cyclictest0-21swapper/212:30:122
1657899381,32cyclictest0-21swapper/212:30:112
1657899380,35cyclictest0-21swapper/211:15:132
1657099383,24cyclictest11776-21ssh09:15:141
1658999374,20cyclictest35966irq/58-eth011:20:133
1657899371,32cyclictest14854-21ssh12:35:122
1657899370,34cyclictest21367-21ntp_192.168.11407:45:232
1657899370,33cyclictest28959-21proc_pri09:00:262
1657899370,21cyclictest0-21swapper/212:16:542
1657099371,28cyclictest26015-21ssh10:35:141
1657099371,28cyclictest26015-21ssh10:35:141
16561993723,5cyclictest12-21ksoftirqd/008:20:130
16578993619,10cyclictest27346irq/40-dwc2_hso12:25:122
1657899360,32cyclictest27895-21ssh11:46:492
1657899360,32cyclictest22771-21apt-get07:50:132
1657899360,28cyclictest6799-21/usr/sbin/munin07:30:202
1657899360,23cyclictest2382-21diskmemload10:27:472
1656199361,31cyclictest29775-21ssh09:25:120
1657899351,31cyclictest19471-21perl08:20:162
1657899351,31cyclictest13076-21munin-node07:40:112
1657899351,28cyclictest24377-21ssh09:40:152
1657899351,28cyclictest24377-21ssh09:40:142
1657899351,21cyclictest0-21swapper/209:39:422
1657899351,21cyclictest0-21swapper/208:10:012
1657899350,32cyclictest17217-21ntp_states10:10:252
1657899350,31cyclictest28941-21apt-get08:30:052
1657899350,31cyclictest0-21swapper/209:50:132
1657099351,28cyclictest7534-21sh11:55:131
1657899342,29cyclictest18190-21timerwakeupswit08:15:292
1657899341,29cyclictest31246-21/usr/sbin/munin07:25:172
1657899341,29cyclictest0-21swapper/208:55:142
1657899341,29cyclictest0-21swapper/208:55:132
1657899341,22cyclictest7640-21ssh12:11:202
1657899340,31cyclictest22124-21ssh11:08:152
1657899340,30cyclictest3169-21interrupts09:10:192
1657099344,19cyclictest21-21ksoftirqd/112:00:141
1657099343,27cyclictest11082-21ssh12:15:131
1657099343,25cyclictest11571-21ssh10:45:141
1656199343,22cyclictest24357-21ssh09:40:130
1656199343,22cyclictest24357-21ssh09:40:120
1658999338,15cyclictest35966irq/58-eth009:10:133
1657899335,25cyclictest1340-21sort08:35:112
1657899331,29cyclictest14124-21kthreadcore09:15:222
1657899331,28cyclictest11062-21ssh11:00:462
1657899331,28cyclictest0-21swapper/211:40:122
1657899330,30cyclictest14178-21/usr/sbin/munin08:10:312
1657899330,30cyclictest0-21swapper/208:45:142
1657899330,27cyclictest20810-21ssh09:20:132
1657899330,26cyclictest12807-21users07:35:322
1656199333,23cyclictest576-21ssh11:15:130
1658999322,15cyclictest35966irq/58-eth011:30:123
1657899321,28cyclictest7123-21cron11:55:012
1657899321,28cyclictest16701-21/usr/sbin/munin07:10:112
1657899320,28cyclictest30156-21ssh11:12:072
1656199322,14cyclictest131rcu_preempt07:50:120
1657899311,26cyclictest29908-21grep07:20:262
1657899311,22cyclictest3053-21rm10:56:542
1657899310,28cyclictest30940-21kthreadcore09:05:192
1657899310,23cyclictest22988-21ssh12:02:272
1657899310,23cyclictest18384-21ssh11:25:042
1656199312,24cyclictest29511-21ssh10:55:140
1658999306,14cyclictest0-21swapper/312:25:103
16589993015,10cyclictest35966irq/58-eth010:15:133
1656199302,22cyclictest18535-21ssh11:25:150
1658999293,15cyclictest35966irq/58-eth011:55:143
1658999291,15cyclictest35966irq/58-eth012:35:113
1657099292,21cyclictest0-21swapper/107:55:151
1656199295,18cyclictest2382-21diskmemload11:45:130
1656199293,20cyclictest844-21munin-node09:10:150
16561992912,10cyclictest16551-21cyclictest09:45:120
1658999282,16cyclictest35966irq/58-eth011:45:133
1657099286,14cyclictest0-21swapper/111:35:141
1656199282,21cyclictest31611-21ssh10:20:130
1658999276,9cyclictest35966irq/58-eth010:55:143
1658999271,17cyclictest35966irq/58-eth011:40:113
16589992711,5cyclictest0-21swapper/310:00:143
1657899271,21cyclictest16584-21wc07:10:012
16589992614,9cyclictest35966irq/58-eth010:10:143
1656199261,8cyclictest5877-21apt-get12:30:140
1656199261,8cyclictest5877-21apt-get12:30:130
1658999251,11cyclictest35966irq/58-eth012:10:133
1657099252,17cyclictest15369-21ssh09:35:151
1657099244,11cyclictest18525-21ssh11:25:141
16570992413,6cyclictest557-21ssh11:15:131
1656199243,15cyclictest7541-21ssh11:55:140
1658999233,9cyclictest35966irq/58-eth009:25:133
1658999233,15cyclictest0-21swapper/311:10:123
1658999231,13cyclictest35966irq/58-eth010:40:143
1657099231,19cyclictest9643-21ssh11:20:131
1657099231,16cyclictest20140-21ssh12:20:121
1656199239,9cyclictest121ksoftirqd/009:30:120
1656199234,16cyclictest151rcuc/010:40:140
16561992316,5cyclictest0-21swapper/010:25:130
16561992316,4cyclictest131rcu_preempt12:15:120
16561992315,4cyclictest0-21swapper/007:20:130
16561992314,6cyclictest30049-21timerwakeupswit10:35:310
16561992314,6cyclictest30049-21timerwakeupswit10:35:300
1656199231,15cyclictest13518-21sh10:10:130
16561992311,3cyclictest131rcu_preempt12:35:230
16570992215,2cyclictest0-21swapper/110:50:111
1656199221,19cyclictest22599-21ls07:50:010
1656199221,16cyclictest19129-21ls08:50:240
1656199221,13cyclictest844-21munin-node09:05:140
16570992111,8cyclictest6371-21ssh09:30:131
16570992111,7cyclictest614-21dbus-daemon10:10:491
1657099211,14cyclictest6070-21ssh11:00:141
1656199214,8cyclictest1172-21ssh11:50:220
1656199211,18cyclictest1797-21ls12:10:020
1656199211,16cyclictest20495-21apt-get10:50:130
1658999209,9cyclictest0-21swapper/309:51:583
1658999204,10cyclictest0-21swapper/309:40:133
1658999204,10cyclictest0-21swapper/309:40:123
1658999204,10cyclictest0-21swapper/309:20:143
16589992013,5cyclictest0-21swapper/310:50:203
1658999201,14cyclictest0-21swapper/312:30:253
1658999201,14cyclictest0-21swapper/312:30:243
16589992010,8cyclictest0-21swapper/310:33:263
16589992010,8cyclictest0-21swapper/310:33:253
1657099204,10cyclictest630-21Async09:36:311
1657099202,7cyclictest24398-21cpu09:40:141
1657099202,7cyclictest24398-21cpu09:40:141
1657099202,15cyclictest30994-21kthreadcore10:55:221
1657099202,12cyclictest30977-21ssh11:50:151
1657099201,8cyclictest26143-21tr11:10:201
16570992011,6cyclictest28023-21/usr/sbin/munin10:00:121
1657099201,16cyclictest12974-21apt-get11:40:121
16570992011,5cyclictest27501-21kthreadcore12:05:221
1657099201,14cyclictest14971-21ssh12:35:141
1657099201,13cyclictest1937-21ls07:25:221
1657099200,16cyclictest21049-21ls10:30:281
1657099200,16cyclictest21049-21ls10:30:271
1656199209,3cyclictest12-21ksoftirqd/009:16:210
1656199204,4cyclictest12-21ksoftirqd/011:20:200
1656199203,14cyclictest26219-21latency_hist07:20:030
1656199203,12cyclictest19003-21ssh09:55:110
16561992013,4cyclictest11865-21if_err_eth010:45:160
16561992011,6cyclictest0-21swapper/007:35:300
1656199201,15cyclictest20673-21apt-get08:55:140
1656199201,15cyclictest20673-21apt-get08:55:130
1658999199,7cyclictest0-21swapper/309:18:383
1658999199,6cyclictest0-21swapper/310:35:073
1658999199,6cyclictest0-21swapper/310:35:063
1658999198,5cyclictest0-21swapper/312:06:473
1658999192,12cyclictest0-21swapper/310:45:133
1658999192,10cyclictest35966irq/58-eth010:20:133
16589991911,5cyclictest35966irq/58-eth012:00:133
1658999191,12cyclictest0-21swapper/307:55:233
16589991910,6cyclictest0-21swapper/311:52:573
16589991910,6cyclictest0-21swapper/309:57:323
16589991910,4cyclictest0-21swapper/311:01:533
1657099198,8cyclictest18418-21sed11:05:231
1657099194,12cyclictest24456-21awk09:20:251
1657099192,9cyclictest2162-21ssh12:10:131
1657099191,9cyclictest26428-21kthreadcore08:25:231
16570991911,6cyclictest7733-21cron10:25:011
16570991911,6cyclictest24158-21sshd09:55:481
16570991911,6cyclictest1101-21sshd08:31:461
16570991911,5cyclictest6256-21ssh09:10:241
1656199199,8cyclictest12-21ksoftirqd/010:15:120
1656199199,7cyclictest9101-21kthreadcore08:40:220
1656199199,7cyclictest4899-21df_inode08:05:130
1656199199,6cyclictest16551-21cyclictest12:20:120
1656199198,3cyclictest131rcu_preempt10:30:210
1656199198,3cyclictest131rcu_preempt10:30:200
1656199197,9cyclictest4595-21apt-get10:05:130
1656199195,4cyclictest131rcu_preempt09:35:260
1656199192,12cyclictest4015-21ssh11:35:150
1656199191,9cyclictest9676-21cpuspeed08:10:140
1656199191,9cyclictest14613-21fschecks_time08:15:160
1656199191,15cyclictest28645-21apt-get08:30:010
1656199191,14cyclictest28906-21cut11:30:200
1656199191,14cyclictest13644-21ntp_192.168.11409:50:250
1656199191,10cyclictest3794-21interrupts07:30:150
1658999189,6cyclictest0-21swapper/311:16:053
1658999188,8cyclictest33-21ksoftirqd/311:25:133
1658999182,10cyclictest0-21swapper/309:05:143
16589991811,5cyclictest35966irq/58-eth009:35:153
1658999181,13cyclictest0-21swapper/308:40:173
1658999181,12cyclictest0-21swapper/307:25:293
1657099187,8cyclictest27491-21sh11:30:141
1657099183,10cyclictest0-21swapper/107:45:121
16570991811,5cyclictest0-21swapper/107:20:171
1657099181,14cyclictest4557-21ls09:45:231
1657099181,12cyclictest1532-21uniq08:35:161
1656199181,7cyclictest21354-21latency_hist07:15:010
1656199181,10cyclictest12991-21ssh11:40:120
16561991810,3cyclictest12-21ksoftirqd/007:25:150
1658999179,6cyclictest0-21swapper/311:36:023
1658999179,5cyclictest35966irq/58-eth012:15:123
1658999172,10cyclictest0-21swapper/307:40:143
16589991712,3cyclictest33-21ksoftirqd/308:30:293
16589991711,4cyclictest0-21swapper/312:20:283
16589991711,4cyclictest0-21swapper/310:05:343
16589991711,3cyclictest0-21swapper/310:25:243
1658999171,13cyclictest0-21swapper/308:05:133
1657099178,3cyclictest2382-21diskmemload11:48:441
1657099175,9cyclictest30452-21fschecks_time09:05:171
1657099174,10cyclictest0-21swapper/109:00:121
1657099172,12cyclictest9590-21latency_hist08:10:021
1657099172,12cyclictest459-21grep12:25:251
1657099172,11cyclictest0-21swapper/107:10:141
1657099172,10cyclictest0-21swapper/110:05:151
1657099172,10cyclictest0-21swapper/108:10:131
1657099172,10cyclictest0-21swapper/107:35:141
1657099171,8cyclictest11885-21kthreadcore08:45:201
16570991711,4cyclictest0-21swapper/108:20:151
1657099171,13cyclictest24025-21kthreadcore07:15:211
1657099170,14cyclictest22762-21find07:50:111
1656199179,5cyclictest4948-21mailstats08:35:260
1656199177,6cyclictest15791-21kthreadcore07:40:200
1656199172,10cyclictest0-21swapper/008:00:130
1656199171,5cyclictest2855-21irqstats12:10:180
1656199171,10cyclictest25795-21irqcore09:00:180
1656199170,11cyclictest16634-21cpuspeed12:00:140
1658999162,9cyclictest0-21swapper/307:20:133
1658999161,9cyclictest0-21swapper/307:30:223
16589991612,2cyclictest33-21ksoftirqd/308:55:313
16589991612,2cyclictest33-21ksoftirqd/308:55:303
16589991611,3cyclictest33-21ksoftirqd/307:35:163
1657099168,5cyclictest569-21irqrtprio08:00:161
1657099164,9cyclictest18048-21kthreadcore08:50:221
1657099161,13cyclictest14412-21/usr/sbin/munin08:15:191
1657099161,12cyclictest930-21snmpd08:55:151
1657099161,12cyclictest930-21snmpd08:55:141
1656199161,12cyclictest14290-21cat08:45:220
1658999152,7cyclictest0-21swapper/308:15:123
1658999151,11cyclictest0-21swapper/308:10:303
1657099151,5cyclictest13246-21/usr/sbin/munin07:40:161
1657099151,11cyclictest16595-21latency_hist07:10:011
16561991510,3cyclictest1-21systemd07:55:120
1658999149,3cyclictest33-21ksoftirqd/307:11:243
1658999142,8cyclictest35966irq/58-eth008:25:153
1658999141,7cyclictest0-21swapper/308:55:013
1658999138,3cyclictest33-21ksoftirqd/308:22:293
1658999133,6cyclictest0-21swapper/307:15:133
1658999132,8cyclictest0-21swapper/308:00:153
1658999131,9cyclictest0-21swapper/308:35:143
1656199133,8cyclictest16551-21cyclictest07:10:020
1658999128,3cyclictest33-21ksoftirqd/307:45:213
1658999127,3cyclictest33-21ksoftirqd/307:50:123
1658999121,9cyclictest0-21swapper/308:45:213
1658999121,8cyclictest0-21swapper/309:00:283
1658999111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional