You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-13 - 18:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Thu Aug 13, 2026 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1940499743,56cyclictest1263-21ssh12:25:121
1941399644,54cyclictest17414-21ssh11:40:132
1939399575,28cyclictest13840-21ssh11:20:130
1939399575,28cyclictest13840-21ssh11:20:120
1941399504,32cyclictest27-21ksoftirqd/209:10:122
1940499492,34cyclictest32718-21ssh09:25:121
1939399492,5cyclictest131rcu_preempt07:15:120
1939399477,8cyclictest131rcu_preempt09:30:100
1939399465,7cyclictest131rcu_preempt09:50:110
1939399465,5cyclictest131rcu_preempt11:05:110
1939399463,37cyclictest20764-21ssh10:30:140
1941399451,26cyclictest6389-21ssh10:40:132
1940499452,30cyclictest5534-21ssh09:10:141
1939399452,31cyclictest4798-21sh11:15:120
1939399451,35cyclictest29780-21ssh10:35:120
1940499432,32cyclictest28203-21ssh11:10:121
1939399431,34cyclictest31815-21ssh11:30:120
1941399422,20cyclictest3041-21ssh11:50:132
19424994121,13cyclictest35966irq/58-eth010:00:123
1940499411,30cyclictest6648-21sh12:10:131
1939399412,32cyclictest10199-21ssh11:00:150
1939399402,30cyclictest22060-21ssh09:55:110
1939399402,30cyclictest22060-21ssh09:55:110
19424993817,10cyclictest35966irq/58-eth012:35:143
19424993817,10cyclictest35966irq/58-eth012:35:133
1941399372,18cyclictest23329-21ssh09:35:422
1941399371,26cyclictest16958-21ssh10:10:142
1941399362,28cyclictest9988-21ssh09:46:522
1941399362,18cyclictest25882-21ssh10:30:502
1942499357,18cyclictest35966irq/58-eth009:25:133
1941399355,20cyclictest4792-21ssh11:15:112
1940499353,25cyclictest2639-21ssh10:20:141
1940499353,25cyclictest2639-21ssh10:20:131
1941399342,17cyclictest131rcu_preempt10:36:512
1939399343,19cyclictest11657-21ssh10:25:130
1939399341,28cyclictest5127-21diskmemload12:05:130
19424993316,12cyclictest35966irq/58-eth009:15:253
1942499331,18cyclictest35966irq/58-eth011:00:143
1941399333,24cyclictest0-21swapper/212:28:512
1942499321,19cyclictest35966irq/58-eth011:40:133
19413993215,11cyclictest19379-21cyclictest07:10:122
19413993117,6cyclictest32727-21ssh09:25:142
19404993113,11cyclictest20747-21apt-get10:30:141
1940499311,20cyclictest16954-21ssh10:10:141
1939399312,22cyclictest8902-21sh12:28:500
1942499301,16cyclictest35966irq/58-eth007:10:113
1941399303,19cyclictest27346irq/40-dwc2_hso09:50:122
1941399302,22cyclictest32484-21ssh09:40:502
19404993013,12cyclictest13009-21apt-get09:50:121
1939399302,22cyclictest23251-21ssh12:18:500
19424992912,11cyclictest35966irq/58-eth009:40:283
1941399294,18cyclictest27346irq/40-dwc2_hso11:35:132
1941399293,21cyclictest0-21swapper/210:18:502
1941399292,20cyclictest0-21swapper/208:35:132
1941399292,20cyclictest0-21swapper/208:05:112
1940499292,21cyclictest8412-21ssh11:35:131
1939399293,21cyclictest24689-21ssh12:20:140
1942499286,11cyclictest35966irq/58-eth010:30:153
1941399283,21cyclictest0-21swapper/212:36:212
1941399283,21cyclictest0-21swapper/212:36:202
1941399281,15cyclictest10281-21ssh12:30:122
1940499284,16cyclictest18311-21ssh09:35:121
1940499282,20cyclictest6405-21ssh10:40:141
19404992818,7cyclictest0-21swapper/111:55:131
19404992811,12cyclictest31226-21ssh10:00:131
1939399282,17cyclictest6666-21ssh12:10:140
19393992815,7cyclictest536-21usb-storage11:45:110
19424992717,6cyclictest0-21swapper/311:15:123
1941399274,15cyclictest27346irq/40-dwc2_hso07:15:122
1941399273,16cyclictest27346irq/40-dwc2_hso07:35:132
1941399273,16cyclictest261rcuc/207:40:132
1941399273,15cyclictest27346irq/40-dwc2_hso10:45:132
19404992721,4cyclictest201rcuc/112:23:171
19404992711,10cyclictest14573-21apt-get09:15:121
1939399277,15cyclictest630-21Async09:36:310
1942499263,15cyclictest0-21swapper/309:46:513
1942499263,15cyclictest0-21swapper/309:35:413
1942499261,14cyclictest35966irq/58-eth011:20:123
1942499261,14cyclictest35966irq/58-eth011:20:113
1941399262,9cyclictest17830-21ssh11:03:502
1941399262,9cyclictest16108-21ssh09:32:492
1941399261,14cyclictest22883-21sh11:25:152
19404992619,4cyclictest13773-21apt-get08:45:111
19404992617,5cyclictest30032-21apt-get12:05:121
1940499261,5cyclictest21-21ksoftirqd/110:55:241
1939399266,11cyclictest23665-21ssh09:20:120
1939399262,18cyclictest7834-21rm10:05:120
1939399261,9cyclictest167-21systemd-network10:43:510
1939399261,19cyclictest21104-21ssh12:00:140
1941399254,15cyclictest27346irq/40-dwc2_hso11:30:122
1941399253,17cyclictest27346irq/40-dwc2_hso12:20:132
1941399253,17cyclictest0-21swapper/212:04:502
1941399253,15cyclictest27346irq/40-dwc2_hso07:45:122
1941399252,8cyclictest9436-21ssh10:22:502
1941399252,8cyclictest9436-21ssh10:22:502
1941399252,8cyclictest16392-21ssh10:09:502
1941399251,8cyclictest7439-21kthreadcore12:10:202
19413992512,9cyclictest5127-21diskmemload11:45:122
1941399251,15cyclictest15782-21rm11:55:272
19413992510,8cyclictest17549-21ssh11:20:262
19413992510,8cyclictest17549-21ssh11:20:262
1940499253,16cyclictest24625-21ssh10:50:141
19404992521,2cyclictest17406-21apt-get11:40:151
1940499251,7cyclictest11962-21grep10:05:301
1942499244,13cyclictest35966irq/58-eth010:40:143
1942499242,16cyclictest0-21swapper/311:45:103
1942499242,15cyclictest0-21swapper/312:05:143
19424992411,8cyclictest35966irq/58-eth012:15:153
1941399242,17cyclictest0-21swapper/210:25:492
1941399241,17cyclictest30055-21ssh12:05:132
1941399241,13cyclictest2914-21switchtime08:30:282
19413992410,6cyclictest30708-21irqcore07:55:162
1940499245,3cyclictest131rcu_preempt09:30:541
1940499244,6cyclictest21-21ksoftirqd/111:34:461
1940499243,4cyclictest21-21ksoftirqd/109:55:311
1940499243,4cyclictest21-21ksoftirqd/109:55:301
19404992418,3cyclictest19284-21apt-get12:35:141
19404992418,3cyclictest19284-21apt-get12:35:141
1940499241,7cyclictest21-21ksoftirqd/111:07:031
1939399242,19cyclictest22884-21ssh11:25:150
1939399241,19cyclictest1187-21ssh10:55:140
1939399241,17cyclictest10269-21apt-get12:30:120
1942499233,16cyclictest33-21ksoftirqd/311:25:123
1942499232,15cyclictest35966irq/58-eth008:40:113
1942499232,14cyclictest0-21swapper/310:15:143
1941399233,16cyclictest0-21swapper/210:04:512
1940499236,8cyclictest21-21ksoftirqd/111:50:511
1940499235,3cyclictest131rcu_preempt09:47:491
1940499233,4cyclictest21-21ksoftirqd/112:15:251
1940499233,4cyclictest21-21ksoftirqd/111:25:251
19404992316,4cyclictest7619-21apt-get08:05:101
19404992315,4cyclictest506-21apt-get09:05:111
19404992315,4cyclictest0-21swapper/108:30:121
1940499231,4cyclictest21-21ksoftirqd/111:15:201
19404992314,4cyclictest2784-21apt-get08:00:141
19404992310,6cyclictest131rcu_preempt12:02:531
1939399231,17cyclictest2989-21ls08:30:290
1939399231,14cyclictest15557-21ssh10:45:130
19393992310,9cyclictest630-21Async09:36:310
19424992212,7cyclictest35966irq/58-eth010:20:123
19424992212,7cyclictest35966irq/58-eth010:20:123
1941399229,11cyclictest22204-21fschecks_count09:55:162
1941399229,11cyclictest22204-21fschecks_count09:55:152
1941399222,17cyclictest19516-21wc12:15:302
1941399221,9cyclictest2369366ssh09:20:132
19413992211,9cyclictest8025-21ssh10:57:542
19413992211,5cyclictest25764-21ssh11:07:342
1940499226,9cyclictest21-21ksoftirqd/111:46:321
1940499226,9cyclictest21-21ksoftirqd/110:15:531
1940499226,5cyclictest21-21ksoftirqd/110:27:571
1940499225,4cyclictest21-21ksoftirqd/111:03:391
1940499223,4cyclictest21-21ksoftirqd/111:20:111
1940499223,4cyclictest21-21ksoftirqd/111:20:101
1940499223,4cyclictest21-21ksoftirqd/109:00:231
1940499223,4cyclictest21-21ksoftirqd/109:00:231
1940499223,11cyclictest0-21swapper/107:25:121
1940499222,14cyclictest10279-21sh12:30:121
19404992217,3cyclictest201rcuc/110:46:401
1940499221,16cyclictest29762-21diskmemload10:35:111
1939399221,17cyclictest2912-21ls10:20:170
1939399221,17cyclictest2912-21ls10:20:160
1942499219,7cyclictest0-21swapper/311:10:133
1942499213,12cyclictest0-21swapper/311:50:143
1942499211,2cyclictest0-21swapper/312:00:133
1941399217,11cyclictest14096-21irqcore08:45:162
1941399213,9cyclictest23241-21latency_hist08:55:012
1941399213,9cyclictest11572-21kthreadcore08:40:212
1941399212,9cyclictest29970-21kthreadcore07:20:192
1941399211,16cyclictest25584-21apt-get07:50:132
19413992110,8cyclictest0-21swapper/209:15:132
1940499219,8cyclictest19532-21apt-get07:10:121
1940499212,13cyclictest22750-21kworker/u8:109:20:141
19404992115,4cyclictest201rcuc/109:43:491
19404992111,6cyclictest0-21swapper/108:20:131
19393992113,5cyclictest0-21swapper/008:20:170
19393992113,2cyclictest0-21swapper/007:55:120
1939399211,16cyclictest17064-21latency_hist08:15:010
1939399211,14cyclictest18327-21ssh09:35:130
1939399211,11cyclictest0-21swapper/009:10:540
1942499209,9cyclictest0-21swapper/310:49:573
1942499204,9cyclictest0-21swapper/311:30:133
1941399209,4cyclictest25336-21kthreadcore08:55:202
1941399206,11cyclictest27397-21kthreadcore10:50:222
1941399205,6cyclictest17615-21irqcore08:15:182
1941399205,12cyclictest27346irq/40-dwc2_hso07:30:132
1941399202,14cyclictest27346irq/40-dwc2_hso08:00:142
1941399201,10cyclictest4061-21grep09:05:252
1941399201,10cyclictest26917-21apt-get08:25:122
1941399201,10cyclictest22292-21fschecks_time08:20:162
1941399201,10cyclictest13564-21kthreadcore08:10:192
1940499207,3cyclictest131rcu_preempt07:20:311
1940499205,9cyclictest0-21swapper/108:50:221
1940499203,4cyclictest21-21ksoftirqd/108:10:141
1939399205,9cyclictest0-21swapper/009:15:130
1939399203,3cyclictest131rcu_preempt11:35:320
1939399201,7cyclictest29314-21kthreadcore11:10:200
1939399201,14cyclictest12051-21ssh11:55:140
1939399200,17cyclictest22884-21ls07:10:210
1942499199,8cyclictest0-21swapper/309:50:353
1942499199,6cyclictest0-21swapper/310:10:233
1942499193,11cyclictest0-21swapper/310:27:503
1942499193,11cyclictest0-21swapper/310:05:463
1942499191,14cyclictest0-21swapper/312:25:283
1942499191,12cyclictest35966irq/58-eth010:35:123
1942499191,12cyclictest0-21swapper/308:45:113
1941399191,9cyclictest2422-21kthreadcore07:25:192
1941399191,15cyclictest28289-21fschecks_count09:00:162
1941399191,15cyclictest28289-21fschecks_count09:00:162
1941399191,15cyclictest28209-21cpu11:10:132
1940499196,3cyclictest21-21ksoftirqd/107:40:161
1940499193,11cyclictest0-21swapper/108:40:131
1940499192,4cyclictest21-21ksoftirqd/108:15:161
1940499192,4cyclictest21-21ksoftirqd/107:30:171
19404991914,3cyclictest30375-21apt-get07:55:121
19404991914,3cyclictest0-21swapper/107:45:111
19404991913,4cyclictest24339-21apt-get07:15:121
1939399198,9cyclictest19549-21sed12:35:170
1939399198,9cyclictest19549-21sed12:35:170
1939399195,11cyclictest26172-21grep10:15:160
1939399192,13cyclictest7803-21uniq08:05:150
1939399191,9cyclictest17440-21cut10:10:200
1939399191,7cyclictest5127-21diskmemload09:48:300
1939399191,15cyclictest29324-21ls07:50:290
1939399191,13cyclictest24614-21ssh10:50:130
1942499188,6cyclictest36066irq/59-eth012:14:503
1942499186,8cyclictest35966irq/58-eth009:30:133
1942499182,11cyclictest35966irq/58-eth011:05:123
1942499182,10cyclictest0-21swapper/308:05:113
1942499182,10cyclictest0-21swapper/307:45:133
1942499181,14cyclictest35966irq/58-eth009:10:123
1942499181,14cyclictest0-21swapper/312:25:023
1942499181,14cyclictest0-21swapper/309:55:173
1942499181,14cyclictest0-21swapper/309:55:173
1942499181,12cyclictest0-21swapper/308:15:283
19424991810,6cyclictest0-21swapper/311:58:563
19424991810,4cyclictest0-21swapper/310:52:563
19404991811,4cyclictest11143-21apt-get07:35:111
1939399188,7cyclictest30932-21missed_timers09:40:260
1939399181,9cyclictest18527-21acpi08:50:100
1939399181,7cyclictest18634-21kthreadcore11:40:200
19393991813,3cyclictest0-21swapper/007:30:130
1939399181,14cyclictest4567-21irqcore08:35:170
1939399181,12cyclictest7391-21munin-run08:05:010
1942499179,6cyclictest0-21swapper/310:58:183
1942499179,6cyclictest0-21swapper/309:21:033
1942499179,5cyclictest0-21swapper/312:30:183
1942499175,9cyclictest0-21swapper/311:35:283
1942499172,10cyclictest0-21swapper/308:10:143
1942499171,10cyclictest0-21swapper/309:00:183
1942499171,10cyclictest0-21swapper/309:00:173
1940499172,3cyclictest0-21swapper/108:55:251
1940499171,3cyclictest131rcu_preempt08:25:261
1939399178,7cyclictest17500-21fw_packets08:15:150
1939399173,11cyclictest23807-21sed08:55:170
1939399172,11cyclictest5125-21latency_hist09:10:020
1939399171,7cyclictest16952-21kthreadcore07:40:200
1939399171,14cyclictest14003-21ls08:45:140
1939399171,13cyclictest32683-21/usr/sbin/munin09:25:170
1939399171,13cyclictest30058-21kthreadcore09:00:210
1939399171,13cyclictest30058-21kthreadcore09:00:200
1939399171,13cyclictest2111-21irqrtprio07:25:190
1942499161,12cyclictest0-21swapper/307:55:213
1941399161,8cyclictest19379-21cyclictest07:10:012
1940499162,9cyclictest0-21swapper/108:35:141
1940499162,10cyclictest0-21swapper/107:50:151
1940499161,6cyclictest19408-21sed07:10:011
1939399164,9cyclictest20787-21apt-get07:45:130
1939399161,13cyclictest8969-21/usr/sbin/munin08:40:170
1939399161,12cyclictest753-21sendmail-mta08:27:510
1939399161,12cyclictest2999-21/usr/sbin/munin11:50:140
1939399161,10cyclictest29197-21sort07:20:140
1942499152,7cyclictest0-21swapper/307:20:123
1942499151,9cyclictest0-21swapper/308:20:143
1942499151,12cyclictest0-21swapper/307:30:013
1942499151,11cyclictest0-21swapper/307:15:173
1942499151,10cyclictest0-21swapper/309:05:123
1942499151,10cyclictest0-21swapper/308:55:173
1942499149,3cyclictest0-21swapper/308:02:433
1942499149,3cyclictest0-21swapper/307:53:203
1942499149,3cyclictest0-21swapper/307:40:263
1942499141,8cyclictest35966irq/58-eth008:50:113
1942499141,7cyclictest0-21swapper/307:30:123
1942499141,10cyclictest0-21swapper/308:35:183
1942499131,8cyclictest35966irq/58-eth008:25:123
1942499127,3cyclictest0-21swapper/308:32:173
1942499121,8cyclictest35966irq/58-eth007:35:113
1939399120,9cyclictest19414-21latency_hist07:10:010
1942499101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional