You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-25 - 23:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Tue Aug 25, 2026 12:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
878099781,32cyclictest19226-21ssh10:35:142
8762996038,11cyclictest19485-21ssh12:05:150
878099521,33cyclictest28441-21ssh12:10:122
878099512,37cyclictest0-21swapper/209:30:162
877099502,40cyclictest19489-21ssh12:05:151
8780994719,21cyclictest27346irq/40-dwc2_hso10:20:102
8780994719,21cyclictest27346irq/40-dwc2_hso10:20:102
8780994527,11cyclictest261rcuc/210:30:102
876299453,36cyclictest26123-21ssh09:45:120
876299450,37cyclictest11776-21ssh09:55:120
877099441,33cyclictest30125-21ssh11:35:131
877099435,5cyclictest131rcu_preempt12:00:131
877099432,5cyclictest131rcu_preempt10:25:141
8762994332,4cyclictest29864-21apt-get10:05:130
878099423,24cyclictest261rcuc/211:10:142
877099427,2cyclictest131rcu_preempt10:35:111
877099412,8cyclictest12179-21ssh11:25:131
8762994125,10cyclictest28397-21apt-get10:40:140
877099402,23cyclictest8600-21ssh11:05:131
878099391,30cyclictest0-21swapper/211:25:152
877099391,3cyclictest131rcu_preempt11:30:121
876299391,28cyclictest0-21swapper/008:35:120
8780993819,14cyclictest0-21swapper/212:25:152
878099381,25cyclictest0-21swapper/210:10:142
877099381,18cyclictest4186-21cut09:15:161
879499372,20cyclictest35966irq/58-eth011:35:133
877099372,12cyclictest6446-21ssh10:10:141
877099361,30cyclictest28456-21ssh12:10:141
877099360,14cyclictest10182-21ls11:40:241
876299362,22cyclictest3097-21sh11:20:140
877099352,28cyclictest4935-21ssh10:45:121
877099352,28cyclictest4935-21ssh10:45:111
877099352,26cyclictest3082-21sh11:20:131
877099351,31cyclictest23178-21ssh12:25:121
877099351,30cyclictest13182-21ssh09:20:121
878099342,26cyclictest32003-21ssh11:00:152
8780993414,13cyclictest27-21ksoftirqd/209:35:132
877099341,26cyclictest26928-21diskmemload10:20:121
877099341,26cyclictest26928-21diskmemload10:20:111
8762993429,3cyclictest10109-21apt-get10:30:110
879499331,8cyclictest0-21swapper/310:35:133
878099334,12cyclictest0-21swapper/212:20:162
877099331,28cyclictest28408-21ssh10:40:141
8762993325,3cyclictest1325-21apt-get11:55:140
8762993324,4cyclictest7890-21apt-get08:50:120
876299332,23cyclictest5069-21ssh12:15:140
876299332,23cyclictest5069-21ssh12:15:130
8780993210,15cyclictest0-21swapper/211:20:132
877099322,26cyclictest1352-21ssh11:55:131
8762993225,3cyclictest17573-21apt-get11:10:130
8762993214,11cyclictest1177-21apt-get10:25:150
879499312,9cyclictest0-21swapper/309:10:133
878099314,17cyclictest27346irq/40-dwc2_hso09:25:142
8770993119,9cyclictest21-21ksoftirqd/109:30:131
877099311,10cyclictest26928-21diskmemload12:24:001
8762993124,3cyclictest13715-21apt-get07:15:120
8762993124,3cyclictest13715-21apt-get07:15:120
8762993123,3cyclictest8833-21apt-get12:35:130
878099303,15cyclictest27346irq/40-dwc2_hso07:30:132
877099302,21cyclictest22229-21ssh09:25:141
876299308,18cyclictest0-21swapper/008:15:130
8762993024,4cyclictest23166-21apt-get12:25:140
8762993022,5cyclictest11406-21apt-get08:20:100
8762993021,5cyclictest31979-21apt-get11:00:150
8762993015,12cyclictest0-21swapper/007:25:130
876299301,15cyclictest24707-21ssh11:50:130
876299301,15cyclictest24707-21ssh11:50:130
879499291,9cyclictest35966irq/58-eth012:10:133
878099293,22cyclictest26138-21sh09:45:142
878099292,11cyclictest16513-21ssh12:02:002
878099282,20cyclictest0-21swapper/211:55:142
876299289,15cyclictest0-21swapper/007:35:110
8762992822,3cyclictest4923-21apt-get10:45:130
8762992822,3cyclictest4923-21apt-get10:45:120
8762992822,2cyclictest6419-21apt-get10:10:140
8762992818,8cyclictest12167-21apt-get11:25:140
8762992816,9cyclictest17117-21ssh09:40:150
8762992816,9cyclictest17117-21ssh09:40:150
8762992814,11cyclictest0-21swapper/008:30:140
8794992722,2cyclictest0-21swapper/307:55:133
8794992722,2cyclictest0-21swapper/307:55:133
877099271,23cyclictest10137-21ssh10:30:131
877099271,20cyclictest31625-21ssh11:53:001
877099271,20cyclictest31625-21ssh11:52:591
876299273,8cyclictest7711-21sed08:50:010
876299272,18cyclictest2741-21ssh09:50:150
8762992720,5cyclictest10386-21apt-get12:00:130
8762992720,4cyclictest31311-21apt-get09:30:140
8762992719,6cyclictest19799-21apt-get07:55:140
8762992719,6cyclictest19799-21apt-get07:55:130
8762992713,9cyclictest0-21swapper/010:15:120
878099263,14cyclictest27346irq/40-dwc2_hso08:50:122
876299264,7cyclictest31492-21ssh11:15:460
876299263,10cyclictest1928-21fschecks_count08:10:150
876299263,10cyclictest14245-21df12:20:150
876299262,19cyclictest21180-21rm11:30:120
876299262,10cyclictest5333-21ssh11:38:520
876299261,8cyclictest0-21swapper/007:30:230
8762992616,5cyclictest8002-21apt-get09:35:120
8762992613,8cyclictest13170-21apt-get09:20:120
879499259,3cyclictest0-21swapper/311:11:443
879499258,6cyclictest35966irq/58-eth009:50:153
879499258,14cyclictest35966irq/58-eth010:30:113
8794992510,10cyclictest35966irq/58-eth010:15:123
878099252,14cyclictest27346irq/40-dwc2_hso07:55:142
878099252,14cyclictest27346irq/40-dwc2_hso07:55:132
878099251,8cyclictest27330-21ssh09:10:142
878099251,21cyclictest9145-21idleruntime12:35:172
878099251,18cyclictest28423-21ssh10:40:142
877099252,15cyclictest630-21Async09:36:311
876299253,8cyclictest24339-21latency_hist11:50:010
876299252,11cyclictest1913-21ssh09:13:020
876299252,10cyclictest28280-21ssh10:56:050
876299251,8cyclictest8573-21apt-get11:05:130
8762992517,4cyclictest24543-21apt-get10:20:120
8762992517,4cyclictest24543-21apt-get10:20:110
879499246,6cyclictest0-21swapper/309:30:083
879499245,4cyclictest35966irq/58-eth011:05:213
879499244,5cyclictest0-21swapper/309:35:283
879499244,14cyclictest36066irq/59-eth012:00:133
879499242,19cyclictest35966irq/58-eth011:30:113
879499241,8cyclictest0-21swapper/311:15:283
879499241,7cyclictest0-21swapper/310:20:273
879499241,7cyclictest0-21swapper/310:20:273
879499241,7cyclictest0-21swapper/310:05:303
879499241,7cyclictest0-21swapper/309:48:303
879499241,6cyclictest0-21swapper/309:25:243
8794992414,7cyclictest35966irq/58-eth010:40:143
8794992410,9cyclictest0-21swapper/311:49:033
878099242,13cyclictest9896-21users12:15:312
878099242,13cyclictest9896-21users12:15:302
878099241,13cyclictest8570-21ssh11:05:112
877099241,8cyclictest28515-21irqcore07:30:171
877099241,17cyclictest844-21munin-node07:15:141
877099241,17cyclictest844-21munin-node07:15:131
877099241,14cyclictest29870-21sh10:05:131
876299246,5cyclictest0-21swapper/009:10:010
876299242,19cyclictest14023-21ssh10:50:150
876299242,10cyclictest29970-21ssh09:28:540
8762992414,8cyclictest4904-21ssh12:30:470
8762992413,9cyclictest13592-21ntp_192.168.11407:45:250
8762992410,11cyclictest0-21swapper/010:00:130
879499238,4cyclictest0-21swapper/310:12:313
879499232,17cyclictest35966irq/58-eth012:30:153
879499231,8cyclictest0-21swapper/312:05:223
879499231,8cyclictest0-21swapper/310:55:233
879499231,6cyclictest36066irq/59-eth011:20:223
879499231,6cyclictest0-21swapper/312:36:463
879499231,6cyclictest0-21swapper/312:25:153
879499231,6cyclictest0-21swapper/311:55:163
879499231,6cyclictest0-21swapper/311:40:303
879499231,6cyclictest0-21swapper/311:00:243
879499231,6cyclictest0-21swapper/310:45:153
879499231,6cyclictest0-21swapper/310:45:143
879499231,6cyclictest0-21swapper/309:55:183
879499231,6cyclictest0-21swapper/309:40:323
879499231,6cyclictest0-21swapper/309:40:313
879499231,6cyclictest0-21swapper/309:20:183
879499231,6cyclictest0-21swapper/307:25:273
879499231,6cyclictest0-21swapper/307:15:153
879499231,6cyclictest0-21swapper/307:15:143
8794992315,5cyclictest35966irq/58-eth012:20:143
879499231,4cyclictest0-21swapper/311:25:173
878099239,11cyclictest0-21swapper/209:10:012
878099236,16cyclictest630-21Async09:36:312
878099236,16cyclictest630-21Async09:36:312
878099236,13cyclictest21179-21diskmemload11:30:122
878099235,7cyclictest20850-21sh10:00:142
878099234,13cyclictest27346irq/40-dwc2_hso08:05:142
878099234,13cyclictest0-21swapper/208:40:112
878099232,16cyclictest0-21swapper/212:09:552
8780992312,9cyclictest18102-21ssh09:20:392
878099231,19cyclictest2929-21fschecks_time09:50:162
8780992311,5cyclictest28395-21ssh10:56:132
878099231,13cyclictest930-21snmpd10:55:012
878099231,13cyclictest15476-21ssh10:15:142
878099231,11cyclictest5119-21latency_hist07:40:012
878099230,19cyclictest16452-21ls08:25:182
877099236,14cyclictest21-21ksoftirqd/109:00:131
877099234,16cyclictest8020-21ssh09:35:131
876299239,11cyclictest30793-21tr08:05:200
876299232,14cyclictest6745-21sh11:40:140
879499229,4cyclictest0-21swapper/310:27:263
879499222,17cyclictest35966irq/58-eth007:40:143
879499221,7cyclictest0-21swapper/310:50:153
879499221,7cyclictest0-21swapper/309:15:083
879499221,6cyclictest0-21swapper/312:15:193
879499221,6cyclictest0-21swapper/312:15:183
879499221,6cyclictest0-21swapper/311:50:213
879499221,6cyclictest0-21swapper/311:50:203
879499221,6cyclictest0-21swapper/309:00:203
879499221,6cyclictest0-21swapper/308:55:163
879499221,6cyclictest0-21swapper/308:55:023
879499221,6cyclictest0-21swapper/308:40:283
879499221,6cyclictest0-21swapper/308:35:203
879499221,6cyclictest0-21swapper/308:25:193
879499221,6cyclictest0-21swapper/308:05:183
879499221,6cyclictest0-21swapper/307:45:273
879499221,6cyclictest0-21swapper/307:35:203
879499221,6cyclictest0-21swapper/307:30:193
878099229,10cyclictest12936-21kthreadcore09:55:192
878099224,15cyclictest27346irq/40-dwc2_hso11:15:132
878099223,15cyclictest27346irq/40-dwc2_hso08:15:142
878099222,6cyclictest0-21swapper/210:25:162
878099222,12cyclictest5083-21grep08:10:222
878099221,8cyclictest0-21swapper/208:35:142
8780992215,4cyclictest17504-21ssh09:40:202
8780992215,4cyclictest17504-21ssh09:40:192
878099221,10cyclictest5217-21hwlatdetect10:45:162
878099221,10cyclictest5217-21hwlatdetect10:45:152
8770992214,5cyclictest13969-21/usr/sbin/munin10:50:171
877099221,14cyclictest844-21munin-node11:00:161
876299229,9cyclictest0-21swapper/008:40:120
8762992216,3cyclictest24627-21apt-get08:00:110
8762992213,6cyclictest0-21swapper/007:10:120
8762992213,6cyclictest0-21swapper/007:10:110
876299221,18cyclictest3999-21apt-get09:15:110
876299221,18cyclictest19433-21cut10:35:160
876299221,16cyclictest17552-21find09:00:130
8762992210,9cyclictest18723-21diskstats07:20:160
879499211,9cyclictest0-21swapper/310:02:543
879499211,6cyclictest0-21swapper/309:05:193
879499211,6cyclictest0-21swapper/308:30:213
879499211,6cyclictest0-21swapper/308:20:213
879499211,6cyclictest0-21swapper/308:10:183
879499211,6cyclictest0-21swapper/308:00:293
879499211,6cyclictest0-21swapper/307:10:193
879499211,6cyclictest0-21swapper/307:10:193
878099215,7cyclictest21354-21kthreadcore07:20:212
878099215,13cyclictest15287-21grep08:20:272
878099215,11cyclictest27346irq/40-dwc2_hso08:45:132
878099214,13cyclictest27346irq/40-dwc2_hso11:45:142
878099212,10cyclictest18846-21kthreadcore09:00:212
878099212,10cyclictest17503-21latency_hist09:00:022
878099211,16cyclictest15349-21sh11:45:012
878099211,11cyclictest4922-21kthreadcore09:15:182
878099211,11cyclictest380-21kthreadcore11:35:222
878099211,11cyclictest18338-21missed_timers07:50:242
877099217,3cyclictest131rcu_preempt10:00:161
877099213,14cyclictest844-21munin-node09:10:131
8770992111,6cyclictest21101-21timerandwakeup09:40:311
8770992111,6cyclictest21101-21timerandwakeup09:40:301
877099211,14cyclictest7802-21ls08:50:021
877099211,10cyclictest19606-21grep10:15:301
876299219,9cyclictest8769-21latency_hist07:10:020
876299214,3cyclictest26928-21diskmemload12:13:290
876299213,13cyclictest664-21ntpd07:50:140
8762992113,4cyclictest5368-21apt-get07:40:160
879499201,6cyclictest0-21swapper/307:50:213
879499201,5cyclictest0-21swapper/308:45:203
878099206,5cyclictest14781-21kthreadcore07:15:192
878099206,5cyclictest14781-21kthreadcore07:15:192
878099201,9cyclictest26181-21kthreadcore08:00:182
878099201,9cyclictest21215-21fschecks_count08:30:182
878099201,15cyclictest23520-21fschecks_count07:25:152
878099201,11cyclictest31986-21ssh10:05:222
878099201,11cyclictest13485-21sh07:15:012
878099201,11cyclictest13485-21sh07:15:002
878099200,10cyclictest2257-21kthreadcore12:30:222
877099206,8cyclictest0-21swapper/111:10:131
877099203,10cyclictest11767-21apt-get09:55:121
877099202,12cyclictest8854-21ssh12:35:141
877099201,7cyclictest5048-21apt12:15:121
877099201,7cyclictest5048-21apt12:15:121
877099200,15cyclictest26457-21find11:15:121
876299209,8cyclictest16309-21/usr/sbin/munin08:25:160
876299205,3cyclictest0-21swapper/008:55:040
878099195,6cyclictest27-21ksoftirqd/207:45:192
878099191,9cyclictest5477-21fschecks_count07:40:182
877099199,7cyclictest1590-21cat08:10:011
877099193,11cyclictest0-21swapper/108:20:131
877099191,9cyclictest7101-21irqrtprio08:15:171
877099191,9cyclictest0-21swapper/109:51:541
877099191,15cyclictest21191-21uniq08:30:191
877099191,14cyclictest16345-21cut08:25:161
877099191,14cyclictest10777-21grep07:10:201
877099191,14cyclictest10777-21grep07:10:191
8770991910,5cyclictest29999-21switchtime09:45:281
877099190,14cyclictest25668-21missed_timers09:05:251
877099187,8cyclictest20364-21cat11:45:301
8770991812,3cyclictest0-21swapper/108:55:031
877099181,14cyclictest3587-21kthreadcore08:10:211
877099181,10cyclictest22999-21ssh10:55:131
8770991810,6cyclictest1-21systemd07:59:161
8770991810,6cyclictest1-21systemd07:59:151
877099180,6cyclictest53646usb-storage07:20:041
877099180,15cyclictest1268-21tr08:40:201
877099180,13cyclictest8513-21ls12:35:011
879499172,9cyclictest0-21swapper/307:20:053
878099171,8cyclictest8788-21ls07:10:012
877099176,9cyclictest24961-21irqcore08:00:161
877099173,10cyclictest8672-21latency_hist07:10:011
877099172,12cyclictest11810-21kthreadcore07:45:211
877099172,10cyclictest0-21swapper/107:35:141
877099171,13cyclictest28065-21ls07:30:021
877099171,13cyclictest17214-21tr07:50:201
877099162,10cyclictest0-21swapper/108:55:141
877099161,11cyclictest30474-21lockfile-create08:40:011
879499111,8cyclictest0-21swapper/308:15:303
879499101,6cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional