You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-07 - 15:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue Apr 07, 2026 12:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27600996428,7cyclictest12-21ksoftirqd/011:25:130
2760099547,5cyclictest131rcu_preempt10:45:130
2761599503,27cyclictest13836-21ssh09:10:132
2760099504,36cyclictest12118-21ssh09:45:130
27606994921,23cyclictest21-21ksoftirqd/109:50:141
2760699491,35cyclictest26944-21ssh11:05:131
2760099496,6cyclictest131rcu_preempt12:10:120
2761599481,28cyclictest19518-21ssh10:25:112
2760699482,39cyclictest21518-21ssh11:20:121
2760099481,39cyclictest8879-21ssh10:55:140
2760699471,35cyclictest3499-21ssh11:10:131
2760699450,32cyclictest28587-21ssh10:30:121
2762499441,22cyclictest28366irq/58-eth012:15:143
2760699441,34cyclictest9079-21rm12:25:131
27606994413,16cyclictest201rcuc/111:40:131
2762499402,24cyclictest28366irq/58-eth012:20:133
2762499402,24cyclictest28366irq/58-eth012:20:133
2760099402,28cyclictest32227-21ssh10:50:110
2762499391,27cyclictest28366irq/58-eth011:30:133
2761599392,35cyclictest12119-21ssh09:45:132
2761599392,20cyclictest9655-21sh10:19:312
2761599373,23cyclictest6840-21ssh10:00:112
27615993710,23cyclictest27-21ksoftirqd/209:25:122
2762499364,18cyclictest28366irq/58-eth011:00:133
27615993626,7cyclictest27-21ksoftirqd/209:00:132
27615993511,10cyclictest0-21swapper/210:45:152
2760099351,6cyclictest131rcu_preempt12:00:130
2761599343,21cyclictest27-21ksoftirqd/209:20:142
2760699333,21cyclictest1673-21ssh11:45:151
2760699333,21cyclictest1673-21ssh11:45:141
2760099331,21cyclictest19810-21ssh11:55:130
2762499322,18cyclictest28366irq/58-eth010:40:133
2762499322,17cyclictest28366irq/58-eth008:55:123
2761599322,20cyclictest9078-21sh12:25:142
27606993217,7cyclictest27588-21cyclictest10:35:141
2760699321,23cyclictest10509-21ssh10:20:141
2762499313,17cyclictest28366irq/58-eth011:35:103
2762499312,17cyclictest28366irq/58-eth010:10:113
2761599314,18cyclictest22346irq/40-dwc2_hso07:50:122
2761599311,17cyclictest1847-21ssh11:25:272
2760699311,3cyclictest0-21swapper/107:15:141
2760099311,23cyclictest26495-21sh09:35:150
27624993016,10cyclictest28366irq/58-eth012:05:253
2761599303,20cyclictest0-21swapper/209:55:392
2760699302,23cyclictest9660-21ssh10:19:311
27600993010,14cyclictest16048-21ssh11:35:110
2761599294,11cyclictest7072-21ssh11:30:132
2761599292,22cyclictest924-21snmpd11:50:132
27615992918,6cyclictest22346irq/40-dwc2_hso08:55:112
2761599291,15cyclictest28611-21ssh10:30:142
2760699292,20cyclictest16068-21ssh11:35:131
27606992918,7cyclictest27588-21cyclictest12:15:141
27624992811,10cyclictest28366irq/58-eth010:30:133
2761599288,12cyclictest10490-21apt-get10:20:122
2761599282,20cyclictest0-21swapper/209:05:122
2761599282,11cyclictest16370-21ssh10:58:422
2761599282,10cyclictest28620-21ssh09:53:382
27615992815,10cyclictest677-21ssh09:37:382
2761599281,21cyclictest924-21snmpd12:15:152
2761599281,16cyclictest23002-21cut09:15:162
2760699283,19cyclictest32530-21ssh12:20:141
2760699283,19cyclictest32530-21ssh12:20:131
2760699282,20cyclictest20806-21ssh11:19:401
2760699282,18cyclictest839-21munin-node08:10:141
2760699282,18cyclictest839-21munin-node08:10:141
2760099282,17cyclictest27195-21ssh12:35:120
2762499273,16cyclictest0-21swapper/309:37:393
2762499272,16cyclictest28366irq/58-eth010:20:133
27624992714,8cyclictest0-21swapper/309:55:223
2761599277,8cyclictest27-21ksoftirqd/210:11:552
2761599276,13cyclictest22346irq/40-dwc2_hso12:30:122
2761599271,22cyclictest29216-21cut07:45:172
2761599271,12cyclictest32514-21ssh12:20:132
2761599271,12cyclictest32514-21ssh12:20:122
2760699274,14cyclictest17881-21ssh11:00:131
2760699272,19cyclictest2120-21ssh09:39:211
2760699272,19cyclictest14443-21ssh12:10:141
2760099273,15cyclictest22806-21ssh09:15:110
2760099272,20cyclictest8786-21ssh10:35:270
2760099272,20cyclictest17397-21ssh09:30:120
2760099272,19cyclictest16440-21ssh09:29:200
2761599268,8cyclictest27-21ksoftirqd/212:36:512
2761599263,16cyclictest22346irq/40-dwc2_hso07:30:142
2761599263,15cyclictest22346irq/40-dwc2_hso07:55:122
2761599262,15cyclictest0-21swapper/211:45:142
2761599262,15cyclictest0-21swapper/211:45:142
27615992614,9cyclictest19559-21ssh12:10:462
27615992612,10cyclictest3112-21apt-get09:40:132
27615992612,10cyclictest3112-21apt-get09:40:122
27615992611,4cyclictest27-21ksoftirqd/212:09:252
2760699262,21cyclictest3128-21ssh09:40:141
2760699262,21cyclictest3128-21ssh09:40:131
27606992616,5cyclictest0-21swapper/109:10:121
27600992619,4cyclictest12-21ksoftirqd/009:05:100
2760099261,20cyclictest30176-21ssh09:55:120
2762499252,13cyclictest0-21swapper/311:25:133
2761599258,9cyclictest17870-21apt-get11:00:142
2761599258,4cyclictest131rcu_preempt11:20:182
2761599253,8cyclictest27-21ksoftirqd/211:14:272
2761599253,14cyclictest0-21swapper/212:00:122
27615992512,6cyclictest27-21ksoftirqd/211:09:382
2760699255,11cyclictest633-21Async12:11:531
2760699252,7cyclictest131rcu_preempt09:25:391
2760699252,3cyclictest131rcu_preempt09:32:511
2760099252,14cyclictest630-21tcf-agent10:15:140
27600992510,9cyclictest27588-21cyclictest09:40:130
27600992510,9cyclictest27588-21cyclictest09:40:130
2762499244,12cyclictest0-21swapper/310:50:123
2762499242,13cyclictest28366irq/58-eth009:30:113
2761599246,15cyclictest1-21systemd09:34:212
2761599244,14cyclictest22346irq/40-dwc2_hso10:35:152
2761599242,15cyclictest0-21swapper/207:40:142
2761599242,12cyclictest131rcu_preempt07:35:282
27615992412,5cyclictest0-21swapper/211:19:092
2761599241,17cyclictest22346irq/40-dwc2_hso07:20:132
27615992410,8cyclictest27-21ksoftirqd/211:55:402
2761599240,15cyclictest32233-21ssh10:50:132
2760099244,12cyclictest0-21swapper/010:00:120
2762499233,13cyclictest0-21swapper/309:47:403
27624992312,8cyclictest28366irq/58-eth010:55:153
27624992310,10cyclictest28366irq/58-eth011:55:153
2761599237,4cyclictest27-21ksoftirqd/210:42:262
2761599237,10cyclictest27-21ksoftirqd/208:35:322
2761599235,8cyclictest27-21ksoftirqd/210:05:592
2761599235,8cyclictest27-21ksoftirqd/210:05:582
27615992312,4cyclictest27-21ksoftirqd/211:35:282
27615992312,4cyclictest0-21swapper/208:00:232
27615992310,4cyclictest27-21ksoftirqd/208:05:322
2760699232,16cyclictest13362-21diskmemload09:49:381
27606992314,5cyclictest27588-21cyclictest10:45:131
2760099236,10cyclictest15808-21ls08:05:020
2760099233,5cyclictest5314-21ntp_192.168.11411:45:260
2760099233,5cyclictest5314-21ntp_192.168.11411:45:250
2760099231,12cyclictest16068-21ssh11:15:250
27600992310,4cyclictest131rcu_preempt12:25:220
2761599225,9cyclictest28258-21kthreadcore08:15:222
2761599223,10cyclictest28484-21kthreadcore08:50:212
2761599221,12cyclictest25203-21diskstats11:40:152
2760699226,12cyclictest31897-21ssh09:20:121
2760699223,12cyclictest48-21kcompactd008:45:131
2760699223,12cyclictest19830-21ssh11:55:141
27606992212,8cyclictest7770-21irqstats11:30:181
27606992212,6cyclictest28980-21usage_Anybus10:10:301
2760099223,15cyclictest12-21ksoftirqd/012:05:120
2760099221,16cyclictest27843-21ls07:10:130
2760099221,16cyclictest26943-21sh11:05:140
27624992111,7cyclictest28366irq/58-eth011:05:133
27624992110,6cyclictest0-21swapper/311:46:293
27624992110,6cyclictest0-21swapper/311:46:283
2761599219,9cyclictest27-21ksoftirqd/208:35:002
2761599211,9cyclictest2460-21gpgv08:25:022
2761599211,11cyclictest1864-21kthreadcore07:15:192
2761599211,10cyclictest2936-21fschecks_time08:25:162
27615992110,4cyclictest0-21swapper/207:25:182
2760699219,8cyclictest0-21swapper/107:40:291
2760699214,12cyclictest30292-21ssh09:55:151
2760699213,13cyclictest14251-21ssh10:40:131
2760699212,15cyclictest3737-21munin-run09:00:011
27606992113,6cyclictest12122-21sshd08:32:401
2760699211,16cyclictest11774-21ssh10:00:331
27606992110,9cyclictest21-21ksoftirqd/112:00:141
2760099216,3cyclictest131rcu_preempt10:20:210
2760099212,13cyclictest0-21swapper/008:40:130
27600992112,6cyclictest21484-21/usr/sbin/munin11:20:160
27600992112,5cyclictest11051-21if_eth011:50:160
2760099211,17cyclictest10839-21latency_hist08:00:020
2760099211,16cyclictest32554-21apt-get07:15:130
2760099211,16cyclictest27527-21latency_hist07:10:020
27600992110,8cyclictest4297-21cat09:20:320
2760099210,17cyclictest8575-21latency_hist09:05:010
2762499209,9cyclictest0-21swapper/309:15:203
27624992012,6cyclictest28366irq/58-eth012:30:133
27624992012,6cyclictest0-21swapper/310:15:333
27624992011,7cyclictest0-21swapper/309:41:053
27624992011,7cyclictest0-21swapper/309:41:053
27624992010,8cyclictest0-21swapper/310:45:293
2761599202,9cyclictest29624-21tail07:10:202
2761599202,15cyclictest20143-21kthreadcore08:40:212
2761599201,10cyclictest22076-21fschecks_count08:45:172
27615992010,6cyclictest0-21swapper/208:10:122
27615992010,6cyclictest0-21swapper/208:10:122
2760699207,10cyclictest32606-21irqcore10:50:171
2760699201,4cyclictest0-21swapper/112:05:111
27606992014,4cyclictest0-21swapper/109:20:011
2760699201,16cyclictest18170-21apt-get12:30:121
27606992011,5cyclictest0-21swapper/111:50:141
2760699200,18cyclictest14699-21ls08:00:281
2760699200,17cyclictest31538-21ls08:55:011
2760099204,12cyclictest0-21swapper/011:30:140
2760099203,11cyclictest0-21swapper/011:00:140
2760099201,8cyclictest3072-21tr12:20:230
2760099201,8cyclictest3072-21tr12:20:220
27600992012,6cyclictest0-21swapper/010:40:410
27600992011,6cyclictest25540-21kthreadcore10:10:190
27600992011,6cyclictest21144-21/usr/sbin/munin09:50:230
2760099201,13cyclictest30032-21latency_hist08:20:010
2760099200,17cyclictest14116-21ls09:10:170
2762499193,9cyclictest28366irq/58-eth012:35:113
2762499193,9cyclictest0-21swapper/308:05:023
2762499193,11cyclictest0-21swapper/312:29:553
2762499193,11cyclictest0-21swapper/311:50:433
2762499193,11cyclictest0-21swapper/309:29:203
2762499193,11cyclictest0-21swapper/309:24:203
2762499192,11cyclictest0-21swapper/311:14:383
2762499191,13cyclictest0-21swapper/309:05:263
2762499191,13cyclictest0-21swapper/307:40:173
27624991910,6cyclictest0-21swapper/311:20:313
2761599190,10cyclictest27639-21latency_hist07:10:022
2760699198,8cyclictest6824-21latency_hist11:30:021
2760699192,9cyclictest131rcu_preempt07:30:211
27606991911,6cyclictest29392-21cat07:45:171
2760699191,16cyclictest19454-21find07:35:111
27606991911,5cyclictest20140-21usage_Anybus10:05:311
27606991911,5cyclictest20140-21usage_Anybus10:05:311
2760699191,14cyclictest12577-21ntp_states10:55:271
2760699191,14cyclictest0-21swapper/108:25:131
2760699190,11cyclictest1620-21grep07:50:151
2760099196,10cyclictest11333-21timerwakeupswit08:30:260
2760099195,10cyclictest3671-21kthreadcore08:25:190
2760099193,13cyclictest4170-21sed11:10:180
2760099193,12cyclictest6175-21grep07:55:000
2760099193,11cyclictest0-21swapper/010:08:400
2760099193,11cyclictest0-21swapper/010:08:400
2760099193,10cyclictest0-21swapper/007:25:130
2760099192,12cyclictest0-21swapper/012:30:110
2760099191,9cyclictest2436-21sh08:25:010
27600991911,6cyclictest4767-21cron10:35:000
2760099191,14cyclictest28662-21ntp_192.168.11411:40:250
2762499189,6cyclictest0-21swapper/312:13:293
2762499183,10cyclictest0-21swapper/309:54:393
2762499182,11cyclictest0-21swapper/310:08:403
2762499182,11cyclictest0-21swapper/310:08:393
2762499182,11cyclictest0-21swapper/309:00:133
2762499182,10cyclictest0-21swapper/310:00:143
2762499182,10cyclictest0-21swapper/308:40:133
2762499181,13cyclictest0-21swapper/309:10:173
2762499181,11cyclictest0-21swapper/308:35:223
2762499181,10cyclictest0-21swapper/307:10:133
27624991810,6cyclictest0-21swapper/311:19:463
27624991810,5cyclictest28366irq/58-eth010:25:123
2760699189,6cyclictest4986-21apt-get07:20:131
2760699189,6cyclictest11492-21kthreadcore07:25:201
2760699187,8cyclictest29078-21cut12:35:191
2760699184,11cyclictest3948-21/usr/sbin/munin09:00:151
27606991813,3cyclictest0-21swapper/107:55:121
27606991812,4cyclictest0-21swapper/110:25:211
2760699181,15cyclictest28625-21kthreadcore08:15:211
27606991810,6cyclictest28082-21awk07:10:171
27606991810,5cyclictest0-21swapper/108:40:171
2760099189,6cyclictest29890-21cut07:45:200
2760099188,7cyclictest23501-21apt-get12:15:130
2760099185,10cyclictest32305-21kthreadcore08:55:170
2760099184,4cyclictest131rcu_preempt08:50:010
2760099184,10cyclictest22005-21kthreadcore10:25:190
2760099183,10cyclictest0-21swapper/008:05:140
27600991811,5cyclictest8382-21ntp_kernel_err07:20:220
2760099181,14cyclictest31523-21latency_hist08:55:020
27600991810,6cyclictest19552-21df07:35:140
2762499179,6cyclictest0-21swapper/311:41:213
2762499172,9cyclictest0-21swapper/310:35:273
2762499172,9cyclictest0-21swapper/308:50:143
2762499172,9cyclictest0-21swapper/307:55:123
2762499172,10cyclictest0-21swapper/308:45:143
2762499172,10cyclictest0-21swapper/308:25:143
2762499172,10cyclictest0-21swapper/307:50:133
2762499171,13cyclictest0-21swapper/308:35:013
2760699171,11cyclictest12435-21ls09:05:261
2760099179,5cyclictest14765-21cut07:30:150
2760099173,4cyclictest131rcu_preempt08:10:280
2760099173,4cyclictest131rcu_preempt08:10:270
2760099171,13cyclictest25054-21kthreadcore07:40:180
2760099171,11cyclictest0-21swapper/008:35:140
2762499162,9cyclictest0-21swapper/312:00:143
2762499161,10cyclictest0-21swapper/307:35:273
2760699161,12cyclictest19154-21cat08:05:241
2760699161,11cyclictest30378-21diskstats08:20:161
2762499151,8cyclictest0-21swapper/307:10:023
2760699151,10cyclictest27633-21latency_hist07:10:011
2762499149,3cyclictest0-21swapper/308:20:253
2762499149,3cyclictest0-21swapper/307:18:393
2762499141,8cyclictest0-21swapper/308:05:133
2762499137,4cyclictest0-21swapper/307:30:123
2762499131,9cyclictest0-21swapper/308:15:233
2762499131,5cyclictest0-21swapper/307:20:183
2762499122,8cyclictest0-21swapper/308:10:143
2762499122,8cyclictest0-21swapper/308:10:143
2762499111,7cyclictest0-21swapper/307:45:183
2762499111,4cyclictest0-21swapper/307:25:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional