You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-10 - 18:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue Mar 10, 2026 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
223999631,43cyclictest27633-21ssh11:20:150
223999621,44cyclictest15421-21ssh09:25:130
226299583,12cyclictest131rcu_preempt09:20:132
226299492,31cyclictest11590-21ssh10:35:122
2262994917,7cyclictest27-21ksoftirqd/209:55:122
2262994917,22cyclictest27-21ksoftirqd/211:40:142
227399468,21cyclictest28366irq/58-eth010:55:133
2273994629,11cyclictest28366irq/58-eth010:20:113
2253994626,3cyclictest21-21ksoftirqd/111:05:131
2262994517,10cyclictest13518-21ssh10:00:142
223999451,37cyclictest9628-21ssh11:10:120
223999451,37cyclictest9628-21ssh11:10:120
227399432,25cyclictest28366irq/58-eth009:50:123
2239994215,4cyclictest0-21swapper/009:30:130
226299412,31cyclictest32529-21cut10:10:202
226299412,31cyclictest32529-21cut10:10:192
226299411,32cyclictest22508-21ssh10:05:122
223999401,31cyclictest13137-21ssh11:30:130
226299392,31cyclictest1873-21ssh09:52:192
226299372,28cyclictest18604-21ssh11:15:142
225399367,16cyclictest0-21swapper/109:40:141
2253993627,5cyclictest19339-21apt-get08:35:131
223999363,22cyclictest12-21ksoftirqd/008:55:140
225399342,22cyclictest20269-21diskmemload12:35:131
223999341,24cyclictest4174-21ssh11:25:130
223999340,23cyclictest22516-21rm10:05:120
227399331,24cyclictest0-21swapper/310:10:203
227399331,24cyclictest0-21swapper/310:10:193
2253993323,7cyclictest12136-21apt-get07:20:111
2253993320,10cyclictest20658-21apt-get09:10:121
227399321,18cyclictest28366irq/58-eth010:35:123
226299328,6cyclictest261rcuc/212:20:122
226299328,6cyclictest261rcuc/212:20:112
225399326,11cyclictest30526-21ssh11:55:571
225399326,10cyclictest28216-21ssh10:06:341
225399325,11cyclictest4398-21ssh11:41:401
225399324,16cyclictest20873-21ssh09:26:121
2253993222,7cyclictest4913-21apt-get08:20:121
225399322,27cyclictest18575-21ssh11:15:111
2253993220,9cyclictest26028-21apt-get10:25:141
223999322,22cyclictest18947-21ssh09:45:110
227399312,15cyclictest28366irq/58-eth012:10:123
226299314,17cyclictest22346irq/40-dwc2_hso12:10:132
226299312,22cyclictest10712-21ssh12:05:142
226299311,19cyclictest22346irq/40-dwc2_hso08:20:122
225399316,8cyclictest17467-21sed10:00:281
225399316,14cyclictest2229-21cyclictest09:50:121
225399313,9cyclictest11732-21ssh09:21:061
2253993120,7cyclictest29637-21apt-get10:45:151
2273993024,3cyclictest28366irq/58-eth011:40:133
226299303,17cyclictest0-21swapper/208:35:142
226299302,13cyclictest27-21ksoftirqd/209:15:112
226299301,8cyclictest0-21swapper/212:15:152
225399303,10cyclictest11054-21ssh11:45:231
2253993018,8cyclictest13125-21apt-get11:30:111
2253993014,12cyclictest31461-21ssh10:10:141
2253993014,12cyclictest31461-21ssh10:10:141
2253993014,11cyclictest10455-21kthreadcore10:15:181
2253993010,15cyclictest0-21swapper/111:00:121
2239993017,9cyclictest2413-21apt-get07:10:130
2239993016,9cyclictest2554-21apt-get10:30:120
223999301,21cyclictest20670-21ssh09:10:140
227399293,17cyclictest0-21swapper/311:10:143
227399293,17cyclictest0-21swapper/311:10:133
227399293,16cyclictest28366irq/58-eth009:40:143
2273992917,9cyclictest28366irq/58-eth009:10:143
226299292,17cyclictest22346irq/40-dwc2_hso09:00:122
225399295,8cyclictest5916-21cron09:20:001
225399294,8cyclictest0-21swapper/112:04:321
225399293,9cyclictest5517-21sed12:20:171
225399293,9cyclictest5517-21sed12:20:171
225399293,9cyclictest12048-21awk07:50:261
225399293,14cyclictest0-21swapper/109:31:051
223999293,19cyclictest7573-21ssh11:45:140
223999292,19cyclictest27945-21ssh09:50:140
227399283,15cyclictest28366irq/58-eth011:15:113
227399282,14cyclictest28366irq/58-eth007:45:133
226299287,13cyclictest27-21ksoftirqd/211:35:132
226299282,21cyclictest15981-21ssh10:19:192
226299282,10cyclictest30879-21ssh12:34:192
2262992812,7cyclictest1-21systemd11:29:172
225399286,9cyclictest3882-21cat11:25:021
225399283,8cyclictest20269-21diskmemload10:42:141
2253992822,3cyclictest4473-21apt-get09:55:131
225399282,23cyclictest16519-21ssh11:50:141
223999284,6cyclictest12-21ksoftirqd/010:45:180
2239992822,4cyclictest151rcuc/009:20:200
223999282,19cyclictest10456-21kthreadcore10:15:180
223999281,4cyclictest12-21ksoftirqd/008:25:010
223999280,18cyclictest12-21ksoftirqd/008:40:110
227399272,13cyclictest28366irq/58-eth010:40:123
227399270,14cyclictest28366irq/58-eth010:50:113
226299273,19cyclictest0-21swapper/209:49:192
226299272,9cyclictest22543-21ssh10:21:192
226299271,20cyclictest20269-21diskmemload09:10:112
225399277,8cyclictest28025-21ssh12:30:481
225399276,7cyclictest13917-21ssh12:25:071
225399275,9cyclictest7822-21ntp_192.168.11411:25:251
225399274,7cyclictest13139-21ntp_192.168.11408:25:251
225399274,16cyclictest2566-21ssh10:30:111
225399274,11cyclictest15238-21sh10:35:241
225399274,10cyclictest5128-21grep08:50:281
225399273,8cyclictest24087-21grep08:05:211
225399273,8cyclictest24087-21grep08:05:201
225399273,8cyclictest17848-21memory08:30:211
225399273,11cyclictest32293-21ntp_192.168.11408:45:251
225399272,9cyclictest17146-21ssh11:13:441
225399272,9cyclictest17146-21ssh11:13:441
2253992720,4cyclictest22072-21sh11:35:151
2253992718,7cyclictest15149-21ssh10:55:141
223999276,3cyclictest131rcu_preempt07:45:270
223999273,9cyclictest12-21ksoftirqd/012:04:060
223999271,18cyclictest1035-21ssh09:35:140
2239992711,3cyclictest12-21ksoftirqd/010:41:490
227399263,17cyclictest0-21swapper/311:55:113
227399262,14cyclictest28366irq/58-eth007:50:113
226299262,18cyclictest0-21swapper/208:30:122
2262992616,6cyclictest27-21ksoftirqd/210:55:132
225399265,7cyclictest22152-21tr09:45:211
225399265,10cyclictest15569-21tr07:55:201
225399263,8cyclictest308-21cat07:40:181
225399263,7cyclictest0-21swapper/109:36:441
225399262,19cyclictest24947-21ssh10:24:121
2253992612,11cyclictest3499-21ntp_192.168.11408:15:231
223999268,4cyclictest131rcu_preempt08:15:270
223999264,5cyclictest12-21ksoftirqd/009:55:140
223999264,5cyclictest12-21ksoftirqd/009:05:280
223999264,4cyclictest12-21ksoftirqd/010:10:220
223999264,4cyclictest12-21ksoftirqd/010:10:220
223999263,7cyclictest12-21ksoftirqd/012:05:270
223999263,7cyclictest12-21ksoftirqd/010:26:270
223999262,5cyclictest12-21ksoftirqd/010:04:000
223999261,18cyclictest28574-21ssh12:15:140
227399253,15cyclictest0-21swapper/309:59:083
227399253,14cyclictest28366irq/58-eth012:20:133
227399253,14cyclictest28366irq/58-eth012:20:133
226299252,8cyclictest6121-21ssh10:50:112
226299252,8cyclictest10611-21ssh10:34:182
226299252,16cyclictest22346irq/40-dwc2_hso10:25:142
226299251,17cyclictest22346irq/40-dwc2_hso10:40:132
225399259,9cyclictest0-21swapper/107:35:151
225399255,7cyclictest20269-21diskmemload12:17:241
225399254,10cyclictest25236-21/usr/sbin/munin07:30:271
225399254,10cyclictest15823-21fschecks_count09:05:151
2253992519,3cyclictest24142-21apt-get08:40:121
2253992517,3cyclictest16944-21apt-get07:25:151
2253992513,9cyclictest0-21swapper/110:50:121
2253992511,12cyclictest19628-21apt-get12:10:131
2253992510,11cyclictest14598-21cut12:05:291
223999257,11cyclictest5060-21sh12:20:130
223999257,11cyclictest5060-21sh12:20:130
223999257,11cyclictest12-21ksoftirqd/007:40:200
223999256,3cyclictest131rcu_preempt12:30:050
223999256,3cyclictest131rcu_preempt11:03:220
223999255,6cyclictest12-21ksoftirqd/011:35:300
223999255,3cyclictest131rcu_preempt12:25:210
223999255,3cyclictest131rcu_preempt12:15:020
223999254,4cyclictest12-21ksoftirqd/011:40:190
223999254,3cyclictest131rcu_preempt11:09:340
223999253,5cyclictest12-21ksoftirqd/009:15:410
223999253,5cyclictest12-21ksoftirqd/008:45:200
223999252,6cyclictest12-21ksoftirqd/011:19:540
223999252,6cyclictest12-21ksoftirqd/010:35:200
223999252,4cyclictest12-21ksoftirqd/008:15:000
223999252,2cyclictest131rcu_preempt08:05:270
223999252,2cyclictest131rcu_preempt08:05:260
227399242,14cyclictest28366irq/58-eth010:00:133
227399241,14cyclictest28366irq/58-eth011:45:143
226299246,8cyclictest4529-21sh11:05:282
226299243,13cyclictest22346irq/40-dwc2_hso08:25:132
226299242,16cyclictest0-21swapper/211:55:122
225399245,4cyclictest7286-21/usr/sbin/munin07:15:341
225399244,8cyclictest6971-21sshd07:13:191
225399244,10cyclictest10959-21awk09:00:131
2253992419,3cyclictest18035-21apt-get08:00:121
223999247,7cyclictest2229-21cyclictest07:25:130
223999245,3cyclictest131rcu_preempt12:35:020
223999244,4cyclictest12-21ksoftirqd/010:20:430
223999243,7cyclictest12-21ksoftirqd/011:50:270
2239992418,4cyclictest151rcuc/010:58:300
223999241,6cyclictest12-21ksoftirqd/011:55:250
223999241,5cyclictest12-21ksoftirqd/010:55:010
227399233,13cyclictest0-21swapper/312:15:143
227399233,13cyclictest0-21swapper/310:30:193
227399231,13cyclictest28366irq/58-eth009:20:123
226299238,7cyclictest30395-21kthreadcore10:45:222
226299237,8cyclictest27011-21grep07:35:192
226299234,15cyclictest22346irq/40-dwc2_hso07:15:142
226299232,16cyclictest0-21swapper/212:29:192
226299232,15cyclictest22346irq/40-dwc2_hso07:45:142
2262992312,9cyclictest19507-21ssh11:32:222
226299231,18cyclictest4951-21ls09:35:282
223999237,3cyclictest131rcu_preempt09:40:250
223999236,4cyclictest12-21ksoftirqd/008:25:180
223999235,3cyclictest131rcu_preempt07:15:220
223999234,3cyclictest131rcu_preempt07:20:140
223999234,2cyclictest131rcu_preempt08:50:150
223999232,6cyclictest12-21ksoftirqd/007:55:120
227399223,12cyclictest0-21swapper/310:15:183
226299228,11cyclictest0-21swapper/211:50:022
226299224,15cyclictest8675-21fschecks_time07:50:162
226299223,13cyclictest22346irq/40-dwc2_hso08:40:112
226299222,16cyclictest24120-21ssh11:00:142
226299222,15cyclictest924-21snmpd07:25:142
226299222,15cyclictest0-21swapper/211:20:152
226299221,7cyclictest0-21swapper/208:10:132
2262992212,7cyclictest32079-21fschecks_time12:35:162
226299221,17cyclictest0-21swapper/208:05:122
226299221,17cyclictest0-21swapper/208:05:122
226299221,13cyclictest2848-21ssh12:00:192
226299221,11cyclictest6228-21fschecks_count08:55:172
2253992213,6cyclictest6162-21df08:55:151
223999225,3cyclictest131rcu_preempt07:30:260
223999224,6cyclictest12-21ksoftirqd/008:35:170
223999224,4cyclictest131rcu_preempt07:35:300
223999224,2cyclictest131rcu_preempt09:00:140
223999224,2cyclictest131rcu_preempt07:53:400
227399215,13cyclictest321rcuc/307:55:133
2273992114,4cyclictest0-21swapper/310:05:133
226299217,7cyclictest4658-21cron08:20:012
226299216,8cyclictest27-21ksoftirqd/209:25:332
226299216,10cyclictest0-21swapper/208:00:132
226299213,13cyclictest0-21swapper/211:10:142
226299213,13cyclictest0-21swapper/211:10:132
226299212,16cyclictest1265-21cpu08:50:142
226299212,14cyclictest0-21swapper/207:30:142
226299211,9cyclictest20397-21timerandwakeup11:50:292
226299211,16cyclictest2595-21fschecks_time07:10:172
226299211,16cyclictest24544-21df_inode09:30:142
226299211,11cyclictest15720-21kthreadcore07:55:212
226299210,11cyclictest13205-21kthreadcore09:40:232
223999219,4cyclictest12-21ksoftirqd/008:30:150
223999213,10cyclictest12-21ksoftirqd/008:00:200
2273992013,5cyclictest0-21swapper/311:50:473
226299205,12cyclictest27-21ksoftirqd/207:40:202
226299201,10cyclictest29811-21kthreadcore08:45:212
226299201,10cyclictest13085-21kthreadcore07:20:192
225399206,3cyclictest0-21swapper/108:10:121
227399192,10cyclictest0-21swapper/308:55:143
227399192,10cyclictest0-21swapper/307:25:143
2273991913,4cyclictest0-21swapper/312:31:003
2273991913,4cyclictest0-21swapper/311:20:103
2273991913,4cyclictest0-21swapper/311:00:233
2273991910,6cyclictest28366irq/58-eth009:35:133
2273991910,6cyclictest0-21swapper/310:26:283
226299194,13cyclictest16021-21ps09:05:172
226299191,15cyclictest2278-21latency_hist07:10:022
225399191,4cyclictest3700-21apt-get07:45:141
227399189,6cyclictest0-21swapper/309:28:453
227399188,6cyclictest0-21swapper/312:26:003
227399183,11cyclictest0-21swapper/309:49:033
227399182,10cyclictest0-21swapper/308:40:123
227399182,10cyclictest0-21swapper/308:30:123
227399182,10cyclictest0-21swapper/308:25:153
227399182,10cyclictest0-21swapper/307:30:143
2273991812,4cyclictest0-21swapper/311:06:213
2273991812,4cyclictest0-21swapper/309:30:253
2273991811,5cyclictest0-21swapper/311:37:133
227399181,12cyclictest0-21swapper/312:05:153
227399181,12cyclictest0-21swapper/307:10:153
2273991810,5cyclictest28366irq/58-eth009:15:113
227399179,3cyclictest0-21swapper/310:46:163
227399178,7cyclictest0-21swapper/311:32:583
227399172,10cyclictest28366irq/58-eth012:00:133
2273991711,4cyclictest0-21swapper/311:25:323
227399171,12cyclictest28366irq/58-eth012:35:133
227399162,9cyclictest0-21swapper/309:00:133
227399162,10cyclictest0-21swapper/308:45:153
227399162,10cyclictest0-21swapper/308:35:143
227399161,9cyclictest0-21swapper/308:10:283
227399152,10cyclictest0-21swapper/307:35:153
227399152,10cyclictest0-21swapper/307:15:043
225399154,7cyclictest2305-21sed07:10:031
223999154,9cyclictest2229-21cyclictest07:10:010
227399149,3cyclictest0-21swapper/308:22:003
227399148,4cyclictest0-21swapper/308:00:303
227399137,4cyclictest28366irq/58-eth007:20:193
227399131,10cyclictest0-21swapper/309:05:203
227399127,3cyclictest0-21swapper/308:07:523
227399127,3cyclictest0-21swapper/308:07:523
227399122,6cyclictest0-21swapper/307:40:123
227399121,6cyclictest0-21swapper/308:15:123
227399111,7cyclictest0-21swapper/307:10:023
227399111,6cyclictest0-21swapper/308:50:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional