You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-15 - 02:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue Sep 15, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1129399746,46cyclictest3628-21ssh22:25:130
1131199676,16cyclictest27-21ksoftirqd/223:35:122
1130299552,35cyclictest21683-21ssh22:35:131
1129399515,32cyclictest131rcu_preempt22:40:130
1129399512,34cyclictest23170-21ssh22:00:130
11302994918,21cyclictest0-21swapper/119:55:131
1130299491,37cyclictest18741-21ssh23:45:121
1131199471,33cyclictest29572-21ssh21:10:142
11302994733,10cyclictest23362-21apt-get20:30:141
11302994639,3cyclictest10179-21apt-get20:50:131
11302994629,13cyclictest946-21ssh21:30:141
11322994439,3cyclictest35966irq/58-eth022:35:133
1132299433,22cyclictest35966irq/58-eth000:35:123
11302994317,11cyclictest13613-21fschecks_time00:00:171
1129399431,30cyclictest532-21ssh23:35:120
11302994216,8cyclictest8622-21ntp_states21:50:271
11302994215,8cyclictest23158-21apt-get22:00:141
11302994135,2cyclictest2985-21apt-get19:35:131
11302994131,7cyclictest5895-21ssh23:20:151
11302994117,9cyclictest31498-21apt-get00:10:141
11302994117,12cyclictest22600-21ssh23:28:571
11302994114,10cyclictest12656-21/usr/sbin/munin22:30:181
1132299403,24cyclictest35966irq/58-eth000:10:143
11302994014,10cyclictest18705-21ssh00:37:541
11302994013,8cyclictest27042-21grep22:20:141
11302994013,11cyclictest22320-21ssh23:10:201
11302993929,7cyclictest32192-21ssh22:05:121
11302993915,10cyclictest3342-21ssh23:17:331
11302993913,9cyclictest3946-21apt23:55:021
11302993913,9cyclictest30608-21sed19:30:021
11302993913,8cyclictest0-21swapper/123:55:351
11302993913,11cyclictest2143-21grep23:00:151
11302993910,11cyclictest21710-21ssh21:58:471
11302993815,10cyclictest28449-21fschecks_time21:45:161
11302993815,10cyclictest28449-21fschecks_time21:45:151
11302993815,10cyclictest12425-21ssh22:30:011
11302993814,9cyclictest8734-21kthreadcore22:45:171
11302993813,11cyclictest20536-21cut22:15:191
11302993813,10cyclictest19931-21cat21:20:301
11302993810,12cyclictest5352-21/usr/sbin/munin20:45:161
1129399382,22cyclictest131rcu_preempt21:15:130
11293993821,10cyclictest131rcu_preempt23:00:120
1129399381,28cyclictest31510-21ssh00:10:130
11302993721,13cyclictest0-21swapper/120:35:121
11302993721,12cyclictest0-21swapper/122:50:121
11302993716,8cyclictest22642-21grep00:05:161
11302993713,8cyclictest13675-21ssh21:18:361
1129399373,23cyclictest8122-21ssh00:15:130
1129399371,26cyclictest10050-21ssh21:35:130
11302993630,3cyclictest9608-21apt-get23:40:131
11302993624,9cyclictest2889-21ssh00:30:141
11302993621,12cyclictest0-21swapper/122:55:131
11302993621,12cyclictest0-21swapper/122:55:121
11302993616,15cyclictest0-21swapper/121:40:131
11302993616,10cyclictest19333-21kthreadcore20:25:191
11302993615,8cyclictest29405-21ssh21:25:491
11302993615,7cyclictest29149-21diskmemload00:28:251
11302993614,8cyclictest8807-21cat20:15:021
11302993613,7cyclictest0-21swapper/122:11:251
11302993611,8cyclictest0-21swapper/123:09:421
1129399363,25cyclictest8867-21sh22:10:120
1129399362,27cyclictest4901-21ssh21:50:130
1132299355,17cyclictest35966irq/58-eth023:05:123
11302993513,10cyclictest16137-21kthreadcore20:55:181
11302993511,8cyclictest14980-21sshd00:17:531
11302993511,10cyclictest22604-21kthreadcore19:20:191
11302993510,11cyclictest15325-21grep19:45:211
1129399355,22cyclictest12-21ksoftirqd/023:20:140
11293993519,6cyclictest0-21swapper/022:15:120
1129399351,26cyclictest7346-21ssh22:45:100
1131199341,26cyclictest0-21swapper/220:25:122
11302993414,8cyclictest522-21apt-get23:35:121
11302993412,8cyclictest2755-21ssh21:11:281
11302993412,10cyclictest10703-21cut19:40:241
11302993411,7cyclictest0-21swapper/121:37:571
11322993321,8cyclictest35966irq/58-eth021:50:133
1131199333,12cyclictest23185-21ssh22:00:152
11311993318,10cyclictest0-21swapper/221:30:152
11302993325,5cyclictest11446-21apt-get19:10:121
11302993322,8cyclictest32705-21apt-get20:40:051
11302993316,13cyclictest0-21swapper/120:05:151
11302993312,8cyclictest23759-21cat21:00:311
11302993312,7cyclictest27856-21cut20:00:211
11302993311,9cyclictest22877-21ssh00:21:321
11302993311,8cyclictest1857-21memory22:40:221
1129399336,13cyclictest131rcu_preempt22:05:150
1131199329,9cyclictest27-21ksoftirqd/223:16:262
1131199327,8cyclictest27-21ksoftirqd/223:50:042
11311993210,5cyclictest27-21ksoftirqd/200:04:002
1130299329,10cyclictest30650-21grep19:30:131
1129399328,19cyclictest12-21ksoftirqd/023:50:120
11322993117,9cyclictest35966irq/58-eth023:30:143
11302993119,9cyclictest29132-21sshd21:07:191
11302993115,11cyclictest0-21swapper/120:20:111
11302993111,9cyclictest31438-21ssh23:33:391
1132299304,16cyclictest35966irq/58-eth022:45:103
1131199304,10cyclictest27-21ksoftirqd/223:04:392
1131199303,17cyclictest27346irq/40-dwc2_hso20:50:112
1131199302,21cyclictest0-21swapper/223:10:132
1131199302,20cyclictest27346irq/40-dwc2_hso20:40:152
1131199301,20cyclictest930-21snmpd19:35:112
11311993011,13cyclictest27-21ksoftirqd/200:25:212
11311993010,9cyclictest16453-21apt-get22:50:112
11302993019,9cyclictest21918-21sshd19:52:391
11302993017,9cyclictest12542-21mailstats20:15:281
1129399302,19cyclictest11874-21ssh00:35:120
1129399301,9cyclictest131rcu_preempt21:40:140
1132299291,22cyclictest35966irq/58-eth023:45:123
1131199293,17cyclictest27346irq/40-dwc2_hso20:20:132
1131199293,11cyclictest27-21ksoftirqd/223:46:322
1131199292,20cyclictest27346irq/40-dwc2_hso22:15:112
11311992912,14cyclictest8473-21cron22:10:022
11311992910,12cyclictest27-21ksoftirqd/221:23:282
1131199290,12cyclictest31497-21/usr/sbin/munin00:10:122
11302992918,8cyclictest16299-21/usr/sbin/munin19:15:151
11293992912,8cyclictest0-21swapper/023:46:560
1132299285,17cyclictest0-21swapper/321:10:153
11322992815,10cyclictest35966irq/58-eth023:25:133
11322992813,10cyclictest35966irq/58-eth000:20:133
1131199288,11cyclictest27-21ksoftirqd/219:30:232
1131199285,7cyclictest131rcu_preempt20:00:302
1131199283,18cyclictest27346irq/40-dwc2_hso19:20:132
1131199283,15cyclictest27-21ksoftirqd/200:05:202
1129399288,9cyclictest0-21swapper/019:45:270
1129399283,17cyclictest23883-21ssh23:30:120
1129399282,19cyclictest24327-21ssh21:25:130
1131199279,12cyclictest27-21ksoftirqd/221:45:222
1131199279,12cyclictest27-21ksoftirqd/221:45:212
1131199278,11cyclictest27-21ksoftirqd/222:59:142
1131199278,11cyclictest27-21ksoftirqd/222:59:132
1131199277,12cyclictest27-21ksoftirqd/220:30:182
1131199275,10cyclictest27-21ksoftirqd/200:34:572
1131199274,18cyclictest27346irq/40-dwc2_hso23:30:122
1131199273,17cyclictest27346irq/40-dwc2_hso19:50:122
1131199273,17cyclictest27346irq/40-dwc2_hso19:15:142
1131199272,9cyclictest27-21ksoftirqd/222:20:112
1131199272,10cyclictest0-21swapper/222:45:122
1129399275,14cyclictest0-21swapper/021:20:130
11293992712,5cyclictest0-21swapper/023:41:490
1131199269,5cyclictest27-21ksoftirqd/221:37:562
1131199268,3cyclictest131rcu_preempt22:13:342
1131199268,11cyclictest0-21swapper/222:26:032
1131199267,8cyclictest27-21ksoftirqd/221:25:462
1131199267,12cyclictest27-21ksoftirqd/222:42:002
1131199266,5cyclictest27-21ksoftirqd/220:05:282
1131199264,3cyclictest131rcu_preempt00:20:272
1131199264,13cyclictest131rcu_preempt21:40:192
1131199264,10cyclictest27-21ksoftirqd/222:35:322
1131199261,3cyclictest8733-21sendmail_mailqu21:50:282
11311992613,6cyclictest19586-21ssh22:32:592
1131199261,15cyclictest14867-21ssh23:25:142
1129399265,14cyclictest27018-21ssh22:20:140
1129399264,11cyclictest0-21swapper/019:30:140
1129399262,21cyclictest29184-21ssh23:15:100
1132299253,12cyclictest35966irq/58-eth022:15:113
1132299252,16cyclictest35966irq/58-eth000:00:143
11322992514,6cyclictest0-21swapper/323:20:173
11322992512,7cyclictest0-21swapper/321:30:153
1131199257,9cyclictest27-21ksoftirqd/223:43:372
1131199255,12cyclictest131rcu_preempt00:37:292
1131199254,5cyclictest27-21ksoftirqd/221:15:172
1131199251,4cyclictest271ksoftirqd/219:55:202
11311992513,6cyclictest27-21ksoftirqd/223:20:292
1129399252,18cyclictest25460-21apt-get22:55:140
1129399252,18cyclictest25460-21apt-get22:55:140
11293992511,2cyclictest12-21ksoftirqd/019:35:110
1131199248,8cyclictest27-21ksoftirqd/219:40:292
1131199247,11cyclictest27-21ksoftirqd/221:55:022
1131199244,5cyclictest27-21ksoftirqd/220:10:232
1131199244,3cyclictest131rcu_preempt23:05:122
1131199244,2cyclictest131rcu_preempt23:55:332
1131199241,5cyclictest27-21ksoftirqd/219:25:202
1131199241,15cyclictest0-21swapper/220:35:152
11302992412,9cyclictest11303-21latency_hist19:10:021
1132299236,9cyclictest35966irq/58-eth000:15:113
1131199234,2cyclictest131rcu_preempt20:45:302
1131199232,5cyclictest27-21ksoftirqd/221:05:012
11293992312,3cyclictest0-21swapper/000:33:450
11322992211,8cyclictest35966irq/58-eth023:00:133
1132299221,11cyclictest35966irq/58-eth022:40:133
1131199227,12cyclictest11421-21latency_hist19:10:022
1131199223,4cyclictest27-21ksoftirqd/219:10:212
1131199222,13cyclictest27-21ksoftirqd/200:15:422
1131199221,10cyclictest10032-21kthreadcore20:15:192
1131199220,14cyclictest27-21ksoftirqd/219:45:112
1129399224,3cyclictest131rcu_preempt20:35:130
11293992210,9cyclictest12951-21ssh22:30:170
1132299213,7cyclictest0-21swapper/323:35:133
11322992115,4cyclictest0-21swapper/322:05:143
11322992115,4cyclictest0-21swapper/321:55:133
11322992114,5cyclictest0-21swapper/322:20:123
11322992112,7cyclictest0-21swapper/321:35:203
1132299211,13cyclictest321rcuc/323:10:103
11322992110,8cyclictest0-21swapper/323:16:073
11322992110,6cyclictest0-21swapper/321:29:403
1129399217,3cyclictest131rcu_preempt23:05:300
1129399215,3cyclictest0-21swapper/021:57:530
1129399212,15cyclictest16465-21ssh22:50:130
1129399211,5cyclictest131rcu_preempt21:15:020
1129399211,4cyclictest131rcu_preempt23:25:210
1129399211,4cyclictest131rcu_preempt20:05:280
1129399211,3cyclictest131rcu_preempt22:35:260
1129399211,14cyclictest17461-21ls00:20:170
1132299202,10cyclictest35966irq/58-eth000:25:123
11322992014,4cyclictest0-21swapper/323:40:263
11322992014,4cyclictest0-21swapper/322:10:513
11322992013,5cyclictest0-21swapper/322:30:023
11322992013,4cyclictest0-21swapper/322:03:293
1132299201,12cyclictest0-21swapper/319:25:173
11322992010,8cyclictest0-21swapper/321:23:113
1131199201,9cyclictest16473-21kthreadcore20:55:192
1131199201,14cyclictest24597-21apt-get21:05:142
1129399203,13cyclictest0-21swapper/020:50:120
1129399201,7cyclictest21039-21cut23:10:170
1129399201,3cyclictest131rcu_preempt19:15:180
1129399201,2cyclictest131rcu_preempt21:00:280
11293992011,6cyclictest22381-21ssh00:05:120
1129399201,13cyclictest0-21swapper/000:25:130
1129399200,3cyclictest131rcu_preempt20:10:310
1129399200,17cyclictest22835-21sed19:55:200
11322991913,4cyclictest0-21swapper/322:56:043
11322991913,4cyclictest0-21swapper/322:56:033
11322991913,4cyclictest0-21swapper/322:50:123
11322991913,4cyclictest0-21swapper/300:06:283
1132299191,12cyclictest0-21swapper/319:40:293
1132299191,11cyclictest0-21swapper/320:05:293
11322991910,6cyclictest35966irq/58-eth022:25:133
1129399198,3cyclictest0-21swapper/020:45:290
1129399191,7cyclictest131rcu_preempt20:40:300
1129399191,3cyclictest131rcu_preempt23:55:290
11293991913,3cyclictest12-21ksoftirqd/021:35:010
1132299188,4cyclictest0-21swapper/323:55:193
1132299182,10cyclictest0-21swapper/320:15:143
1132299182,10cyclictest0-21swapper/319:20:143
1132299182,10cyclictest0-21swapper/319:10:133
11322991812,4cyclictest0-21swapper/323:53:383
11322991812,4cyclictest0-21swapper/321:41:593
11322991812,4cyclictest0-21swapper/320:35:293
11322991812,4cyclictest0-21swapper/319:35:143
11322991812,4cyclictest0-21swapper/300:30:203
1132299181,11cyclictest0-21swapper/320:55:013
11322991810,5cyclictest35966irq/58-eth021:45:173
11322991810,5cyclictest35966irq/58-eth021:45:163
1129399185,3cyclictest0-21swapper/019:10:200
1129399182,2cyclictest131rcu_preempt20:30:210
11293991811,5cyclictest29672-21grep19:25:280
11293991811,5cyclictest0-21swapper/021:48:130
11293991811,5cyclictest0-21swapper/021:48:120
1129399180,5cyclictest0-21swapper/000:00:170
1129399180,3cyclictest131rcu_preempt21:05:270
1132299172,9cyclictest0-21swapper/320:30:143
11322991711,4cyclictest0-21swapper/321:15:183
1132299171,12cyclictest0-21swapper/321:00:253
1129399179,5cyclictest12-21ksoftirqd/019:20:130
1129399173,4cyclictest0-21swapper/019:50:210
1129399171,8cyclictest17205-21ntpq20:20:230
1129399171,6cyclictest27494-21tr20:00:190
1129399171,6cyclictest19041-21irqcore20:25:170
11293991712,3cyclictest0-21swapper/019:40:160
1132299162,10cyclictest0-21swapper/320:55:133
1132299162,10cyclictest0-21swapper/320:45:143
1132299162,10cyclictest0-21swapper/319:45:143
1132299161,10cyclictest0-21swapper/319:55:133
1129399162,12cyclictest15087-21fschecks_count20:55:150
1132299152,9cyclictest0-21swapper/320:40:153
1132299152,7cyclictest321rcuc/320:25:133
1129399153,4cyclictest9254-21sed20:15:180
1132299149,3cyclictest0-21swapper/321:06:073
1132299148,4cyclictest0-21swapper/319:30:273
1132299142,8cyclictest0-21swapper/320:10:143
1129399147,5cyclictest0-21swapper/019:10:010
1132299131,10cyclictest0-21swapper/320:20:193
1132299123,6cyclictest0-21swapper/319:15:133
1132299121,9cyclictest0-21swapper/320:00:293
1132299111,7cyclictest0-21swapper/319:10:023
1132299111,5cyclictest0-21swapper/319:50:133
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional