You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 03:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Mon Nov 24, 2025 00:45:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
9704995423,22cyclictest21-21ksoftirqd/122:05:121
9704995220,20cyclictest21-21ksoftirqd/123:28:251
9704995210,32cyclictest131rcu_preempt23:00:121
9715994927,13cyclictest28436-21diskmemload22:20:142
972499464,32cyclictest29566irq/58-eth023:50:143
972499454,26cyclictest29566irq/58-eth023:05:133
9724994522,18cyclictest0-21swapper/322:35:143
972499434,22cyclictest29566irq/58-eth000:15:163
969699433,6cyclictest131rcu_preempt22:00:140
972499422,24cyclictest29566irq/58-eth022:20:163
969699412,25cyclictest23610-21ssh00:00:130
969699402,33cyclictest2854-21ssh22:50:120
969699401,29cyclictest12379-21ssh23:15:120
972499393,19cyclictest29566irq/58-eth000:30:143
972499391,22cyclictest29566irq/58-eth020:20:143
971599397,9cyclictest27-21ksoftirqd/220:35:132
969699393,33cyclictest8508-21ssh00:30:130
969699391,34cyclictest7208-21ssh21:55:140
972499370,21cyclictest29566irq/58-eth023:00:123
9704993719,10cyclictest21-21ksoftirqd/122:25:151
972499367,19cyclictest29566irq/58-eth023:35:123
971599362,18cyclictest31261-21sh22:26:452
970499362,28cyclictest4552-21ssh21:15:131
972499351,16cyclictest29566irq/58-eth021:05:133
971599356,22cyclictest27-21ksoftirqd/222:15:212
971599356,22cyclictest27-21ksoftirqd/222:15:212
971599354,12cyclictest27-21ksoftirqd/223:25:132
970499357,4cyclictest131rcu_preempt00:20:131
9704993513,6cyclictest131rcu_preempt22:10:151
9704993513,6cyclictest131rcu_preempt22:10:151
969699343,8cyclictest12-21ksoftirqd/022:05:120
969699342,26cyclictest24840-21ssh00:20:130
972499332,19cyclictest29566irq/58-eth021:55:143
972499323,21cyclictest29566irq/58-eth019:30:133
969699322,24cyclictest18409-21ssh22:40:130
969699322,18cyclictest28436-21diskmemload22:55:140
972499313,18cyclictest29566irq/58-eth020:45:123
972499312,15cyclictest29566irq/58-eth022:05:113
969699313,20cyclictest5851-21ssh21:35:150
972499303,18cyclictest29566irq/58-eth022:25:153
9724993018,8cyclictest29566irq/58-eth021:40:133
971599302,21cyclictest5134-21ntp_192.168.11422:30:242
971599302,11cyclictest29798-21ssh21:30:132
971599302,11cyclictest16062-21ssh00:34:552
971599302,10cyclictest13461-21memory22:35:242
971599300,10cyclictest16462-21rm00:15:162
970499303,7cyclictest131rcu_preempt00:00:141
970499303,18cyclictest441-21ssh00:25:151
969699303,19cyclictest29317-21ssh23:25:140
972499295,19cyclictest29566irq/58-eth020:15:143
971599294,9cyclictest27-21ksoftirqd/222:49:062
971599294,15cyclictest7991-21ssh00:10:162
971599292,11cyclictest932-21snmpd22:05:212
9715992911,14cyclictest6835-21apt-get23:50:152
972499283,18cyclictest0-21swapper/300:29:003
9724992815,8cyclictest29566irq/58-eth000:00:143
971599288,5cyclictest131rcu_preempt21:25:142
971599282,19cyclictest0-21swapper/219:35:132
970499282,18cyclictest19043-21grep00:15:261
969699285,6cyclictest12-21ksoftirqd/023:20:150
969699282,17cyclictest14315-21ssh21:40:120
969699281,6cyclictest12-21ksoftirqd/022:15:130
969699281,6cyclictest12-21ksoftirqd/022:15:130
9696992810,4cyclictest12-21ksoftirqd/019:10:280
972499273,15cyclictest0-21swapper/321:29:153
971599273,19cyclictest0-21swapper/221:51:142
971599272,20cyclictest22497-21ssh22:03:212
971599272,18cyclictest27-21ksoftirqd/200:35:312
971599272,10cyclictest0-21swapper/221:00:162
9715992719,6cyclictest27-21ksoftirqd/223:15:112
971599271,19cyclictest22748-21sh21:45:142
970499277,14cyclictest131rcu_preempt20:25:121
970499274,3cyclictest131rcu_preempt22:00:141
970499271,4cyclictest131rcu_preempt22:20:151
969699277,8cyclictest131rcu_preempt00:05:160
969699276,3cyclictest131rcu_preempt23:05:230
969699276,3cyclictest131rcu_preempt22:30:170
969699273,6cyclictest12-21ksoftirqd/021:10:140
969699272,5cyclictest12-21ksoftirqd/021:45:160
969699272,19cyclictest7475-21ssh23:10:240
969699270,7cyclictest32269-21fschecks_count20:40:160
972499269,11cyclictest0-21swapper/323:10:243
9715992615,8cyclictest0-21swapper/223:55:232
9715992614,4cyclictest0-21swapper/221:10:262
9715992613,4cyclictest27-21ksoftirqd/223:15:012
9715992610,13cyclictest27-21ksoftirqd/222:50:142
970499262,18cyclictest856-21sshd23:11:591
970499262,18cyclictest32093-21ssh00:05:171
9704992612,5cyclictest0-21swapper/121:54:561
969699264,5cyclictest12-21ksoftirqd/022:45:280
969699264,3cyclictest131rcu_preempt22:10:170
969699264,3cyclictest131rcu_preempt22:10:170
969699263,6cyclictest12-21ksoftirqd/021:20:190
969699261,7cyclictest12-21ksoftirqd/000:15:250
9696992612,7cyclictest0-21swapper/022:26:300
972499258,11cyclictest0-21swapper/323:42:253
9724992511,9cyclictest29566irq/58-eth022:55:153
9715992512,4cyclictest27-21ksoftirqd/221:35:152
9715992512,11cyclictest0-21swapper/200:10:012
9715992511,12cyclictest27-21ksoftirqd/223:20:182
9715992510,12cyclictest131rcu_preempt00:25:362
970499254,13cyclictest131rcu_preempt23:30:141
970499251,17cyclictest20903-21ssh23:20:171
970499251,16cyclictest1522-21ssh22:30:121
969699256,3cyclictest131rcu_preempt21:25:160
969699255,7cyclictest12-21ksoftirqd/021:30:280
969699255,3cyclictest131rcu_preempt00:35:310
969699254,4cyclictest12-21ksoftirqd/023:30:200
969699254,4cyclictest12-21ksoftirqd/000:00:010
969699254,10cyclictest12-21ksoftirqd/021:15:410
969699253,5cyclictest12-21ksoftirqd/019:30:020
969699252,4cyclictest12-21ksoftirqd/021:05:160
969699251,6cyclictest12-21ksoftirqd/020:05:270
972499243,14cyclictest0-21swapper/323:17:243
971599249,6cyclictest27-21ksoftirqd/223:32:472
971599248,4cyclictest27-21ksoftirqd/222:10:172
971599248,4cyclictest27-21ksoftirqd/222:10:172
971599247,4cyclictest131rcu_preempt23:05:142
971599246,8cyclictest4922-21grep21:15:172
971599246,11cyclictest131rcu_preempt21:55:192
971599245,11cyclictest131rcu_preempt21:20:312
971599242,16cyclictest26346irq/40-dwc2_hso20:10:142
971599241,7cyclictest0-21swapper/219:20:302
9715992414,8cyclictest0-21swapper/200:00:182
971599241,10cyclictest131rcu_preempt19:15:132
970499242,6cyclictest131rcu_preempt21:21:031
970499242,17cyclictest28126-21ssh21:46:161
970499242,16cyclictest17591-21ntp_states23:35:251
970499242,11cyclictest0-21swapper/122:46:251
969699247,3cyclictest131rcu_preempt23:40:240
969699246,3cyclictest131rcu_preempt00:10:290
969699245,6cyclictest12-21ksoftirqd/000:25:330
969699245,10cyclictest131rcu_preempt22:21:230
969699244,5cyclictest12-21ksoftirqd/021:50:170
969699244,4cyclictest12-21ksoftirqd/023:51:570
969699243,6cyclictest12-21ksoftirqd/019:15:240
969699242,5cyclictest12-21ksoftirqd/022:36:500
969699242,19cyclictest13913-21ssh23:35:120
969699242,18cyclictest30852-21ssh23:45:130
969699241,15cyclictest629-21Async14:00:480
969699240,6cyclictest11905-21uname19:45:200
972499233,13cyclictest0-21swapper/322:30:143
972499233,13cyclictest0-21swapper/322:01:443
972499231,11cyclictest29566irq/58-eth000:10:163
971599239,3cyclictest131rcu_preempt23:04:432
971599239,12cyclictest131rcu_preempt22:40:252
971599238,13cyclictest131rcu_preempt23:45:282
971599237,7cyclictest27-21ksoftirqd/223:44:252
971599237,5cyclictest131rcu_preempt21:44:032
971599236,8cyclictest27-21ksoftirqd/220:05:122
9715992312,8cyclictest27-21ksoftirqd/220:55:212
9715992312,4cyclictest0-21swapper/220:30:172
970499231,5cyclictest0-21swapper/100:30:231
970499231,2cyclictest131rcu_preempt23:50:001
970499231,2cyclictest131rcu_preempt22:16:581
970499231,2cyclictest131rcu_preempt22:16:581
970499231,14cyclictest131rcu_preempt21:00:281
970499231,12cyclictest29787-21ssh21:30:121
970499230,13cyclictest13309-21sh22:55:221
969699236,7cyclictest22903-21ntpq19:20:260
969699235,2cyclictest131rcu_preempt20:00:120
969699235,2cyclictest131rcu_preempt20:00:120
969699234,5cyclictest12-21ksoftirqd/019:50:270
969699234,4cyclictest151rcuc/020:55:010
969699234,4cyclictest12-21ksoftirqd/019:55:200
969699234,4cyclictest12-21ksoftirqd/019:40:350
969699231,7cyclictest12-21ksoftirqd/020:25:290
9696992310,4cyclictest12-21ksoftirqd/023:00:190
9724992214,6cyclictest29566irq/58-eth021:46:473
9724992211,8cyclictest29566irq/58-eth021:50:143
971599228,3cyclictest131rcu_preempt19:40:212
971599228,12cyclictest131rcu_preempt20:40:122
971599227,12cyclictest131rcu_preempt22:56:442
971599227,12cyclictest131rcu_preempt00:20:392
971599225,7cyclictest131rcu_preempt20:20:232
9715992214,5cyclictest19391-21sshd23:36:252
9715992212,8cyclictest27-21ksoftirqd/220:15:322
970499229,4cyclictest0-21swapper/121:43:151
970499223,8cyclictest0-21swapper/122:44:441
970499223,2cyclictest131rcu_preempt21:10:311
970499222,14cyclictest27464-21ssh21:27:151
970499221,9cyclictest131rcu_preempt00:36:351
970499221,2cyclictest131rcu_preempt21:57:331
9704992211,9cyclictest0-21swapper/122:54:031
969699225,3cyclictest131rcu_preempt20:10:190
969699223,6cyclictest12-21ksoftirqd/020:45:150
969699223,4cyclictest12-21ksoftirqd/020:55:180
969699223,4cyclictest12-21ksoftirqd/020:30:300
969699222,5cyclictest12-21ksoftirqd/020:35:260
969699222,4cyclictest12-21ksoftirqd/020:15:220
971599219,8cyclictest0-21swapper/219:10:332
971599217,12cyclictest131rcu_preempt19:25:272
9715992112,7cyclictest0-21swapper/220:00:282
9715992112,7cyclictest0-21swapper/220:00:282
971599211,12cyclictest10030-21tail20:50:192
9715992110,9cyclictest0-21swapper/220:25:182
970499219,9cyclictest20455-21cron23:20:011
970499212,8cyclictest131rcu_preempt21:35:041
970499212,3cyclictest131rcu_preempt23:05:141
970499212,14cyclictest15100-21ssh00:13:431
970499211,3cyclictest131rcu_preempt20:45:131
970499210,9cyclictest9729-21latency_hist19:10:031
970499210,13cyclictest131rcu_preempt23:54:531
969699214,3cyclictest131rcu_preempt20:25:020
969699213,5cyclictest12-21ksoftirqd/019:35:130
969699211,5cyclictest12-21ksoftirqd/021:00:200
972499203,14cyclictest0-21swapper/300:35:133
972499203,11cyclictest0-21swapper/323:32:253
9724992011,6cyclictest29566irq/58-eth020:05:133
972499201,12cyclictest0-21swapper/321:35:013
971599209,7cyclictest27-21ksoftirqd/219:55:142
971599206,3cyclictest131rcu_preempt19:30:132
971599203,14cyclictest26346irq/40-dwc2_hso19:50:142
971599201,9cyclictest25390-21kthreadcore21:05:212
970499202,5cyclictest131rcu_preempt23:59:591
970499201,9cyclictest17234-21kthreadcore19:50:201
970499201,8cyclictest11362-21kthreadcore20:50:211
970499201,2cyclictest131rcu_preempt23:40:231
970499201,2cyclictest131rcu_preempt21:05:301
970499201,15cyclictest29129-21apt-get19:30:131
9704992011,5cyclictest211ksoftirqd/120:35:131
970499201,12cyclictest131rcu_preempt22:37:491
972499193,11cyclictest0-21swapper/322:10:213
972499193,11cyclictest0-21swapper/322:10:213
972499192,12cyclictest0-21swapper/322:15:163
972499192,12cyclictest0-21swapper/322:15:153
972499192,11cyclictest0-21swapper/322:45:053
972499192,10cyclictest0-21swapper/319:40:043
972499191,12cyclictest0-21swapper/319:35:153
972499191,12cyclictest0-21swapper/319:30:023
9724991910,6cyclictest29566irq/58-eth023:25:143
971599192,14cyclictest9728-21latency_hist19:10:022
971599191,8cyclictest8396-21grep20:45:282
970499192,4cyclictest131rcu_preempt19:35:211
970499191,9cyclictest131rcu_preempt19:20:171
970499191,2cyclictest131rcu_preempt20:05:151
9704991912,5cyclictest0-21swapper/119:45:301
970499191,15cyclictest14817-21uniq19:15:151
970499191,12cyclictest0-21swapper/120:40:171
972499189,6cyclictest0-21swapper/322:42:433
972499183,11cyclictest0-21swapper/323:21:243
972499182,10cyclictest0-21swapper/320:50:143
972499182,10cyclictest0-21swapper/320:00:153
972499182,10cyclictest0-21swapper/320:00:143
972499181,13cyclictest0-21swapper/320:25:173
972499181,11cyclictest0-21swapper/319:10:233
9724991810,6cyclictest0-21swapper/323:55:143
9724991810,6cyclictest0-21swapper/323:45:023
9724991810,6cyclictest0-21swapper/321:23:313
9724991810,5cyclictest0-21swapper/321:20:023
971599181,8cyclictest12223-21kthreadcore19:45:212
970499183,7cyclictest131rcu_preempt20:00:201
970499183,7cyclictest131rcu_preempt20:00:201
970499183,3cyclictest0-21swapper/119:10:191
970499181,9cyclictest8164-21fschecks_time20:15:181
970499181,9cyclictest6841-21sed19:40:201
970499181,3cyclictest131rcu_preempt19:25:261
9704991812,4cyclictest0-21swapper/120:55:151
9704991810,5cyclictest0-21swapper/120:10:171
970499180,2cyclictest131rcu_preempt20:30:191
972499179,6cyclictest0-21swapper/321:10:173
972499172,7cyclictest0-21swapper/300:20:153
972499172,11cyclictest0-21swapper/300:05:173
972499172,10cyclictest0-21swapper/321:00:163
972499172,10cyclictest0-21swapper/320:35:153
972499172,10cyclictest0-21swapper/319:50:153
972499172,10cyclictest0-21swapper/319:20:153
972499171,12cyclictest0-21swapper/321:35:153
972499171,10cyclictest0-21swapper/319:55:053
970499171,3cyclictest131rcu_preempt20:20:221
972499166,7cyclictest29566irq/58-eth022:50:133
970499161,8cyclictest21109-21fschecks_count19:55:161
972499151,12cyclictest0-21swapper/320:35:013
972499151,11cyclictest0-21swapper/320:55:163
969699158,5cyclictest12-21ksoftirqd/019:10:020
972499143,7cyclictest0-21swapper/319:15:123
972499132,6cyclictest0-21swapper/320:10:153
972499131,10cyclictest0-21swapper/320:40:213
972499112,6cyclictest0-21swapper/319:45:153
97249991,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional