You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 04:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sun Nov 30, 2025 00:45:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
489099684,54cyclictest3395-21ssh23:40:162
488299652,47cyclictest20441-21ssh22:30:131
489099561,31cyclictest11697-21ssh22:45:152
4882995241,3cyclictest23479-21diskmemload21:25:131
489899463,28cyclictest29566irq/58-eth023:20:133
489899463,28cyclictest29566irq/58-eth023:20:133
489099462,29cyclictest3666-21ssh23:00:142
487099461,31cyclictest11742-21ssh00:05:140
489099442,33cyclictest19651-21ssh23:10:132
487099443,28cyclictest2763666ssh00:15:130
488299425,21cyclictest131rcu_preempt22:10:131
4890994114,19cyclictest27-21ksoftirqd/221:00:162
488299413,25cyclictest27590-21ssh23:35:131
488299412,8cyclictest0-21swapper/121:55:151
487099411,35cyclictest29244-21ssh22:15:160
489099404,19cyclictest26346irq/40-dwc2_hso22:35:132
487099403,33cyclictest27615-21ssh23:55:130
4882993722,6cyclictest131rcu_preempt19:55:131
4870993721,14cyclictest12-21ksoftirqd/000:10:140
4898993628,5cyclictest29566irq/58-eth019:35:163
488299363,22cyclictest3163-21ssh00:20:141
4882993620,10cyclictest0-21swapper/123:20:131
4882993620,10cyclictest0-21swapper/123:20:131
487099362,4cyclictest131rcu_preempt20:40:170
487099362,4cyclictest131rcu_preempt20:40:160
4870993618,13cyclictest22354-21apt-get21:50:140
489899353,20cyclictest29566irq/58-eth000:00:163
489899351,19cyclictest29566irq/58-eth023:45:153
4870993519,5cyclictest131rcu_preempt23:45:140
487099351,29cyclictest19657-21ssh23:10:140
489899343,21cyclictest29566irq/58-eth021:15:153
489899342,18cyclictest29566irq/58-eth021:25:133
4890993416,13cyclictest0-21swapper/220:15:132
488299343,24cyclictest4581-21ssh22:20:141
489099332,15cyclictest13934-21ssh21:44:322
487099338,13cyclictest12-21ksoftirqd/019:55:150
487099335,20cyclictest27725-21ssh23:15:130
489099323,20cyclictest26346irq/40-dwc2_hso19:20:132
489099322,23cyclictest31972-21ssh21:15:152
489099322,16cyclictest20452-21ssh22:30:142
489899313,19cyclictest29566irq/58-eth022:30:123
4898993116,7cyclictest29566irq/58-eth023:40:163
4890993111,14cyclictest27-21ksoftirqd/222:25:122
4870993119,7cyclictest0-21swapper/019:15:160
489899306,17cyclictest29566irq/58-eth000:30:143
489899301,14cyclictest29566irq/58-eth022:20:143
489099303,18cyclictest26346irq/40-dwc2_hso22:10:132
4890993024,2cyclictest27-21ksoftirqd/222:15:152
4890993017,8cyclictest27-21ksoftirqd/200:27:332
4890993011,16cyclictest27600-21ssh23:35:142
487099307,7cyclictest12-21ksoftirqd/023:25:230
489099293,21cyclictest0-21swapper/221:56:322
488299298,13cyclictest0-21swapper/123:30:111
488299292,22cyclictest24476-21ssh23:10:321
4882992920,7cyclictest21-21ksoftirqd/100:30:141
487099295,5cyclictest8504-21find22:20:300
487099294,6cyclictest12-21ksoftirqd/023:30:190
4898992814,9cyclictest29566irq/58-eth021:55:153
489099285,13cyclictest26346irq/40-dwc2_hso20:20:162
489099282,21cyclictest932-21snmpd00:34:322
489099282,21cyclictest776-21ssh23:16:322
489099282,19cyclictest19581-21ssh00:10:042
489099282,19cyclictest0-21swapper/220:35:142
489099282,11cyclictest10985-21ssh22:44:322
4890992810,15cyclictest31630-21cron00:40:012
488299282,20cyclictest17647-21ssh23:27:311
487099283,7cyclictest12-21ksoftirqd/023:05:130
487099283,5cyclictest12-21ksoftirqd/000:35:020
487099283,18cyclictest19461-21ssh23:50:140
487099282,20cyclictest7884-21usage_EthcTask23:20:330
487099282,20cyclictest7884-21usage_EthcTask23:20:320
487099282,20cyclictest15843-21grep22:45:320
4870992816,4cyclictest3392-21apt-get23:40:160
489899273,16cyclictest0-21swapper/322:45:323
489099274,15cyclictest26346irq/40-dwc2_hso19:25:142
489099273,9cyclictest0-21swapper/200:23:322
489099273,15cyclictest26346irq/40-dwc2_hso00:00:152
489099272,19cyclictest0-21swapper/223:25:162
489099272,19cyclictest0-21swapper/220:50:152
489099272,17cyclictest26346irq/40-dwc2_hso19:10:142
488299274,10cyclictest21-21ksoftirqd/121:35:141
488299272,19cyclictest9154-21rm23:01:311
487099279,12cyclictest6406-21apt-get19:45:150
487099274,8cyclictest22919-21ntp_192.168.11419:25:240
487099274,5cyclictest12-21ksoftirqd/021:35:260
487099274,4cyclictest12-21ksoftirqd/000:25:290
487099272,19cyclictest10198-21sh22:43:180
4870992717,7cyclictest0-21swapper/020:05:150
489899261,14cyclictest29566irq/58-eth023:50:153
489099263,17cyclictest0-21swapper/223:20:332
489099263,17cyclictest0-21swapper/223:20:332
489099262,9cyclictest23479-21diskmemload22:52:322
489099262,21cyclictest0-21swapper/200:17:322
4890992613,10cyclictest27256-21ssh23:35:012
488299262,18cyclictest19405-21users21:45:321
488299260,19cyclictest21-21ksoftirqd/122:35:131
487099264,5cyclictest20962-21ntp_192.168.11420:30:270
487099263,5cyclictest12-21ksoftirqd/022:35:140
489899254,13cyclictest0-21swapper/323:10:143
489099256,13cyclictest26346irq/40-dwc2_hso19:45:152
489099253,15cyclictest26346irq/40-dwc2_hso20:05:162
489099251,9cyclictest304-21ssh22:55:322
489099251,14cyclictest842-21munin-node21:35:142
488299252,20cyclictest3705-21ssh22:40:131
487099256,5cyclictest12-21ksoftirqd/022:30:160
487099255,6cyclictest12-21ksoftirqd/022:05:180
487099253,5cyclictest12-21ksoftirqd/021:25:130
487099253,5cyclictest12-21ksoftirqd/020:50:120
487099253,5cyclictest12-21ksoftirqd/000:00:260
487099252,4cyclictest12-21ksoftirqd/000:25:000
487099251,8cyclictest23770-21grep22:10:220
487099251,6cyclictest12-21ksoftirqd/022:50:170
4870992510,11cyclictest31950-21apt-get21:15:140
487099250,18cyclictest0-21swapper/019:10:130
489099245,13cyclictest26346irq/40-dwc2_hso22:20:152
489099244,15cyclictest26346irq/40-dwc2_hso21:30:132
489099244,14cyclictest0-21swapper/219:30:162
489099243,17cyclictest26346irq/40-dwc2_hso21:10:152
489099243,15cyclictest26346irq/40-dwc2_hso23:05:132
489099243,14cyclictest26346irq/40-dwc2_hso20:00:162
489099240,16cyclictest31987-21find20:45:142
4882992416,5cyclictest21-21ksoftirqd/100:00:181
4882992415,6cyclictest0-21swapper/123:50:151
4882992414,3cyclictest21-21ksoftirqd/123:15:221
488299241,20cyclictest0-21swapper/121:15:131
487099245,5cyclictest12-21ksoftirqd/022:55:180
487099244,6cyclictest12-21ksoftirqd/022:25:300
487099244,6cyclictest12-21ksoftirqd/021:21:380
487099244,4cyclictest12-21ksoftirqd/021:59:120
487099243,7cyclictest12-21ksoftirqd/019:50:200
487099243,5cyclictest12-21ksoftirqd/021:40:210
487099243,4cyclictest12-21ksoftirqd/021:30:170
487099242,5cyclictest151rcuc/021:47:120
4870992417,3cyclictest27579-21apt-get23:35:140
4870992417,3cyclictest24318-21apt-get19:30:150
4870992410,5cyclictest0-21swapper/022:00:150
489099233,11cyclictest0-21swapper/220:25:132
489099232,16cyclictest0-21swapper/221:53:322
489099232,15cyclictest26346irq/40-dwc2_hso20:10:162
489099231,3cyclictest0-21swapper/223:45:152
489099231,18cyclictest14582-21acpi21:45:132
488299236,8cyclictest0-21swapper/122:55:141
488299231,16cyclictest3409-21ssh23:40:171
488299231,14cyclictest131rcu_preempt21:32:211
4882992310,7cyclictest0-21swapper/100:05:151
487099234,4cyclictest12-21ksoftirqd/020:45:220
487099234,14cyclictest629-21Async14:00:480
487099234,14cyclictest629-21Async14:00:480
487099232,5cyclictest12-21ksoftirqd/020:15:170
487099232,4cyclictest12-21ksoftirqd/019:20:230
487099232,4cyclictest12-21ksoftirqd/000:35:240
4870992316,3cyclictest22364-21apt-get20:35:140
4870992314,6cyclictest4054-21sh23:00:190
487099231,19cyclictest78681apt20:20:140
487099230,6cyclictest12-21ksoftirqd/021:10:150
489899227,10cyclictest0-21swapper/323:25:323
489899223,12cyclictest0-21swapper/323:07:323
4898992212,6cyclictest29566irq/58-eth021:40:143
4898992211,6cyclictest0-21swapper/322:50:323
489099223,16cyclictest10299-21kthreadcore20:55:212
489099222,14cyclictest26346irq/40-dwc2_hso20:30:162
489099221,18cyclictest15273-21wc00:05:252
489099221,11cyclictest16397-21ssh21:25:202
488299222,14cyclictest7770-21ssh21:20:161
488299221,2cyclictest131rcu_preempt23:46:431
488299221,2cyclictest131rcu_preempt22:26:291
4882992212,3cyclictest0-21swapper/122:18:141
488299221,18cyclictest17431-21timerandwakeup22:05:301
488299221,15cyclictest22376-21ssh21:50:151
488299221,15cyclictest14868-21ls19:50:291
488299221,11cyclictest0-21swapper/123:56:401
488299220,3cyclictest131rcu_preempt20:55:171
487099224,6cyclictest12-21ksoftirqd/020:10:150
487099224,6cyclictest12-21ksoftirqd/020:00:220
487099223,4cyclictest12-21ksoftirqd/020:55:290
487099221,6cyclictest12-21ksoftirqd/021:00:160
489899213,13cyclictest0-21swapper/323:32:313
4898992112,6cyclictest0-21swapper/321:50:293
489899211,15cyclictest0-21swapper/320:30:143
4898992110,8cyclictest29566irq/58-eth023:35:143
489099218,5cyclictest18956-21cpuspeed21:05:172
489099218,5cyclictest18956-21cpuspeed21:05:162
489099215,13cyclictest5340-21timerwakeupswit19:40:312
489099215,13cyclictest5340-21timerwakeupswit19:40:302
489099211,17cyclictest31150-21ssh23:55:242
489099211,10cyclictest8162-21kthreadcore22:00:242
489099210,19cyclictest19589-21fschecks_count23:50:172
489099210,14cyclictest7897-21cut21:20:182
488299211,8cyclictest17453-21latency_hist00:30:021
488299211,16cyclictest19883-21sh22:50:151
488299210,9cyclictest131rcu_preempt22:49:551
487099213,4cyclictest12-21ksoftirqd/021:05:280
487099213,4cyclictest12-21ksoftirqd/021:05:280
487099212,5cyclictest12-21ksoftirqd/019:35:180
487099210,4cyclictest0-21swapper/020:25:220
489899209,6cyclictest0-21swapper/300:11:043
489899203,9cyclictest29566irq/58-eth023:00:153
489099209,8cyclictest0-21swapper/219:15:212
489099203,14cyclictest4924-21latency_hist19:10:022
489099202,9cyclictest17456-21kthreadcore19:55:212
489099202,4cyclictest0-21swapper/222:05:142
489099202,12cyclictest0-21swapper/219:35:142
488299209,2cyclictest21-21ksoftirqd/120:20:291
488299205,13cyclictest201rcuc/121:40:161
488299203,13cyclictest201rcuc/122:00:151
488299201,3cyclictest0-21swapper/120:50:151
4882992013,4cyclictest21-21ksoftirqd/121:10:221
4882992011,6cyclictest0-21swapper/100:10:281
489899199,8cyclictest0-21swapper/322:07:133
489899199,6cyclictest0-21swapper/300:20:143
489899194,8cyclictest0-21swapper/322:55:153
489899193,11cyclictest0-21swapper/321:48:323
489899193,11cyclictest0-21swapper/300:28:423
489899193,11cyclictest0-21swapper/300:09:333
489899192,12cyclictest0-21swapper/323:16:323
489899192,11cyclictest0-21swapper/319:10:153
4898991911,6cyclictest0-21swapper/322:40:213
4898991911,6cyclictest0-21swapper/300:15:163
4898991911,5cyclictest29566irq/58-eth021:35:143
489899191,13cyclictest0-21swapper/319:20:153
489899191,12cyclictest0-21swapper/322:35:163
4898991910,6cyclictest0-21swapper/322:01:243
4898991910,6cyclictest0-21swapper/321:32:573
489099191,9cyclictest11747-21irqrtprio19:50:202
489099191,14cyclictest27391-21tune2fs20:40:192
489099191,14cyclictest27391-21tune2fs20:40:192
488299193,3cyclictest131rcu_preempt20:10:181
488299192,4cyclictest131rcu_preempt20:05:231
488299192,3cyclictest131rcu_preempt23:05:191
488299191,9cyclictest4839-21latency19:40:251
488299191,9cyclictest4839-21latency19:40:241
488299191,3cyclictest0-21swapper/100:38:251
488299191,16cyclictest697-21kthreadcore20:45:201
488299190,2cyclictest131rcu_preempt20:25:261
489899189,6cyclictest0-21swapper/322:18:493
489899189,6cyclictest0-21swapper/321:21:553
489899182,11cyclictest29566irq/58-eth022:25:143
489899182,10cyclictest0-21swapper/320:45:163
489899182,10cyclictest0-21swapper/320:25:153
489899182,10cyclictest0-21swapper/320:05:163
489899182,10cyclictest0-21swapper/319:50:153
489899181,10cyclictest0-21swapper/321:05:023
488299187,3cyclictest21-21ksoftirqd/119:15:011
488299185,3cyclictest20005-21kthreadcore20:30:231
488299181,2cyclictest131rcu_preempt19:30:201
4882991810,6cyclictest1-21systemd19:22:341
4882991810,5cyclictest21-21ksoftirqd/120:35:201
488299180,3cyclictest131rcu_preempt19:40:011
488299180,11cyclictest131rcu_preempt00:17:141
489899179,5cyclictest29566irq/58-eth023:55:133
489899175,8cyclictest29566irq/58-eth020:40:173
489899175,8cyclictest29566irq/58-eth020:40:173
489899172,9cyclictest0-21swapper/319:30:173
489899172,11cyclictest0-21swapper/300:35:153
489899172,10cyclictest0-21swapper/322:10:143
489899172,10cyclictest0-21swapper/320:15:153
489899172,10cyclictest0-21swapper/319:15:173
4898991711,4cyclictest0-21swapper/321:10:193
489899171,12cyclictest0-21swapper/320:40:013
489899171,10cyclictest0-21swapper/320:55:163
489899171,10cyclictest0-21swapper/319:45:013
489899171,10cyclictest0-21swapper/319:45:013
489899171,10cyclictest0-21swapper/319:25:153
488299178,6cyclictest4829-21kthreadcore20:15:211
488299172,10cyclictest0-21swapper/119:15:171
488299171,5cyclictest20145-21irqstats19:25:191
4882991712,3cyclictest0-21swapper/120:40:171
4882991712,3cyclictest0-21swapper/120:40:161
488299171,13cyclictest6391-21find19:45:141
488299171,12cyclictest23476-21cron21:10:011
488299171,12cyclictest23476-21cron21:10:011
487099173,11cyclictest4917-21cat19:10:020
489899163,10cyclictest0-21swapper/320:20:043
488299162,2cyclictest131rcu_preempt21:00:151
488299161,9cyclictest20898-21find20:00:151
489899152,10cyclictest0-21swapper/320:10:163
489899151,11cyclictest0-21swapper/319:45:183
489899151,10cyclictest0-21swapper/321:10:023
489899142,10cyclictest0-21swapper/321:05:173
488299141,10cyclictest4956-21cat19:10:011
489899132,7cyclictest0-21swapper/320:50:143
489899121,9cyclictest0-21swapper/320:00:013
489899121,5cyclictest0-21swapper/320:00:303
48989991,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional