You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 10:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Wed Nov 19, 2025 00:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
898699603,43cyclictest30684-21ssh21:50:150
8986995918,29cyclictest12-21ksoftirqd/000:05:140
9011995633,14cyclictest29566irq/58-eth022:30:163
900399553,34cyclictest8192-21ssh22:15:152
900399522,27cyclictest22530-21ssh00:00:142
898699501,4cyclictest131rcu_preempt19:45:160
898699501,4cyclictest131rcu_preempt19:45:160
899399491,43cyclictest18081-21ssh22:40:161
899399443,32cyclictest13108-21ssh23:35:161
898699442,7cyclictest131rcu_preempt22:50:140
898699441,38cyclictest12293-21ssh21:20:140
898699440,10cyclictest131rcu_preempt19:15:150
899399432,32cyclictest5202-21ssh21:35:151
899399431,32cyclictest27432-21ssh23:05:151
898699432,34cyclictest20679-21ssh21:25:160
899399426,4cyclictest0-21swapper/121:40:151
899399425,29cyclictest16683-21ssh22:20:151
901199413,23cyclictest29566irq/58-eth023:50:133
900399411,21cyclictest11861-21ssh23:15:172
898699413,30cyclictest16674-21ssh22:20:140
8986994010,21cyclictest12-21ksoftirqd/020:20:140
898699392,28cyclictest21520-21ssh23:40:160
8986993919,13cyclictest12-21ksoftirqd/021:10:120
899399389,12cyclictest21-21ksoftirqd/122:15:151
899399381,27cyclictest26582-21ssh22:45:181
901199376,15cyclictest29566irq/58-eth023:25:153
901199375,24cyclictest29566irq/58-eth023:55:173
900399372,25cyclictest29170-21ssh21:30:162
900399372,18cyclictest18682-21ssh23:36:362
8986993523,5cyclictest12-21ksoftirqd/023:00:150
899399342,24cyclictest22520-21ssh00:20:151
8986993415,15cyclictest12-21ksoftirqd/021:05:160
901199332,15cyclictest29566irq/58-eth000:10:143
900399332,28cyclictest3380-21ssh00:05:362
900399332,18cyclictest30754-21ssh22:08:362
9003993311,18cyclictest27-21ksoftirqd/200:35:152
898699331,23cyclictest11841-21ssh23:15:150
901199322,21cyclictest29566irq/58-eth000:20:133
900399322,26cyclictest13137-21rm21:57:352
8993993222,5cyclictest0-21swapper/121:20:131
899399322,23cyclictest3439-21ssh22:10:291
898699326,11cyclictest26566-21ssh22:45:170
900399313,22cyclictest0-21swapper/223:43:362
900399313,18cyclictest26346irq/40-dwc2_hso22:10:142
900399312,7cyclictest0-21swapper/200:22:452
899399313,21cyclictest3487-21ssh23:10:221
899399312,25cyclictest5361-21ssh00:30:141
899399312,22cyclictest20089-21ssh00:16:361
899399311,27cyclictest14150-21ssh23:55:171
899399310,3cyclictest131rcu_preempt23:15:171
898699314,17cyclictest28250-21kworker/0:119:30:170
898699312,22cyclictest27427-21ssh00:00:360
901199303,10cyclictest0-21swapper/322:08:363
9003993012,12cyclictest261rcuc/222:30:142
8993993020,4cyclictest0-21swapper/100:10:161
898699302,22cyclictest11224-21ssh21:37:110
8986993014,8cyclictest8976-21cyclictest22:05:150
9011992917,8cyclictest29566irq/58-eth022:50:153
9011992916,10cyclictest29566irq/58-eth022:00:183
9011992916,10cyclictest29566irq/58-eth022:00:173
900399296,14cyclictest27-21ksoftirqd/220:00:172
900399293,21cyclictest261rcuc/221:10:142
900399293,21cyclictest0-21swapper/223:48:362
900399292,18cyclictest26346irq/40-dwc2_hso19:30:182
8993992918,6cyclictest201rcuc/122:30:151
898699294,19cyclictest20272-21sh23:20:160
898699294,19cyclictest20272-21sh23:20:150
898699292,18cyclictest30061-21ssh00:25:160
900399284,15cyclictest26346irq/40-dwc2_hso00:15:152
900399283,20cyclictest0-21swapper/221:38:362
900399283,18cyclictest26346irq/40-dwc2_hso19:15:152
900399282,10cyclictest11653-21ssh00:10:372
900399281,18cyclictest0-21swapper/222:25:142
900399281,15cyclictest10730-21ssh22:55:152
900399281,11cyclictest2452566ssh21:45:242
899399282,21cyclictest30248-21ssh00:04:361
899399282,20cyclictest15300-21ssh22:00:181
899399282,20cyclictest15300-21ssh22:00:181
899399282,16cyclictest21-21ksoftirqd/121:30:161
8993992812,11cyclictest28718-21ssh23:25:151
898699284,18cyclictest12049-21sh23:52:390
898699282,21cyclictest25110-21ssh22:25:150
898699282,20cyclictest0-21swapper/000:17:360
898699282,19cyclictest3846-21ssh21:15:150
901199273,16cyclictest0-21swapper/321:52:143
901199273,16cyclictest0-21swapper/300:05:363
900399272,18cyclictest26346irq/40-dwc2_hso20:05:142
900399272,16cyclictest19114-21ssh23:00:152
899399272,19cyclictest13136-21sh21:57:361
898699273,18cyclictest13150-21ssh00:35:170
898699272,19cyclictest28532-21ssh00:22:440
898699272,19cyclictest2754-21ssh21:32:090
898699272,19cyclictest20795-21ssh23:39:360
901199269,11cyclictest29566irq/58-eth023:15:173
901199263,16cyclictest0-21swapper/321:57:363
901199263,15cyclictest0-21swapper/321:37:113
901199263,15cyclictest0-21swapper/321:30:163
900399265,15cyclictest26346irq/40-dwc2_hso20:35:152
900399263,17cyclictest0-21swapper/221:22:092
900399263,16cyclictest26346irq/40-dwc2_hso23:25:152
900399263,15cyclictest26346irq/40-dwc2_hso21:00:142
900399263,14cyclictest26346irq/40-dwc2_hso20:10:162
900399262,16cyclictest26346irq/40-dwc2_hso20:40:182
900399261,16cyclictest26346irq/40-dwc2_hso23:10:212
900399261,15cyclictest9649-21ssh22:35:162
8986992611,8cyclictest12-21ksoftirqd/019:20:150
898699261,17cyclictest18057-21ssh22:40:140
901199252,15cyclictest29666irq/59-eth023:05:143
900399254,15cyclictest26346irq/40-dwc2_hso20:20:142
901199242,12cyclictest29566irq/58-eth021:20:133
901199241,16cyclictest0-21swapper/322:15:153
900399248,13cyclictest271ksoftirqd/221:40:142
900399242,17cyclictest26346irq/40-dwc2_hso23:50:132
900399242,14cyclictest26346irq/40-dwc2_hso19:45:162
900399242,14cyclictest26346irq/40-dwc2_hso19:45:162
900399242,13cyclictest0-21swapper/219:25:152
900399241,20cyclictest19315-21sed19:20:202
8993992412,4cyclictest21-21ksoftirqd/123:43:181
899399241,18cyclictest12026-21ssh23:52:381
899399241,15cyclictest662-21ntpd21:00:151
8993992411,4cyclictest21-21ksoftirqd/119:35:161
898699242,8cyclictest2690-21ssh23:09:360
901199239,9cyclictest0-21swapper/322:45:263
901199234,15cyclictest29566irq/58-eth019:25:153
901199234,13cyclictest0-21swapper/323:35:163
901199233,13cyclictest0-21swapper/322:40:163
9011992316,5cyclictest29566irq/58-eth000:00:143
900399233,16cyclictest26346irq/40-dwc2_hso23:05:142
900399232,6cyclictest0-21swapper/200:29:362
900399232,16cyclictest0-21swapper/223:55:352
900399232,16cyclictest0-21swapper/222:44:352
900399232,13cyclictest27625-21diskmemload21:15:142
9003992312,5cyclictest25148-21ssh23:20:382
9003992312,5cyclictest25148-21ssh23:20:382
900399230,9cyclictest24730-21latency_hist22:25:012
899399239,4cyclictest21-21ksoftirqd/121:53:031
898699231,17cyclictest23409-21ls22:05:010
898699231,17cyclictest23409-21ls22:05:010
901199223,13cyclictest0-21swapper/300:15:153
901199223,11cyclictest0-21swapper/322:20:163
901199222,13cyclictest29566irq/58-eth023:45:153
9011992210,7cyclictest321rcuc/321:25:163
900399229,10cyclictest5342-21cut23:30:192
900399228,11cyclictest11738-21cron20:20:012
900399224,8cyclictest32061-21sh21:50:222
900399222,16cyclictest26346irq/40-dwc2_hso21:25:162
900399221,17cyclictest5794-21ssh22:50:232
900399221,17cyclictest19394-21grep19:50:312
900399221,11cyclictest12974-21grep19:10:282
899399223,9cyclictest21-21ksoftirqd/100:39:051
8993992211,3cyclictest0-21swapper/121:27:571
8993992210,3cyclictest21-21ksoftirqd/123:30:231
898699221,15cyclictest21403-21latency_hist20:30:020
898699221,14cyclictest12-21ksoftirqd/021:55:130
901199219,9cyclictest0-21swapper/322:11:083
901199212,15cyclictest29566irq/58-eth022:25:143
9011992116,3cyclictest321rcuc/323:12:083
9011992112,6cyclictest29566irq/58-eth023:40:163
9011992111,6cyclictest0-21swapper/321:40:233
900399213,15cyclictest26756-21sort22:45:202
900399211,9cyclictest7587-21grep20:45:312
900399211,15cyclictest27618-21munin-plugin-st21:10:012
900399211,11cyclictest6949-21kthreadcore00:30:222
900399211,11cyclictest5652-21latency_hist19:40:022
899399218,9cyclictest29806-21apt23:45:151
899399218,5cyclictest21-21ksoftirqd/120:20:231
899399218,4cyclictest21-21ksoftirqd/122:35:311
899399217,8cyclictest21-21ksoftirqd/122:54:321
899399217,6cyclictest21-21ksoftirqd/122:25:181
899399217,4cyclictest21-21ksoftirqd/120:40:201
899399215,6cyclictest21-21ksoftirqd/119:15:251
899399213,4cyclictest21-21ksoftirqd/100:06:321
899399213,3cyclictest131rcu_preempt19:25:261
899399212,16cyclictest21865-21ls20:30:181
899399212,16cyclictest21865-21ls20:30:181
8993992111,7cyclictest27052-21munin-run23:05:021
899399211,17cyclictest26701-21ls20:35:181
899399211,16cyclictest27976-21find21:10:131
898699217,4cyclictest131rcu_preempt23:25:190
898699211,16cyclictest840-21ls23:45:230
901199202,15cyclictest29566irq/58-eth021:10:123
901199201,9cyclictest0-21swapper/319:40:133
9011992015,3cyclictest321rcuc/322:37:163
9011992015,3cyclictest321rcuc/321:19:593
900399209,9cyclictest22878-21ssh22:04:272
900399209,9cyclictest22878-21ssh22:04:272
900399201,9cyclictest26234-21cron20:35:012
900399201,9cyclictest26234-21cron20:35:012
900399201,9cyclictest21007-21kthreadcore19:55:202
900399201,16cyclictest12436-21head20:50:312
900399201,10cyclictest16984-21cut20:25:162
899399209,9cyclictest0-21swapper/123:21:121
899399209,9cyclictest0-21swapper/123:21:111
899399205,3cyclictest131rcu_preempt20:50:291
8993992014,4cyclictest0-21swapper/119:30:191
8993992012,6cyclictest3589-21sshd00:27:361
8993992011,6cyclictest14403-21ssh22:55:261
899399200,14cyclictest3789-21ls20:45:161
8986992011,7cyclictest8942-21ntp_kernel_pll_00:30:270
8986992011,6cyclictest0-21swapper/000:10:250
898699201,13cyclictest11651-21ls22:15:250
901199192,14cyclictest0-21swapper/320:10:163
9011991914,3cyclictest321rcuc/300:33:453
9011991913,4cyclictest321rcuc/321:48:033
9011991913,4cyclictest0-21swapper/319:10:263
901199191,13cyclictest0-21swapper/320:15:313
9011991910,6cyclictest0-21swapper/323:33:533
9011991910,6cyclictest0-21swapper/322:55:323
900399192,14cyclictest13550-21cut20:55:182
900399191,8cyclictest9015-21latency_hist19:10:012
900399191,14cyclictest5951-21sort19:40:162
899399192,5cyclictest21-21ksoftirqd/120:25:181
8993991913,4cyclictest0-21swapper/121:45:151
8993991911,5cyclictest27795-21grep22:05:311
899399190,16cyclictest13854-21ls19:15:001
898699198,8cyclictest7164-21ntpq23:10:350
898699193,9cyclictest0-21swapper/021:42:120
898699191,9cyclictest20666-21irqcore19:55:180
898699191,6cyclictest27042-21tail20:00:240
898699191,6cyclictest1146-21diskstats22:30:170
8986991911,6cyclictest1-21systemd23:57:380
8986991911,5cyclictest12-21ksoftirqd/020:05:150
898699191,13cyclictest9648-21sh22:35:160
898699190,16cyclictest26240-21latency_hist20:35:010
898699190,16cyclictest26240-21latency_hist20:35:010
901199187,7cyclictest0-21swapper/323:20:153
901199187,7cyclictest0-21swapper/323:20:153
901199185,9cyclictest29566irq/58-eth020:05:143
901199183,8cyclictest0-21swapper/300:35:173
901199181,13cyclictest0-21swapper/319:45:313
901199181,13cyclictest0-21swapper/319:45:313
9011991810,6cyclictest0-21swapper/323:04:473
899399187,8cyclictest18897-21df19:20:161
899399184,7cyclictest0-21swapper/120:15:161
8993991812,4cyclictest21-21ksoftirqd/119:50:161
8993991812,4cyclictest0-21swapper/120:55:171
8993991811,5cyclictest0-21swapper/119:55:261
8993991810,6cyclictest10344-21ssh21:17:501
898699189,6cyclictest18698-21sendmail23:00:010
898699182,14cyclictest6578-21kthreadcore19:40:190
898699182,13cyclictest11750-21munin-run20:20:020
8986991811,5cyclictest0-21swapper/019:50:280
898699181,13cyclictest8362-21ssh23:30:260
898699181,12cyclictest5622-21awk19:40:000
8986991810,6cyclictest28253-21cron19:30:000
901199177,4cyclictest0-21swapper/319:15:153
901199172,9cyclictest0-21swapper/320:20:153
901199172,9cyclictest0-21swapper/319:35:043
901199172,11cyclictest0-21swapper/300:25:163
9011991712,3cyclictest0-21swapper/319:50:243
899399177,7cyclictest7778-21kthreadcore19:40:211
899399172,11cyclictest593-21kthreadcore20:05:231
899399172,10cyclictest0-21swapper/120:10:171
8993991712,3cyclictest0-21swapper/121:05:211
898699178,7cyclictest7797-21cron22:15:010
898699173,12cyclictest28881-21kthreadcore20:35:230
898699173,10cyclictest31303-21cpu20:40:180
898699170,10cyclictest23282-21kthreadcore21:45:220
901199162,9cyclictest0-21swapper/320:40:183
901199162,9cyclictest0-21swapper/320:30:153
901199162,9cyclictest0-21swapper/320:30:153
901199162,9cyclictest0-21swapper/320:00:183
901199162,10cyclictest0-21swapper/320:50:163
9011991611,3cyclictest0-21swapper/321:08:363
9011991611,3cyclictest0-21swapper/321:00:303
9011991610,4cyclictest0-21swapper/320:35:293
9011991610,4cyclictest0-21swapper/319:55:183
8993991611,3cyclictest0-21swapper/120:00:191
8993991611,3cyclictest0-21swapper/119:45:181
8993991611,3cyclictest0-21swapper/119:45:181
898699167,7cyclictest9346-21perl19:10:160
898699165,8cyclictest20731-21cut21:00:220
898699163,10cyclictest2756-21irqcore20:10:210
898699161,7cyclictest13564-21fschecks_time20:55:180
898699161,11cyclictest8617-21diskstats20:50:180
901199158,5cyclictest0-21swapper/319:30:473
9011991510,3cyclictest0-21swapper/320:55:173
9011991510,3cyclictest0-21swapper/320:45:273
9011991510,3cyclictest0-21swapper/320:25:303
9011991510,3cyclictest0-21swapper/319:20:313
898699156,7cyclictest3770-21sort20:45:160
898699152,11cyclictest9042-21latency_hist19:10:010
899399142,10cyclictest8976-21cyclictest19:10:011
901199111,7cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional