You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-16 - 04:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun Nov 16, 2025 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
118699603,45cyclictest18179-21ssh00:05:152
116899593,43cyclictest23839-21ssh22:50:160
1186995435,13cyclictest8976-21sh23:20:152
118699542,35cyclictest15441-21ssh22:45:142
118699542,35cyclictest15441-21ssh22:45:132
118699541,34cyclictest32188-21ssh22:55:162
1186994512,25cyclictest27-21ksoftirqd/223:00:132
119599425,24cyclictest29566irq/58-eth022:10:143
116899402,24cyclictest11946-21ssh21:25:120
119599395,21cyclictest29566irq/58-eth000:05:153
119599393,22cyclictest29566irq/58-eth000:35:153
118699392,30cyclictest14566-21ssh22:25:162
116899391,32cyclictest20192-21ssh21:30:140
117799384,17cyclictest8732-21ssh00:20:141
118699374,20cyclictest26346irq/40-dwc2_hso19:55:162
118699362,18cyclictest19806-21diskmemload22:41:482
117799363,24cyclictest4266-21ssh21:40:171
117799362,30cyclictest29229-21ssh21:55:151
117799362,30cyclictest29229-21ssh21:55:151
119599353,22cyclictest29566irq/58-eth000:00:143
119599342,19cyclictest29566irq/58-eth023:40:143
116899344,19cyclictest12-21ksoftirqd/023:05:120
1195993318,8cyclictest29566irq/58-eth000:25:163
119599331,20cyclictest29566irq/58-eth022:30:133
118699332,26cyclictest4459-21ssh22:18:052
117799332,4cyclictest131rcu_preempt20:15:151
117799331,10cyclictest21-21ksoftirqd/123:40:131
116899337,18cyclictest12-21ksoftirqd/019:10:150
116899332,26cyclictest8737-21ssh00:20:150
116899331,29cyclictest0-21swapper/022:20:160
119599323,12cyclictest29566irq/58-eth023:00:143
118699323,20cyclictest26346irq/40-dwc2_hso21:00:132
118699323,17cyclictest26346irq/40-dwc2_hso20:40:152
116899322,21cyclictest12-21ksoftirqd/019:45:160
118699314,18cyclictest26346irq/40-dwc2_hso19:20:142
118699311,18cyclictest20827-21ssh21:50:152
116899314,18cyclictest12-21ksoftirqd/019:30:130
116899311,21cyclictest558-21ssh23:15:140
1195993015,10cyclictest29566irq/58-eth022:40:153
118699303,22cyclictest0-21swapper/222:38:482
118699302,20cyclictest26346irq/40-dwc2_hso23:45:152
1177993019,7cyclictest21-21ksoftirqd/100:15:151
1168993010,6cyclictest131rcu_preempt21:00:130
116899300,22cyclictest1732-21ssh23:55:150
119599294,14cyclictest29566irq/58-eth019:35:163
119599291,11cyclictest29566irq/58-eth022:05:143
118699294,15cyclictest26346irq/40-dwc2_hso00:35:152
118699293,11cyclictest0-21swapper/221:43:482
118699292,21cyclictest30943-21ssh22:53:492
118699292,21cyclictest0-21swapper/222:10:152
118699292,20cyclictest0-21swapper/200:15:162
116899296,3cyclictest131rcu_preempt00:00:130
119599283,19cyclictest0-21swapper/322:53:093
118699283,19cyclictest0-21swapper/223:50:152
118699283,19cyclictest0-21swapper/223:50:152
118699283,15cyclictest26346irq/40-dwc2_hso19:40:162
118699282,21cyclictest27922-21ssh22:30:482
118699282,10cyclictest15860-21ssh00:24:482
117799286,14cyclictest201rcuc/122:10:141
117799281,24cyclictest31234-21ssh22:35:141
116899281,19cyclictest8041-21sh23:00:130
116899281,14cyclictest0-21swapper/020:15:150
119599272,18cyclictest0-21swapper/300:19:483
118699274,16cyclictest26346irq/40-dwc2_hso23:05:122
118699272,9cyclictest0-21swapper/220:45:172
118699272,9cyclictest0-21swapper/220:45:172
118699272,19cyclictest0-21swapper/220:05:152
118699272,19cyclictest0-21swapper/219:50:142
117799275,18cyclictest25805-21ssh00:10:171
116899275,5cyclictest131rcu_preempt22:00:160
116899275,12cyclictest131rcu_preempt21:45:160
116899272,19cyclictest29527-21rm22:33:060
116899271,19cyclictest14555-21ssh22:25:150
119599265,15cyclictest0-21swapper/300:10:173
119599262,14cyclictest29566irq/58-eth020:40:153
118699269,8cyclictest4202-21rm23:55:222
118699262,18cyclictest0-21swapper/220:55:182
118699262,15cyclictest27-21ksoftirqd/220:30:182
118699262,15cyclictest26346irq/40-dwc2_hso20:00:122
118699261,15cyclictest20313-21ssh21:30:172
117799268,3cyclictest21-21ksoftirqd/121:28:311
117799263,19cyclictest30779-21ssh00:35:151
1177992614,3cyclictest21-21ksoftirqd/119:50:311
1177992611,3cyclictest131rcu_preempt23:00:131
119599252,17cyclictest0-21swapper/322:28:053
118699254,16cyclictest26346irq/40-dwc2_hso23:10:152
118699252,19cyclictest0-21swapper/221:15:152
118699251,9cyclictest12371-21ssh21:45:042
118699251,15cyclictest26346irq/40-dwc2_hso00:30:162
118699251,14cyclictest16342-21diskmemload00:25:162
118699250,22cyclictest942-21ls23:35:022
118699250,20cyclictest17433-21cut23:25:202
116899254,15cyclictest12-21ksoftirqd/022:10:140
119599248,11cyclictest0-21swapper/322:15:483
119599243,13cyclictest0-21swapper/321:32:483
119599242,15cyclictest0-21swapper/323:55:153
118699243,16cyclictest26346irq/40-dwc2_hso21:20:142
118699243,16cyclictest26346irq/40-dwc2_hso21:20:142
118699243,13cyclictest26346irq/40-dwc2_hso19:35:162
118699242,20cyclictest0-21swapper/221:25:142
118699242,17cyclictest0-21swapper/223:38:342
118699242,16cyclictest0-21swapper/220:20:162
118699242,16cyclictest0-21swapper/220:20:162
118699242,13cyclictest26346irq/40-dwc2_hso19:15:182
1186992411,8cyclictest6441-21fschecks_time22:20:192
117799246,9cyclictest21-21ksoftirqd/122:41:001
117799243,15cyclictest20711-21find19:30:121
117799242,20cyclictest8977-21ssh23:20:151
117799242,17cyclictest21-21ksoftirqd/120:45:151
117799242,17cyclictest21-21ksoftirqd/120:45:151
1177992413,3cyclictest21-21ksoftirqd/122:35:021
1177992412,8cyclictest0-21swapper/121:35:141
1177992411,4cyclictest19806-21diskmemload22:56:261
1177992410,4cyclictest21-21ksoftirqd/122:09:501
116899242,17cyclictest1294-21ssh23:35:170
119599233,9cyclictest0-21swapper/322:55:163
118699237,13cyclictest14865-21cut22:05:202
118699232,6cyclictest0-21swapper/221:39:482
118699232,16cyclictest0-21swapper/219:30:142
118699231,18cyclictest15983-21fschecks_count19:25:202
118699231,12cyclictest21817-21kthreadcore20:35:252
118699231,12cyclictest13752-21head23:40:302
118699231,11cyclictest10149-21ls00:00:172
117799239,3cyclictest21-21ksoftirqd/122:20:151
117799237,2cyclictest21-21ksoftirqd/119:10:331
1177992315,5cyclictest21-21ksoftirqd/123:55:131
1177992313,3cyclictest21-21ksoftirqd/123:27:281
116899234,15cyclictest13889-21kworker/u8:022:45:150
116899234,15cyclictest13889-21kworker/u8:022:45:150
116899231,13cyclictest25360-21acpi23:30:120
1168992310,8cyclictest12-21ksoftirqd/021:50:150
1195992212,7cyclictest29566irq/58-eth022:00:183
1195992211,8cyclictest0-21swapper/321:48:533
119599221,18cyclictest0-21swapper/319:55:203
119599221,13cyclictest0-21swapper/321:50:153
118699221,12cyclictest12925-21head20:25:312
118699221,11cyclictest3041-21sh23:15:222
118699221,10cyclictest5350-21fschecks_count22:00:192
118699220,18cyclictest29221-21apt-get21:55:152
118699220,18cyclictest29221-21apt-get21:55:152
118699220,11cyclictest20180-21ssh21:10:132
117799229,4cyclictest21-21ksoftirqd/123:05:191
117799225,2cyclictest21-21ksoftirqd/120:25:191
1177992212,3cyclictest21-21ksoftirqd/121:00:181
117799221,15cyclictest23794-21ssh21:30:271
117799221,13cyclictest30245-21ssh22:15:121
1177992210,9cyclictest21-21ksoftirqd/123:18:251
1177992210,9cyclictest10908-21cat00:00:201
1177992210,3cyclictest21-21ksoftirqd/123:35:211
117799220,3cyclictest8798-21ntp_192.168.11422:00:281
116899226,4cyclictest131rcu_preempt22:35:150
116899222,14cyclictest3888-21sh00:37:470
116899221,9cyclictest131rcu_preempt23:50:300
116899221,9cyclictest131rcu_preempt23:50:300
1168992213,6cyclictest12-21ksoftirqd/019:20:130
1168992213,6cyclictest121ksoftirqd/021:10:120
1168992210,8cyclictest13637-21diskstats22:05:160
119599213,6cyclictest0-21swapper/323:38:343
119599213,12cyclictest0-21swapper/300:24:483
1195992112,6cyclictest29566irq/58-eth021:55:153
1195992112,6cyclictest29566irq/58-eth021:55:153
118699211,16cyclictest880-21fschecks_time20:50:162
118699211,16cyclictest27122-21grep20:10:192
118699211,16cyclictest1332-21latency_hist19:10:022
118699211,15cyclictest3458-21kthreadcore19:10:242
118699211,11cyclictest5372-21kthreadcore19:45:242
118699211,11cyclictest25964-21sort00:10:202
118699211,10cyclictest986-21kthreadcore20:15:232
118699211,10cyclictest932-21snmpd21:05:182
117799219,3cyclictest21-21ksoftirqd/119:20:151
117799219,2cyclictest21-21ksoftirqd/119:45:241
117799218,3cyclictest21-21ksoftirqd/123:49:441
117799211,4cyclictest131rcu_preempt20:05:151
1177992113,5cyclictest3836-21sh21:20:121
1177992113,5cyclictest3836-21sh21:20:121
117799211,17cyclictest30090-21cron19:40:011
117799211,17cyclictest18170-21ssh00:05:141
116899215,9cyclictest1160-21cyclictest21:40:160
116899215,11cyclictest12-21ksoftirqd/021:55:150
116899215,11cyclictest12-21ksoftirqd/021:55:150
116899212,14cyclictest15924-21ssh23:23:320
116899211,4cyclictest131rcu_preempt22:40:250
116899211,4cyclictest131rcu_preempt19:40:300
116899211,4cyclictest131rcu_preempt00:25:150
116899210,14cyclictest13573-21ls23:40:300
1195992012,4cyclictest0-21swapper/322:35:143
117799209,3cyclictest21-21ksoftirqd/121:12:501
117799209,2cyclictest21-21ksoftirqd/122:26:181
117799209,2cyclictest21-21ksoftirqd/119:55:021
117799208,8cyclictest21-21ksoftirqd/120:10:151
117799208,3cyclictest21-21ksoftirqd/121:48:591
117799208,2cyclictest21-21ksoftirqd/120:35:151
117799207,4cyclictest21-21ksoftirqd/120:20:191
117799207,4cyclictest21-21ksoftirqd/120:20:191
117799206,11cyclictest21-21ksoftirqd/123:10:151
117799204,2cyclictest131rcu_preempt19:15:161
117799201,15cyclictest28984-21ntp_192.168.11423:30:251
117799201,14cyclictest23795-21ssh00:30:181
1177992010,2cyclictest21-21ksoftirqd/121:50:271
1177992010,2cyclictest21-21ksoftirqd/121:15:291
116899207,9cyclictest2956-21ssh21:38:480
116899202,8cyclictest131rcu_preempt00:30:150
1168992013,5cyclictest0-21swapper/020:35:180
116899201,2cyclictest131rcu_preempt22:19:110
1168992012,3cyclictest12-21ksoftirqd/022:55:310
1168992011,5cyclictest0-21swapper/020:00:220
116899201,14cyclictest7783-21timerandwakeup21:20:290
116899201,14cyclictest7783-21timerandwakeup21:20:290
116899201,12cyclictest18162-21apt-get00:05:140
119599197,7cyclictest41-21kworker/3:119:30:143
119599193,11cyclictest0-21swapper/323:21:483
119599193,11cyclictest0-21swapper/322:21:483
119599193,11cyclictest0-21swapper/321:40:483
119599192,10cyclictest0-21swapper/321:00:143
119599191,12cyclictest0-21swapper/323:10:153
119599191,12cyclictest0-21swapper/320:55:323
1195991910,6cyclictest0-21swapper/323:19:543
1177991914,3cyclictest21-21ksoftirqd/122:50:241
1177991913,3cyclictest21-21ksoftirqd/122:45:151
1177991913,3cyclictest21-21ksoftirqd/122:45:151
1177991913,3cyclictest21-21ksoftirqd/120:40:271
117799191,2cyclictest131rcu_preempt00:25:481
117799191,15cyclictest15263-21cpuspeed21:05:151
117799191,10cyclictest29840-21ntp_states23:50:271
117799191,10cyclictest29840-21ntp_states23:50:271
116899199,8cyclictest793-21df20:50:150
116899199,8cyclictest12700-21perl19:55:200
116899199,7cyclictest17241-21apt-get23:25:170
116899197,9cyclictest17966-21apt-get23:45:140
116899192,11cyclictest1160-21cyclictest20:20:140
116899192,11cyclictest1160-21cyclictest20:20:140
116899191,3cyclictest131rcu_preempt00:15:270
1168991913,4cyclictest0-21swapper/019:50:130
116899191,14cyclictest32649-21apt23:15:010
119599189,6cyclictest0-21swapper/323:53:413
119599189,6cyclictest0-21swapper/323:53:413
119599189,6cyclictest0-21swapper/321:37:073
119599182,11cyclictest29566irq/58-eth021:10:133
119599182,10cyclictest0-21swapper/320:45:173
119599182,10cyclictest0-21swapper/320:45:173
119599182,10cyclictest0-21swapper/320:30:193
119599182,10cyclictest0-21swapper/320:20:163
119599182,10cyclictest0-21swapper/320:20:163
119599182,10cyclictest0-21swapper/320:10:173
119599182,10cyclictest0-21swapper/320:00:133
119599182,10cyclictest0-21swapper/319:15:173
119599181,9cyclictest0-21swapper/319:50:133
119599181,9cyclictest0-21swapper/319:40:163
1195991811,4cyclictest29566irq/58-eth021:15:153
119599181,11cyclictest0-21swapper/323:45:173
1195991810,6cyclictest0-21swapper/323:32:453
1195991810,6cyclictest0-21swapper/322:45:193
1195991810,6cyclictest0-21swapper/322:45:193
1195991810,6cyclictest0-21swapper/321:21:023
1195991810,6cyclictest0-21swapper/321:21:023
1195991810,5cyclictest29566irq/58-eth023:05:123
117799187,8cyclictest17853-21sed20:00:171
117799181,9cyclictest30565-21sed19:40:191
1177991812,3cyclictest21-21ksoftirqd/120:55:191
117799181,15cyclictest17534-21kthreadcore19:25:251
1177991811,4cyclictest21-21ksoftirqd/120:55:011
1177991810,6cyclictest18397-21sshd20:31:471
116899182,9cyclictest131rcu_preempt20:40:220
116899181,9cyclictest0-21swapper/020:30:200
116899181,5cyclictest131rcu_preempt00:10:170
116899181,4cyclictest0-21swapper/019:15:160
1168991812,4cyclictest0-21swapper/021:16:080
1168991812,4cyclictest0-21swapper/019:25:330
116899181,14cyclictest9386-21irqcore20:25:200
1168991810,6cyclictest12-21ksoftirqd/020:10:020
116899180,2cyclictest131rcu_preempt20:10:150
119599179,6cyclictest0-21swapper/323:26:113
119599172,9cyclictest0-21swapper/319:25:183
119599172,10cyclictest0-21swapper/320:50:143
119599172,10cyclictest0-21swapper/320:25:163
119599172,10cyclictest0-21swapper/320:05:153
119599172,10cyclictest0-21swapper/319:20:153
1195991712,3cyclictest0-21swapper/321:25:133
1195991711,4cyclictest0-21swapper/300:30:243
116899171,11cyclictest131rcu_preempt20:55:190
116899170,2cyclictest131rcu_preempt21:05:160
116899161,5cyclictest25553-21grep19:35:170
116899161,10cyclictest28724-21irqcore20:45:210
116899161,10cyclictest28724-21irqcore20:45:210
119599151,8cyclictest0-21swapper/320:15:163
117799151,10cyclictest1208-21cat19:10:011
116899151,11cyclictest1209-21latency_hist19:10:010
119599141,6cyclictest0-21swapper/319:45:293
119599131,5cyclictest0-21swapper/319:10:303
119599122,7cyclictest0-21swapper/319:10:023
119599121,4cyclictest0-21swapper/321:05:153
119599111,7cyclictest0-21swapper/320:35:193
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional