You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-06 - 20:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sat Dec 06, 2025 12:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2500799655,48cyclictest7739-21ssh10:10:121
25023995335,12cyclictest29566irq/58-eth009:35:153
25023995335,12cyclictest29566irq/58-eth009:35:153
2499899529,6cyclictest131rcu_preempt09:15:140
2499899512,40cyclictest31843-21ssh12:25:160
2500799481,33cyclictest3438-21ssh11:10:141
2499899483,35cyclictest8313-21ssh12:30:160
2499899483,35cyclictest8313-21ssh12:30:150
2500799462,32cyclictest27212-21ssh11:25:271
2501599441,25cyclictest13909-21ssh10:35:142
2500799431,34cyclictest17392-21ssh12:35:291
2500799431,2cyclictest0-21swapper/109:55:131
25007994311,17cyclictest21-21ksoftirqd/110:30:151
2502399414,20cyclictest29566irq/58-eth012:15:133
2502399403,21cyclictest29566irq/58-eth012:30:163
2502399403,21cyclictest29566irq/58-eth012:30:163
2500799401,29cyclictest29264-21ssh10:45:121
2500799400,5cyclictest131rcu_preempt09:05:151
24998993915,18cyclictest12-21ksoftirqd/007:25:130
2499899391,30cyclictest864-21ssh09:45:140
2499899381,33cyclictest10230-21ssh11:35:160
2500799371,29cyclictest8831-21ssh09:50:121
2499899373,26cyclictest17491-21ssh09:35:160
2499899373,26cyclictest17491-21ssh09:35:150
2500799352,5cyclictest131rcu_preempt11:45:151
2502399342,19cyclictest29566irq/58-eth011:15:123
2499899345,21cyclictest22894-21diskmemload12:20:110
2499899345,14cyclictest4372-21ssh10:50:140
2499899341,23cyclictest11340-21diskmemload10:15:140
2502399332,20cyclictest29566irq/58-eth011:50:143
25023993314,9cyclictest29566irq/58-eth011:10:123
2500799324,17cyclictest21-21ksoftirqd/107:45:131
2500799324,17cyclictest21-21ksoftirqd/107:45:121
2499899324,18cyclictest12-21ksoftirqd/011:50:130
2501599316,15cyclictest26346irq/40-dwc2_hso09:55:132
2501599315,16cyclictest26346irq/40-dwc2_hso10:45:142
2500799314,9cyclictest131rcu_preempt07:50:141
2500799311,25cyclictest18368-21ssh11:40:141
2501599303,21cyclictest0-21swapper/211:20:142
2501599303,16cyclictest26346irq/40-dwc2_hso11:40:142
2502399294,14cyclictest0-21swapper/312:25:163
2502399292,14cyclictest29566irq/58-eth009:20:133
2501599294,10cyclictest0-21swapper/211:30:132
25015992920,5cyclictest26346irq/40-dwc2_hso07:25:132
25015992912,11cyclictest0-21swapper/207:45:132
25015992912,11cyclictest0-21swapper/207:45:122
2500799294,16cyclictest21-21ksoftirqd/111:00:171
2500799292,20cyclictest14025-21ssh12:15:141
2500799291,20cyclictest10219-21ssh11:35:141
24998992918,7cyclictest11340-21diskmemload09:20:140
25023992815,9cyclictest29566irq/58-eth012:20:133
25023992811,10cyclictest29566irq/58-eth009:45:143
2501599282,20cyclictest0-21swapper/208:20:152
2501599281,12cyclictest27-21ksoftirqd/211:36:442
24998992812,8cyclictest629-21Async14:00:480
2499899281,22cyclictest12268-21ssh10:55:180
2501599276,7cyclictest27-21ksoftirqd/211:15:002
2501599273,11cyclictest27-21ksoftirqd/210:15:412
2500799273,16cyclictest21-21ksoftirqd/108:45:141
2500799272,20cyclictest13914-21ssh10:35:161
2499899272,19cyclictest2797-21ssh12:07:450
2499899272,19cyclictest2797-21ssh12:07:440
2502399262,11cyclictest0-21swapper/311:04:443
25015992612,7cyclictest0-21swapper/209:37:142
25015992612,7cyclictest0-21swapper/209:37:132
2501599256,6cyclictest131rcu_preempt10:57:542
2501599253,10cyclictest27-21ksoftirqd/212:22:082
25015992513,5cyclictest0-21swapper/212:01:252
25015992512,11cyclictest0-21swapper/210:33:412
2501599251,19cyclictest8322-21ssh12:30:172
2501599251,19cyclictest8322-21ssh12:30:162
25007992512,11cyclictest0-21swapper/112:13:201
2501599247,11cyclictest0-21swapper/208:55:152
2501599244,9cyclictest27-21ksoftirqd/211:51:162
2501599241,15cyclictest20055-21apt-get11:00:152
25015992411,4cyclictest0-21swapper/208:00:292
2501599241,13cyclictest15276-21mailstats11:15:292
25015992410,12cyclictest27-21ksoftirqd/212:18:212
2500799248,3cyclictest21-21ksoftirqd/110:40:221
2500799248,3cyclictest21-21ksoftirqd/110:40:211
2500799243,14cyclictest0-21swapper/108:25:141
2500799243,14cyclictest0-21swapper/108:25:141
25007992411,11cyclictest0-21swapper/111:32:381
2499899242,17cyclictest23233-21apt-get08:15:130
24998992412,9cyclictest12-21ksoftirqd/008:50:170
2499899241,15cyclictest25311-21ssh09:40:140
2499899241,10cyclictest20409-21ntp_states09:55:270
2502399234,11cyclictest0-21swapper/309:25:123
2501599239,11cyclictest0-21swapper/208:15:202
2501599238,7cyclictest30070-21cut11:25:372
2501599238,6cyclictest19881-21kthreadcore08:10:192
2501599237,8cyclictest1-21systemd11:56:262
2501599234,15cyclictest27497-21irqrtprio09:20:182
2501599234,12cyclictest27-21ksoftirqd/209:10:272
2501599232,6cyclictest0-21swapper/208:35:042
2501599232,15cyclictest27-21ksoftirqd/210:29:352
2501599231,4cyclictest575766ssh09:25:232
2501599231,19cyclictest7723-21find10:10:132
2501599231,18cyclictest28697-21ssh11:05:192
2500799232,12cyclictest0-21swapper/109:44:011
25007992317,4cyclictest0-21swapper/109:45:131
2500799231,17cyclictest8295-21apt-get12:30:151
2500799231,17cyclictest8295-21apt-get12:30:141
25007992310,4cyclictest131rcu_preempt07:35:151
25007992310,11cyclictest0-21swapper/110:50:151
2499899232,6cyclictest0-21swapper/011:12:430
2499899232,13cyclictest0-21swapper/010:20:150
2499899231,17cyclictest25300-21ssh12:01:280
2502399226,12cyclictest0-21swapper/311:35:153
2502399224,11cyclictest29566irq/58-eth011:25:273
25023992213,5cyclictest29566irq/58-eth010:35:163
2501599229,8cyclictest471-21apt-get08:25:162
2501599229,8cyclictest471-21apt-get08:25:152
2501599229,6cyclictest24739-21sshd09:16:022
2501599229,10cyclictest28985-21cut12:05:202
2501599229,10cyclictest28985-21cut12:05:192
2501599229,10cyclictest27032-21fschecks_time11:45:192
2501599225,9cyclictest9644-21kthreadcore09:50:202
2501599223,7cyclictest131rcu_preempt08:30:242
25015992213,7cyclictest0-21swapper/209:45:242
25015992212,3cyclictest0-21swapper/209:40:282
2501599221,12cyclictest7973-21ssh10:50:262
2500799221,9cyclictest131rcu_preempt11:55:171
25007992215,4cyclictest23612-21sh10:20:171
2500799221,15cyclictest28092-21find08:20:121
2500799221,15cyclictest26960-21sh09:20:131
25007992211,3cyclictest21-21ksoftirqd/109:34:021
25007992211,3cyclictest21-21ksoftirqd/109:34:011
25007992210,4cyclictest21-21ksoftirqd/110:00:261
25007992210,10cyclictest7887-21cron12:30:001
25007992210,10cyclictest0-21swapper/110:19:011
2500799220,14cyclictest131rcu_preempt09:14:031
2499899225,11cyclictest17383-21ssh12:35:300
2499899223,13cyclictest1023-21powerfaild09:25:130
24998992213,6cyclictest6254-21grep07:20:280
2499899221,17cyclictest11719-21ssh09:10:160
2502399213,6cyclictest0-21swapper/308:00:143
2501599216,2cyclictest131rcu_preempt07:55:212
2501599215,6cyclictest27-21ksoftirqd/212:39:172
2501599213,9cyclictest29472-21sh10:00:332
2501599213,15cyclictest31671-21fschecks_time07:50:162
2501599212,8cyclictest22782-21cut10:40:222
2501599212,8cyclictest22782-21cut10:40:212
25015992111,8cyclictest0-21swapper/212:29:422
25015992111,8cyclictest0-21swapper/210:22:242
25015992111,8cyclictest0-21swapper/209:32:592
25015992111,8cyclictest0-21swapper/209:32:582
2501599211,12cyclictest26346irq/40-dwc2_hso07:40:132
2501599211,12cyclictest26346irq/40-dwc2_hso07:40:122
2501599211,11cyclictest2274-21sed09:00:192
2501599211,10cyclictest29948-21cat07:15:022
2500799217,4cyclictest21-21ksoftirqd/112:04:471
25007992114,4cyclictest21-21ksoftirqd/109:25:141
25007992112,6cyclictest0-21swapper/108:35:291
2499899213,15cyclictest708-21cron09:10:020
24998992114,4cyclictest2173-21grep09:00:180
24998992112,6cyclictest2167-21sendmail07:20:020
24998992111,6cyclictest21854-21sh10:40:170
24998992111,6cyclictest21854-21sh10:40:160
2499899211,15cyclictest10933-21ssh11:55:130
2502399203,12cyclictest0-21swapper/312:01:283
2502399203,11cyclictest0-21swapper/311:48:443
2502399202,13cyclictest0-21swapper/312:07:453
2502399202,13cyclictest0-21swapper/312:07:443
25023992012,5cyclictest29566irq/58-eth010:15:143
2502399201,11cyclictest29566irq/58-eth010:40:153
2502399201,11cyclictest29566irq/58-eth010:40:143
2501599203,11cyclictest26346irq/40-dwc2_hso09:05:162
2501599201,8cyclictest21545-21awk07:40:022
2501599201,14cyclictest5087-21ssh12:10:122
2501599201,11cyclictest17826-21kthreadcore08:40:212
25015992010,7cyclictest32445-21apt-get10:05:132
2500799208,10cyclictest21-21ksoftirqd/111:17:551
2500799204,2cyclictest21-21ksoftirqd/110:59:011
25007992012,6cyclictest0-21swapper/107:30:191
25007992011,6cyclictest0-21swapper/108:15:211
2500799201,15cyclictest824-21ls07:15:221
2500799201,14cyclictest32471-21ssh10:05:151
2500799201,10cyclictest131rcu_preempt12:20:571
25007992010,7cyclictest1102-21sshd11:06:441
2499899209,9cyclictest12-21ksoftirqd/010:05:160
2499899202,12cyclictest8451-21ssh11:32:430
2499899201,7cyclictest18617-21hwlatdetect11:40:170
24998992012,6cyclictest24477-21sshd08:47:440
2499899201,16cyclictest9631-21ssh09:30:190
2499899201,16cyclictest9631-21ssh09:30:180
2499899201,16cyclictest28184-21fschecks_time11:05:180
2499899201,15cyclictest27203-21apt-get11:25:260
2499899201,14cyclictest19348-21ssh11:20:130
2502399193,11cyclictest0-21swapper/311:56:273
2502399193,11cyclictest0-21swapper/311:40:453
2502399192,13cyclictest0-21swapper/307:25:013
2502399192,11cyclictest0-21swapper/309:30:203
2502399192,11cyclictest0-21swapper/309:30:193
2502399192,10cyclictest0-21swapper/308:35:143
2502399191,13cyclictest0-21swapper/310:15:013
25023991910,6cyclictest0-21swapper/310:50:323
2501599195,11cyclictest0-21swapper/208:50:212
2501599191,9cyclictest30703-21kthreadcore07:15:182
2501599191,9cyclictest12815-21kthreadcore07:30:202
2501599191,15cyclictest15102-21kthreadcore08:05:202
2501599191,11cyclictest20017-21cut08:45:182
2501599190,11cyclictest2917-21irqrtprio07:20:182
2500799199,8cyclictest19423-21cat11:20:161
2500799199,2cyclictest131rcu_preempt12:07:091
2500799199,2cyclictest131rcu_preempt12:07:081
2500799197,4cyclictest21-21ksoftirqd/111:50:251
2500799197,4cyclictest21-21ksoftirqd/109:36:001
2500799197,4cyclictest21-21ksoftirqd/109:35:591
25007991912,5cyclictest0-21swapper/110:25:271
25007991912,5cyclictest0-21swapper/109:15:251
25007991911,5cyclictest0-21swapper/108:10:161
2500799191,10cyclictest5333-21/usr/sbin/munin08:30:311
2499899199,8cyclictest25113-21memory07:40:230
2499899199,8cyclictest25113-21memory07:40:220
2499899198,8cyclictest18970-21cron11:20:020
2499899193,12cyclictest20076-21ssh11:00:160
2499899191,8cyclictest30785-21timerandwakeup11:45:300
24998991912,4cyclictest0-21swapper/007:35:300
24998991911,5cyclictest8853-21wc09:50:150
2499899191,14cyclictest32203-21kthreadcore10:45:230
2499899191,12cyclictest6531-21sh10:30:170
2499899191,10cyclictest0-21swapper/012:10:110
24998991910,4cyclictest26736-21kthreadcore10:00:210
2502399189,6cyclictest0-21swapper/311:23:313
2502399187,7cyclictest0-21swapper/310:30:163
2502399182,13cyclictest0-21swapper/310:05:233
2502399182,10cyclictest0-21swapper/309:15:153
2502399182,10cyclictest0-21swapper/308:55:153
2502399182,10cyclictest0-21swapper/307:25:133
2502399181,12cyclictest0-21swapper/310:30:013
25023991810,6cyclictest0-21swapper/312:10:173
25023991810,6cyclictest0-21swapper/310:46:073
25023991810,5cyclictest29566irq/58-eth011:30:153
25023991810,5cyclictest29566irq/58-eth011:05:153
25023991810,3cyclictest0-21swapper/310:23:353
2502399180,14cyclictest29566irq/58-eth012:35:313
2501599181,8cyclictest25043-21sort07:10:012
2500799183,8cyclictest21-21ksoftirqd/107:40:171
2500799183,8cyclictest21-21ksoftirqd/107:40:161
2500799181,9cyclictest0-21swapper/107:25:241
2500799181,8cyclictest29871-21grep07:15:021
25007991812,4cyclictest0-21swapper/108:00:301
2500799180,14cyclictest16933-21ls08:05:231
2499899189,7cyclictest8868-21diskstats08:00:170
2499899188,7cyclictest7733-21apt-get10:10:130
2499899183,12cyclictest725-21grep08:25:170
2499899183,12cyclictest725-21grep08:25:170
2499899183,12cyclictest31704-21tune2fs07:50:180
2499899183,12cyclictest17902-21sort10:35:300
2499899180,3cyclictest131rcu_preempt12:15:490
2502399179,6cyclictest0-21swapper/309:54:363
2502399179,6cyclictest0-21swapper/309:41:173
2502399179,5cyclictest29566irq/58-eth009:10:153
2502399172,10cyclictest0-21swapper/308:50:193
2502399172,10cyclictest0-21swapper/307:10:143
2502399171,9cyclictest0-21swapper/310:00:303
2502399171,10cyclictest0-21swapper/307:35:023
2500799173,9cyclictest21-21ksoftirqd/108:50:171
2500799172,2cyclictest131rcu_preempt07:55:271
2500799171,13cyclictest2265-21irqcore09:00:201
25007991710,4cyclictest21-21ksoftirqd/108:40:261
2499899178,6cyclictest6178-21cut07:55:220
2499899174,10cyclictest28537-21kthreadcore07:45:210
2499899174,10cyclictest28537-21kthreadcore07:45:200
2499899174,10cyclictest28512-21irqcore08:20:190
2499899174,10cyclictest27573-21kthreadcore07:10:210
2499899173,11cyclictest20125-21kthreadcore08:10:200
2499899173,11cyclictest12584-21irqcore07:30:190
2499899171,8cyclictest31442-21irqcore10:25:190
2499899171,12cyclictest8774-21netstat08:30:260
2499899171,11cyclictest30038-21irqcore08:55:200
24998991710,3cyclictest0-21swapper/008:05:120
2502399168,6cyclictest0-21swapper/309:57:203
2502399162,10cyclictest0-21swapper/310:55:183
2502399162,10cyclictest0-21swapper/309:00:163
2500799166,2cyclictest21-21ksoftirqd/108:55:251
2500799164,6cyclictest0-21swapper/107:20:131
2499899163,10cyclictest15079-21cpu08:40:150
2502399151,8cyclictest0-21swapper/308:15:313
2502399151,7cyclictest0-21swapper/309:10:013
25023991510,3cyclictest0-21swapper/307:45:153
25023991510,3cyclictest0-21swapper/307:45:143
2502399146,5cyclictest0-21swapper/308:50:013
2502399143,9cyclictest0-21swapper/307:50:183
2502399142,10cyclictest0-21swapper/307:15:143
2502399141,11cyclictest0-21swapper/308:10:153
2502399141,11cyclictest0-21swapper/307:55:133
2500799144,8cyclictest25127-21latency_hist07:10:011
2499899141,10cyclictest24927-21latency_hist07:10:020
2502399132,8cyclictest0-21swapper/307:40:003
2502399131,9cyclictest0-21swapper/307:40:203
2502399131,9cyclictest0-21swapper/307:40:193
2502399131,5cyclictest0-21swapper/308:25:253
2502399131,5cyclictest0-21swapper/308:25:243
2502399131,10cyclictest0-21swapper/308:05:203
2502399122,7cyclictest0-21swapper/308:40:143
2502399122,7cyclictest0-21swapper/308:25:003
2502399121,9cyclictest0-21swapper/308:30:193
2502399101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional