You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-22 - 17:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun Mar 22, 2026 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1588799703,48cyclictest5812-21ssh09:30:131
1590099622,7cyclictest131rcu_preempt08:20:122
1587899591,46cyclictest25695-21ssh12:05:130
1590099582,32cyclictest6212-21ssh11:00:112
1587899561,45cyclictest20834-21ssh10:50:140
1587899561,45cyclictest20834-21ssh10:50:130
1587899521,39cyclictest2175-21ssh12:10:130
1588799507,5cyclictest131rcu_preempt07:10:141
1590099493,25cyclictest27-21ksoftirqd/212:30:132
15906994841,5cyclictest28366irq/58-eth010:50:153
15906994841,5cyclictest28366irq/58-eth010:50:143
1588799482,36cyclictest29310-21ssh12:25:141
1590099461,30cyclictest24071-21ssh09:40:132
1590699432,28cyclictest28366irq/58-eth012:15:133
1587899431,37cyclictest10996-21ssh09:15:130
15878994215,16cyclictest12-21ksoftirqd/008:20:120
1587899413,27cyclictest6227-21ssh11:00:120
1588799401,32cyclictest31494-21ssh10:20:131
1588799395,23cyclictest29125-21ssh09:25:101
1588799391,33cyclictest4426-21ssh10:05:111
1590699383,23cyclictest28366irq/58-eth009:40:113
1590699383,22cyclictest28366irq/58-eth009:20:123
1590699383,22cyclictest28366irq/58-eth009:20:113
1588799381,26cyclictest9696-21ssh09:50:131
1590699373,20cyclictest28366irq/58-eth011:25:133
15906993729,6cyclictest28466irq/59-eth010:00:103
1590099372,18cyclictest17473-21ssh09:53:572
1588799377,21cyclictest21-21ksoftirqd/112:10:141
15878993718,15cyclictest16724-21ssh12:00:140
1590099362,27cyclictest31148-21rm11:50:142
1588799364,20cyclictest4377-21ssh11:35:131
1590099353,21cyclictest655-21ssh11:15:132
1590099352,25cyclictest9281-21apt-get11:20:042
15878993513,17cyclictest12-21ksoftirqd/009:20:110
15878993513,17cyclictest12-21ksoftirqd/009:20:110
1590099342,26cyclictest14999-21ssh10:27:572
1590099341,21cyclictest20815-21ssh10:50:122
1590099341,21cyclictest20815-21ssh10:50:122
1588799341,21cyclictest20261-21ssh12:20:131
1587899341,26cyclictest29120-21ssh09:25:090
15900993324,3cyclictest13430-21ssh10:10:122
1590699320,12cyclictest28366irq/58-eth010:05:123
1587899323,17cyclictest5777-21ssh12:30:130
1590699312,14cyclictest28366irq/58-eth008:25:123
1590099315,7cyclictest131rcu_preempt10:32:572
1590099311,19cyclictest22221-21ssh11:45:122
1590699303,20cyclictest0-21swapper/309:53:573
1590699301,16cyclictest28366irq/58-eth007:45:143
1590099305,16cyclictest22346irq/40-dwc2_hso09:20:122
1590099305,16cyclictest22346irq/40-dwc2_hso09:20:112
1590099304,16cyclictest22346irq/40-dwc2_hso09:00:122
1590099302,11cyclictest27-21ksoftirqd/211:05:142
15900993010,15cyclictest27-21ksoftirqd/208:00:122
1587899303,21cyclictest27823-21ssh10:00:100
1587899303,17cyclictest0-21swapper/012:15:140
1587899301,18cyclictest15159-21ssh11:05:130
1590699292,16cyclictest28366irq/58-eth007:50:133
1590099292,9cyclictest11149-21ssh12:15:132
1590099292,20cyclictest0-21swapper/207:55:132
15900992913,7cyclictest22346irq/40-dwc2_hso08:35:142
15900992913,10cyclictest261rcuc/207:35:112
1588799292,20cyclictest17170-21rm10:30:131
1590699283,16cyclictest0-21swapper/311:37:573
1590699282,15cyclictest28366irq/58-eth009:55:113
1590699282,14cyclictest28366irq/58-eth010:40:133
1590099283,8cyclictest0-21swapper/211:29:572
1590099283,20cyclictest0-21swapper/210:18:572
1590099283,16cyclictest22346irq/40-dwc2_hso08:25:122
1590099282,19cyclictest0-21swapper/212:25:062
1590099282,10cyclictest21833-21ssh09:37:572
1590099282,10cyclictest1428-21ssh10:38:572
1588799282,15cyclictest18865-21ssh11:25:151
1588799281,19cyclictest26181-21ssh10:35:151
1588799281,19cyclictest11826-21ssh10:45:161
15878992818,5cyclictest1593-21diskmemload09:40:130
1590099277,3cyclictest131rcu_preempt08:10:132
1590099273,15cyclictest0-21swapper/212:05:132
1590099273,14cyclictest0-21swapper/208:05:132
1588799272,19cyclictest22416-21ssh10:15:131
1588799271,20cyclictest11108-21ssh09:15:141
1587899272,19cyclictest30178-21ssh10:18:570
1587899272,19cyclictest27319-21ssh11:29:570
15878992714,8cyclictest12-21ksoftirqd/009:51:570
1590099261,21cyclictest7588-21ssh10:24:532
1588799262,18cyclictest15179-21ssh11:05:141
15887992619,3cyclictest0-21swapper/107:50:131
1588799261,16cyclictest13348-21ssh11:40:111
1590699252,17cyclictest0-21swapper/309:39:573
1590099251,8cyclictest1593-21diskmemload11:14:572
1590099251,18cyclictest13254-21ssh11:40:062
1590099251,13cyclictest2191-21ssh12:10:142
1588799253,10cyclictest19829-21kthreadcore12:00:221
1588799252,19cyclictest3155-21ssh10:03:571
1588799251,19cyclictest27678-21ssh09:40:241
1587899253,7cyclictest0-21swapper/010:09:570
1590699248,9cyclictest28366irq/58-eth010:55:133
1590699247,10cyclictest28366irq/58-eth012:00:133
1590699243,11cyclictest28366irq/58-eth012:25:133
1590099244,14cyclictest22346irq/40-dwc2_hso08:30:122
1590099244,14cyclictest22346irq/40-dwc2_hso08:30:122
1590099242,6cyclictest0-21swapper/211:30:062
1590099242,18cyclictest5756-21ssh10:59:572
15900992419,3cyclictest0-21swapper/210:45:182
15900992416,5cyclictest924-21snmpd07:20:142
15900992413,9cyclictest0-21swapper/211:55:262
15900992413,9cyclictest0-21swapper/210:00:122
1590099240,18cyclictest23774-21missed_timers12:20:232
1588799242,16cyclictest13445-21ssh10:10:131
15887992412,6cyclictest0-21swapper/109:24:571
15887992412,6cyclictest0-21swapper/109:24:571
1588799241,18cyclictest7713-21ssh11:55:131
1587899244,16cyclictest131rcu_preempt08:05:140
15878992410,10cyclictest9462-21gpgv11:20:020
1590699231,14cyclictest0-21swapper/312:10:133
1590099239,2cyclictest27-21ksoftirqd/209:45:262
1590099239,2cyclictest10993-21sh09:15:122
1590099234,11cyclictest131rcu_preempt09:10:562
1590099232,9cyclictest16723-21sh12:00:132
1590099232,6cyclictest0-21swapper/211:39:562
1590099232,10cyclictest18928-21fschecks_time09:55:152
1590099231,11cyclictest14468-21cron09:35:012
1588799237,11cyclictest15870-21cyclictest08:40:121
1588799232,15cyclictest14877-21ssh09:35:141
15887992314,6cyclictest0-21swapper/108:05:271
1588799231,16cyclictest21-21ksoftirqd/109:55:121
1587899237,4cyclictest12-21ksoftirqd/008:30:130
1587899237,4cyclictest12-21ksoftirqd/008:30:130
1587899232,17cyclictest6413-21ssh10:40:260
15878992313,7cyclictest5516-21sed09:10:220
1587899231,17cyclictest24148-21ssh11:10:150
15878992310,7cyclictest0-21swapper/010:25:130
1590699224,10cyclictest0-21swapper/311:00:123
1590699223,13cyclictest0-21swapper/309:32:573
1590099226,6cyclictest131rcu_preempt07:40:142
1590099222,14cyclictest22346irq/40-dwc2_hso07:15:132
1590099222,11cyclictest20790-21ps07:45:282
1590099221,11cyclictest23097-21awk12:40:012
1588799221,17cyclictest29670-21ssh10:55:131
1588799221,17cyclictest14880-21cut12:35:171
15887992211,4cyclictest11397-21timerwakeupswit07:35:261
1588799220,15cyclictest31935-21cut08:00:161
1587899226,4cyclictest131rcu_preempt09:40:030
1587899224,3cyclictest131rcu_preempt11:39:020
1587899221,6cyclictest1865966ssh10:11:000
15906992110,9cyclictest0-21swapper/310:11:593
1590099217,10cyclictest0-21swapper/208:45:282
1590099211,9cyclictest24420-21cron09:00:002
1590099211,9cyclictest24420-21cron09:00:002
15900992111,8cyclictest0-21swapper/207:30:302
1590099211,17cyclictest8545-21unixbench_singl10:05:302
1590099211,11cyclictest5339-21kthreadcore10:40:222
1590099211,11cyclictest17867-21kthreadcore07:10:222
15900992110,9cyclictest4141-21ssh09:28:322
15887992113,6cyclictest21-21ksoftirqd/112:20:011
1588799211,16cyclictest2065-21ls09:10:131
1587899217,11cyclictest12-21ksoftirqd/010:32:570
1587899212,7cyclictest131rcu_preempt11:58:270
1587899211,8cyclictest131rcu_preempt11:42:350
1587899211,6cyclictest13369-21awk11:20:240
1587899211,4cyclictest131rcu_preempt08:45:140
15878992110,3cyclictest12-21ksoftirqd/010:49:410
1590699209,8cyclictest28366irq/58-eth012:05:133
1590699207,8cyclictest28366irq/58-eth011:40:133
1590699204,11cyclictest28366irq/58-eth011:10:153
1590699203,12cyclictest0-21swapper/310:27:573
1590699203,10cyclictest0-21swapper/309:26:573
1590699202,11cyclictest28366irq/58-eth010:45:163
1590699202,11cyclictest0-21swapper/308:00:043
15906992013,5cyclictest0-21swapper/312:34:483
15906992012,5cyclictest28366irq/58-eth011:15:133
1590699201,14cyclictest0-21swapper/307:20:213
15906992010,8cyclictest0-21swapper/311:05:343
15906992010,7cyclictest28366irq/58-eth012:20:123
1590099202,9cyclictest30691-21tail07:25:192
15900992011,7cyclictest18400-21idleruntime-cro08:20:012
1590099201,10cyclictest25520-21ntpq07:50:252
1590099200,9cyclictest31264-21kthreadcore09:05:222
1588799208,9cyclictest9264-21cat12:30:241
1588799202,13cyclictest0-21swapper/110:40:141
1588799201,4cyclictest21-21ksoftirqd/107:40:281
15887992013,4cyclictest29377-21ntp_states12:05:271
15887992012,5cyclictest0-21swapper/109:45:281
15887992011,6cyclictest24215-21sed11:45:201
1587899202,2cyclictest131rcu_preempt10:20:150
1587899201,9cyclictest131rcu_preempt10:55:240
1587899201,2cyclictest131rcu_preempt11:48:270
1587899201,2cyclictest131rcu_preempt09:45:030
15878992011,7cyclictest12181-21sshd07:35:580
1587899201,13cyclictest0-21swapper/007:55:130
1590699198,7cyclictest0-21swapper/309:10:233
1590699194,11cyclictest0-21swapper/310:20:173
1590699193,11cyclictest0-21swapper/311:33:573
1590699193,11cyclictest0-21swapper/310:32:573
1590699193,11cyclictest0-21swapper/309:18:573
1590699192,10cyclictest0-21swapper/307:10:143
1590699191,13cyclictest0-21swapper/309:45:273
1590699191,12cyclictest28366irq/58-eth011:45:123
1590699191,12cyclictest0-21swapper/307:15:153
1590099191,9cyclictest5552-21fschecks_count08:40:152
1590099191,9cyclictest15929-21latency_hist07:10:022
1590099191,15cyclictest15279-21grep08:50:182
15887991911,5cyclictest24421-21cron09:00:001
15887991911,5cyclictest24421-21cron09:00:001
1588799191,15cyclictest13930-21awk08:45:291
15887991911,5cyclictest0-21swapper/109:00:251
1588799190,7cyclictest14312-21irqrtprio08:15:181
1588799190,14cyclictest29246-21ls07:20:271
1587899196,9cyclictest5811-21diskmemload09:30:130
1587899195,4cyclictest12-21ksoftirqd/012:20:150
1587899193,3cyclictest131rcu_preempt10:36:420
1587899192,2cyclictest131rcu_preempt12:35:330
1587899192,2cyclictest131rcu_preempt12:29:240
1587899192,12cyclictest28900-21missed_timers07:20:210
1587899192,11cyclictest0-21swapper/008:25:140
1587899191,8cyclictest0-21swapper/011:50:100
1587899191,8cyclictest0-21swapper/011:34:170
1587899191,6cyclictest0-21swapper/008:00:140
1590699189,6cyclictest28366irq/58-eth010:35:133
1590699188,3cyclictest0-21swapper/311:56:503
1590699187,3cyclictest0-21swapper/311:50:323
1590699183,10cyclictest0-21swapper/310:19:523
1590699182,11cyclictest28366irq/58-eth012:35:283
1590699182,10cyclictest0-21swapper/311:20:043
1590699182,10cyclictest0-21swapper/308:10:133
1590699182,10cyclictest0-21swapper/307:40:143
1590699181,11cyclictest0-21swapper/309:05:253
1590699181,10cyclictest0-21swapper/308:55:173
1590699181,10cyclictest0-21swapper/308:55:163
1588799182,13cyclictest8187-21kthreadcore11:00:201
1588799181,14cyclictest24040-21kthreadcore10:50:221
1588799181,14cyclictest24040-21kthreadcore10:50:221
1588799181,14cyclictest18618-21apt-get08:20:121
1588799181,13cyclictest27852-21rm11:10:271
1588799181,13cyclictest1786-21ls09:10:021
1588799181,13cyclictest11715-21ntp_192.168.11410:25:241
15887991810,4cyclictest0-21swapper/111:30:141
1587899189,7cyclictest20592-21memory07:45:250
1587899189,6cyclictest5703-21cut07:30:210
1587899182,9cyclictest0-21swapper/008:10:110
1587899181,9cyclictest16167-21fschecks_time07:10:160
1587899181,12cyclictest22257-21ls09:55:210
1590699172,10cyclictest0-21swapper/308:45:163
1590699172,10cyclictest0-21swapper/308:40:133
1590699172,10cyclictest0-21swapper/308:20:133
1590699172,10cyclictest0-21swapper/307:25:163
1590699171,13cyclictest0-21swapper/307:30:153
1590699171,10cyclictest0-21swapper/308:35:163
1588799173,12cyclictest28518-21kthreadcore07:55:201
1588799171,3cyclictest131rcu_preempt08:35:141
15887991711,4cyclictest0-21swapper/111:50:291
1588799171,13cyclictest9063-21diskstats08:10:151
1588799171,12cyclictest9281-21apt-get11:20:021
1588799171,12cyclictest1356-21cut11:15:181
1587899179,6cyclictest16294-21cut08:15:220
1587899173,5cyclictest22572-21sed07:50:180
1587899172,2cyclictest131rcu_preempt09:00:230
1587899172,11cyclictest0-21swapper/007:40:140
1587899172,10cyclictest151rcuc/008:35:140
1587899171,2cyclictest131rcu_preempt09:05:180
1590699162,9cyclictest0-21swapper/308:50:153
1590699161,12cyclictest0-21swapper/309:00:143
1590699161,12cyclictest0-21swapper/307:55:173
1588799168,5cyclictest15497-21irqcore08:50:191
1588799168,4cyclictest17905-21irqrtprio07:45:201
1588799165,8cyclictest0-21swapper/107:25:141
1588799164,9cyclictest30072-21kthreadcore08:30:191
1588799164,9cyclictest30072-21kthreadcore08:30:191
1588799163,10cyclictest27118-21timerandwakeup08:25:271
1588799162,9cyclictest0-21swapper/107:30:131
1588799161,12cyclictest20940-21tune2fs07:15:171
1587899169,5cyclictest23829-21cat08:55:290
1587899169,5cyclictest23829-21cat08:55:290
1587899168,6cyclictest854-21tr07:25:240
1587899163,10cyclictest17122-21kthreadcore08:50:220
1590699152,9cyclictest0-21swapper/308:15:143
1587899151,11cyclictest21401-21sed07:15:190
1587899151,10cyclictest0-21swapper/008:40:130
1590699141,10cyclictest0-21swapper/307:35:163
1588799141,9cyclictest15923-21latency_hist07:10:021
1587899131,9cyclictest15870-21cyclictest07:10:020
1590699122,7cyclictest0-21swapper/308:30:283
1590699122,7cyclictest0-21swapper/308:30:273
1590699121,5cyclictest0-21swapper/308:05:213
1590699101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional