You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-10 - 10:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun May 10, 2026 00:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17862995335,14cyclictest18009-21apt-get19:10:130
1787199512,36cyclictest1942-21ssh21:45:131
17896994932,9cyclictest35966irq/58-eth021:40:153
1787199471,37cyclictest19872-21ssh21:55:131
1787199453,4cyclictest21-21ksoftirqd/120:50:121
1786299454,12cyclictest131rcu_preempt00:05:130
1787199432,31cyclictest30257-21ssh22:55:131
17862994220,16cyclictest0-21swapper/023:05:140
17862994213,11cyclictest27873-21ssh23:45:460
17862994134,2cyclictest4795-21apt-get19:30:110
17862994129,9cyclictest8002-21apt-get23:55:050
17862994127,10cyclictest27624-21apt-get19:20:110
17871994017,16cyclictest13794-21sshd00:33:251
17862994034,4cyclictest0-21swapper/019:55:120
17862994013,8cyclictest6417-21ntp_192.168.11400:10:230
17862994012,8cyclictest3622-21diskmemload21:52:080
17862994012,11cyclictest7221-21rm22:23:400
17862994012,11cyclictest6474-21ssh23:53:130
17862994012,11cyclictest16797-21ssh22:29:280
1786299394,9cyclictest131rcu_preempt00:20:140
17862993928,8cyclictest14262-21apt-get22:10:140
1786299392,33cyclictest17669-21ssh22:30:130
17862993915,10cyclictest541-21rm21:43:500
17862993913,10cyclictest2616-21ssh00:25:470
17862993912,11cyclictest23749-21ssh00:39:330
17886993822,12cyclictest30167-21apt-get20:30:132
17862993814,8cyclictest12157-21ssh00:31:290
17862993814,7cyclictest28449-21ssh21:22:240
17862993814,11cyclictest26941-21ssh21:58:140
17862993813,9cyclictest19020-21ssh21:16:580
17862993812,8cyclictest31980-21ssh23:13:240
17862993812,10cyclictest12535-21ssh21:14:580
17862993811,9cyclictest3622-21diskmemload22:01:030
17862993811,11cyclictest28519-21rm22:16:050
17862993811,11cyclictest28519-21rm22:16:040
17886993728,4cyclictest15531-21apt-get00:35:142
17886993721,12cyclictest4012-21apt-get21:10:142
17886993719,10cyclictest0-21swapper/219:20:122
1788699371,16cyclictest10883-21ssh21:50:112
17862993729,4cyclictest10944-21apt-get20:10:110
17862993713,9cyclictest22303-21ssh21:36:430
17862993713,8cyclictest18116-21ssh22:46:460
17862993713,8cyclictest18116-21ssh22:46:450
17862993713,11cyclictest2967-21ssh22:55:490
17862993712,9cyclictest9205-21ssh21:48:260
17862993712,10cyclictest22723-21ntp_states19:45:250
17862993710,12cyclictest8431-21ssh23:18:190
17886993618,11cyclictest10269-21apt-get23:20:152
17886993618,11cyclictest10269-21apt-get23:20:142
1787199365,4cyclictest131rcu_preempt20:05:121
1787199365,4cyclictest131rcu_preempt20:05:111
1787199365,27cyclictest12986-21ssh21:15:141
17862993628,4cyclictest10291-21ssh23:20:150
17862993628,4cyclictest10291-21ssh23:20:140
17862993627,6cyclictest31465-21apt-get21:05:140
17862993624,9cyclictest21277-21ssh22:50:110
17862993615,8cyclictest4054-21ssh21:26:190
17862993612,9cyclictest3622-21diskmemload00:16:010
17862993612,8cyclictest3622-21diskmemload22:37:240
17862993612,11cyclictest13819-21ssh23:03:080
17862993611,9cyclictest3622-21diskmemload23:35:050
17862993611,9cyclictest27517-21ssh23:29:410
17862993611,9cyclictest20993-21ntp_192.168.11400:00:250
17862993611,9cyclictest17096-21tune2fs20:50:150
17862993523,8cyclictest13741-21apt-get23:40:110
17862993512,8cyclictest3622-21diskmemload21:31:390
17862993511,8cyclictest3622-21diskmemload22:05:500
17862993511,8cyclictest10309-21ssh22:43:110
17862993426,5cyclictest14469-21apt-get19:40:140
17862993412,8cyclictest3622-21diskmemload23:31:440
17862993410,9cyclictest5785-21users20:00:300
17862993410,8cyclictest19137-21ntp_192.168.11420:15:240
1789699331,18cyclictest35966irq/58-eth020:45:133
17886993320,8cyclictest21246-21apt-get22:50:112
17871993317,11cyclictest17848-21cyclictest22:15:131
17871993317,11cyclictest17848-21cyclictest22:15:121
1786299339,8cyclictest7617-21/usr/sbin/munin20:40:180
17862993313,6cyclictest13147-21proc_pri19:35:270
17862993313,6cyclictest13147-21proc_pri19:35:260
1788699328,11cyclictest0-21swapper/221:00:132
17886993216,4cyclictest28511-21/usr/sbin/munin23:30:182
17886993215,14cyclictest0-21swapper/222:10:142
1788699321,15cyclictest27271-21sh00:22:252
1787199322,22cyclictest15857-21sh23:21:261
1787199322,22cyclictest15857-21sh23:21:251
1786299329,8cyclictest28777-21ntpq20:25:250
17862993221,7cyclictest6052-21apt-get20:05:130
17862993221,7cyclictest6052-21apt-get20:05:120
17862993211,4cyclictest26201-21ntp_192.168.11419:15:240
17862993210,8cyclictest12071-21/usr/sbin/munin20:45:290
1788699314,24cyclictest0-21swapper/221:40:142
1788699311,13cyclictest10836-21ssh22:43:492
17871993116,11cyclictest21-21ksoftirqd/123:25:141
17862993119,9cyclictest26024-21cat20:55:300
17886993014,14cyclictest26693-21ssh22:35:132
1788699300,13cyclictest3622-21diskmemload21:56:262
1786299307,9cyclictest2508-21/usr/sbin/munin20:35:150
17862993010,8cyclictest24049-21acpi19:50:120
1789699299,16cyclictest0-21swapper/323:05:133
1788699298,18cyclictest0-21swapper/221:30:142
1788699292,9cyclictest0-21swapper/200:34:142
17886992922,3cyclictest2526-21apt-get20:35:112
1788699292,19cyclictest10471-21ssh23:00:252
1788699292,11cyclictest20426-21ssh00:19:562
17886992919,4cyclictest0-21swapper/221:20:132
1788699291,13cyclictest19481-21ntp_states23:05:262
1788699291,12cyclictest16695-21ssh22:29:202
1788699290,13cyclictest4729-21ssh00:28:202
1787199292,21cyclictest25000-21sh22:33:261
17871992915,5cyclictest21-21ksoftirqd/120:00:211
1788699283,8cyclictest31560-21entropy21:05:162
1788699281,7cyclictest91-21pr/ttyS022:20:012
1788699281,7cyclictest91-21pr/ttyS022:20:002
1788699281,13cyclictest9670-21ssh23:35:352
1788699281,12cyclictest3622-21diskmemload22:59:102
17871992810,9cyclictest21-21ksoftirqd/122:51:221
17862992815,9cyclictest32547-21fschecks_count19:25:160
1788699276,4cyclictest31695-21/usr/sbin/munin21:25:172
1788699276,4cyclictest0-21swapper/222:49:292
1788699276,4cyclictest0-21swapper/222:49:282
1788699275,20cyclictest261rcuc/222:34:182
1788699273,8cyclictest27-21ksoftirqd/223:50:352
1788699273,13cyclictest27-21ksoftirqd/222:23:342
1788699271,21cyclictest29994-21ssh23:48:182
1788699271,21cyclictest0-21swapper/200:13:222
1787199272,4cyclictest131rcu_preempt19:45:151
1787199272,19cyclictest26499-21ntp_states23:45:251
1787199272,19cyclictest19996-21ssh00:19:261
17871992712,6cyclictest131rcu_preempt20:35:121
1787199271,18cyclictest4016-21ssh21:10:141
17871992711,8cyclictest21-21ksoftirqd/123:05:011
17871992710,8cyclictest21-21ksoftirqd/123:34:171
17862992721,3cyclictest26681-21apt-get21:00:120
17862992713,10cyclictest0-21swapper/020:30:130
1788699268,5cyclictest0-21swapper/223:40:112
1788699265,12cyclictest131rcu_preempt23:16:522
1788699264,8cyclictest27-21ksoftirqd/223:56:102
1788699261,20cyclictest15880-21cut20:15:172
17886992611,7cyclictest23561-21ssh23:25:292
1788699261,11cyclictest29665-21ssh23:10:362
1787199269,9cyclictest21-21ksoftirqd/123:15:451
1787199269,6cyclictest21-21ksoftirqd/121:24:511
1787199262,18cyclictest27270-21sh00:22:251
17871992617,6cyclictest0-21swapper/121:25:221
17871992611,8cyclictest21-21ksoftirqd/123:09:541
17862992618,6cyclictest20787-21cut20:20:160
17896992514,5cyclictest0-21swapper/300:20:153
1789699251,18cyclictest0-21swapper/323:23:253
1789699251,18cyclictest0-21swapper/323:23:253
1788699253,8cyclictest0-21swapper/200:03:432
1788699251,5cyclictest27-21ksoftirqd/221:35:542
1788699251,5cyclictest0-21swapper/220:20:122
17886992512,9cyclictest28781-21ssh22:00:142
1788699251,17cyclictest0-21swapper/221:17:542
17886992511,6cyclictest22688-21sshd19:45:252
1788699251,15cyclictest27-21ksoftirqd/200:08:032
1788699251,15cyclictest0-21swapper/219:35:042
1788699251,15cyclictest0-21swapper/219:35:042
1787199259,9cyclictest21-21ksoftirqd/100:06:511
1787199257,4cyclictest21-21ksoftirqd/122:00:331
1787199256,3cyclictest131rcu_preempt22:35:211
1787199253,13cyclictest8435-21ssh23:55:111
1787199251,8cyclictest131rcu_preempt21:40:251
1787199251,7cyclictest21-21ksoftirqd/119:35:141
1787199251,7cyclictest21-21ksoftirqd/119:35:131
17871992511,3cyclictest21-21ksoftirqd/122:08:031
17871992510,8cyclictest21-21ksoftirqd/122:28:391
17871992510,5cyclictest21-21ksoftirqd/121:39:351
17896992418,4cyclictest321rcuc/322:00:143
17896992414,8cyclictest35966irq/58-eth022:25:123
1789699241,16cyclictest0-21swapper/322:35:133
1788699248,4cyclictest0-21swapper/220:50:142
1788699245,5cyclictest27346irq/40-dwc2_hso22:05:212
1788699243,16cyclictest27346irq/40-dwc2_hso19:55:142
1788699242,8cyclictest271ksoftirqd/221:46:482
1788699241,8cyclictest0-21swapper/220:45:142
1787199249,6cyclictest21-21ksoftirqd/121:51:161
1787199248,9cyclictest21-21ksoftirqd/122:21:561
1787199248,9cyclictest21-21ksoftirqd/100:26:311
1787199248,7cyclictest21-21ksoftirqd/100:38:571
1787199248,10cyclictest21-21ksoftirqd/121:30:511
1787199248,10cyclictest21-21ksoftirqd/100:01:501
17871992414,7cyclictest9439-21ssh23:35:321
17871992411,4cyclictest21-21ksoftirqd/122:10:331
1789699235,13cyclictest35966irq/58-eth022:50:113
1789699234,15cyclictest35966irq/58-eth019:25:143
1788699231,15cyclictest0-21swapper/219:50:122
1788699231,11cyclictest22772-21kthreadcore20:55:202
1788699231,10cyclictest0-21swapper/219:10:122
1788699230,9cyclictest1912-21ps20:00:172
1787199238,5cyclictest21-21ksoftirqd/100:13:301
1787199237,10cyclictest21-21ksoftirqd/122:42:251
1787199236,8cyclictest21-21ksoftirqd/123:43:341
1787199235,8cyclictest21-21ksoftirqd/122:45:111
1787199235,8cyclictest21-21ksoftirqd/122:45:101
17871992318,3cyclictest0-21swapper/119:10:121
17871992310,6cyclictest21-21ksoftirqd/123:13:271
17871992310,5cyclictest21-21ksoftirqd/123:54:241
17862992310,9cyclictest17957-21grep19:10:020
1789699229,10cyclictest0-21swapper/300:10:203
1789699224,15cyclictest35966irq/58-eth023:25:123
17896992218,3cyclictest321rcuc/321:00:243
17896992216,4cyclictest36066irq/59-eth021:10:303
1788699225,6cyclictest27-21ksoftirqd/219:40:132
1788699224,14cyclictest9606-21sendmail_mailst20:05:272
1788699224,14cyclictest9606-21sendmail_mailst20:05:262
1788699224,12cyclictest27-21ksoftirqd/220:10:212
1788699222,6cyclictest27-21ksoftirqd/220:40:242
1788699221,10cyclictest26371-21kthreadcore20:25:202
1787199228,7cyclictest21-21ksoftirqd/120:30:131
1787199228,6cyclictest131rcu_preempt19:50:271
1787199227,3cyclictest131rcu_preempt20:15:261
1787199221,6cyclictest21-21ksoftirqd/119:20:211
1787199221,3cyclictest0-21swapper/119:25:251
17871992210,4cyclictest21-21ksoftirqd/119:40:251
1789699215,12cyclictest35966irq/58-eth023:00:133
17896992115,4cyclictest321rcuc/320:40:133
17896992115,4cyclictest321rcuc/319:40:143
17896992114,5cyclictest36066irq/59-eth023:10:143
17896992114,5cyclictest35966irq/58-eth021:55:123
17896992113,5cyclictest35966irq/58-eth023:30:143
1789699211,13cyclictest0-21swapper/321:20:143
1789699211,11cyclictest0-21swapper/321:30:143
17896992110,8cyclictest35966irq/58-eth022:05:133
1788699214,5cyclictest0-21swapper/219:30:122
1788699213,12cyclictest27-21ksoftirqd/219:15:312
1788699211,16cyclictest27346irq/40-dwc2_hso19:25:122
17871992110,4cyclictest21-21ksoftirqd/120:20:221
17871992110,3cyclictest131rcu_preempt20:45:161
17871992110,2cyclictest21-21ksoftirqd/121:05:161
17896992015,3cyclictest321rcuc/320:15:253
17896992014,4cyclictest321rcuc/319:15:143
17896992014,4cyclictest321rcuc/319:10:113
17896992014,4cyclictest321rcuc/300:00:133
17896992013,5cyclictest0-21swapper/321:35:123
17896992012,5cyclictest35966irq/58-eth023:50:123
17896992011,4cyclictest0-21swapper/321:19:033
1789699201,13cyclictest0-21swapper/300:28:233
1789699201,12cyclictest0-21swapper/323:40:253
1787199208,6cyclictest131rcu_preempt20:40:161
1787199208,3cyclictest131rcu_preempt19:30:211
1787199202,14cyclictest28834-21apt-get19:55:121
17871992015,3cyclictest0-21swapper/120:25:121
1789699198,9cyclictest0-21swapper/300:38:223
17896991914,3cyclictest321rcuc/321:45:133
17896991914,3cyclictest321rcuc/320:55:153
17896991914,3cyclictest321rcuc/319:30:113
17896991913,4cyclictest321rcuc/320:50:133
17896991913,4cyclictest321rcuc/320:10:123
17896991913,4cyclictest321rcuc/319:20:123
17896991913,4cyclictest0-21swapper/323:16:253
17896991911,5cyclictest35966irq/58-eth023:35:143
1789699191,13cyclictest0-21swapper/321:25:203
1789699191,12cyclictest0-21swapper/323:45:163
17896991910,7cyclictest0-21swapper/322:15:143
17896991910,7cyclictest0-21swapper/322:15:143
1787199196,4cyclictest21-21ksoftirqd/119:15:171
1787199192,8cyclictest27333-21kernelversion21:00:181
17871991914,3cyclictest0-21swapper/120:55:181
1789699188,6cyclictest35966irq/58-eth022:10:153
17896991813,3cyclictest36066irq/59-eth000:17:343
17896991813,3cyclictest321rcuc/322:47:343
17896991813,3cyclictest321rcuc/322:47:333
17896991812,4cyclictest0-21swapper/322:56:493
17896991812,4cyclictest0-21swapper/300:05:143
17896991811,4cyclictest0-21swapper/323:55:233
17896991810,6cyclictest0-21swapper/321:51:083
1789699180,10cyclictest35966irq/58-eth021:05:133
1788699181,9cyclictest17938-21latency_hist19:10:022
17871991813,3cyclictest0-21swapper/120:10:111
1789699178,6cyclictest0-21swapper/300:30:193
17896991712,3cyclictest0-21swapper/320:25:323
17896991712,3cyclictest0-21swapper/320:20:143
17896991711,4cyclictest321rcuc/320:00:123
17896991711,4cyclictest321rcuc/319:50:133
17896991711,4cyclictest0-21swapper/322:30:023
17896991711,4cyclictest0-21swapper/322:20:023
17896991711,4cyclictest0-21swapper/319:45:233
1789699171,11cyclictest0-21swapper/322:40:123
1789699171,11cyclictest0-21swapper/320:05:133
1789699171,11cyclictest0-21swapper/320:05:133
1789699169,4cyclictest321rcuc/319:55:133
17896991611,3cyclictest0-21swapper/320:30:263
17896991610,4cyclictest321rcuc/319:35:143
17896991610,4cyclictest321rcuc/319:35:133
17896991510,3cyclictest0-21swapper/320:35:293
1787199142,4cyclictest131rcu_preempt19:10:021
1789699101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional