You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-27 - 16:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Fri Mar 27, 2026 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1635499745,56cyclictest25711-21ssh10:35:131
16342996627,9cyclictest12-21ksoftirqd/008:55:130
1636799592,32cyclictest924-21snmpd10:50:122
1634299544,31cyclictest9760-21ssh09:50:130
1635499523,37cyclictest22213-21ssh10:15:131
1635499523,37cyclictest22213-21ssh10:15:131
16377995116,19cyclictest28366irq/58-eth011:50:133
1635499503,36cyclictest22258-21ssh11:45:121
1634299502,41cyclictest4242-21rm11:35:160
1634299493,37cyclictest2436-21ssh12:10:110
16354994728,14cyclictest211ksoftirqd/108:25:121
16367994413,19cyclictest131rcu_preempt08:05:132
16367994413,19cyclictest131rcu_preempt08:05:122
1635499442,32cyclictest780-21sh09:45:141
1637799435,24cyclictest28366irq/58-eth012:30:143
1637799432,21cyclictest28366irq/58-eth009:20:113
1636799431,29cyclictest15186-21ssh09:35:122
16354994316,20cyclictest21-21ksoftirqd/107:45:121
1637799373,18cyclictest28366irq/58-eth012:35:123
1635499373,26cyclictest13254-21sh11:40:131
1635499373,26cyclictest13254-21sh11:40:121
16354993717,13cyclictest16335-21cyclictest12:20:131
1637799362,17cyclictest28366irq/58-eth009:15:113
1636799362,25cyclictest22346irq/40-dwc2_hso11:15:122
1636799362,20cyclictest20472-21ssh09:20:112
1634299361,26cyclictest2276-21ssh10:40:120
1636799342,20cyclictest924-21snmpd08:00:102
16354993419,9cyclictest15526-21apt-get08:50:111
16342993420,6cyclictest121ksoftirqd/012:35:130
1636799322,15cyclictest15065-21ssh10:28:342
1635499322,23cyclictest6141-21ssh09:30:141
1636799312,11cyclictest11997-21kthreadcore11:20:212
16367993117,9cyclictest2865-21ssh09:26:342
16367993117,10cyclictest0-21swapper/210:10:142
1636799310,20cyclictest785246ssh11:55:142
1637799303,20cyclictest0-21swapper/310:14:483
1636799301,20cyclictest27-21ksoftirqd/208:30:132
1635499302,22cyclictest9776-21ssh11:36:221
1635499302,21cyclictest31278-21rm11:50:131
1635499302,20cyclictest11609-21ssh12:15:161
1636799292,8cyclictest0-21swapper/207:40:132
1636799292,8cyclictest0-21swapper/207:40:122
1636799292,13cyclictest0-21swapper/210:06:332
1636799292,12cyclictest11044-21ssh09:30:342
1636799292,11cyclictest1-21systemd10:59:592
16367992914,12cyclictest22319-21df_inode10:15:162
16367992914,12cyclictest22319-21df_inode10:15:152
1636799291,22cyclictest0-21swapper/212:15:222
16354992922,3cyclictest21298-21apt-get07:15:131
16354992921,4cyclictest10717-21apt-get08:45:131
16354992921,4cyclictest10717-21apt-get08:45:121
1635499290,8cyclictest13469-21ntp_states09:50:251
16342992924,2cyclictest12-21ksoftirqd/008:15:120
1636799283,18cyclictest22346irq/40-dwc2_hso09:50:132
1636799282,21cyclictest9146-21ssh09:49:432
1636799282,20cyclictest16712-21ssh10:30:132
1636799282,20cyclictest0-21swapper/208:40:132
1636799282,11cyclictest16876-21apt-get12:00:132
16367992814,11cyclictest7582-21ssh12:30:212
1635499282,21cyclictest31611-21sh09:43:341
16354992821,3cyclictest20343-21apt-get08:55:141
1634299287,11cyclictest666-21ntpd08:45:130
1634299287,11cyclictest666-21ntpd08:45:120
16342992821,3cyclictest12-21ksoftirqd/011:30:120
16342992818,5cyclictest0-21swapper/009:20:110
1634299280,3cyclictest131rcu_preempt09:55:140
1637799271,13cyclictest28366irq/58-eth008:40:123
1636799272,20cyclictest6059-21ssh11:53:222
1635499275,15cyclictest0-21swapper/111:30:121
1635499272,19cyclictest5316-21ssh09:29:341
1635499272,18cyclictest469-21/usr/sbin/munin11:15:231
16354992718,4cyclictest15038-21apt-get12:35:121
1634299272,19cyclictest21631-21ssh10:14:480
1637799262,18cyclictest0-21swapper/310:07:343
1637799262,12cyclictest28366irq/58-eth012:10:123
1636799263,8cyclictest0-21swapper/211:36:222
1636799262,19cyclictest0-21swapper/212:09:222
1636799261,18cyclictest14918-21ssh11:05:132
1636799261,11cyclictest31486-21ls11:30:282
1635499268,8cyclictest0-21swapper/109:16:581
1635499267,12cyclictest0-21swapper/107:35:121
1635499262,18cyclictest0-21swapper/110:45:131
16354992617,4cyclictest13176-21apt-get10:10:141
16354992616,6cyclictest12992-21apt-get07:40:131
16354992616,6cyclictest12992-21apt-get07:40:121
1634299264,15cyclictest15171-21ssh09:35:110
1634299263,16cyclictest16703-21ssh10:30:130
1634299262,7cyclictest131rcu_preempt11:10:030
16342992612,8cyclictest2161-21diskmemload09:49:130
1634299261,20cyclictest2547-21ssh09:10:110
16342992610,10cyclictest121ksoftirqd/008:40:120
16377992515,8cyclictest0-21swapper/310:25:003
1636799259,13cyclictest0-21swapper/209:15:112
1636799254,14cyclictest22346irq/40-dwc2_hso12:10:112
1636799252,16cyclictest0-21swapper/209:00:052
1636799252,16cyclictest0-21swapper/209:00:042
1636799252,12cyclictest22346irq/40-dwc2_hso08:20:112
1636799251,9cyclictest26881-21ssh09:59:342
16367992514,8cyclictest0-21swapper/210:20:292
1635499257,14cyclictest0-21swapper/110:55:151
16354992516,6cyclictest7834-21apt-get11:55:151
1634299253,18cyclictest9560-21sh11:20:130
1634299253,17cyclictest25912-21ssh12:05:140
1634299252,17cyclictest11513-21ssh09:15:110
16342992519,3cyclictest0-21swapper/007:25:130
1637799243,13cyclictest28366irq/58-eth007:40:133
1637799243,13cyclictest28366irq/58-eth007:40:123
1636799242,8cyclictest0-21swapper/211:10:042
1636799241,7cyclictest18925-21ls08:20:012
1636799241,18cyclictest839-21munin-node12:35:132
1636799241,11cyclictest2275-21sh10:40:112
1635499246,5cyclictest0-21swapper/110:30:111
1635499246,4cyclictest0-21swapper/111:02:211
1635499246,4cyclictest0-21swapper/110:27:531
1635499243,5cyclictest21-21ksoftirqd/110:20:161
1635499243,14cyclictest4218-21ssh10:05:121
16354992416,4cyclictest9537-21apt-get11:20:121
16354992416,4cyclictest20461-21apt-get09:20:111
16354992414,7cyclictest20204-21apt-get10:50:121
1635499240,9cyclictest4575-21open_inodes08:35:251
1634299248,6cyclictest131rcu_preempt12:15:140
16342992419,3cyclictest16335-21cyclictest07:20:120
16342992410,10cyclictest12-21ksoftirqd/010:45:110
1637799239,9cyclictest28366irq/58-eth010:35:123
16377992315,5cyclictest28366irq/58-eth012:25:133
1636799237,13cyclictest27981-21ssh10:00:172
1636799234,16cyclictest3035-21cron07:30:002
1636799232,6cyclictest0-21swapper/210:47:342
1636799232,15cyclictest22346irq/40-dwc2_hso07:15:132
1636799231,18cyclictest23517-21users11:25:312
1636799231,17cyclictest13272-21ssh11:40:142
1636799231,17cyclictest13272-21ssh11:40:132
16367992311,10cyclictest0-21swapper/210:35:322
1636799230,18cyclictest27650-21latency09:40:242
1636799230,10cyclictest22273-21ssh11:45:142
1635499234,4cyclictest21-21ksoftirqd/110:00:021
1635499233,5cyclictest21-21ksoftirqd/112:09:201
1635499233,4cyclictest21-21ksoftirqd/112:04:041
1635499233,4cyclictest21-21ksoftirqd/111:06:381
1635499233,10cyclictest21-21ksoftirqd/112:32:221
1635499231,15cyclictest28797-21ls08:30:151
1634299239,10cyclictest18123-21kthreadcore11:05:220
1634299232,15cyclictest31210-21ssh10:20:140
1634299231,8cyclictest16306-21munin-run10:30:020
16342992317,4cyclictest25547-21kthreadcore11:45:210
16342992312,8cyclictest19381-21irqstats11:25:190
16377992212,7cyclictest0-21swapper/312:20:003
1636799223,14cyclictest22346irq/40-dwc2_hso08:50:132
1636799222,9cyclictest8061-21latency_hist07:35:022
1636799222,15cyclictest0-21swapper/212:29:222
16367992211,9cyclictest12551-21ssh11:02:412
16367992210,5cyclictest9068-21ssh09:12:292
1635499229,8cyclictest29547-21ssh12:25:131
1635499227,3cyclictest0-21swapper/109:35:061
1635499226,4cyclictest0-21swapper/110:42:191
1635499226,4cyclictest0-21swapper/109:10:571
1635499224,5cyclictest21-21ksoftirqd/112:14:111
1635499224,4cyclictest21-21ksoftirqd/111:10:031
1635499223,4cyclictest21-21ksoftirqd/107:30:021
16354992210,7cyclictest16335-21cyclictest08:10:131
1634299222,8cyclictest9553-21memory12:30:230
16342992217,3cyclictest16245-21latency_hist07:10:020
16342992213,6cyclictest21620-21irqcore07:15:160
1634299221,19cyclictest10080-21ls09:30:300
1637799219,9cyclictest0-21swapper/309:35:243
1637799213,10cyclictest0-21swapper/309:50:133
16377992112,7cyclictest28366irq/58-eth009:40:143
16377992112,6cyclictest28366irq/58-eth011:00:123
1636799219,9cyclictest0-21swapper/207:35:182
1636799218,10cyclictest27516-21df_inode07:55:152
1636799211,17cyclictest2798-21kthreadcore08:35:202
1635499216,4cyclictest0-21swapper/108:00:261
1635499214,4cyclictest21-21ksoftirqd/110:00:171
16354992114,3cyclictest5909-21apt-get08:40:131
1635499211,17cyclictest18674-21rm11:25:131
1634299212,15cyclictest29567-21ssh12:25:150
16342992113,5cyclictest4032-21/usr/sbin/munin11:15:230
16342992112,7cyclictest29632-21diskstats09:25:140
16342992112,6cyclictest12-21ksoftirqd/007:50:150
1634299211,14cyclictest29169-21ls12:25:000
1634299211,10cyclictest31320-21unixbench_singl07:55:280
1634299210,18cyclictest5872-21ssh11:00:130
1637799203,11cyclictest28366irq/58-eth010:00:133
1637799202,13cyclictest0-21swapper/310:45:133
1637799201,16cyclictest28366irq/58-eth012:00:123
1637799201,14cyclictest28366irq/58-eth009:30:133
16377992010,7cyclictest0-21swapper/311:35:223
1636799209,8cyclictest0-21swapper/207:45:162
1636799209,4cyclictest24923-21kthreadcore08:25:182
1636799206,5cyclictest22778-21fschecks_count07:50:162
1636799201,16cyclictest11509-21kthreadcore08:10:222
1636799201,15cyclictest21547-21kthreadcore08:55:192
1636799201,10cyclictest21226-21irqstats12:20:182
16354992012,6cyclictest25338-21df_inode09:00:151
16354992012,6cyclictest25338-21df_inode09:00:141
16354992012,5cyclictest18086-21unixbench_singl08:15:281
1634299206,3cyclictest131rcu_preempt11:50:230
1634299203,14cyclictest27665-21ssh10:00:130
1634299202,15cyclictest16908-21ssh12:00:140
1634299201,9cyclictest2173-21cut09:10:000
16342992012,6cyclictest30348-21cat10:35:300
16342992012,5cyclictest29778-21cron09:05:010
16342992012,5cyclictest29778-21cron09:05:000
1634299201,16cyclictest32256-21apt-get08:00:110
1634299201,16cyclictest29215-21ssh10:55:120
1634299201,15cyclictest7857-21ssh11:55:140
1637799199,6cyclictest28366irq/58-eth011:07:343
1637799193,11cyclictest0-21swapper/311:10:033
1637799192,14cyclictest0-21swapper/309:10:113
1637799192,13cyclictest0-21swapper/309:25:153
1637799192,11cyclictest0-21swapper/311:20:053
1637799192,11cyclictest0-21swapper/308:50:133
1637799192,10cyclictest0-21swapper/307:30:043
1637799192,10cyclictest0-21swapper/307:15:143
16377991913,4cyclictest0-21swapper/311:45:013
16377991913,4cyclictest0-21swapper/311:45:003
16377991911,5cyclictest28366irq/58-eth012:20:123
1637799191,14cyclictest28366irq/58-eth010:55:163
1637799191,13cyclictest0-21swapper/311:15:163
1637799191,13cyclictest0-21swapper/308:45:273
1637799191,13cyclictest0-21swapper/308:45:263
1637799191,12cyclictest0-21swapper/310:40:123
1637799191,11cyclictest0-21swapper/309:00:123
1637799191,11cyclictest0-21swapper/309:00:113
1636799195,6cyclictest12609-21kthreadcore08:45:202
1636799195,6cyclictest12609-21kthreadcore08:45:192
1636799191,8cyclictest18444-21kthreadcore07:10:212
1636799191,15cyclictest26303-21fschecks_time07:20:162
1636799191,15cyclictest1235-21switchtime09:05:272
1635499194,12cyclictest19052-21apt-get08:20:101
16354991911,5cyclictest7190-21grep07:30:281
1634299192,15cyclictest8458-21unixbench_singl08:05:290
1634299192,15cyclictest8458-21unixbench_singl08:05:280
1634299191,8cyclictest19162-21fschecks_count08:20:140
16342991913,4cyclictest16805-21irqcore07:10:180
1634299191,14cyclictest20226-21ssh10:50:140
16342991910,6cyclictest25055-21kthreadcore09:40:200
1637799187,9cyclictest0-21swapper/312:05:063
1637799182,11cyclictest0-21swapper/311:48:223
16377991811,4cyclictest0-21swapper/310:25:183
16377991811,4cyclictest0-21swapper/310:15:223
16377991811,4cyclictest0-21swapper/310:15:213
1637799181,11cyclictest0-21swapper/308:30:023
1637799181,10cyclictest0-21swapper/307:10:113
1637799180,11cyclictest0-21swapper/307:35:123
1636799181,8cyclictest16398-21latency_hist07:10:012
1635499185,10cyclictest30677-21cron07:25:011
1635499182,4cyclictest21-21ksoftirqd/108:00:021
1635499181,9cyclictest25594-21kthreadcore07:50:221
1634299187,8cyclictest13322-21cpuspeed11:40:140
1634299187,8cyclictest13322-21cpuspeed11:40:130
1634299187,6cyclictest0-21swapper/007:30:040
1634299184,12cyclictest11612-21kthreadcore08:10:210
1634299181,9cyclictest29039-21ps08:30:170
1634299181,3cyclictest0-21swapper/008:50:140
16342991812,4cyclictest4954-21kthreadcore10:05:200
1634299181,14cyclictest17812-21apt-get07:45:110
1634299181,13cyclictest22963-21kthreadcore10:15:200
1634299181,13cyclictest22963-21kthreadcore10:15:200
16342991810,6cyclictest24254-21awk08:25:170
1637799179,6cyclictest0-21swapper/311:25:273
1637799179,6cyclictest0-21swapper/309:56:123
1637799179,6cyclictest0-21swapper/309:45:283
1637799173,5cyclictest0-21swapper/311:55:223
1637799172,7cyclictest0-21swapper/311:30:123
1637799172,10cyclictest0-21swapper/308:55:153
1637799172,10cyclictest0-21swapper/308:20:123
16377991712,2cyclictest0-21swapper/310:30:143
1637799171,13cyclictest0-21swapper/309:05:113
1635499172,12cyclictest790-21kthreadcore09:05:211
1635499172,10cyclictest0-21swapper/107:10:131
1635499171,11cyclictest0-21swapper/108:05:141
1635499171,11cyclictest0-21swapper/108:05:131
1634299174,10cyclictest9587-21kthreadcore07:35:200
1634299172,10cyclictest0-21swapper/007:40:140
1634299172,10cyclictest0-21swapper/007:40:130
1634299171,13cyclictest1599-21irqrtprio08:35:180
1637799168,6cyclictest0-21swapper/310:52:103
1637799161,12cyclictest0-21swapper/308:05:293
1637799161,12cyclictest0-21swapper/308:05:283
1635499162,10cyclictest16360-21latency_hist07:10:021
1637799159,4cyclictest33-21ksoftirqd/308:10:163
1637799152,9cyclictest0-21swapper/307:25:153
1637799151,9cyclictest0-21swapper/308:15:203
1637799141,7cyclictest0-21swapper/308:30:123
1637799141,7cyclictest0-21swapper/307:50:133
1637799141,3cyclictest0-21swapper/307:20:123
1637799141,10cyclictest0-21swapper/307:55:153
1637799138,3cyclictest0-21swapper/308:00:253
1637799135,4cyclictest0-21swapper/308:35:123
1637799132,9cyclictest0-21swapper/307:50:013
1637799101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional