You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-19 - 06:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Thu Mar 19, 2026 00:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1151999635,20cyclictest12-21ksoftirqd/022:45:130
1152499574,10cyclictest131rcu_preempt23:05:121
11541995448,4cyclictest28366irq/58-eth022:00:133
11524995434,7cyclictest21-21ksoftirqd/121:00:121
1151999536,7cyclictest131rcu_preempt21:45:130
1153299511,3cyclictest131rcu_preempt20:30:112
1152499511,39cyclictest30227-21ssh21:10:131
11532995023,23cyclictest0-21swapper/223:15:122
1153299481,30cyclictest6788-21ssh21:15:112
11519994835,8cyclictest12-21ksoftirqd/000:35:130
11541994425,11cyclictest321rcuc/323:50:133
1152499443,30cyclictest20963-21ssh00:05:121
1151999423,30cyclictest19123-21ssh23:10:130
11524994115,19cyclictest3030-21ssh23:35:261
1153299401,29cyclictest9947-21ssh00:35:132
11524994011,17cyclictest1224-21ssh23:00:131
1152499393,5cyclictest131rcu_preempt22:55:131
1151999393,33cyclictest6798-21ssh21:15:130
11519993917,17cyclictest12-21ksoftirqd/019:20:120
11541993824,10cyclictest28366irq/58-eth020:30:103
11519993815,17cyclictest12-21ksoftirqd/023:25:120
11519993714,17cyclictest12-21ksoftirqd/000:25:130
1154199353,20cyclictest28366irq/58-eth021:55:123
1153299353,25cyclictest0-21swapper/219:10:112
1151999352,24cyclictest12-21ksoftirqd/019:30:120
1153299342,25cyclictest3326-21mailstats22:05:272
1153299342,25cyclictest3326-21mailstats22:05:262
1152499342,4cyclictest131rcu_preempt21:35:111
1151999342,23cyclictest4840-21rm21:50:130
1151999342,23cyclictest4840-21rm21:50:120
1151999342,22cyclictest2880-21ssh23:55:130
11541993320,10cyclictest0-21swapper/323:30:143
1154199331,25cyclictest28366irq/58-eth023:10:143
1152499332,24cyclictest21-21ksoftirqd/122:50:111
1152499332,23cyclictest20840-21cat19:15:281
1152499331,23cyclictest0-21swapper/122:15:111
1151999331,23cyclictest29946-21ssh22:40:110
1154199323,18cyclictest28366irq/58-eth021:50:123
1154199323,18cyclictest28366irq/58-eth021:50:113
1153299321,6cyclictest0-21swapper/222:45:132
1151999325,23cyclictest31937-21sh22:05:120
1151999325,23cyclictest31937-21sh22:05:120
1151999324,21cyclictest13877-21ssh21:55:110
1151999321,23cyclictest12-21ksoftirqd/020:10:120
1153299319,14cyclictest0-21swapper/221:55:142
1153299312,11cyclictest2158-21ssh22:24:272
1152499312,21cyclictest1327-21ssh21:30:131
11524993117,10cyclictest0-21swapper/122:20:131
1151999312,23cyclictest22759-21ssh23:30:130
11519993111,6cyclictest0-21swapper/020:30:110
11541993017,8cyclictest0-21swapper/322:05:133
11541993017,8cyclictest0-21swapper/322:05:133
1153299309,7cyclictest0-21swapper/219:45:132
1153299309,7cyclictest0-21swapper/219:45:122
1153299303,5cyclictest131rcu_preempt00:25:122
1152499301,22cyclictest1009-21ssh00:30:131
1151999302,22cyclictest28164-21ssh23:15:120
11519993014,10cyclictest12-21ksoftirqd/019:45:120
11519993014,10cyclictest12-21ksoftirqd/019:45:120
1153299292,16cyclictest4839-21diskmemload21:50:122
1153299292,16cyclictest4839-21diskmemload21:50:122
11532992912,7cyclictest0-21swapper/220:05:022
1152499292,20cyclictest20961-21ssh22:35:121
1151999292,21cyclictest19265-21ssh22:33:280
1151999292,21cyclictest19265-21ssh22:33:270
1151999291,22cyclictest8498-21ssh22:10:120
11541992813,11cyclictest0-21swapper/322:10:113
1153299284,18cyclictest22346irq/40-dwc2_hso19:20:122
1153299282,19cyclictest0-21swapper/222:40:122
1153299282,18cyclictest17291-21ssh23:45:112
1153299282,14cyclictest22346irq/40-dwc2_hso21:45:132
1152499282,18cyclictest26286-21ssh23:50:121
11519992816,3cyclictest12-21ksoftirqd/022:29:210
1151999281,19cyclictest26518-21ssh22:20:140
1153299274,16cyclictest22346irq/40-dwc2_hso22:25:142
1153299272,9cyclictest26387-21ssh23:13:272
1153299272,6cyclictest32417-21sshd20:05:122
1152499272,19cyclictest5669-21ssh00:14:271
1152499272,19cyclictest5669-21ssh00:14:271
1152499272,19cyclictest30071-21ssh23:33:261
1151999272,18cyclictest22840-21ssh22:00:130
11519992714,4cyclictest12-21ksoftirqd/021:35:310
11519992714,3cyclictest12-21ksoftirqd/023:46:450
1154199262,11cyclictest0-21swapper/321:15:123
1154199261,15cyclictest28366irq/58-eth023:45:113
1153299262,8cyclictest15607-21ssh23:43:272
1153299261,9cyclictest2654066ssh22:00:242
11532992613,10cyclictest5667-21sshd00:14:282
11532992613,10cyclictest5667-21sshd00:14:272
11532992610,13cyclictest0-21swapper/222:32:112
11532992610,13cyclictest0-21swapper/222:32:102
1151999260,11cyclictest26100-21sendmail22:20:010
1154199252,17cyclictest0-21swapper/322:20:263
1154199252,11cyclictest0-21swapper/320:05:133
11541992515,7cyclictest28366irq/58-eth022:50:163
11541992510,10cyclictest0-21swapper/323:35:263
1153299259,13cyclictest14007-21swap23:05:272
1153299253,9cyclictest27-21ksoftirqd/222:55:132
1153299252,17cyclictest22346irq/40-dwc2_hso20:10:132
1153299251,12cyclictest31789-21ssh23:35:152
11532992510,5cyclictest24966-21cat22:35:292
11532992510,11cyclictest6947-21timerwakeupswit23:55:292
1152499256,7cyclictest131rcu_preempt21:25:111
1151999252,19cyclictest4732-21apt-get23:20:110
11519992510,5cyclictest12-21ksoftirqd/021:22:270
1153299249,12cyclictest19512-21ssh22:50:272
1153299248,3cyclictest27-21ksoftirqd/200:00:032
1153299242,15cyclictest22346irq/40-dwc2_hso19:30:112
1153299241,8cyclictest8474-21sh22:10:122
11532992418,3cyclictest1759-21ssh23:00:172
11532992413,9cyclictest0-21swapper/223:50:222
11532992412,9cyclictest23791-21cron20:30:002
1153299241,19cyclictest474746ssh23:20:122
1153299241,18cyclictest924-21snmpd23:25:142
1153299241,17cyclictest10543-21ssh00:15:282
1153299241,17cyclictest10543-21ssh00:15:272
1153299241,13cyclictest1293-21/usr/sbin/munin21:30:152
11532992410,6cyclictest1470-21sshd21:10:262
1151999242,17cyclictest16435-21ssh23:44:270
11519992414,7cyclictest7931-21cron23:40:010
1154199234,10cyclictest28366irq/58-eth022:25:143
1154199232,10cyclictest0-21swapper/322:33:283
1154199232,10cyclictest0-21swapper/322:33:273
1153299234,9cyclictest25013-21cron21:05:022
1153299232,12cyclictest20973-21ssh00:05:112
1153299232,11cyclictest13539-21kthreadcore21:35:212
1153299231,17cyclictest20652-21kthreadcore21:40:192
11532992311,10cyclictest24338-21ssh22:17:562
1152499233,2cyclictest131rcu_preempt23:15:111
1152499233,13cyclictest0-21swapper/119:35:131
1152499232,16cyclictest4792-21apt-get20:10:131
1152499231,2cyclictest131rcu_preempt23:55:131
1151999238,8cyclictest0-21swapper/020:15:280
1151999233,14cyclictest0-21swapper/019:10:110
1151999232,3cyclictest131rcu_preempt00:10:260
1151999232,3cyclictest131rcu_preempt00:10:260
11519992311,9cyclictest30217-21apt-get21:10:130
1154199222,12cyclictest0-21swapper/323:15:063
11541992216,4cyclictest28466irq/59-eth023:40:173
11541992212,6cyclictest0-21swapper/321:20:133
1153299229,10cyclictest17809-21kthreadcore19:15:202
1153299221,17cyclictest24766-21ssh21:25:102
1153299221,17cyclictest22713-21rm23:30:112
1153299221,17cyclictest13079-21grep20:15:262
1153299221,17cyclictest11144-21ps20:50:182
1153299221,17cyclictest11144-21ps20:50:172
1153299221,16cyclictest16144-21if_eth121:20:172
11532992210,5cyclictest17515-21kthreadcore00:20:212
1153299220,7cyclictest25231-21/usr/sbin/munin21:05:242
1152499227,9cyclictest11991-21ssh00:00:131
1152499221,2cyclictest0-21swapper/123:40:141
11524992212,3cyclictest29822-21diskmemload00:21:441
1151999227,4cyclictest12-21ksoftirqd/022:55:270
1151999225,9cyclictest32415-21sshd20:05:130
1151999221,3cyclictest131rcu_preempt20:45:200
1151999221,16cyclictest19205-21apt-get20:25:130
1154199213,11cyclictest28366irq/58-eth023:25:123
1154199211,9cyclictest28366irq/58-eth000:17:353
1154199211,9cyclictest28366irq/58-eth000:17:343
11541992112,7cyclictest0-21swapper/300:25:163
1153299214,4cyclictest27-21ksoftirqd/220:35:162
1153299214,14cyclictest4793-21kthreadcore19:35:202
1153299211,2cyclictest131rcu_preempt20:55:232
11532992111,8cyclictest0-21swapper/200:32:292
1153299211,11cyclictest10695-21kthreadcore19:40:202
1153299211,10cyclictest29266-21cut19:25:222
1153299211,10cyclictest27483-21latency_hist20:00:012
11532992110,9cyclictest0-21swapper/220:45:142
1152499215,5cyclictest21-21ksoftirqd/122:30:201
1152499215,5cyclictest21-21ksoftirqd/122:30:201
1152499211,7cyclictest24091-21cat21:40:321
11524992114,5cyclictest0-21swapper/100:17:341
11524992114,5cyclictest0-21swapper/100:17:341
11524992112,6cyclictest20188-21sh21:00:001
11524992112,5cyclictest211ksoftirqd/100:25:111
1152499211,15cyclictest15817-21ssh21:20:141
11524992110,8cyclictest3102-21awk22:25:141
1151999215,3cyclictest12-21ksoftirqd/000:18:010
1151999215,3cyclictest12-21ksoftirqd/000:18:010
1151999215,11cyclictest12-21ksoftirqd/019:16:280
1151999212,2cyclictest131rcu_preempt23:00:190
1151999211,6cyclictest25284-21irqcore21:25:180
1154199203,12cyclictest0-21swapper/322:17:283
11541992010,8cyclictest0-21swapper/321:35:173
1153299202,9cyclictest17911-21fschecks_count19:50:162
1153299202,9cyclictest14817-21irqcore20:20:172
1153299201,11cyclictest1909-21kthreadcore20:40:152
1152499202,6cyclictest28290-21apt-get21:45:121
1152499201,13cyclictest9963-21ssh00:35:131
1152499201,12cyclictest5943-21ls20:45:021
1151999208,2cyclictest12-21ksoftirqd/022:39:480
1151999206,8cyclictest12-21ksoftirqd/020:40:210
1151999204,3cyclictest131rcu_preempt19:35:200
1151999202,2cyclictest131rcu_preempt23:05:540
1151999201,9cyclictest0-21swapper/000:08:510
1151999201,6cyclictest0-21swapper/021:30:130
1151999201,4cyclictest131rcu_preempt20:50:160
1151999201,4cyclictest131rcu_preempt20:50:160
1151999201,4cyclictest0-21swapper/023:54:210
1154199197,7cyclictest28366irq/58-eth022:45:123
11541991914,3cyclictest28466irq/59-eth023:20:133
1154199191,12cyclictest0-21swapper/321:50:023
11541991910,6cyclictest28366irq/58-eth021:30:233
1152499199,8cyclictest8100-21cron22:10:011
1152499199,8cyclictest8100-21cron22:10:001
1152499191,8cyclictest30059-21sed19:25:281
11524991912,4cyclictest13988-21df_inode21:55:131
1152499190,13cyclictest31136-21switchtime20:00:281
1152499190,12cyclictest17906-21cut19:50:151
1151999199,8cyclictest12-21ksoftirqd/021:05:130
1151999195,3cyclictest12-21ksoftirqd/020:05:000
1151999194,3cyclictest131rcu_preempt21:00:000
11519991913,4cyclictest0-21swapper/000:00:140
11519991913,3cyclictest19310-21sendmail_mailst00:20:260
1151999191,14cyclictest4154-21cut00:30:210
1151999191,14cyclictest22842-21memory21:40:220
11519991910,6cyclictest19919-21cat22:50:290
1154199189,6cyclictest28366irq/58-eth022:55:123
1154199189,6cyclictest28366irq/58-eth000:00:133
1154199183,10cyclictest0-21swapper/323:03:283
1154199183,10cyclictest0-21swapper/319:17:283
1154199182,11cyclictest0-21swapper/319:25:133
1154199182,10cyclictest0-21swapper/320:35:143
1154199182,10cyclictest0-21swapper/319:45:143
1154199182,10cyclictest0-21swapper/319:45:133
1154199182,10cyclictest0-21swapper/319:10:123
1154199182,10cyclictest0-21swapper/300:11:273
1154199182,10cyclictest0-21swapper/300:11:263
1154199181,12cyclictest0-21swapper/320:10:223
11541991810,6cyclictest0-21swapper/322:39:413
1153299181,14cyclictest11588-21latency_hist19:10:022
1152499184,10cyclictest750-21sendmail-mta20:15:111
1152499182,11cyclictest12989-21apt-get19:45:121
1152499182,11cyclictest12989-21apt-get19:45:111
1152499181,9cyclictest18711-21kthreadcore23:45:191
1152499181,7cyclictest10796-21unixbench_singl22:45:291
1152499181,4cyclictest0-21swapper/122:10:091
11524991811,5cyclictest16225-21sshd19:14:291
11524991811,4cyclictest24488-21kthreadcore22:00:191
1152499181,13cyclictest10514-21ssh21:15:251
11524991810,6cyclictest0-21swapper/123:25:181
1151999185,4cyclictest12-21ksoftirqd/019:55:240
1151999183,13cyclictest14370-21/usr/sbin/munin20:20:250
1151999183,12cyclictest0-21swapper/021:00:190
1151999182,11cyclictest151rcuc/020:35:140
1151999181,7cyclictest19208-21kthreadcore19:50:190
11519991810,5cyclictest1-21systemd19:40:080
1154199179,6cyclictest0-21swapper/323:55:023
1154199179,6cyclictest0-21swapper/323:05:513
1154199179,6cyclictest0-21swapper/321:11:293
1154199179,6cyclictest0-21swapper/300:22:083
1154199179,5cyclictest28366irq/58-eth022:40:123
1154199178,5cyclictest28366irq/58-eth021:25:123
1154199172,10cyclictest0-21swapper/321:40:133
1154199172,10cyclictest0-21swapper/319:30:143
1154199172,10cyclictest0-21swapper/300:38:573
11541991710,5cyclictest0-21swapper/300:06:493
1152499178,7cyclictest21956-21cut23:10:211
1152499178,6cyclictest5336-21sed21:50:171
1152499178,6cyclictest5336-21sed21:50:171
1152499178,6cyclictest31975-21tr22:40:201
1152499173,11cyclictest29973-21cat21:10:011
1152499172,6cyclictest131rcu_preempt23:20:121
1152499171,6cyclictest17166systemd-network19:43:201
1152499171,2cyclictest0-21swapper/119:55:271
1152499171,14cyclictest1860-21grep20:05:191
1151999172,10cyclictest0-21swapper/019:25:130
1154199169,5cyclictest0-21swapper/300:30:083
1154199168,6cyclictest0-21swapper/319:52:223
1154199162,9cyclictest0-21swapper/320:15:143
1154199161,12cyclictest0-21swapper/320:55:293
1154199161,10cyclictest0-21swapper/319:20:133
1152499169,4cyclictest9257-21kthreadcore20:45:201
1152499167,6cyclictest31031-21cpuspeed19:30:131
1152499162,11cyclictest15064-21kthreadcore20:20:181
1152499161,7cyclictest28969-21sort20:35:161
1152499161,12cyclictest24177-21fschecks_count20:30:141
1152499161,12cyclictest10952-21fschecks_count20:50:151
1152499161,12cyclictest10952-21fschecks_count20:50:151
11524991610,4cyclictest25610-21usage_EthcTask19:20:291
1151999160,2cyclictest131rcu_preempt19:10:010
1152499159,4cyclictest19594-21irqcore20:25:171
1152499151,11cyclictest11606-21latency_hist19:10:011
1154199149,3cyclictest0-21swapper/320:40:153
1154199142,8cyclictest0-21swapper/319:40:143
1154199142,6cyclictest0-21swapper/320:50:143
1154199142,6cyclictest0-21swapper/320:50:133
1154199141,7cyclictest0-21swapper/320:50:013
1154199138,3cyclictest0-21swapper/321:00:133
1154199138,3cyclictest0-21swapper/320:25:123
1154199132,7cyclictest0-21swapper/319:55:043
1154199131,9cyclictest0-21swapper/321:10:023
1154199131,9cyclictest0-21swapper/320:00:203
1154199131,10cyclictest0-21swapper/319:35:183
1154199127,3cyclictest0-21swapper/320:20:153
1154199101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional