You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-28 - 14:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Aug 28, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11212996523,9cyclictest131rcu_preempt22:20:131
1121299563,46cyclictest11086-21ssh00:35:151
1121299563,46cyclictest11086-21ssh00:35:141
1120399514,32cyclictest32077-21ssh22:05:150
1120399513,10cyclictest131rcu_preempt20:25:130
1121299503,9cyclictest131rcu_preempt19:25:121
1120399455,5cyclictest0-21swapper/019:40:140
11203994220,15cyclictest12-21ksoftirqd/023:15:140
1120399421,35cyclictest16072-21ssh22:50:140
1120399412,11cyclictest29320-21diskmemload21:40:130
1120399412,11cyclictest29320-21diskmemload21:40:120
1120399411,5cyclictest3346-21apt-get22:25:150
1120399411,5cyclictest0-21swapper/000:25:120
11203994111,23cyclictest12-21ksoftirqd/000:00:120
1122499403,24cyclictest35966irq/58-eth021:35:123
11212994023,10cyclictest0-21swapper/120:55:131
11212994023,10cyclictest0-21swapper/120:55:121
1121299401,30cyclictest9070-21ssh23:40:101
1122499374,22cyclictest35966irq/58-eth023:35:133
1122499373,26cyclictest35966irq/58-eth020:35:143
1120399371,9cyclictest18504-21head23:25:290
1121299365,24cyclictest8674-21ssh22:10:141
1121299363,26cyclictest12306-21ssh22:30:131
1120399362,13cyclictest25555-21ntp_states00:05:260
1122499351,23cyclictest35966irq/58-eth021:45:133
1120399355,24cyclictest30401-21ssh22:40:130
1120399353,7cyclictest9060-21ps22:10:180
1120399353,28cyclictest29320-21diskmemload00:30:120
1120399353,26cyclictest0-21swapper/022:45:140
1120399351,12cyclictest12602-21kthreadcore20:50:200
1122499343,12cyclictest35966irq/58-eth000:30:123
11224993422,7cyclictest35966irq/58-eth022:15:113
1122499342,21cyclictest35966irq/58-eth021:15:153
1122499342,21cyclictest35966irq/58-eth021:15:143
1121299345,20cyclictest2001-21ssh00:30:111
1120399346,24cyclictest3647-21kthreadcore20:40:220
1120399343,23cyclictest17693-21ssh22:15:120
1120399341,9cyclictest21738-21kthreadcore23:10:200
1120399341,8cyclictest26708-21kthreadcore19:25:180
1120399341,8cyclictest25586-21ntp_states19:55:250
1120399341,8cyclictest13674-21tr19:10:210
1120399341,6cyclictest23946-21sed20:30:180
1120399341,5cyclictest6090-21users21:30:330
11221993321,9cyclictest261rcuc/219:50:112
11221993321,9cyclictest261rcuc/219:50:102
1122199331,19cyclictest930-21snmpd21:10:142
1121299332,20cyclictest23075-21ssh22:00:131
1120399332,7cyclictest25478-21kthreadcore23:30:210
1120399331,9cyclictest6474-21mailstats19:35:270
1120399331,9cyclictest29770-21kthreadcore20:35:210
1120399331,9cyclictest15415-21/usr/sbin/munin21:20:150
1120399331,9cyclictest12467-21grep20:15:270
1120399331,8cyclictest5044-21df_inode21:50:150
1120399331,8cyclictest18281-21fschecks_count23:45:160
1120399331,7cyclictest7195-21tr21:15:210
1120399331,7cyclictest7195-21tr21:15:200
1120399331,7cyclictest27034-21timerandwakeup22:00:300
1120399331,6cyclictest8969-21ssh23:20:240
1120399331,5cyclictest1486-21cut00:10:230
1120399331,20cyclictest3797-21ssh23:55:150
1120399331,10cyclictest26728-21kthreadcore21:25:200
1120399330,9cyclictest29693-21/usr/sbin/munin21:10:160
1120399330,8cyclictest10332-21latency_hist23:05:010
1120399330,7cyclictest30191-21latency_hist22:40:010
1122499321,19cyclictest35966irq/58-eth022:00:143
11212993227,3cyclictest3796-21sh23:55:141
11212993217,10cyclictest29320-21diskmemload21:15:161
11212993217,10cyclictest29320-21diskmemload21:15:161
1120399328,21cyclictest15175-21fschecks_count20:55:160
1120399328,21cyclictest15175-21fschecks_count20:55:160
1120399328,20cyclictest14673-21cut22:30:220
1120399321,9cyclictest4225-21df_inode20:10:140
1120399321,9cyclictest29076-21/usr/sbin/munin21:45:210
1120399321,9cyclictest26760-21/usr/sbin/munin22:20:290
1120399321,8cyclictest21880-21kthreadcore21:00:190
1120399321,8cyclictest13530-21ssh00:35:230
1120399321,8cyclictest13530-21ssh00:35:220
1120399321,7cyclictest7394-21/usr/sbin/munin00:15:270
1120399321,7cyclictest21609-21irqstats19:20:180
11203993217,10cyclictest12-21ksoftirqd/021:55:140
1120399320,9cyclictest29403-21latency_hist21:10:020
1120399320,6cyclictest13006-21find23:40:260
1122499312,18cyclictest35966irq/58-eth000:35:153
1122499312,18cyclictest35966irq/58-eth000:35:143
1122199312,18cyclictest1681-21ssh23:00:142
1121299311,25cyclictest3369-21sh22:25:181
1120399318,20cyclictest17487-21fschecks_count19:50:150
1120399318,20cyclictest17487-21fschecks_count19:50:140
1120399311,9cyclictest15188-21kthreadcore20:20:200
1120399311,8cyclictest26382-21kthreadcore22:55:210
1120399311,7cyclictest10353-21diskstats21:35:150
1120399311,5cyclictest20319-21ssh00:20:300
1120399310,9cyclictest5966-21irqrtprio20:45:190
1120399310,8cyclictest16004-21ntp_states19:45:250
1122499304,19cyclictest0-21swapper/322:25:173
1122199304,10cyclictest11195-21cyclictest19:15:142
11212993012,8cyclictest21-21ksoftirqd/100:20:141
1120399301,9cyclictest31069-21timerandwakeup23:50:290
1120399301,8cyclictest16552-21irqcore19:15:170
1120399301,5cyclictest3548-21cat23:35:240
1120399300,5cyclictest30632-21sed20:00:280
1122499297,16cyclictest35966irq/58-eth021:20:133
1122199292,20cyclictest0-21swapper/222:40:142
11212992910,13cyclictest21-21ksoftirqd/123:00:141
1120399291,7cyclictest30533-21/usr/sbin/munin19:30:310
11203992916,9cyclictest12-21ksoftirqd/023:05:150
1120399290,6cyclictest31738-21find20:05:120
1122499283,13cyclictest0-21swapper/323:00:143
11224992819,6cyclictest35966irq/58-eth022:05:153
1122199283,10cyclictest0-21swapper/222:07:302
1122199282,19cyclictest0-21swapper/222:10:132
1122199282,15cyclictest27346irq/40-dwc2_hso20:15:112
11212992813,11cyclictest19980-21ssh23:45:201
1122499273,16cyclictest0-21swapper/323:07:363
1122499273,15cyclictest0-21swapper/323:40:093
1122499272,17cyclictest35966irq/58-eth022:30:133
1122499271,14cyclictest35966irq/58-eth023:50:133
1122199274,16cyclictest27346irq/40-dwc2_hso19:25:122
1122199271,13cyclictest0-21swapper/223:50:132
1121299275,15cyclictest30392-21ssh22:40:111
1121299273,16cyclictest28751-21apt-get23:15:131
1122499263,16cyclictest0-21swapper/323:32:433
1122499263,16cyclictest0-21swapper/322:45:153
11224992610,12cyclictest35966irq/58-eth023:55:133
1122199265,13cyclictest27346irq/40-dwc2_hso23:10:122
1122199262,14cyclictest27346irq/40-dwc2_hso20:05:122
1122199261,18cyclictest9064-21ssh23:40:092
1120399260,6cyclictest11195-21cyclictest19:10:020
1120399260,6cyclictest11195-21cyclictest19:10:010
1122499254,13cyclictest0-21swapper/322:20:163
1122499252,18cyclictest0-21swapper/321:52:123
1122199255,14cyclictest27346irq/40-dwc2_hso19:55:102
1122199254,12cyclictest27346irq/40-dwc2_hso00:35:142
1122199254,12cyclictest27346irq/40-dwc2_hso00:35:132
1122199252,18cyclictest906-21ssh00:29:112
11221992516,7cyclictest261rcuc/221:15:152
11221992516,7cyclictest261rcuc/221:15:142
1122199251,18cyclictest628-21tcf-agent20:10:132
1122199251,11cyclictest93066snmpd22:30:302
1122199250,18cyclictest19586-21ls21:40:172
1122199250,18cyclictest19586-21ls21:40:162
11212992512,3cyclictest21-21ksoftirqd/121:35:131
1122499244,11cyclictest35966irq/58-eth020:05:133
1122499242,17cyclictest0-21swapper/322:50:133
11221992414,6cyclictest29320-21diskmemload22:25:182
1122199241,17cyclictest21459-21cpuspeed22:35:162
1121299241,18cyclictest18566-21apt-get20:25:111
1122499231,13cyclictest0-21swapper/321:40:133
1122499231,13cyclictest0-21swapper/321:40:123
1122199232,6cyclictest0-21swapper/221:55:052
1122199232,6cyclictest0-21swapper/221:35:112
1122199232,6cyclictest0-21swapper/221:28:122
1122199232,16cyclictest0-21swapper/222:48:112
1122199232,16cyclictest0-21swapper/221:23:112
1122199232,16cyclictest0-21swapper/200:33:112
1122199232,15cyclictest0-21swapper/223:45:122
1122199232,15cyclictest0-21swapper/223:30:112
1122199231,9cyclictest24146-21kthreadcore20:30:202
1122199231,13cyclictest12530-21apt-get19:45:122
1121299233,17cyclictest7487-1kworker/u9:123:50:131
11212992315,6cyclictest13520-21kworker/1:120:20:131
1122499229,9cyclictest35966irq/58-eth022:35:163
1122499223,14cyclictest0-21swapper/321:55:123
1122199228,10cyclictest7535-21df00:15:152
1122199226,13cyclictest6434-21tr21:50:202
1122199222,6cyclictest0-21swapper/223:20:112
1122199222,5cyclictest11195-21cyclictest00:10:132
1122199221,7cyclictest21398-21ls00:05:002
1122199221,16cyclictest27346irq/40-dwc2_hso00:20:142
1122199221,14cyclictest131rcu_preempt22:20:022
11221992210,9cyclictest27382-21sh22:00:312
1121299221,18cyclictest15444-21ssh21:20:121
11212992211,8cyclictest14483-21forks23:25:141
1122499218,7cyclictest0-21swapper/319:25:113
1122499216,10cyclictest35966irq/58-eth023:45:143
1122499213,14cyclictest0-21swapper/322:40:133
1122499212,8cyclictest0-21swapper/320:50:123
1122499211,17cyclictest35966irq/58-eth019:10:133
11224992111,7cyclictest0-21swapper/300:20:223
1122499211,15cyclictest0-21swapper/300:05:023
1122499211,14cyclictest35966irq/58-eth023:20:113
1122199218,4cyclictest0-21swapper/223:25:122
1122199217,9cyclictest0-21swapper/220:50:122
1122199216,12cyclictest614-21dbus-daemon20:57:222
1122199216,12cyclictest614-21dbus-daemon20:57:212
1122199213,12cyclictest0-21swapper/223:35:132
1122199212,15cyclictest0-21swapper/223:08:122
11221992112,6cyclictest0-21swapper/223:15:272
1122199211,16cyclictest3120-21fschecks_time19:35:172
1122199211,11cyclictest24825-21ls21:05:152
1122199211,10cyclictest2939-21kthreadcore20:40:222
11221992110,9cyclictest0-21swapper/223:55:212
1122199210,7cyclictest24595-21switchtime19:20:272
1121299216,10cyclictest0-21swapper/123:35:131
11212992115,3cyclictest0-21swapper/122:35:161
11212992113,6cyclictest21-21ksoftirqd/123:15:021
11212992112,6cyclictest21308-21ssh22:15:241
1121299211,17cyclictest6982-21ssh22:45:141
1122499202,12cyclictest0-21swapper/323:25:123
1122499202,11cyclictest0-21swapper/322:10:123
1122199208,8cyclictest0-21swapper/220:25:012
1122199205,12cyclictest0-21swapper/219:10:172
1122199202,15cyclictest29785-21kthreadcore20:00:222
1122199201,9cyclictest29331-21kthreadcore21:45:222
1122199201,16cyclictest9970-21latency_hist21:35:012
1122199201,15cyclictest22381-21irqcore00:05:182
1121299208,8cyclictest752-21cron21:30:021
1121299203,14cyclictest7642-21ps20:10:261
1121299203,14cyclictest7449-21ssh00:15:141
1121299202,14cyclictest25085-21ssh22:55:131
11212992014,4cyclictest7742-21apt19:40:131
11212992013,5cyclictest1317-21ssh21:10:291
11212992012,6cyclictest5741-21ssh22:07:001
11212992011,6cyclictest23754-21ssh22:53:551
1121299201,15cyclictest21392-21apt-get00:05:051
1121299201,15cyclictest19410-21df21:40:151
1121299201,15cyclictest19410-21df21:40:141
1122499193,11cyclictest0-21swapper/321:28:113
1122499193,11cyclictest0-21swapper/300:26:113
1122499193,10cyclictest0-21swapper/321:30:123
1122499192,13cyclictest35966irq/58-eth021:00:113
1122499192,11cyclictest0-21swapper/319:35:143
1122499191,14cyclictest0-21swapper/320:00:193
1122499191,12cyclictest0-21swapper/323:10:123
11224991910,6cyclictest0-21swapper/300:12:543
1122199191,9cyclictest25310-21fschecks_time22:55:162
1122199191,8cyclictest18732-21fschecks_time20:25:152
1122199191,8cyclictest16257-21fschecks_time22:50:162
1122199191,8cyclictest10975-21cat19:40:222
1122199191,10cyclictest5426-21/usr/sbin/munin20:45:332
1122199191,10cyclictest28334-21fschecks_time20:35:162
11221991910,6cyclictest0-21swapper/221:00:222
1122199190,9cyclictest26984-21grep22:20:172
1121299196,3cyclictest131rcu_preempt21:55:131
1121299193,9cyclictest21-21ksoftirqd/119:15:121
1121299193,13cyclictest6760-21grep19:35:291
1121299192,14cyclictest10728-21sh23:05:141
11212991912,5cyclictest31675-21sshd00:27:111
11212991911,6cyclictest8928-21ssh23:20:241
11212991911,6cyclictest13506-21sshd20:16:091
1121299191,12cyclictest14007-21timerwakeupswit20:50:281
11212991910,6cyclictest0-21swapper/120:45:181
1122499189,6cyclictest0-21swapper/323:16:033
1122499182,10cyclictest0-21swapper/319:15:133
1122499181,12cyclictest0-21swapper/300:15:233
11224991810,5cyclictest35966irq/58-eth021:10:133
1122199181,9cyclictest30661-21fschecks_count19:30:142
1121299189,6cyclictest6280-21kthreadcore21:50:211
1121299188,7cyclictest23017-21kthreadcore21:00:211
1121299185,10cyclictest9843-21cron21:35:011
1121299184,10cyclictest17087-21kworker/1:219:50:111
1121299184,10cyclictest17087-21kworker/1:219:50:101
1121299181,9cyclictest32152-21cut20:05:181
1121299181,14cyclictest20957-21/usr/sbin/munin19:20:241
1121299181,13cyclictest12833-21ssh00:00:151
1121299181,11cyclictest30280-21missed_timers20:00:221
1122499179,6cyclictest0-21swapper/300:07:343
1122499172,9cyclictest0-21swapper/319:45:133
1122499172,11cyclictest0-21swapper/320:25:143
1122499172,10cyclictest0-21swapper/322:55:143
1122499172,10cyclictest0-21swapper/320:40:133
1122499172,10cyclictest0-21swapper/320:15:143
1122499172,10cyclictest0-21swapper/319:30:143
1122499172,10cyclictest0-21swapper/319:20:133
1122499171,13cyclictest0-21swapper/321:05:183
1122499171,11cyclictest0-21swapper/319:55:273
1121299179,6cyclictest32257-21kthreadcore19:30:191
1121299174,10cyclictest3736-21cut20:40:231
1121299171,5cyclictest30858-21apt-get00:10:131
1121299171,14cyclictest23384-21apt-get23:30:131
11212991711,4cyclictest0-21swapper/121:05:281
1121299171,11cyclictest28378-21sh21:45:141
1122499162,10cyclictest0-21swapper/320:20:143
1122499161,12cyclictest0-21swapper/321:00:013
1122499161,12cyclictest0-21swapper/321:00:003
1121299168,5cyclictest24209-21kthreadcore20:30:201
1121299161,11cyclictest11504-21fschecks_count19:10:161
1122499151,9cyclictest0-21swapper/320:30:163
1121299157,5cyclictest28340-21cut20:35:161
1121299156,7cyclictest12611-21df19:45:131
1121299150,13cyclictest23160-21kthreadcore19:55:191
1122499144,6cyclictest35966irq/58-eth020:10:133
1122499141,7cyclictest0-21swapper/320:50:023
1122199141,5cyclictest0-21swapper/219:10:022
1122199141,5cyclictest0-21swapper/219:10:012
1121299141,9cyclictest11265-21latency_hist19:10:021
1121299141,9cyclictest11265-21latency_hist19:10:021
1122499131,7cyclictest0-21swapper/319:50:263
1122499131,7cyclictest0-21swapper/319:50:253
1122499131,10cyclictest0-21swapper/319:40:223
1122499113,6cyclictest0-21swapper/319:10:023
1122499113,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional