You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-15 - 05:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Mon Dec 15, 2025 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
563699250,24cyclictest19480-21apt-get20:49:230
563699250,23cyclictest313-21systemd-journal21:45:010
563799241,0cyclictest151rcu_preempt20:04:241
5639992323,0cyclictest0-21swapper/320:44:233
5639992321,1cyclictest0-21swapper/320:55:013
563999230,22cyclictest6154-21basename21:25:003
5638992320,2cyclictest9876-21apt-get20:29:222
563899230,2cyclictest20023-21grep19:40:012
563899230,22cyclictest27044-21wc23:15:002
5637992321,1cyclictest22-21ksoftirqd/121:15:011
5637992321,1cyclictest21159-21munin-run00:10:011
5637992321,1cyclictest0-21swapper/123:55:001
563799230,2cyclictest7996-21sh19:15:011
563799230,22cyclictest16264-21munin-run00:00:001
563799230,22cyclictest0-21swapper/123:30:011
563699230,22cyclictest0-21swapper/021:19:410
563699230,22cyclictest0-21swapper/021:04:230
5639992221,1cyclictest0-21swapper/300:00:003
5639992220,2cyclictest30194-21basename22:15:003
5639992220,1cyclictest0-21swapper/300:10:003
563999220,21cyclictest27870-21apt-get19:59:233
563999220,21cyclictest26083-21idleruntime-cro21:00:013
563999220,21cyclictest15190-21apt-get22:54:243
563999220,21cyclictest0-21swapper/321:19:413
563999220,0cyclictest0-21swapper/300:14:253
5638992220,1cyclictest13943-21apt-get21:44:242
563899220,21cyclictest24211-21awk23:10:002
563899220,21cyclictest19414-21sendmail-msp23:00:012
563899220,21cyclictest10401-21apt-get22:44:252
563899220,21cyclictest0-21swapper/221:39:232
563899220,21cyclictest0-21swapper/220:09:242
5637992220,1cyclictest17613-21turbostat.cron19:35:011
5637992220,1cyclictest12838-21turbostat19:25:011
5637992220,1cyclictest0-21swapper/119:54:241
5637992219,2cyclictest21275-21basename20:50:001
563799220,21cyclictest7983-21apt-get22:39:241
563799220,21cyclictest0-21swapper/123:44:241
563799220,21cyclictest0-21swapper/123:04:231
563799220,1cyclictest0-21swapper/119:50:001
5636992222,0cyclictest0-21swapper/023:19:240
5636992221,0cyclictest20000-21apt-get23:04:230
5636992220,1cyclictest6900-21munin-run20:20:010
5636992220,1cyclictest0-21swapper/019:19:240
5636992220,1cyclictest0-21swapper/000:35:010
563699220,21cyclictest0-21swapper/019:34:230
563699220,21cyclictest0-21swapper/000:39:240
563699220,1cyclictest0-21swapper/023:44:240
563699220,1cyclictest0-21swapper/022:44:250
563699220,1cyclictest0-21swapper/020:55:010
5639992121,0cyclictest27635-21apt-get23:19:243
5639992120,1cyclictest34-21ksoftirqd/300:25:013
5639992120,1cyclictest12278-21apt-get20:34:243
5639992119,0cyclictest151rcu_preempt20:04:243
5639992118,2cyclictest8518-21idleruntime-cro21:30:003
563999210,20cyclictest22394-21apt-get23:09:253
563999210,20cyclictest0-21swapper/323:49:253
563999210,20cyclictest0-21swapper/323:14:243
563999210,0cyclictest0-21swapper/320:24:243
5638992120,1cyclictest9647-21apt-get23:49:252
5638992120,1cyclictest19339-21apt-get00:09:232
5638992119,1cyclictest0-21swapper/219:29:232
563899210,20cyclictest31739-21apt-get00:34:252
563899210,20cyclictest0-21swapper/222:24:242
563899210,20cyclictest0-21swapper/221:34:252
563899210,0cyclictest0-21swapper/220:34:242
5637992121,0cyclictest0-21swapper/121:09:241
5637992120,0cyclictest32590-21perf22:20:001
5637992119,2cyclictest18736-21cron00:05:001
5637992119,1cyclictest15740-21basename21:45:011
5637992119,1cyclictest13338-21lockfile-create21:40:001
563799210,20cyclictest5067-21apt-get20:19:291
563799210,20cyclictest0-21swapper/123:34:241
563799210,20cyclictest0-21swapper/122:54:241
563799210,20cyclictest0-21swapper/121:59:251
563799210,20cyclictest0-21swapper/100:24:241
563799210,0cyclictest0-21swapper/122:14:251
563799210,0cyclictest0-21swapper/100:39:241
5636992121,0cyclictest0-21swapper/022:19:240
5636992120,1cyclictest0-21swapper/022:59:240
5636992120,1cyclictest0-21swapper/022:50:010
5636992120,1cyclictest0-21swapper/021:09:240
5636992119,1cyclictest606-21dbus-daemon21:00:010
563699210,21cyclictest0-21swapper/021:14:230
563699210,21cyclictest0-21swapper/020:44:230
563699210,21cyclictest0-21swapper/000:25:020
563699210,1cyclictest16254-21lockfile-create00:00:000
563699210,0cyclictest0-21swapper/022:39:230
5639992020,0cyclictest18216-21apt-get19:39:253
5639992020,0cyclictest1124-21perf00:35:013
5639992020,0cyclictest0-21swapper/322:19:243
5639992020,0cyclictest0-21swapper/319:29:233
5639992020,0cyclictest0-21swapper/319:14:233
563999200,20cyclictest0-21swapper/300:19:273
563999200,0cyclictest0-21swapper/322:44:253
563999200,0cyclictest0-21swapper/320:09:243
563999200,0cyclictest0-21swapper/319:54:243
5638992020,0cyclictest0-21swapper/223:19:242
5638992020,0cyclictest0-21swapper/222:39:242
5638992020,0cyclictest0-21swapper/222:29:232
5638992020,0cyclictest0-21swapper/222:04:252
5638992020,0cyclictest0-21swapper/221:54:252
5638992020,0cyclictest0-21swapper/220:59:242
5638992020,0cyclictest0-21swapper/220:14:222
5638992020,0cyclictest0-21swapper/200:25:022
5638992019,1cyclictest20000-21apt-get23:04:232
5638992019,0cyclictest7484-21apt-get20:24:242
5638992018,1cyclictest6667-21munin-run23:40:012
5638992018,1cyclictest31863-21cut23:25:012
5638992018,1cyclictest0-21swapper/220:00:002
5638992018,1cyclictest0-21swapper/219:19:242
563899200,20cyclictest0-21swapper/220:49:242
563899200,20cyclictest0-21swapper/219:24:222
563899200,0cyclictest0-21swapper/222:54:242
5637992020,0cyclictest23546-21apt-get22:04:241
5637992020,0cyclictest0-21swapper/122:09:221
5637992020,0cyclictest0-21swapper/100:14:241
5637992019,1cyclictest22-21ksoftirqd/123:20:011
5637992019,1cyclictest0-21swapper/122:44:251
5637992019,1cyclictest0-21swapper/120:34:241
5637992018,2cyclictest24195-21apt-get23:10:001
5637992018,2cyclictest22-21ksoftirqd/121:25:001
563799200,20cyclictest0-21swapper/123:39:251
563799200,20cyclictest0-21swapper/121:29:261
563799200,19cyclictest0-21swapper/120:09:241
563799200,0cyclictest0-21swapper/121:49:401
5636992020,0cyclictest0-21swapper/023:29:260
5636992020,0cyclictest0-21swapper/023:14:240
5636992020,0cyclictest0-21swapper/021:49:400
5636992020,0cyclictest0-21swapper/020:24:240
5636992020,0cyclictest0-21swapper/019:54:240
5636992020,0cyclictest0-21swapper/000:09:220
5636992019,1cyclictest6735-21apt-get21:29:260
5636992019,1cyclictest0-21swapper/022:04:250
5636992019,0cyclictest2432-21apt-get23:34:240
5636992019,0cyclictest0-21swapper/021:59:250
5636992019,0cyclictest0-21swapper/021:34:250
563699200,0cyclictest0-21swapper/023:09:250
563699200,0cyclictest0-21swapper/022:34:240
563699200,0cyclictest0-21swapper/020:34:240
5639991919,0cyclictest11028-21apt-get19:24:233
5639991919,0cyclictest0-21swapper/323:44:243
5639991919,0cyclictest0-21swapper/323:24:233
5639991919,0cyclictest0-21swapper/322:59:243
5639991919,0cyclictest0-21swapper/321:09:233
5639991919,0cyclictest0-21swapper/320:49:243
5639991919,0cyclictest0-21swapper/319:44:223
5639991919,0cyclictest0-21swapper/300:29:263
5639991918,1cyclictest16357-21apt-get21:49:403
5638991919,0cyclictest26742-21apt-get21:04:242
5638991919,0cyclictest0-21swapper/220:44:232
5638991919,0cyclictest0-21swapper/220:04:242
5638991918,1cyclictest16949-21apt-get00:04:262
563899190,18cyclictest25956-21turbostat00:20:012
5637991919,0cyclictest0-21swapper/123:49:251
5637991919,0cyclictest0-21swapper/100:29:261
5637991918,1cyclictest15219-21turbostat19:30:011
5637991917,2cyclictest28549-21latency_hist21:05:011
563799190,19cyclictest0-21swapper/119:44:221
563799190,0cyclictest0-21swapper/119:39:251
5636991919,0cyclictest24142-21apt-get00:19:270
5636991919,0cyclictest0-21swapper/023:54:240
5636991919,0cyclictest0-21swapper/022:54:240
5636991919,0cyclictest0-21swapper/021:24:230
5636991919,0cyclictest0-21swapper/019:49:240
5636991917,1cyclictest2073-21perf20:10:010
5636991917,1cyclictest20015-21sh19:40:010
5636991917,1cyclictest0-21swapper/000:05:000
5639991818,0cyclictest5067-21apt-get20:19:293
5639991817,0cyclictest0-21swapper/321:04:243
5638991818,0cyclictest0-21swapper/222:19:252
5638991818,0cyclictest0-21swapper/221:59:252
563799184,9cyclictest151rcu_preempt00:19:271
5637991818,0cyclictest0-21swapper/120:24:241
5637991818,0cyclictest0-21swapper/119:59:231
5637991818,0cyclictest0-21swapper/100:34:251
5637991817,1cyclictest22-21ksoftirqd/122:30:431
5637991817,1cyclictest0-21swapper/120:44:231
563799180,17cyclictest0-21swapper/122:49:231
5636991817,1cyclictest0-21swapper/019:44:220
5636991817,0cyclictest6214-21apt-get19:14:230
5636991816,1cyclictest29676-21latency_hist20:00:000
5639991717,0cyclictest5588-21apt-get22:34:253
5639991717,0cyclictest31927-21apt-get21:14:233
5639991717,0cyclictest0-21swapper/323:39:253
5639991717,0cyclictest0-21swapper/321:44:243
5639991717,0cyclictest0-21swapper/320:29:223
5639991717,0cyclictest0-21swapper/319:34:243
563999170,0cyclictest764-21apt-get22:24:243
5638991717,0cyclictest0-21swapper/219:44:222
5637991717,0cyclictest0-21swapper/122:59:241
5637991717,0cyclictest0-21swapper/121:34:251
5637991717,0cyclictest0-21swapper/120:59:241
5636991717,0cyclictest0-21swapper/023:49:250
5636991717,0cyclictest0-21swapper/020:14:220
5639991616,0cyclictest21128-21apt-get21:59:253
5639991616,0cyclictest0-21swapper/323:54:243
5639991616,0cyclictest0-21swapper/321:39:243
5639991615,1cyclictest1816-21sh23:30:013
5639991615,1cyclictest0-21swapper/300:40:003
563999160,15cyclictest0-21swapper/300:04:263
5638991616,0cyclictest0-21swapper/223:29:262
5638991616,0cyclictest0-21swapper/221:09:242
5638991616,0cyclictest0-21swapper/220:19:292
5638991615,1cyclictest7250-21apt-get23:44:242
5638991615,0cyclictest21874-21apt-get20:54:232
5638991614,1cyclictest30158-21cron22:15:002
563899160,2cyclictest5825-21sed21:24:312
563899160,16cyclictest0-21swapper/221:49:402
563899160,15cyclictest11697-21runrttasks23:50:232
5636991616,0cyclictest11527-21apt-get21:39:240
5636991616,0cyclictest0-21swapper/022:29:230
5636991616,0cyclictest0-21swapper/020:04:240
5636991614,1cyclictest0-21swapper/019:29:320
563699160,15cyclictest28795-21expr22:14:260
5639991515,0cyclictest24839-21awk19:50:003
5639991515,0cyclictest0-21swapper/322:10:003
5639991514,1cyclictest0-21swapper/321:52:363
5638991515,0cyclictest0-21swapper/222:09:222
5638991515,0cyclictest0-21swapper/221:14:232
563899150,1cyclictest0-21swapper/221:19:452
563899150,14cyclictest15311-21hddtemp_smartct20:39:262
563899150,14cyclictest0-21swapper/223:31:582
5637991515,0cyclictest0-21swapper/123:24:231
5637991515,0cyclictest0-21swapper/123:14:241
5637991514,1cyclictest22-21ksoftirqd/120:51:131
5637991513,1cyclictest10514-21expr20:29:251
5636991515,0cyclictest25936-21apt-get22:09:220
5636991515,0cyclictest0-21swapper/023:24:230
5636991513,1cyclictest0-21swapper/020:29:310
563699150,1cyclictest19377-21hddtemp_smartct21:54:280
5639991414,0cyclictest0-21swapper/322:35:053
5639991414,0cyclictest0-21swapper/321:31:023
5639991414,0cyclictest0-21swapper/319:15:063
563999140,14cyclictest0-21swapper/323:30:103
563999140,14cyclictest0-21swapper/322:45:113
563999140,14cyclictest0-21swapper/320:37:073
563999140,14cyclictest0-21swapper/320:12:593
563999140,0cyclictest0-21swapper/323:00:353
563999140,0cyclictest0-21swapper/322:25:303
563999140,0cyclictest0-21swapper/322:01:073
5638991414,0cyclictest0-21swapper/222:30:242
5638991414,0cyclictest0-21swapper/221:28:442
5638991414,0cyclictest0-21swapper/200:20:492
563899140,14cyclictest0-21swapper/223:55:082
563899140,14cyclictest0-21swapper/219:50:132
563899140,14cyclictest0-21swapper/219:47:062
563899140,14cyclictest0-21swapper/219:30:142
563899140,14cyclictest0-21swapper/219:10:482
563899140,14cyclictest0-21swapper/200:35:472
563899140,14cyclictest0-21swapper/200:10:102
563899140,0cyclictest0-21swapper/222:45:142
5637991414,0cyclictest0-21swapper/122:25:491
5637991414,0cyclictest0-21swapper/121:15:011
563799140,14cyclictest0-21swapper/121:50:401
563799140,14cyclictest0-21swapper/120:35:011
563799140,14cyclictest0-21swapper/119:15:151
563799140,0cyclictest0-21swapper/122:20:321
5636991414,0cyclictest0-21swapper/000:20:150
5636991414,0cyclictest0-21swapper/000:10:130
563699140,14cyclictest0-21swapper/022:20:250
563699140,14cyclictest0-21swapper/019:20:130
563699140,0cyclictest0-21swapper/023:35:140
563699140,0cyclictest0-21swapper/020:35:050
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional