You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-20 - 05:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Fri Mar 20, 2026 00:44:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
300521994440,3cyclictest586080-21kworker/u8:3+events_unbound23:50:141
300521994037,2cyclictest367873-21kworker/u8:2+events_unbound20:25:241
300521993835,2cyclictest554790-21kworker/u8:1+events_unbound23:30:131
300521993835,2cyclictest529181-21kworker/u8:2+events_unbound00:00:251
300521993835,2cyclictest352603-21kworker/u8:0+events_unbound20:10:231
300521993632,3cyclictest555137-21/usr/sbin/munin23:15:131
300521993632,2cyclictest294974-21kworker/u8:2+events_unbound19:20:211
300521993632,2cyclictest294974-21kworker/u8:2+events_unbound19:20:211
300518993633,2cyclictest294974-21kworker/u8:2+flush-259:019:40:150
300518993631,4cyclictest315937-21apt-get19:25:000
300518993631,4cyclictest315937-21apt-get19:24:590
300527993530,4cyclictest352603-21kworker/u8:0+flush-259:020:10:163
300521993532,2cyclictest519066-21kworker/u8:0+events_unbound22:50:161
300521993532,2cyclictest404099-21kworker/u8:2+events_unbound21:05:151
300521993431,2cyclictest529181-21kworker/u8:2+events_unbound23:25:151
300521993429,4cyclictest622441-21apt-get00:20:001
300518993428,4cyclictest580685-21latency_hist23:40:000
300527993328,3cyclictest429966-21latency_hist21:15:003
300527993228,3cyclictest476855-21kworker/u8:0+events_unbound22:20:183
300521993229,2cyclictest476855-21kworker/u8:0+events_unbound22:15:131
300521993229,2cyclictest367873-21kworker/u8:2+events_unbound20:30:281
300518993227,4cyclictest352638-21apt-get20:00:130
300527993126,3cyclictest477095-21apt-get22:00:103
300525993127,3cyclictest404099-21kworker/u8:2+flush-259:020:55:172
300521993128,2cyclictest294974-21kworker/u8:2+events_unbound19:40:111
300521993126,2cyclictest294974-21kworker/u8:2+events_unbound19:25:251
300521993027,2cyclictest253005-21kworker/u8:1+events_unbound19:52:261
300518993025,4cyclictest590123-21ntpq23:45:210
300527992924,3cyclictest388433-21kworker/u8:1+flush-259:020:45:163
300521992927,2cyclictest404099-21kworker/u8:2+events_unbound21:15:151
300521992926,2cyclictest529181-21kworker/u8:2+events_unbound00:05:221
300521992926,2cyclictest529181-21kworker/u8:2+events_unbound00:05:221
300521992926,2cyclictest294974-21kworker/u8:2+events_unbound19:35:261
300521992925,3cyclictest424848-21kworker/u8:3+events_unbound21:54:481
300521992925,2cyclictest596359-21kworker/u8:1+events_unbound23:55:231
300518992923,4cyclictest606927-21latency_hist00:05:000
300527992823,3cyclictest300722-21apt-get19:10:103
300521992826,2cyclictest424848-21kworker/u8:3+events_unbound23:00:151
300521992825,2cyclictest404099-21kworker/u8:2+events_unbound20:55:281
300521992825,2cyclictest367873-21kworker/u8:2+events_unbound20:20:141
300521992825,2cyclictest352603-21kworker/u8:0+events_unbound20:35:211
300518992824,3cyclictest607189-21apt-get00:05:100
300518992824,3cyclictest607189-21apt-get00:05:100
300518992823,4cyclictest419947-21apt-get21:05:120
300518992816,11cyclictest0-21swapper/023:55:000
300527992723,3cyclictest367873-21kworker/u8:2+events_unbound20:30:163
300521992724,2cyclictest502560-21kworker/u8:2+events_unbound22:35:221
300521992724,2cyclictest424848-21kworker/u8:3+events_unbound21:45:231
300521992724,2cyclictest424848-21kworker/u8:3+events_unbound21:45:221
300521992723,3cyclictest424848-21kworker/u8:3+events_unbound22:25:141
300521992723,3cyclictest352603-21kworker/u8:0+events_unbound20:47:381
300521992722,2cyclictest352603-21kworker/u8:0+events_unbound20:06:481
300521992721,4cyclictest414357-21/usr/sbin/munin21:00:131
300521992721,4cyclictest397800-21ntpq20:40:221
300521992721,4cyclictest252-21systemd-journal21:31:451
300527992622,3cyclictest529181-21kworker/u8:2+flush-259:000:15:003
300527992622,3cyclictest424848-21kworker/u8:3+flush-259:021:35:233
300521992624,2cyclictest476855-21kworker/u8:0+events_unbound22:10:131
300521992623,2cyclictest253005-21kworker/u8:1+events_unbound19:45:261
300521992620,4cyclictest637521-21ntpq00:30:221
300527992521,2cyclictest424848-21kworker/u8:3+flush-259:022:55:233
300521992522,2cyclictest519066-21kworker/u8:0+events_unbound23:05:251
300521992522,2cyclictest424848-21kworker/u8:3+events_unbound22:44:471
300521992522,2cyclictest424848-21kworker/u8:3+events_unbound21:25:221
300521992522,2cyclictest424848-21kworker/u8:3+events_unbound21:20:481
300521992522,2cyclictest404099-21kworker/u8:2+events_unbound21:10:271
300521992522,2cyclictest253005-21kworker/u8:1+events_unbound20:01:481
300521992521,3cyclictest575520-21kworker/u8:3+events_unbound23:35:251
300521992521,3cyclictest519066-21kworker/u8:0+events_unbound22:46:461
300518992511,12cyclictest455373-21ntpq21:35:220
30051899250,3cyclictest368397-21hddtemp_smartct20:15:160
300527992420,3cyclictest456267-21kworker/u8:1+flush-259:021:40:243
300527992419,4cyclictest435818-21apt-get21:20:113
300527992418,4cyclictest596169-21http23:55:003
300525992418,4cyclictest252-21systemd-journal00:20:022
300521992421,3cyclictest294974-21kworker/u8:2+events_unbound19:56:471
300521992421,2cyclictest529181-21kworker/u8:2+events_unbound23:43:151
300521992419,3cyclictest471910-21/usr/sbin/munin21:55:141
300521992320,2cyclictest529181-21kworker/u8:2+events_unbound00:12:471
300521992320,2cyclictest476855-21kworker/u8:0+events_unbound22:30:471
300521992320,2cyclictest456267-21kworker/u8:1+events_unbound22:05:271
300521992319,3cyclictest586113-21apt-get23:45:121
300521992319,3cyclictest519066-21kworker/u8:0+events_unbound00:21:481
300521992319,3cyclictest424848-21kworker/u8:3+events_unbound22:01:471
300518992317,4cyclictest326919-21apt-get19:35:100
300527992219,2cyclictest554790-21kworker/u8:1+flush-259:023:30:243
300527992218,3cyclictest554790-21kworker/u8:1+flush-259:023:28:113
300527992217,3cyclictest612134-21kworker/u8:3+flush-259:000:20:163
300525992218,3cyclictest294974-21kworker/u8:2+flush-259:019:50:152
300525992217,4cyclictest575520-21kworker/u8:3+flush-259:023:35:162
300518992217,4cyclictest633447-21apt-get00:30:110
30051899220,4cyclictest518281-21timerwakeupswit22:35:240
30051899220,2cyclictest325724-21/usr/sbin/munin19:30:180
30051899220,2cyclictest325724-21/usr/sbin/munin19:30:170
300527992118,2cyclictest456267-21kworker/u8:1+flush-259:021:50:163
300527992115,4cyclictest377254-21ntpq20:20:223
30052599215,5cyclictest517924-21ntpq22:35:202
300521992118,2cyclictest628022-21kworker/u8:2+events_unbound00:29:471
300521992118,2cyclictest529181-21kworker/u8:2+events_unbound23:23:061
30051899211,6cyclictest341197-21ntpq19:45:200
30051899211,6cyclictest320669-21ntpq19:25:230
300518992111,9cyclictest61150irq/133-enp0s29f1:tx-000:15:140
30051899210,3cyclictest472709-21sed21:55:170
30051899210,19cyclictest585006-21ntpq23:40:210
300527992017,2cyclictest352603-21kworker/u8:0+flush-259:020:40:163
300527992016,3cyclictest529181-21kworker/u8:2+flush-259:023:39:593
300527992016,3cyclictest519066-21kworker/u8:0+flush-259:022:45:233
300527992016,2cyclictest347210-21idleruntime-cro19:55:013
300527992013,5cyclictest419447-21cp21:05:003
300521992017,2cyclictest612134-21kworker/u8:3+flush-259:000:35:261
300521992014,2cyclictest191rcu_preempt23:00:011
300518992017,2cyclictest414382-21apt-get21:00:080
30051899201,6cyclictest409251-21taskset20:55:130
300518992016,3cyclictest502531-21latency_hist22:25:000
300518992016,3cyclictest424848-21kworker/u8:3+flush-259:022:05:000
30051899201,5cyclictest534256-21awk22:55:000
30051899201,3cyclictest575550-21turbostat.cron23:35:000
30051899201,3cyclictest402989-21ntpq20:45:200
30051899201,18cyclictest0-21swapper/021:25:220
30051899200,5cyclictest544610-21sh23:05:000
300527991916,2cyclictest456267-21kworker/u8:1+flush-259:022:05:223
300527991915,3cyclictest529181-21kworker/u8:2+flush-259:000:05:203
300527991915,3cyclictest529181-21kworker/u8:2+flush-259:000:05:203
300527991913,4cyclictest252-21systemd-journal23:10:453
30052599195,6cyclictest445882-21munin-run21:30:002
30052599193,4cyclictest560154-21sed23:20:002
300525991917,1cyclictest0-21swapper/223:55:002
300525991913,4cyclictest331770-21latency_hist19:40:012
300521991916,2cyclictest424848-21kworker/u8:3+events_unbound22:23:021
300518991917,1cyclictest383296-21cron20:30:010
30051899191,6cyclictest559163-21ntpq23:15:190
30051899191,5cyclictest534781-21fschecks_time22:55:140
30051899191,5cyclictest362692-21latency_hist20:10:000
30051899191,4cyclictest565927-21sed23:25:160
30051899191,4cyclictest487113-21latency_hist22:09:590
30051899191,4cyclictest471670-21latency_hist21:54:590
30051899191,4cyclictest440754-21sh21:25:000
30051899191,2cyclictest460483-21ntpq21:40:210
30051899191,2cyclictest397783-21ntpq20:40:220
30051899191,2cyclictest387549-21ntpq20:30:230
30051899191,2cyclictest304750-21ntpq19:10:200
30051899190,5cyclictest497396-21cron22:20:000
30051899190,3cyclictest300520-21sed19:10:000
30051899190,2cyclictest612636-21fschecks_time00:10:130
30051899190,2cyclictest450162-21ntpq21:30:220
30051899190,1cyclictest598662-21grep23:55:190
30051899190,17cyclictest393568-21latency_hist20:40:000
300527991815,2cyclictest612134-21kworker/u8:3+events_unbound00:15:183
300527991815,2cyclictest519066-21kworker/u8:0+events_unbound23:05:213
300527991815,2cyclictest424848-21kworker/u8:3+events_unbound21:45:193
300527991815,2cyclictest424848-21kworker/u8:3+events_unbound21:45:183
300527991814,3cyclictest294974-21kworker/u8:2+events_unbound19:55:213
300527991814,3cyclictest253005-21kworker/u8:1+flush-259:020:00:183
300527991814,2cyclictest0-21swapper/319:50:003
30052599186,3cyclictest427507-21kthreadcore21:10:182
30052599184,5cyclictest503549-21cat22:25:162
30052599184,4cyclictest312398-21kthreadcore19:20:162
30052599184,4cyclictest312398-21kthreadcore19:20:152
30052599183,5cyclictest324034-21cut19:30:162
30052599183,5cyclictest324034-21cut19:30:152
30052599182,6cyclictest364517-21kthreadcore20:10:172
30052599182,6cyclictest356679-21ntpq20:00:212
300525991816,1cyclictest481968-21sh22:05:002
300525991813,3cyclictest300432-21latency_hist19:10:012
300525991813,3cyclictest252-21systemd-journal21:20:002
300521991817,1cyclictest424848-21kworker/u8:3+events_unbound21:42:461
300521991814,3cyclictest529181-21kworker/u8:2+events_unbound23:13:471
300521991812,4cyclictest408104-21ntpq20:50:211
30051899181,4cyclictest623376-21fschecks_time00:20:140
30051899181,4cyclictest524383-21tr22:45:120
30051899181,4cyclictest487776-21grep22:10:160
30051899181,4cyclictest373335-21cpuspeed_turbos20:20:120
30051899181,3cyclictest564339-21ntpq23:20:190
30051899181,3cyclictest408988-21turbostat.cron20:55:010
30051899181,3cyclictest367815-21cron20:15:010
30051899181,2cyclictest632416-21ps00:25:240
30051899181,2cyclictest512411-21ntpq22:30:210
30051899181,2cyclictest429064-21ntpq21:10:210
30051899181,2cyclictest309861-21ntpq19:15:200
30051899180,6cyclictest352375-21dump-pmu-power20:00:000
30051899180,5cyclictest643548-21dump-pmu-power00:40:000
30051899180,5cyclictest544960-21tr23:05:120
30051899180,5cyclictest435702-21latency_hist21:20:000
30051899180,5cyclictest342506-21df19:50:110
30051899180,4cyclictest550981-21sed23:10:160
300527991714,2cyclictest586080-21kworker/u8:3+flush-259:023:50:003
300527991713,3cyclictest476855-21kworker/u8:0+flush-259:022:10:163
300527991713,3cyclictest456267-21kworker/u8:1+events_unbound21:55:163
300527991713,3cyclictest294974-21kworker/u8:2+flush-259:019:40:233
30052599173,5cyclictest439835-21ntpq21:20:212
30052599173,4cyclictest465626-21ntpq21:45:202
30052599173,4cyclictest465626-21ntpq21:45:202
30052599173,3cyclictest616345-21ntpq00:10:192
30052599173,3cyclictest523086-21ntpq22:40:202
30052599173,2cyclictest392672-21ntpq20:35:202
30052599172,4cyclictest0-21swapper/221:30:222
300525991715,1cyclictest0-21swapper/219:45:002
300525991711,4cyclictest628425-21diskstats00:25:142
300521991715,1cyclictest294974-21kworker/u8:2+flush-259:019:34:561
300521991715,1cyclictest294974-21kworker/u8:2+flush-259:019:34:561
300521991714,2cyclictest305851-21kworker/u8:3+events_unbound19:16:471
300521991713,3cyclictest242092-21kworker/u8:0+events_unbound19:14:471
30051899171,4cyclictest519271-21/usr/sbin/munin22:40:140
30051899171,4cyclictest508232-21latency_hist22:30:010
30051899171,4cyclictest466112-21grep21:45:250
30051899171,4cyclictest466112-21grep21:45:240
30052799169,5cyclictest252-21systemd-journal22:20:003
300527991613,2cyclictest628022-21kworker/u8:2+flush-259:000:35:193
300527991612,3cyclictest502560-21kworker/u8:2+flush-259:022:35:143
300527991612,3cyclictest404099-21kworker/u8:2+flush-259:020:50:173
300527991612,3cyclictest294974-21kworker/u8:2+events_unbound19:25:183
30052599163,3cyclictest387501-21ntpq20:30:222
30052599162,5cyclictest637490-21ld00:30:212
30052599162,5cyclictest309946-21ntpq19:15:212
30052599161,4cyclictest418382-21ld21:00:192
30052199168,1cyclictest0-21swapper/120:15:191
300527991512,2cyclictest529181-21kworker/u8:2+events_unbound00:00:233
300527991512,2cyclictest519066-21kworker/u8:0+flush-259:000:30:193
300527991511,3cyclictest424848-21kworker/u8:3+flush-259:023:00:243
300527991511,3cyclictest404099-21kworker/u8:2+flush-259:021:30:223
30052599155,9cyclictest513279-21latency_hist22:35:012
30052599154,6cyclictest372171-21ntpq20:15:232
30052599153,6cyclictest408058-21ntpq20:50:202
30052599153,2cyclictest574724-21ntpq23:30:222
30052599153,2cyclictest569484-21ntpq23:25:202
30052599152,2cyclictest460492-21ntpq21:40:212
30052599151,5cyclictest455328-21ldconfig21:35:202
30052599151,4cyclictest639629-21irqstats00:35:172
30052599151,4cyclictest470821-21ld21:50:212
300525991512,2cyclictest528174-21needreboot22:45:182
300525991512,2cyclictest503-21tpm2-abrmd00:20:002
30052199156,2cyclictest191rcu_preempt19:10:011
30052799149,4cyclictest253005-21kworker/u8:1+events_unbound20:15:223
30052799149,3cyclictest306044-21diskstats19:15:123
30052799148,4cyclictest435551-21latency_hist21:20:013
300527991411,2cyclictest476855-21kworker/u8:0+flush-259:022:30:243
300527991411,2cyclictest327934-21kthreadcore19:35:153
30052599144,5cyclictest539830-21df23:00:132
30052599144,5cyclictest424422-21users21:05:252
30052599143,3cyclictest316929-21hddtemp_smartct19:25:162
30052599142,6cyclictest373559-21hddtemp_smartct20:20:152
30052599142,2cyclictest580925-21/usr/sbin/munin23:40:192
30052599141,4cyclictest382406-21ntpq20:25:222
300525991412,1cyclictest0-21swapper/219:15:002
300525991411,2cyclictest0-21swapper/221:55:222
300525991410,3cyclictest529181-21kworker/u8:2+flush-259:000:05:152
300525991410,3cyclictest529181-21kworker/u8:2+flush-259:000:05:152
30052599140,4cyclictest341270-21ntpq19:45:212
30052199145,1cyclictest191rcu_preempt21:35:171
30052799139,3cyclictest519066-21kworker/u8:0+events_unbound23:15:203
30052799139,3cyclictest424848-21kworker/u8:3+flush-259:022:25:163
30052799139,3cyclictest404099-21kworker/u8:2+flush-259:021:05:213
30052799137,4cyclictest362678-21cron20:10:013
300527991310,2cyclictest596359-21kworker/u8:1+flush-259:023:55:263
30052599135,6cyclictest397942-21ps20:40:232
30052599133,8cyclictest565255-21latency_hist23:25:002
30052599132,6cyclictest545024-21diskstats23:05:122
30052599131,7cyclictest534059-21taskset22:52:512
30052599131,3cyclictest486773-21runrttasks22:06:082
30052599131,2cyclictest402943-21ntpq20:45:202
30052599130,4cyclictest596655-21apt-config23:55:012
30052799129,2cyclictest529181-21kworker/u8:2+events_unbound23:20:123
30052799129,2cyclictest424848-21kworker/u8:3+flush-259:022:40:203
30052799128,3cyclictest519066-21kworker/u8:0+flush-259:022:50:233
30052599129,2cyclictest0-21swapper/222:10:162
30052599122,6cyclictest538340-21memory22:55:192
30052599121,6cyclictest587095-21irqstats23:45:172
30052799118,2cyclictest352603-21kworker/u8:0+flush-259:020:35:123
30052799117,2cyclictest383300-21latency_hist20:30:003
30052599111,2cyclictest401ktimers/223:10:142
30052799108,1cyclictest409659-21tr20:55:163
30052799100,9cyclictest586056-21cat23:45:003
30052599101,2cyclictest358102-21fw_conntrack20:05:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional