You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-26 - 10:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sun Jul 26, 2026 00:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
303756299125115,7cyclictest12-21ksoftirqd/000:40:020
3037562991111,3cyclictest0-21swapper/000:05:020
3037562998939,4cyclictest16850irq/134-800000023:05:000
303756299822,14cyclictest3042458-21munin-run20:45:010
303756299791,40cyclictest3069884-21ssh22:05:340
3037562997826,11cyclictest3107972-21ssh23:26:150
303756299741,71cyclictest0-21swapper/000:15:020
303756299741,71cyclictest0-21swapper/000:15:020
3037562997412,31cyclictest16850irq/134-800000022:44:030
3037562996923,13cyclictest3073020-21ssh22:12:180
3037562996920,32cyclictest12-21ksoftirqd/021:40:530
3037562996920,14cyclictest3048189-21ssh21:19:430
303756299687,33cyclictest16850irq/134-800000022:19:320
3037571996742,7cyclictest3041451-21kworker/2:321:40:452
3037562996725,37cyclictest1-21systemd00:00:010
3037573996656,4cyclictest37-21ksoftirqd/323:25:023
3037571996642,7cyclictest3041451-21kworker/2:321:30:232
3037562996622,10cyclictest3132728-21ssh00:20:070
303756299659,32cyclictest16850irq/134-800000000:15:320
3037562996414,30cyclictest16850irq/134-800000022:50:000
3037571996340,7cyclictest3041451-21kworker/2:321:38:422
3037571996340,7cyclictest3041451-21kworker/2:321:38:422
3037562996027,8cyclictest12-21ksoftirqd/023:15:170
3037571995838,15cyclictest3467602-21systemd-journal23:46:292
3037562995828,14cyclictest3045145-21ssh21:13:040
303756299582,40cyclictest3081265-21ssh22:30:000
303756299582,40cyclictest3081265-21ssh22:30:000
3037571995749,5cyclictest3041451-21kworker/2:300:27:212
3037571995649,5cyclictest30-21ksoftirqd/222:56:172
3037571995647,5cyclictest3076562-21kworker/2:222:26:272
3037571995647,5cyclictest3076562-21kworker/2:222:26:272
3037573995535,16cyclictest275-21dbus-daemon22:40:003
3037571995546,5cyclictest3076562-21kworker/2:222:34:312
3037566995538,12cyclictest23-21ksoftirqd/121:20:001
3037562995528,12cyclictest3106930-21ssh23:24:220
3037562995528,11cyclictest3116045-21ssh23:43:430
3037562995527,19cyclictest0-21swapper/022:56:020
3037562995525,15cyclictest3063670-21ssh21:52:340
303757399543,46cyclictest3123063-40mandb00:00:043
3037571995446,4cyclictest3076562-21kworker/2:222:55:032
3037571995445,6cyclictest3076562-21kworker/2:200:15:212
3037566995431,12cyclictest3139448-21rm00:35:011
3037562995427,12cyclictest3099651-21ssh23:08:490
3037562995427,11cyclictest3120147-21ssh23:53:320
3037562995427,11cyclictest3110537-21ssh23:31:580
3037562995419,15cyclictest3060504-21ssh21:45:260
3037562995419,15cyclictest3060504-21ssh21:45:250
303757399536,7cyclictest37-21ksoftirqd/321:45:563
303757399536,7cyclictest37-21ksoftirqd/321:45:553
3037571995345,5cyclictest3041451-21kworker/2:322:04:032
303756699539,6cyclictest131rcu_preempt22:21:001
303756299538,14cyclictest131rcu_preempt00:33:120
3037562995326,23cyclictest3113305-21ssh23:37:470
3037562995325,13cyclictest3051591-21ssh21:26:460
3037562995325,12cyclictest3065073-21ssh21:55:250
303756299532,12cyclictest3078868-21modprobe22:25:000
3037562995320,23cyclictest3068846-21ssh22:03:420
3037571995244,5cyclictest3041451-21kworker/2:323:36:492
3037571995244,5cyclictest3041451-21kworker/2:323:15:252
3037571995243,5cyclictest3041451-21kworker/2:322:05:592
3037571995241,6cyclictest3076562-21kworker/2:200:00:302
3037571995144,4cyclictest3041451-21kworker/2:323:50:132
3037571995143,5cyclictest3076562-21kworker/2:223:55:462
3037571995143,5cyclictest3041451-21kworker/2:323:21:102
3037571995143,5cyclictest3041451-21kworker/2:323:09:012
3037571995143,5cyclictest3041451-21kworker/2:321:55:012
3037571995143,5cyclictest3041451-21kworker/2:321:47:302
3037571995143,5cyclictest3041451-21kworker/2:321:47:292
3037571995143,5cyclictest3041451-21kworker/2:321:27:232
3037571995142,6cyclictest3037146-21kworker/2:121:15:012
3037571995141,6cyclictest3041451-21kworker/2:322:17:452
3037571995141,6cyclictest3041451-21kworker/2:322:15:012
3037571995130,15cyclictest275-21dbus-daemon19:20:012
3037566995115,10cyclictest23-21ksoftirqd/121:35:251
3037566995115,10cyclictest23-21ksoftirqd/121:35:251
3037562995136,11cyclictest3043760-21ssh21:10:000
3037562995128,7cyclictest12-21ksoftirqd/022:40:010
3037562995125,11cyclictest3083364-21ssh22:34:300
3037562995124,12cyclictest3125943-21ssh00:05:380
3037562995123,12cyclictest3054525-21ssh21:33:070
303757399505,3cyclictest131rcu_preempt19:25:023
3037573995035,11cyclictest37-21ksoftirqd/321:25:243
3037573995034,11cyclictest3467602-21systemd-journal22:51:323
303757399502,5cyclictest131rcu_preempt20:20:013
303757399502,5cyclictest131rcu_preempt20:20:003
3037571995043,4cyclictest3037146-21kworker/2:120:45:282
3037571995043,4cyclictest3037146-21kworker/2:120:45:282
3037571995042,5cyclictest3076562-21kworker/2:200:10:492
3037571995042,5cyclictest3076562-21kworker/2:200:10:492
3037571995042,5cyclictest3076562-21kworker/2:200:09:172
3037571995042,5cyclictest3041451-21kworker/2:300:36:072
3037571995042,5cyclictest3041451-21kworker/2:300:33:332
3037571995041,6cyclictest3041451-21kworker/2:323:41:532
3037571995041,6cyclictest3041451-21kworker/2:323:12:212
3037571995041,5cyclictest3041451-21kworker/2:323:31:362
3037571995041,5cyclictest3037146-21kworker/2:120:10:032
3037566995031,14cyclictest0-21swapper/100:40:001
3037562995028,7cyclictest3135070-21ssh00:25:090
3037562995021,14cyclictest3100985-21ssh23:11:290
3037573994929,16cyclictest37-21ksoftirqd/323:40:093
3037573994925,16cyclictest3043211-21modprobe21:00:003
3037573994922,20cyclictest275-21dbus-daemon20:05:013
3037571994941,5cyclictest3076562-21kworker/2:222:42:122
3037571994941,5cyclictest3076562-21kworker/2:222:39:592
3037571994941,5cyclictest3037146-21kworker/2:120:00:012
3037571994940,6cyclictest3041451-21kworker/2:322:24:162
3037571994940,5cyclictest3037146-21kworker/2:121:21:592
3037571994940,5cyclictest3037146-21kworker/2:121:17:022
3037566994934,11cyclictest1-21systemd23:25:011
303756299492,10cyclictest3118940-21cron23:50:010
3037573994833,12cyclictest3104889-21systemctl23:20:003
3037573994833,11cyclictest1-21systemd22:00:013
3037573994830,14cyclictest3467602-21systemd-journal19:35:013
3037573994830,12cyclictest3050631-21cron21:25:013
3037573994824,5cyclictest37-21ksoftirqd/322:10:013
303757399482,11cyclictest3095427-21latency_hist23:00:023
3037571994840,5cyclictest3076562-21kworker/2:222:47:522
3037571994840,5cyclictest3041451-21kworker/2:323:00:292
3037571994840,5cyclictest3041451-21kworker/2:321:55:242
3037571994840,5cyclictest3041451-21kworker/2:300:25:032
3037571994840,5cyclictest3037146-21kworker/2:120:28:272
3037571994840,5cyclictest3037146-21kworker/2:120:14:192
3037571994839,6cyclictest3037146-21kworker/2:119:40:022
3037571994839,6cyclictest3037146-21kworker/2:119:35:012
303756699489,6cyclictest131rcu_preempt23:34:011
3037566994821,6cyclictest23-21ksoftirqd/120:50:011
3037566994821,6cyclictest23-21ksoftirqd/120:50:011
303756699482,15cyclictest3044464-21ssh21:11:241
3037566994814,5cyclictest131rcu_preempt21:49:111
3037566994814,5cyclictest131rcu_preempt21:49:111
303756699481,3cyclictest131rcu_preempt23:25:541
3037566994813,5cyclictest131rcu_preempt23:15:441
3037562994810,19cyclictest12-21ksoftirqd/021:40:000
3037562994810,19cyclictest12-21ksoftirqd/021:39:590
3037573994731,11cyclictest3467602-21systemd-journal23:55:003
3037571994739,5cyclictest3037146-21kworker/2:120:05:012
3037566994728,14cyclictest0-21swapper/122:40:001
3037566994725,15cyclictest0-21swapper/119:15:011
3037566994713,5cyclictest131rcu_preempt00:11:451
3037566994713,5cyclictest131rcu_preempt00:11:451
3037566994710,5cyclictest131rcu_preempt00:22:171
3037562994739,4cyclictest16850irq/134-800000022:55:020
303756299472,7cyclictest3038553-21munin-plugin-st19:30:010
303756299472,7cyclictest3038553-21munin-plugin-st19:30:010
3037562994712,31cyclictest16850irq/134-800000021:22:100
303757399463,8cyclictest3114305-21latency_hist23:40:023
303757399463,18cyclictest131rcu_preempt00:24:213
3037573994630,12cyclictest3467602-21systemd-journal21:32:283
3037573994628,4cyclictest37-21ksoftirqd/320:30:023
3037573994623,16cyclictest275-21dbus-daemon20:00:013
3037573994616,20cyclictest37-21ksoftirqd/322:50:013
303757399461,5cyclictest131rcu_preempt23:10:053
3037573994610,5cyclictest131rcu_preempt22:30:503
3037571994638,5cyclictest3041451-21kworker/2:323:29:062
3037571994637,5cyclictest3037146-21kworker/2:121:10:002
3037566994634,9cyclictest3038782-21sshd19:33:281
3037566994630,12cyclictest3467602-21systemd-journal19:27:051
3037566994630,12cyclictest3467602-21systemd-journal19:27:051
3037566994629,12cyclictest1-21systemd19:59:591
303756699462,3cyclictest131rcu_preempt22:34:221
3037566994610,7cyclictest131rcu_preempt19:10:011
3037562994629,14cyclictest3040124-21sendmail-msp20:00:000
3037562994622,20cyclictest3041415-21latency_hist20:25:010
3037573994522,6cyclictest37-21ksoftirqd/300:00:233
303757399452,15cyclictest37-21ksoftirqd/323:35:013
3037571994532,9cyclictest3042950-21cron20:55:002
3037571994530,10cyclictest1-21systemd20:20:012
3037571994530,10cyclictest1-21systemd20:20:002
303756699452,6cyclictest3115746-21ssh23:43:051
3037562994537,4cyclictest16950irq/224-800000019:35:000
3037562994528,14cyclictest3040595-21sshd20:06:280
3037573994430,11cyclictest275-21dbus-daemon23:50:023
3037573994427,4cyclictest37-21ksoftirqd/322:20:013
3037573994427,14cyclictest37-21ksoftirqd/321:45:003
3037573994421,4cyclictest37-21ksoftirqd/323:30:033
3037573994420,17cyclictest275-21dbus-daemon19:30:013
3037573994420,17cyclictest275-21dbus-daemon19:30:003
3037571994434,6cyclictest309-21in:imuxsock20:45:012
303756699446,9cyclictest23-21ksoftirqd/123:10:031
3037566994436,5cyclictest3060114-21latency_hist21:45:001
3037566994434,7cyclictest23-21ksoftirqd/100:07:121
3037566994433,8cyclictest3041652-21sshd20:28:281
3037566994429,12cyclictest23-21ksoftirqd/119:50:291
3037566994429,11cyclictest275-21dbus-daemon00:05:011
3037566994429,11cyclictest23-21ksoftirqd/123:40:001
3037566994426,4cyclictest23-21ksoftirqd/100:27:211
303756699441,7cyclictest120-21kswapd023:00:011
303756699441,13cyclictest3467602-21systemd-journal20:10:271
3037562994431,9cyclictest3039440-21grep19:45:030
3037573994332,6cyclictest3042183-21munin-plugin-st20:40:003
3037573994324,4cyclictest37-21ksoftirqd/322:02:563
3037573994324,14cyclictest275-21dbus-daemon21:15:013
3037573994320,5cyclictest37-21ksoftirqd/300:13:563
3037573994320,5cyclictest37-21ksoftirqd/300:13:563
3037571994334,5cyclictest3037146-21kworker/2:120:35:012
3037571994334,5cyclictest3037146-21kworker/2:120:35:012
3037571994333,6cyclictest3037146-21kworker/2:119:45:012
3037571994330,9cyclictest30-21ksoftirqd/220:25:002
3037571994316,7cyclictest131rcu_preempt19:25:012
303756699439,6cyclictest131rcu_preempt22:18:381
303756699439,5cyclictest131rcu_preempt23:55:311
3037562994336,4cyclictest16950irq/224-800000020:55:000
3037562994334,4cyclictest16950irq/224-800000021:00:000
303757399424,23cyclictest3467602-21systemd-journal00:35:003
3037573994226,4cyclictest37-21ksoftirqd/300:26:433
3037573994225,15cyclictest37-21ksoftirqd/321:20:003
303757399422,4cyclictest131rcu_preempt00:15:403
3037573994215,18cyclictest37-21ksoftirqd/321:05:013
3037566994228,9cyclictest275-21dbus-daemon22:10:011
303756699422,3cyclictest131rcu_preempt21:57:141
303756699421,5cyclictest3118039-21ssh23:47:571
3037562994235,4cyclictest16950irq/224-800000021:05:010
3037562994227,11cyclictest275-21dbus-daemon20:39:590
303757399416,6cyclictest131rcu_preempt00:40:013
303757399414,3cyclictest131rcu_preempt21:37:063
303757399414,3cyclictest131rcu_preempt21:37:063
3037573994127,10cyclictest275-21dbus-daemon23:05:013
303757399411,4cyclictest131rcu_preempt22:43:493
303757199415,3cyclictest131rcu_preempt20:39:592
3037571994130,8cyclictest309-21in:imuxsock19:25:032
3037571994130,8cyclictest309-21in:imuxsock19:25:022
303756699418,5cyclictest131rcu_preempt21:26:461
3037566994129,9cyclictest23-21ksoftirqd/121:31:501
3037566994129,10cyclictest23-21ksoftirqd/100:19:031
303756699412,10cyclictest3089883-21ssh22:48:101
3037566994119,4cyclictest23-21ksoftirqd/122:44:461
303756699411,3cyclictest131rcu_preempt22:54:071
303756699411,3cyclictest131rcu_preempt20:55:001
303756299413,3cyclictest131rcu_preempt19:10:020
303756299412,9cyclictest3041149-21munin-run20:20:010
303756299412,9cyclictest3041149-21munin-run20:20:000
3037562994126,11cyclictest3039021-21sshd19:35:270
303757399405,26cyclictest3043736-21modprobe21:10:003
3037573994024,4cyclictest37-21ksoftirqd/323:07:043
303756699404,4cyclictest131rcu_preempt21:20:281
303756699403,3cyclictest131rcu_preempt20:45:011
3037566994031,6cyclictest3040625-21munin-plugin-st20:10:011
303756699402,9cyclictest3096446-21ssh23:01:541
3037566994027,8cyclictest275-21dbus-daemon22:30:011
3037566994027,8cyclictest275-21dbus-daemon22:30:011
303756299406,6cyclictest12-21ksoftirqd/019:55:020
303757399398,5cyclictest131rcu_preempt22:28:433
303757399398,5cyclictest131rcu_preempt22:28:423
303757399398,20cyclictest3467602-21systemd-journal00:10:003
303757399393,31cyclictest3039594-21munin-plugin-st19:50:003
3037573993928,8cyclictest3042721-21systemd-run20:50:013
3037573993928,8cyclictest3042721-21systemd-run20:50:013
303757399392,3cyclictest131rcu_preempt22:11:563
3037573993923,13cyclictest37-21ksoftirqd/322:22:503
3037566993929,7cyclictest23-21ksoftirqd/123:55:021
3037566993925,9cyclictest23-21ksoftirqd/119:45:001
3037566993925,10cyclictest3074099-21ssh22:15:011
3037562993928,7cyclictest3042697-21cron20:50:010
3037562993928,7cyclictest3042697-21cron20:50:000
3037562993923,10cyclictest1-21systemd19:25:010
303756299391,9cyclictest3041084-21date20:15:020
303756299391,7cyclictest3037922-21latency_hist19:15:030
3037573993827,8cyclictest37-21ksoftirqd/319:55:013
3037573993821,12cyclictest3467602-21systemd-journal20:55:003
3037571993830,5cyclictest3037146-21kworker/2:119:51:092
3037571993829,6cyclictest309-21in:imuxsock21:05:012
303756699387,3cyclictest131rcu_preempt19:50:011
3037566993829,7cyclictest23-21ksoftirqd/120:05:021
3037566993825,10cyclictest23-21ksoftirqd/120:20:011
3037566993825,10cyclictest23-21ksoftirqd/120:20:001
3037562993827,8cyclictest12-21ksoftirqd/019:45:030
303756299381,33cyclictest0-21swapper/019:20:000
303757399378,17cyclictest3038007-21modprobe19:19:593
303757399373,6cyclictest131rcu_preempt20:15:003
3037573993726,9cyclictest37-21ksoftirqd/321:53:413
3037573993710,22cyclictest1-21systemd20:25:013
3037566993730,4cyclictest3063352-21sh21:51:491
3037566993726,9cyclictest23-21ksoftirqd/123:12:351
3037566993726,8cyclictest23-21ksoftirqd/119:40:001
303757399364,25cyclictest3040628-21munin-plugin-st20:10:003
303757399364,24cyclictest3037770-21unin-run19:15:013
3037573993623,10cyclictest37-21ksoftirqd/320:44:593
303756699366,19cyclictest131rcu_preempt20:55:021
3037566993629,4cyclictest3043747-21sh21:09:591
3037566993623,9cyclictest3042163-21cron20:40:001
3037566993619,13cyclictest3467602-21systemd-journal20:35:011
3037566993619,13cyclictest3467602-21systemd-journal20:35:001
3037562993623,10cyclictest3041899-21sshd20:32:280
3037562993623,10cyclictest3041899-21sshd20:32:280
303756299361,31cyclictest150-21pr/ttyS220:00:290
303756299361,10cyclictest1-21systemd20:26:280
303757399354,27cyclictest284-21systemd-logind19:40:013
303757199353,26cyclictest3037759-21cut19:15:002
303756699353,27cyclictest150-21pr/ttyS219:21:321
3037566993524,9cyclictest23-21ksoftirqd/122:04:401
3037566993523,8cyclictest23-21ksoftirqd/119:20:001
303757399341,5cyclictest131rcu_preempt20:35:023
303757399341,5cyclictest131rcu_preempt20:35:013
303757199343,3cyclictest131rcu_preempt20:59:592
3037571993422,9cyclictest30-21ksoftirqd/219:10:022
3037571993421,10cyclictest3039575-21sshd19:49:282
303757399337,18cyclictest309-21in:imuxsock19:45:013
303756699336,3cyclictest150-21pr/ttyS220:25:001
303756699332,3cyclictest131rcu_preempt21:05:011
3037573993124,4cyclictest3037683-21latency_hist19:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional