You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-15 - 19:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Fri May 15, 2026 12:45:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
880927999614,37cyclictest0-21swapper/012:25:020
880927998625,15cyclictest882033-21cat07:30:020
880927998325,32cyclictest150-21pr/ttyS209:24:300
880927998325,31cyclictest150-21pr/ttyS210:22:380
88092799819,36cyclictest0-21swapper/010:20:010
880927998073,3cyclictest131rcu_preempt10:40:030
880927998015,34cyclictest0-21swapper/011:51:430
880927997922,17cyclictest984156-21ssh12:30:510
880927997812,33cyclictest0-21swapper/011:20:180
880927997727,16cyclictest883197-21latency_hist07:55:050
880927997723,16cyclictest967846-21ssh11:57:000
880927997715,26cyclictest882743-21latency_hist07:45:010
880927997537,4cyclictest935485-21munin-run10:50:010
880927997537,4cyclictest935485-21munin-run10:50:010
880927997432,11cyclictest945684-21ssh11:10:560
880927997430,12cyclictest958444-21ssh11:37:400
880927997324,44cyclictest926850-21ssh10:32:040
880927997224,14cyclictest976354-21munin-run12:15:010
880927997211,35cyclictest0-21swapper/011:26:440
880927997211,35cyclictest0-21swapper/011:26:440
880927997123,24cyclictest150-21pr/ttyS210:13:330
880927997116,24cyclictest885951-21latency_hist08:45:020
880927997024,15cyclictest931433-21ssh10:41:200
880927997022,15cyclictest986887-21ssh12:36:360
880927996929,35cyclictest48-21kcompactd011:05:170
880927996929,35cyclictest48-21kcompactd011:05:170
880927996929,10cyclictest977134-21sh12:16:230
880927996926,13cyclictest0-21swapper/009:26:080
880927996923,14cyclictest901323-21ssh09:39:380
880927996922,15cyclictest881391-21/usr/sbin/munin07:20:000
880927996824,22cyclictest0-21swapper/011:43:010
880927996823,15cyclictest942653-21munin-run11:05:000
880927996815,22cyclictest911168-21ssh09:59:530
880927996656,5cyclictest880930-21latency_hist07:10:020
880927996526,35cyclictest0-21swapper/010:07:310
880927996526,35cyclictest0-21swapper/010:07:310
880927996524,12cyclictest981935-21ssh12:26:150
880927996522,39cyclictest889652-21latency_hist09:15:020
880927996522,11cyclictest949876-21munin-run11:20:010
880927996427,4cyclictest971806-21ssh12:05:060
880927996427,4cyclictest971806-21ssh12:05:060
880927996416,16cyclictest904038-21latency_hist09:45:010
880927996342,17cyclictest884807-21sshd08:24:430
880927996320,38cyclictest0-21swapper/008:19:210
880927996317,13cyclictest940384-21cut11:00:020
880927996229,17cyclictest898298-21ssh09:33:000
880927996221,36cyclictest150-21pr/ttyS212:00:280
880927996142,16cyclictest883326-21cat07:55:020
880927996132,4cyclictest364-21snmpd08:06:380
880927996127,29cyclictest0-21swapper/011:45:180
880927996120,12cyclictest12-21ksoftirqd/008:05:000
880927996119,38cyclictest0-21swapper/010:55:020
880927996112,22cyclictest0-21swapper/010:29:370
880927996033,22cyclictest886603-21cron09:00:000
880927996028,16cyclictest906892-21ssh09:50:380
880927996013,14cyclictest886363-21/usr/sbin/munin08:55:010
880927996010,15cyclictest905753-21ssh09:48:350
880936995910,18cyclictest966932-21ssh11:55:022
880927995918,4cyclictest1-21systemd10:05:000
880927995743,11cyclictest882380-21sshd07:35:430
880927995743,10cyclictest882132-21sshd07:31:440
880927995742,12cyclictest886874-21sshd09:04:180
880927995715,12cyclictest887151-21diskmemload09:19:560
880936995648,5cyclictest874333-21kworker/2:109:13:242
880936995646,6cyclictest959557-21kworker/2:012:40:002
880936995545,6cyclictest906478-21kworker/2:211:00:022
880936995545,6cyclictest874333-21kworker/2:107:20:022
880927995540,12cyclictest885292-21sshd08:30:440
880927995519,11cyclictest0-21swapper/011:32:130
880936995446,5cyclictest906478-21kworker/2:212:03:032
880936995446,5cyclictest906478-21kworker/2:211:49:202
880936995445,6cyclictest874333-21kworker/2:110:54:082
880936995445,5cyclictest959557-21kworker/2:012:20:262
880936995443,6cyclictest906478-21kworker/2:211:20:282
880927995439,11cyclictest881675-21munin-run07:25:010
880927995439,11cyclictest881675-21munin-run07:25:010
880927995420,30cyclictest0-21swapper/007:47:180
880941995319,7cyclictest131rcu_preempt10:35:023
880936995345,5cyclictest906478-21kworker/2:211:14:202
880936995337,11cyclictest1-21systemd10:40:012
880936995335,13cyclictest275-21dbus-daemon12:35:022
880927995346,5cyclictest19350irq/34-mmc109:05:020
880927995340,10cyclictest886087-21sshd08:48:430
880936995244,5cyclictest906478-21kworker/2:211:30:182
880936995244,5cyclictest874333-21kworker/2:110:33:592
880936995244,5cyclictest874333-21kworker/2:109:15:102
880936995243,6cyclictest906478-21kworker/2:211:59:382
880936995243,6cyclictest874333-21kworker/2:109:31:592
880936995243,5cyclictest874333-21kworker/2:109:05:022
88093699521,16cyclictest887151-21diskmemload10:21:362
880932995238,8cyclictest23-21ksoftirqd/111:20:021
880927995241,7cyclictest881095-21sshd07:12:430
880941995132,15cyclictest275-21dbus-daemon11:55:003
880936995143,5cyclictest906478-21kworker/2:211:18:212
880936995143,5cyclictest906478-21kworker/2:211:10:022
880936995143,5cyclictest906478-21kworker/2:211:10:022
880936995142,6cyclictest959557-21kworker/2:012:27:462
880936995142,6cyclictest874333-21kworker/2:109:56:472
880927995140,9cyclictest885553-21sshd08:37:430
880927995135,11cyclictest364-21snmpd07:55:050
880941995033,13cyclictest275-21dbus-daemon08:45:013
880936995042,5cyclictest959557-21kworker/2:012:10:362
880936995042,5cyclictest906478-21kworker/2:211:03:462
880936995042,5cyclictest906478-21kworker/2:210:12:042
880936995042,5cyclictest874333-21kworker/2:109:26:432
880932995036,11cyclictest275-21dbus-daemon10:25:011
880927995043,4cyclictest885053-21sshd08:28:420
880941994938,7cyclictest885323-21sh08:35:003
880941994933,11cyclictest275-21dbus-daemon11:20:013
880936994941,5cyclictest959557-21kworker/2:012:19:562
880936994941,5cyclictest874333-21kworker/2:109:37:052
880936994941,5cyclictest874333-21kworker/2:109:23:322
880936994940,5cyclictest906478-21kworker/2:211:37:232
880936994938,7cyclictest874333-21kworker/2:109:53:352
880936994937,8cyclictest1-21systemd07:50:012
880936994933,12cyclictest3467602-21systemd-journal07:25:012
880936994933,12cyclictest3467602-21systemd-journal07:25:002
880927994936,9cyclictest364-21snmpd08:14:300
88094199487,6cyclictest131rcu_preempt11:40:003
880941994831,13cyclictest37-21ksoftirqd/309:30:023
880941994828,14cyclictest3467602-21systemd-journal10:29:433
880941994822,21cyclictest37-21ksoftirqd/311:01:193
880941994815,6cyclictest131rcu_preempt12:40:033
880936994840,5cyclictest906478-21kworker/2:211:44:202
880936994840,5cyclictest906478-21kworker/2:211:25:482
880936994840,5cyclictest906478-21kworker/2:211:25:482
880936994840,5cyclictest906478-21kworker/2:210:09:192
880936994840,5cyclictest906478-21kworker/2:210:09:192
880936994840,5cyclictest874333-21kworker/2:110:40:512
880936994839,6cyclictest874333-21kworker/2:110:00:052
880936994828,15cyclictest883510-21systemctl07:59:592
880941994735,8cyclictest883502-21sendmail08:00:003
880941994729,14cyclictest275-21dbus-daemon10:45:013
880941994727,14cyclictest1-21systemd10:05:023
880936994739,5cyclictest874333-21kworker/2:109:48:082
880936994739,5cyclictest874333-21kworker/2:108:50:022
880936994738,6cyclictest874333-21kworker/2:109:43:502
880932994735,9cyclictest23-21ksoftirqd/110:45:221
880932994735,9cyclictest23-21ksoftirqd/110:45:221
880932994729,14cyclictest3467602-21systemd-journal11:40:001
880932994729,13cyclictest886627-21systemctl09:00:001
880941994631,10cyclictest961928-21grep11:45:003
880941994629,12cyclictest275-21dbus-daemon07:25:013
880941994629,12cyclictest275-21dbus-daemon07:25:013
880941994621,21cyclictest37-21ksoftirqd/309:40:003
880936994638,5cyclictest906478-21kworker/2:210:25:252
880936994638,5cyclictest874333-21kworker/2:110:48:182
880936994638,5cyclictest874333-21kworker/2:110:48:182
880936994638,5cyclictest874333-21kworker/2:108:55:032
880932994631,12cyclictest901630-21sed09:40:021
880932994630,12cyclictest945008-21cron11:10:001
880932994630,12cyclictest945008-21cron11:10:001
880932994630,12cyclictest1-21systemd11:00:011
88094199458,5cyclictest131rcu_preempt09:25:013
880941994529,10cyclictest309-21in:imuxsock07:20:003
880941994524,16cyclictest275-21dbus-daemon09:15:013
880941994520,21cyclictest37-21ksoftirqd/312:23:413
880941994520,21cyclictest37-21ksoftirqd/312:11:543
880941994511,6cyclictest131rcu_preempt08:50:023
880941994510,6cyclictest131rcu_preempt08:15:003
880936994536,6cyclictest906478-21kworker/2:212:06:582
880936994536,6cyclictest906478-21kworker/2:212:06:582
88093299453,3cyclictest131rcu_preempt12:00:171
880932994533,9cyclictest961371-21sshd11:43:431
88093299453,30cyclictest892173-21ssh09:20:021
880932994532,10cyclictest1-21systemd12:35:001
880932994512,6cyclictest131rcu_preempt11:25:011
880941994429,11cyclictest3467602-21systemd-journal10:19:433
880941994428,11cyclictest0-21swapper/309:45:003
880941994425,14cyclictest1-21systemd08:10:013
880941994412,7cyclictest131rcu_preempt10:00:013
880936994436,5cyclictest874333-21kworker/2:107:51:592
880936994436,5cyclictest874333-21kworker/2:107:51:592
88093699442,3cyclictest131rcu_preempt07:35:022
880932994431,10cyclictest23-21ksoftirqd/108:34:121
88093299442,3cyclictest131rcu_preempt12:25:301
880932994423,16cyclictest942676-21unin-run11:05:001
88093299441,3cyclictest131rcu_preempt09:44:151
880941994333,6cyclictest887151-21diskmemload11:49:453
880941994329,10cyclictest1-21systemd10:25:003
88094199431,5cyclictest131rcu_preempt07:35:023
880936994325,12cyclictest275-21dbus-daemon08:25:002
88093299433,3cyclictest131rcu_preempt11:51:221
880932994328,10cyclictest1-21systemd11:30:011
880932994328,10cyclictest1-21systemd11:30:011
880932994327,11cyclictest275-21dbus-daemon08:50:001
88093299432,3cyclictest131rcu_preempt10:35:161
88093299431,3cyclictest131rcu_preempt12:38:291
88093299431,3cyclictest131rcu_preempt10:40:441
88093299431,3cyclictest131rcu_preempt09:27:441
88094199425,6cyclictest37-21ksoftirqd/312:26:573
880941994231,9cyclictest37-21ksoftirqd/312:06:493
880941994231,9cyclictest37-21ksoftirqd/312:06:493
880941994229,10cyclictest37-21ksoftirqd/311:57:203
880941994229,10cyclictest3467602-21systemd-journal07:42:433
880941994224,12cyclictest1-21systemd08:20:003
880941994219,15cyclictest275-21dbus-daemon08:30:013
880936994234,5cyclictest874333-21kworker/2:108:32:182
880936994234,5cyclictest874333-21kworker/2:108:02:522
880932994229,10cyclictest23-21ksoftirqd/110:06:191
880932994229,10cyclictest23-21ksoftirqd/110:06:191
880932994227,12cyclictest23-21ksoftirqd/109:15:331
880932994214,8cyclictest131rcu_preempt10:16:161
88094199417,7cyclictest37-21ksoftirqd/311:30:233
88094199412,5cyclictest131rcu_preempt12:17:143
880941994119,18cyclictest37-21ksoftirqd/309:49:403
88094199411,7cyclictest0-21swapper/311:22:433
880941994116,6cyclictest131rcu_preempt11:11:043
88094199411,3cyclictest131rcu_preempt11:08:483
88094199411,3cyclictest131rcu_preempt11:08:483
88094199411,35cyclictest48-21kcompactd008:55:293
880941994110,6cyclictest131rcu_preempt10:50:023
880941994110,6cyclictest131rcu_preempt10:50:023
880936994133,5cyclictest874333-21kworker/2:110:17:222
880936994128,9cyclictest3467602-21systemd-journal07:15:002
880936994125,10cyclictest885580-21modprobe08:40:012
88093299413,3cyclictest131rcu_preempt09:49:231
880932994129,9cyclictest23-21ksoftirqd/112:00:001
880932994129,9cyclictest23-21ksoftirqd/111:34:101
880932994128,9cyclictest275-21dbus-daemon10:30:011
880932994128,9cyclictest23-21ksoftirqd/109:30:431
880932994128,10cyclictest23-21ksoftirqd/112:20:271
880932994127,7cyclictest23-21ksoftirqd/110:35:011
880932994127,11cyclictest23-21ksoftirqd/109:13:181
880932994127,10cyclictest275-21dbus-daemon10:05:021
880932994126,11cyclictest23-21ksoftirqd/108:42:271
88093299411,3cyclictest131rcu_preempt12:19:041
88094199403,5cyclictest131rcu_preempt12:00:573
880941994029,8cyclictest37-21ksoftirqd/309:34:443
880941994029,6cyclictest930590-21sh10:40:013
880941994028,8cyclictest37-21ksoftirqd/310:15:003
880941994027,9cyclictest37-21ksoftirqd/311:25:573
880941994027,9cyclictest37-21ksoftirqd/311:25:573
880941994026,10cyclictest275-21dbus-daemon11:00:013
88094199402,5cyclictest131rcu_preempt09:55:013
88094199402,5cyclictest131rcu_preempt09:10:033
880941994023,13cyclictest1-21systemd09:20:003
880941994017,18cyclictest275-21dbus-daemon10:10:013
880941994017,18cyclictest275-21dbus-daemon10:10:013
880941994010,5cyclictest131rcu_preempt10:52:353
880932994031,5cyclictest886349-21cron08:55:011
880932994029,9cyclictest23-21ksoftirqd/112:09:121
880932994029,9cyclictest23-21ksoftirqd/112:09:121
880932994028,10cyclictest23-21ksoftirqd/110:14:141
880932994028,10cyclictest23-21ksoftirqd/109:55:531
880932994027,9cyclictest3467602-21systemd-journal07:51:431
880932994027,9cyclictest3467602-21systemd-journal07:51:421
880932994022,13cyclictest1-21systemd08:40:011
88094199394,25cyclictest37-21ksoftirqd/308:40:023
880941993916,4cyclictest37-21ksoftirqd/312:32:563
880936993929,6cyclictest874333-21kworker/2:108:15:002
880936993925,11cyclictest886740-21wc09:00:022
880936993924,11cyclictest30-21ksoftirqd/208:29:122
880932993911,24cyclictest1-21systemd11:50:011
88094199385,13cyclictest37-21ksoftirqd/307:30:013
880941993829,5cyclictest882420-21sendmail07:40:013
88093699388,6cyclictest131rcu_preempt09:07:432
880936993830,5cyclictest874333-21kworker/2:107:29:052
880936993829,5cyclictest874333-21kworker/2:108:20:022
880936993829,5cyclictest874333-21kworker/2:108:10:032
880936993824,11cyclictest886013-21sed08:45:022
88093299386,9cyclictest23-21ksoftirqd/107:36:431
880932993827,8cyclictest1-21systemd08:25:001
880932993826,9cyclictest23-21ksoftirqd/112:12:191
880932993826,10cyclictest23-21ksoftirqd/110:50:361
880932993825,9cyclictest309-21in:imuxsock07:30:001
880932993824,10cyclictest23-21ksoftirqd/108:08:251
88093299381,3cyclictest131rcu_preempt11:12:041
88094199379,7cyclictest37-21ksoftirqd/307:55:013
88094199379,7cyclictest37-21ksoftirqd/307:55:013
880941993726,8cyclictest37-21ksoftirqd/308:05:023
880941993723,12cyclictest37-21ksoftirqd/307:15:023
88093299375,9cyclictest23-21ksoftirqd/108:00:001
880932993728,7cyclictest23-21ksoftirqd/109:51:451
880932993725,9cyclictest23-21ksoftirqd/107:33:261
880932993725,9cyclictest23-21ksoftirqd/107:13:381
880932993724,10cyclictest23-21ksoftirqd/107:20:001
880932993723,10cyclictest23-21ksoftirqd/109:03:111
880932993723,10cyclictest23-21ksoftirqd/108:15:391
88093299371,29cyclictest882407-21kworker/u8:107:45:011
880941993621,10cyclictest275-21dbus-daemon09:05:013
88093299364,5cyclictest131rcu_preempt07:50:021
880932993627,6cyclictest885075-21munin-plugin-st08:30:001
880932993622,11cyclictest880948-21sed07:10:011
880932993620,12cyclictest884257-21sshd08:12:431
880941993525,7cyclictest37-21ksoftirqd/308:51:443
880941993521,11cyclictest37-21ksoftirqd/307:50:023
880936993525,6cyclictest882875-21wc07:45:022
88093699351,3cyclictest131rcu_preempt07:40:012
880932993524,8cyclictest23-21ksoftirqd/109:05:431
880932993521,11cyclictest3467602-21systemd-journal08:02:431
880941993325,5cyclictest881013-21latency_hist07:10:023
880936993325,5cyclictest874333-21kworker/2:107:10:022
880932993321,9cyclictest881752-21sed07:25:021
880932993321,9cyclictest881752-21sed07:25:011
880941993221,9cyclictest37-21ksoftirqd/308:25:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional