You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-26 - 07:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue May 26, 2026 00:45:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1530119912295,13cyclictest30-21ksoftirqd/219:25:012
153005998617,4cyclictest3467602-21systemd-journal22:40:030
153005997322,21cyclictest0-21swapper/023:06:470
153008997258,10cyclictest23-21ksoftirqd/120:10:021
153011997129,38cyclictest157862-21sshd20:42:162
153005997125,16cyclictest131rcu_preempt21:46:560
153011997021,9cyclictest248562-21wc00:15:032
153011996857,8cyclictest30-21ksoftirqd/222:15:022
153011996825,5cyclictest162000-21sh21:15:292
15300599675,41cyclictest131rcu_preempt21:45:010
153011996622,6cyclictest186427-21sh22:06:032
153011996622,6cyclictest186427-21sh22:06:032
153011996525,5cyclictest216228-21sh23:08:092
153011996320,11cyclictest275-21dbus-daemon21:30:012
153005996318,36cyclictest16850irq/134-800000022:48:210
153011996223,10cyclictest3467602-21systemd-journal21:14:022
153011996223,10cyclictest3467602-21systemd-journal21:14:012
153011996221,10cyclictest30-21ksoftirqd/220:35:022
153011996220,8cyclictest193988-1kworker/u9:022:28:562
153011996220,8cyclictest193988-1kworker/u9:022:28:562
15300899629,33cyclictest197985-21ssh22:30:031
153005996223,6cyclictest16850irq/134-800000023:20:020
153005996219,34cyclictest16850irq/134-800000022:14:010
153005996219,34cyclictest16850irq/134-800000000:10:370
153005996218,35cyclictest16850irq/134-800000021:24:500
153005996217,36cyclictest16850irq/134-800000022:05:570
153005996217,36cyclictest16850irq/134-800000022:05:570
153015996112,7cyclictest131rcu_preempt22:50:003
153011996127,9cyclictest30-21ksoftirqd/223:26:462
153005996120,32cyclictest16850irq/134-800000021:11:390
153005996120,32cyclictest16850irq/134-800000021:11:390
153005996118,34cyclictest16850irq/134-800000023:59:580
153005996118,34cyclictest16850irq/134-800000023:40:010
153005996118,34cyclictest16850irq/134-800000023:40:010
153005996117,35cyclictest16850irq/134-800000022:59:520
153005996117,35cyclictest16850irq/134-800000022:37:580
153015996053,5cyclictest37-21ksoftirqd/320:40:033
153011996022,5cyclictest227969-21sh23:32:332
153011996022,5cyclictest227969-21sh23:32:332
153005996019,32cyclictest16850irq/134-800000023:52:590
153005996019,32cyclictest16850irq/134-800000023:28:160
153005996019,32cyclictest16850irq/134-800000021:17:140
153005996018,34cyclictest16850irq/134-800000021:31:050
153005996018,33cyclictest16850irq/134-800000023:34:220
153005996018,33cyclictest16850irq/134-800000023:34:220
153005996018,33cyclictest16850irq/134-800000022:30:420
153005996018,33cyclictest16850irq/134-800000000:26:250
153005996018,33cyclictest16850irq/134-800000000:26:250
153005996018,32cyclictest16850irq/134-800000000:34:540
153005996017,34cyclictest16850irq/134-800000023:23:040
153005996017,34cyclictest16850irq/134-800000023:23:040
153005996017,34cyclictest16850irq/134-800000000:01:390
153005996016,35cyclictest16850irq/134-800000022:51:000
153015995915,6cyclictest131rcu_preempt21:19:593
153011995936,19cyclictest131rcu_preempt22:35:022
153005995920,30cyclictest16850irq/134-800000022:19:300
153005995919,33cyclictest16850irq/134-800000023:49:400
153005995919,33cyclictest16850irq/134-800000023:49:390
153005995918,33cyclictest16850irq/134-800000022:27:580
153005995918,33cyclictest16850irq/134-800000022:27:580
153005995918,33cyclictest16850irq/134-800000021:29:060
153005995918,32cyclictest16850irq/134-800000023:14:480
153005995918,32cyclictest16850irq/134-800000023:01:050
153005995916,34cyclictest16850irq/134-800000021:38:150
153015995849,7cyclictest37-21ksoftirqd/323:55:013
153011995823,7cyclictest30-21ksoftirqd/223:44:222
153005995818,31cyclictest16850irq/134-800000022:20:480
153005995818,31cyclictest16850irq/134-800000021:52:520
153005995818,31cyclictest16850irq/134-800000000:36:340
153005995818,31cyclictest16850irq/134-800000000:36:340
153005995817,32cyclictest16850irq/134-800000000:23:030
153005995816,33cyclictest16850irq/134-800000022:04:210
153005995816,33cyclictest16850irq/134-800000000:17:500
153005995816,33cyclictest16850irq/134-800000000:17:500
153011995725,4cyclictest252358-21sh00:22:582
153005995718,31cyclictest16850irq/134-800000023:41:050
153005995717,31cyclictest16850irq/134-800000021:55:370
153005995717,31cyclictest16850irq/134-800000021:55:370
153015995621,30cyclictest37-21ksoftirqd/322:35:023
153005995611,13cyclictest1-21systemd20:25:010
153011995528,6cyclictest30-21ksoftirqd/200:03:522
153011995522,8cyclictest30-21ksoftirqd/200:40:012
153011995522,8cyclictest30-21ksoftirqd/200:40:012
153011995446,5cyclictest152966-21kworker/2:021:35:222
153011995424,7cyclictest30-21ksoftirqd/222:44:222
153011995423,8cyclictest30-21ksoftirqd/222:48:402
153011995422,7cyclictest30-21ksoftirqd/200:26:432
153011995422,7cyclictest30-21ksoftirqd/200:26:432
153011995421,8cyclictest30-21ksoftirqd/222:55:002
153005995415,31cyclictest16850irq/134-800000000:09:330
153005995410,12cyclictest0-21swapper/020:40:010
153011995345,5cyclictest231547-21kworker/2:300:00:012
153011995323,6cyclictest30-21ksoftirqd/200:05:592
153011995321,9cyclictest30-21ksoftirqd/221:53:352
153011995319,10cyclictest30-21ksoftirqd/223:19:242
153015995233,14cyclictest1-21systemd22:00:013
153015995233,14cyclictest1-21systemd22:00:013
153011995243,5cyclictest155590-21kworker/2:321:32:372
153011995238,8cyclictest183321-21cron22:00:002
153011995238,8cyclictest183321-21cron22:00:002
153011995233,13cyclictest275-21dbus-daemon22:15:012
153011995221,9cyclictest30-21ksoftirqd/223:21:402
153011995221,9cyclictest30-21ksoftirqd/223:21:402
153011995221,8cyclictest30-21ksoftirqd/222:00:082
153011995220,8cyclictest30-21ksoftirqd/220:20:292
153008995235,13cyclictest1-21systemd21:10:011
153005995231,15cyclictest1-21systemd20:35:000
15301199513,42cyclictest176134-21sh21:45:022
15300599515,38cyclictest154231-21modprobe19:35:000
153015995034,12cyclictest275-21dbus-daemon00:40:013
153015995034,12cyclictest275-21dbus-daemon00:40:013
153011995042,5cyclictest152966-21kworker/2:023:10:032
153011995040,6cyclictest200250-21kworker/2:123:03:282
153011995034,13cyclictest159179-21cron21:10:002
153011995022,7cyclictest30-21ksoftirqd/223:50:312
153011995020,11cyclictest30-21ksoftirqd/200:34:342
15300599503,14cyclictest154479-21cron19:39:590
153015994929,16cyclictest275-21dbus-daemon22:05:003
153011994941,5cyclictest155590-21kworker/2:321:21:322
153011994941,5cyclictest152966-21kworker/2:021:45:362
153011994940,5cyclictest231547-21kworker/2:323:47:512
153011994940,5cyclictest231547-21kworker/2:323:47:512
153011994936,9cyclictest3467602-21systemd-journal19:10:162
153011994936,9cyclictest3467602-21systemd-journal19:10:162
153008994933,12cyclictest275-21dbus-daemon23:50:011
153008994933,12cyclictest275-21dbus-daemon23:50:011
15300899491,12cyclictest212929-21ssh23:01:001
15300599499,35cyclictest0-21swapper/021:10:020
15301599482,3cyclictest131rcu_preempt22:20:013
153011994840,5cyclictest246008-21kworker/2:200:19:522
153011994840,5cyclictest246008-21kworker/2:200:19:522
153011994840,5cyclictest152966-21kworker/2:023:35:562
153011994840,5cyclictest152966-21kworker/2:023:35:562
153011994832,11cyclictest3467602-21systemd-journal20:08:162
15300899484,4cyclictest131rcu_preempt22:16:251
15300899481,10cyclictest170630-21ssh21:33:351
153005994810,6cyclictest131rcu_preempt19:15:010
153005994810,6cyclictest131rcu_preempt19:15:000
15301599477,5cyclictest131rcu_preempt00:35:003
153015994727,16cyclictest275-21dbus-daemon21:55:013
153015994727,15cyclictest275-21dbus-daemon00:25:483
153015994727,15cyclictest275-21dbus-daemon00:25:483
15301599472,3cyclictest131rcu_preempt22:07:573
15301599472,3cyclictest131rcu_preempt22:07:573
153015994719,4cyclictest37-21ksoftirqd/300:00:033
15301599471,4cyclictest131rcu_preempt22:24:023
15300899479,6cyclictest131rcu_preempt21:51:221
15300599474,8cyclictest156060-21munin-run20:10:000
15301599463,4cyclictest131rcu_preempt23:35:023
15301599463,4cyclictest131rcu_preempt23:35:013
153015994630,9cyclictest0-21swapper/321:40:003
153011994638,5cyclictest144914-21kworker/2:319:35:002
153011994637,5cyclictest155590-21kworker/2:321:00:012
153011994631,11cyclictest155271-21cron19:55:002
153008994631,11cyclictest195696-21sshd22:25:171
153008994631,11cyclictest195696-21sshd22:25:161
153008994630,12cyclictest275-21dbus-daemon23:00:001
153008994630,12cyclictest275-21dbus-daemon22:00:011
153008994630,12cyclictest275-21dbus-daemon22:00:011
153008994613,6cyclictest131rcu_preempt00:36:051
153008994613,6cyclictest131rcu_preempt00:36:051
15300599464,36cyclictest153451-21modprobe19:20:000
153005994631,12cyclictest154980-21cut19:45:030
15300599461,12cyclictest158670-21sendmail20:59:590
153015994530,11cyclictest3467602-21systemd-journal20:32:173
153015994530,11cyclictest275-21dbus-daemon23:10:013
153015994529,12cyclictest309-21in:imuxsock22:25:023
153015994529,12cyclictest309-21in:imuxsock22:25:023
15301599452,5cyclictest171147-21rm21:34:493
153011994538,4cyclictest152966-21kworker/2:022:36:572
153011994537,5cyclictest144914-21kworker/2:319:30:012
153011994520,12cyclictest275-21dbus-daemon20:30:002
15300899451,8cyclictest242334-21ssh00:02:071
153005994538,4cyclictest16950irq/224-800000020:00:020
15300599451,4cyclictest131rcu_preempt20:41:100
15301599443,3cyclictest131rcu_preempt23:35:433
15301599443,3cyclictest131rcu_preempt23:35:433
153015994426,12cyclictest3467602-21systemd-journal22:55:013
15301599442,3cyclictest131rcu_preempt23:22:383
15301599442,3cyclictest131rcu_preempt23:22:383
153015994420,6cyclictest37-21ksoftirqd/321:10:003
153015994420,5cyclictest37-21ksoftirqd/319:40:023
153015994417,5cyclictest37-21ksoftirqd/321:45:013
15300899444,4cyclictest131rcu_preempt23:55:011
15300899441,12cyclictest177250-21ssh21:47:091
153005994432,7cyclictest309-21in:imuxsock19:25:000
153005994428,12cyclictest155855-21cat20:05:010
153005994427,13cyclictest158899-21sshd21:02:170
153005994427,13cyclictest158899-21sshd21:02:170
15301599438,6cyclictest131rcu_preempt23:50:023
15301599438,6cyclictest131rcu_preempt23:50:023
15301599438,4cyclictest131rcu_preempt23:45:023
153015994330,9cyclictest1-21systemd23:05:013
153015994330,10cyclictest275-21dbus-daemon23:29:593
153015994329,11cyclictest37-21ksoftirqd/322:13:023
15301599431,3cyclictest131rcu_preempt20:25:023
153011994335,5cyclictest155590-21kworker/2:322:24:222
153011994334,6cyclictest155590-21kworker/2:320:55:032
15300899438,6cyclictest131rcu_preempt22:39:511
15300899433,3cyclictest131rcu_preempt21:24:541
153008994326,11cyclictest275-21dbus-daemon20:25:001
15300899431,8cyclictest185030-21rm22:03:201
153005994330,10cyclictest1-21systemd20:50:010
153005994328,12cyclictest3467602-21systemd-journal19:30:010
153005994325,14cyclictest156592-21tr20:19:590
153005994324,15cyclictest155244-21sshd19:51:160
15301599424,4cyclictest131rcu_preempt22:39:013
153015994230,8cyclictest37-21ksoftirqd/319:20:003
153015994229,10cyclictest156037-21sshd20:08:163
15301599422,4cyclictest131rcu_preempt21:13:253
15301599422,4cyclictest131rcu_preempt21:13:253
153015994220,16cyclictest275-21dbus-daemon19:30:003
153015994218,4cyclictest37-21ksoftirqd/321:21:513
153015994210,6cyclictest131rcu_preempt00:03:343
15300899423,3cyclictest0-21swapper/123:10:021
153008994225,13cyclictest3467602-21systemd-journal22:44:591
15300899421,12cyclictest189640-21ssh22:12:571
15300899421,11cyclictest219878-21ssh23:15:311
15301599416,8cyclictest37-21ksoftirqd/300:11:433
15301599416,4cyclictest131rcu_preempt23:00:023
15301599414,3cyclictest131rcu_preempt00:17:363
15301599414,3cyclictest131rcu_preempt00:17:363
15301599413,4cyclictest131rcu_preempt23:20:003
15301599413,3cyclictest131rcu_preempt00:21:553
153015994129,9cyclictest154894-21sed19:45:023
153015994125,12cyclictest275-21dbus-daemon21:30:013
153015994119,5cyclictest37-21ksoftirqd/323:11:513
15301599411,5cyclictest131rcu_preempt19:15:033
153011994134,4cyclictest200250-21kworker/2:122:55:302
153011994113,5cyclictest131rcu_preempt19:49:172
15300899417,4cyclictest150-21pr/ttyS200:30:311
15300899413,4cyclictest131rcu_preempt21:20:001
153008994123,14cyclictest23-21ksoftirqd/120:00:001
153008994123,13cyclictest275-21dbus-daemon19:15:011
153008994123,13cyclictest275-21dbus-daemon19:15:011
15300899411,3cyclictest131rcu_preempt22:09:301
15300899411,3cyclictest131rcu_preempt22:09:291
153005994122,4cyclictest12-21ksoftirqd/020:15:010
15301599406,6cyclictest131rcu_preempt19:25:003
15301599405,26cyclictest156571-21modprobe20:20:003
15301599404,4cyclictest131rcu_preempt22:44:373
15301599404,4cyclictest131rcu_preempt21:45:323
153015994028,7cyclictest157893-21munin-run20:45:003
153015994014,5cyclictest37-21ksoftirqd/300:06:393
15300899408,4cyclictest131rcu_preempt21:11:121
15300899408,4cyclictest131rcu_preempt21:11:121
153008994028,9cyclictest168074-21sshd21:28:171
153008994027,9cyclictest23-21ksoftirqd/122:22:461
153008994026,10cyclictest1-21systemd23:45:011
153008994025,11cyclictest23-21ksoftirqd/121:41:381
15300599402,34cyclictest157133-21sort20:30:010
15301599395,3cyclictest131rcu_preempt19:59:163
15301599395,26cyclictest158659-21grep21:00:003
153015993928,9cyclictest37-21ksoftirqd/320:50:003
153015993926,8cyclictest37-21ksoftirqd/320:15:013
153011993928,8cyclictest30-21ksoftirqd/219:40:012
153011993926,10cyclictest1-21systemd19:55:482
15300899396,3cyclictest131rcu_preempt19:45:011
153008993929,7cyclictest158899-21sshd21:02:171
153008993929,7cyclictest158899-21sshd21:02:171
153008993927,9cyclictest0-21swapper/122:55:021
153008993924,12cyclictest23-21ksoftirqd/119:20:011
15300899391,4cyclictest249909-21ssh00:17:541
15300899391,4cyclictest249909-21ssh00:17:541
15300899391,4cyclictest131rcu_preempt22:47:281
15300899391,3cyclictest131rcu_preempt23:24:481
15300899391,3cyclictest131rcu_preempt23:24:481
15301599383,3cyclictest131rcu_preempt19:55:013
15301599382,4cyclictest131rcu_preempt19:15:023
153015993822,12cyclictest1-21systemd20:55:013
15301599381,3cyclictest131rcu_preempt19:35:023
153015993811,4cyclictest37-21ksoftirqd/320:05:023
15301199382,27cyclictest291rcuc/219:10:022
15300899383,4cyclictest131rcu_preempt20:05:011
153008993825,11cyclictest23-21ksoftirqd/100:25:281
153008993825,11cyclictest23-21ksoftirqd/100:25:281
153008993825,10cyclictest23-21ksoftirqd/123:56:161
153008993825,10cyclictest23-21ksoftirqd/123:33:221
153008993825,10cyclictest23-21ksoftirqd/123:33:211
153008993824,11cyclictest23-21ksoftirqd/100:23:241
153008993823,11cyclictest23-21ksoftirqd/120:40:011
15300899381,5cyclictest0-21swapper/121:37:241
153005993829,6cyclictest19350irq/34-mmc120:50:060
15301599377,3cyclictest131rcu_preempt20:30:013
15301599373,4cyclictest131rcu_preempt19:10:023
153015993727,7cyclictest158908-21sshd21:03:173
153015993727,7cyclictest158908-21sshd21:03:163
15301599372,3cyclictest131rcu_preempt19:50:003
153011993729,5cyclictest156540-21sh20:17:002
153011993722,11cyclictest3467602-21systemd-journal20:36:182
153011993721,12cyclictest275-21dbus-daemon19:45:012
15301199371,3cyclictest131rcu_preempt21:05:022
15301199371,3cyclictest131rcu_preempt21:05:022
15300899374,5cyclictest131rcu_preempt19:10:021
153008993726,8cyclictest23-21ksoftirqd/123:15:021
153008993724,10cyclictest23-21ksoftirqd/100:13:511
153008993719,13cyclictest0-21swapper/120:20:001
15300599373,3cyclictest131rcu_preempt19:50:020
153005993727,8cyclictest12-21ksoftirqd/019:10:020
153008993629,4cyclictest245219-21sshd00:08:181
153008993624,8cyclictest0-21swapper/123:40:021
153008993624,8cyclictest0-21swapper/123:40:021
15300899361,9cyclictest224682-21ssh23:25:261
15300899361,4cyclictest131rcu_preempt19:50:021
153008993528,4cyclictest1-21systemd21:00:011
153008993524,6cyclictest309-21in:imuxsock19:25:001
153008993520,11cyclictest23-21ksoftirqd/119:53:561
153011993427,4cyclictest3467562-21sshd20:12:172
153011993426,5cyclictest3467562-21sshd19:15:172
15300899348,3cyclictest131rcu_preempt19:30:021
153008993425,6cyclictest309-21in:imuxsock20:55:011
153008993422,9cyclictest23-21ksoftirqd/119:40:011
153008993422,8cyclictest275-21dbus-daemon20:50:011
15300899341,11cyclictest157359-21latency_hist20:35:001
153011993326,5cyclictest152966-21kworker/2:020:05:002
15300899334,3cyclictest131rcu_preempt20:15:011
15300899333,4cyclictest131rcu_preempt19:35:011
153008993325,5cyclictest157106-21munin-plugin-st20:30:001
153011993221,9cyclictest30-21ksoftirqd/220:50:022
153008993223,5cyclictest309-21in:imuxsock20:45:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional