You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-04 - 10:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sat Jul 04, 2026 00:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2188699913,5cyclictest131rcu_preempt00:10:021
21890997830,14cyclictest108085-21ssh23:58:322
21890997827,14cyclictest117615-21ssh00:18:362
21890997630,14cyclictest125693-21ssh00:35:152
21890997618,18cyclictest23702-21munin-run19:45:012
21890997525,15cyclictest93602-21ssh23:28:032
21890997524,15cyclictest98689-21ssh23:38:512
21890997519,20cyclictest82287-21ssh23:04:252
21890997519,20cyclictest82287-21ssh23:04:252
21890997427,16cyclictest84964-21ssh23:10:002
21890997420,16cyclictest75880-21ssh22:50:322
21890997323,9cyclictest121789-21ssh00:27:182
21890997322,17cyclictest100457-21ssh23:42:212
21890997317,18cyclictest41157-21ssh21:37:302
21890997222,18cyclictest75427-21ssh22:49:552
21890997122,16cyclictest54629-21ssh22:05:482
21881997166,3cyclictest150-21pr/ttyS220:15:020
21881997166,3cyclictest150-21pr/ttyS220:15:010
21890997017,17cyclictest29605-21ssh21:13:112
21890997017,15cyclictest34953-21ssh21:24:382
21890997016,16cyclictest62847-21ssh22:23:292
21890996925,13cyclictest60475-21ssh22:18:292
21890996919,14cyclictest58524-21ssh22:14:232
2188699691,38cyclictest49500-21ssh21:55:031
21890996827,16cyclictest51088-21ssh21:58:402
21890996821,15cyclictest27202-21date20:50:022
21881996828,15cyclictest131rcu_preempt23:37:510
21890996720,15cyclictest110967-21sh00:04:412
21890996621,15cyclictest28108-21diskmemload21:16:032
21890996616,16cyclictest91853-21ssh23:24:242
21890996615,17cyclictest45288-21ssh21:45:592
21890996614,14cyclictest36523-21ssh21:27:472
2188199664,42cyclictest131rcu_preempt00:17:530
21890996321,15cyclictest73041-21latency_hist22:45:022
21890996318,14cyclictest112674-21ssh00:08:092
21890996312,19cyclictest88068-21rm23:16:162
21886996331,9cyclictest3467602-21systemd-journal21:10:011
21890996220,13cyclictest103883-21ssh23:49:532
21890996220,13cyclictest103883-21ssh23:49:532
21890996216,13cyclictest67121-21ssh22:32:172
21890996123,11cyclictest85462-21ssh23:10:452
21890996123,11cyclictest85462-21ssh23:10:442
21881996123,19cyclictest12-21ksoftirqd/022:54:460
21881996122,35cyclictest1-21systemd20:20:010
21890996020,12cyclictest95407-21ssh23:31:432
21890996019,12cyclictest69713-21ssh22:37:522
21881996040,17cyclictest16850irq/134-800000022:00:000
21890995912,15cyclictest78367-21ssh22:55:412
21881995928,15cyclictest54796-21ssh22:06:100
21881995919,19cyclictest16850irq/134-800000023:25:020
21881995913,4cyclictest131rcu_preempt00:40:020
21890995846,9cyclictest120138-21ssh00:23:592
21890995840,15cyclictest43272-21ssh21:41:552
21890995742,11cyclictest105161-21ssh23:52:172
21890995741,12cyclictest47702-21ssh21:51:212
21890995739,14cyclictest114821-21ssh00:12:412
21890995713,12cyclictest52039-21ssh22:00:152
21881995729,13cyclictest90889-21ssh23:22:130
21881995728,13cyclictest72557-21ssh22:44:010
21881995719,33cyclictest16850irq/134-800000021:20:030
21890995646,6cyclictest24425-21kworker/2:122:25:342
21890995643,10cyclictest38851-21ssh21:32:362
21890995643,10cyclictest38851-21ssh21:32:352
21890995639,13cyclictest123874-21ssh00:31:322
2188199564,15cyclictest44641-21latency_hist21:45:010
21881995626,26cyclictest114911-21ssh00:12:520
2188199562,28cyclictest131rcu_preempt21:12:020
21881995619,32cyclictest16850irq/134-800000023:11:100
21881995619,32cyclictest16850irq/134-800000023:11:090
21881995618,33cyclictest16850irq/134-800000023:01:470
21881995618,33cyclictest16850irq/134-800000023:01:460
21890995541,10cyclictest27539-21sshd20:55:212
21890995533,18cyclictest25799-21munin-run20:25:002
21881995535,5cyclictest0-21swapper/000:31:490
21881995527,24cyclictest107680-21ssh23:57:310
21881995527,13cyclictest95733-21ssh23:32:270
21881995519,32cyclictest16850irq/134-800000022:15:270
21881995519,31cyclictest16850irq/134-800000021:36:490
21881995518,32cyclictest16850irq/134-800000022:47:530
21881995518,32cyclictest16850irq/134-800000021:52:290
21881995518,32cyclictest16850irq/134-800000021:25:050
21890995438,13cyclictest22634-21sshd19:21:212
21890995436,13cyclictest3467602-21systemd-journal20:55:002
21890995412,14cyclictest1-21systemd20:05:002
21881995428,11cyclictest61979-21ssh22:21:190
21881995426,13cyclictest105783-21ssh23:53:410
21881995418,32cyclictest16850irq/134-800000022:26:170
21881995418,32cyclictest16850irq/134-800000000:22:470
21881995418,32cyclictest16850irq/134-800000000:09:100
21881995418,31cyclictest16850irq/134-800000022:01:120
2188199541,37cyclictest34346-21ssh21:23:130
21890995344,6cyclictest13265-21kworker/2:319:15:012
21890995339,10cyclictest23390-21sshd19:36:202
21881995318,6cyclictest16850irq/134-800000022:12:040
21881995318,31cyclictest16850irq/134-800000023:06:430
21881995318,31cyclictest16850irq/134-800000021:31:170
21881995318,31cyclictest16850irq/134-800000021:31:160
21881995317,7cyclictest16850irq/134-800000023:16:120
21881995317,32cyclictest16850irq/134-800000000:29:230
21890995240,9cyclictest48-21kcompactd020:08:082
21890995223,24cyclictest28180-21sed21:10:012
21881995229,7cyclictest103834-21ssh23:49:380
21881995229,7cyclictest103834-21ssh23:49:380
21881995218,30cyclictest16850irq/134-800000000:02:560
21881995217,6cyclictest16850irq/134-800000022:37:340
21881995217,31cyclictest16850irq/134-800000022:55:340
21881995217,31cyclictest16850irq/134-800000022:33:060
21886995110,33cyclictest92254-21ssh23:25:031
21881995143,5cyclictest16950irq/224-800000020:59:590
21881995134,13cyclictest275-21dbus-daemon20:45:000
21881995118,30cyclictest16850irq/134-800000023:42:180
21890995043,4cyclictest27823-21sshd21:01:222
21881995015,31cyclictest16850irq/134-800000021:47:090
21892994930,15cyclictest1-21systemd20:00:013
21892994930,15cyclictest1-21systemd20:00:013
21892994914,6cyclictest131rcu_preempt23:43:353
21890994940,5cyclictest23182-21munin-run19:35:022
21890994939,6cyclictest24170-21sshd19:50:202
21890994937,9cyclictest26505-21sshd20:35:202
21890994930,16cyclictest364-21snmpd19:17:452
21890994926,18cyclictest25712-21latency_hist20:20:022
2188199492,15cyclictest28137-21munin-run21:10:010
21892994815,3cyclictest150-21pr/ttyS221:30:473
21892994815,3cyclictest150-21pr/ttyS221:30:463
21890994834,11cyclictest26802-21sshd20:44:212
21890994831,13cyclictest364-21snmpd19:58:252
21890994831,13cyclictest364-21snmpd19:58:252
21886994829,15cyclictest3467602-21systemd-journal00:00:001
2188699481,7cyclictest0-21swapper/121:35:261
21881994841,4cyclictest16950irq/224-800000019:20:010
21881994828,17cyclictest26009-21sshd20:26:400
21892994713,6cyclictest131rcu_preempt21:18:263
21890994734,9cyclictest364-21snmpd20:33:132
21886994732,11cyclictest1-21systemd23:15:011
21886994732,11cyclictest1-21systemd23:15:011
21886994730,13cyclictest3467602-21systemd-journal19:45:011
21892994629,13cyclictest275-21dbus-daemon21:15:003
21892994627,14cyclictest3467602-21systemd-journal20:20:003
2188699469,7cyclictest131rcu_preempt21:34:091
2188699469,7cyclictest131rcu_preempt21:34:091
21886994626,14cyclictest1-21systemd22:25:011
2189299457,5cyclictest131rcu_preempt21:53:423
2189299455,4cyclictest131rcu_preempt19:13:213
2188699456,4cyclictest131rcu_preempt22:28:101
2188699454,3cyclictest131rcu_preempt22:05:541
21886994531,11cyclictest275-21dbus-daemon23:45:011
21886994530,12cyclictest24789-21cat20:05:021
21886994529,13cyclictest3467602-21systemd-journal22:44:591
2188699452,4cyclictest131rcu_preempt22:30:581
2188699452,4cyclictest131rcu_preempt21:21:451
2188699452,3cyclictest131rcu_preempt23:09:221
2188699452,3cyclictest131rcu_preempt20:19:591
2189299447,5cyclictest131rcu_preempt19:20:003
2189299444,8cyclictest82880-21ssh23:05:113
21892994429,10cyclictest68282-21systemd-run22:35:003
21892994427,13cyclictest1-21systemd19:25:013
21892994421,5cyclictest37-21ksoftirqd/321:21:473
21892994411,6cyclictest131rcu_preempt22:10:113
21892994411,5cyclictest131rcu_preempt00:36:553
21890994432,9cyclictest364-21snmpd19:46:272
21890994422,18cyclictest3467602-21systemd-journal20:30:012
2188699444,9cyclictest53943-21ssh22:04:381
2188699444,4cyclictest131rcu_preempt21:45:141
2188699443,3cyclictest131rcu_preempt22:52:221
21886994432,9cyclictest3467602-21systemd-journal00:40:011
21886994431,9cyclictest3467602-21systemd-journal20:35:001
2188699442,3cyclictest131rcu_preempt22:16:531
21881994427,12cyclictest275-21dbus-daemon20:10:000
2189299438,5cyclictest131rcu_preempt22:50:313
2189299435,4cyclictest131rcu_preempt00:12:283
21892994330,10cyclictest37-21ksoftirqd/322:46:553
2189299432,37cyclictest37-21ksoftirqd/320:59:593
21892994310,5cyclictest131rcu_preempt22:25:583
21890994330,10cyclictest364-21snmpd19:25:462
21890994329,11cyclictest3467562-21sshd20:12:202
21890994329,11cyclictest3467562-21sshd20:12:202
2188699434,3cyclictest131rcu_preempt23:21:461
2188699433,3cyclictest131rcu_preempt23:48:361
2188699433,3cyclictest131rcu_preempt23:48:351
21886994331,9cyclictest43890-21sshd21:43:221
21886994328,10cyclictest23-21ksoftirqd/121:15:001
2188699432,4cyclictest131rcu_preempt23:37:401
2188699432,3cyclictest131rcu_preempt23:32:291
2188699432,3cyclictest131rcu_preempt22:35:161
21886994322,14cyclictest275-21dbus-daemon20:50:001
2188699431,23cyclictest0-21swapper/100:22:041
2188699431,23cyclictest0-21swapper/100:13:401
21881994324,14cyclictest275-21dbus-daemon20:00:020
21881994324,14cyclictest275-21dbus-daemon20:00:010
2189299428,6cyclictest131rcu_preempt23:12:083
2189299428,6cyclictest131rcu_preempt23:12:073
2189299427,4cyclictest131rcu_preempt20:25:213
2189299426,3cyclictest131rcu_preempt23:55:543
21892994228,10cyclictest3467602-21systemd-journal23:05:023
21892994228,10cyclictest3467602-21systemd-journal23:05:023
2189299422,4cyclictest131rcu_preempt20:55:013
2189299422,3cyclictest131rcu_preempt00:06:423
2189299421,6cyclictest0-21swapper/322:06:183
2189299421,6cyclictest0-21swapper/300:26:273
2189299421,22cyclictest0-21swapper/323:27:073
21892994210,5cyclictest131rcu_preempt23:34:503
2188699428,5cyclictest131rcu_preempt23:19:591
2188699424,3cyclictest131rcu_preempt21:18:071
2188699422,5cyclictest131rcu_preempt00:03:251
21886994225,13cyclictest3467602-21systemd-journal19:47:201
2188699422,3cyclictest131rcu_preempt22:49:301
2188699422,3cyclictest131rcu_preempt21:26:201
2188199421,6cyclictest22917-21munin-run19:30:000
2189299417,5cyclictest131rcu_preempt21:26:273
2189299417,3cyclictest131rcu_preempt21:38:373
21892994129,9cyclictest1-21systemd20:25:013
21892994128,9cyclictest275-21dbus-daemon23:40:013
21892994127,9cyclictest37-21ksoftirqd/323:50:133
2189299411,4cyclictest131rcu_preempt00:21:143
2189299411,4cyclictest131rcu_preempt00:05:013
2188699417,6cyclictest23-21ksoftirqd/120:58:211
2188699413,3cyclictest131rcu_preempt23:04:371
2188699413,3cyclictest131rcu_preempt23:04:371
2188699412,4cyclictest131rcu_preempt23:50:541
2188699412,4cyclictest131rcu_preempt00:16:171
21886994124,11cyclictest309-21in:imuxsock20:30:011
2188699411,3cyclictest131rcu_preempt00:30:421
21881994125,13cyclictest25762-21sshd20:22:210
2188199411,36cyclictest0-21swapper/019:25:020
2189299407,3cyclictest131rcu_preempt21:56:423
2189299406,5cyclictest131rcu_preempt21:48:563
2189299405,3cyclictest131rcu_preempt22:57:583
2189299405,29cyclictest125562-21grep00:35:033
2189299404,3cyclictest131rcu_preempt00:20:023
2189299403,4cyclictest131rcu_preempt23:17:063
21892994025,12cyclictest28114-21latency_hist21:10:033
21892994025,11cyclictest44625-21systemd-run21:45:023
21892994024,11cyclictest275-21dbus-daemon20:15:013
21892994024,11cyclictest275-21dbus-daemon20:15:013
21892994016,3cyclictest150-21pr/ttyS222:18:423
2188699405,6cyclictest131rcu_preempt21:52:051
2188699403,3cyclictest131rcu_preempt22:13:421
2188699403,3cyclictest131rcu_preempt20:54:221
21886994026,11cyclictest23401-21sshd19:38:211
2188699402,4cyclictest131rcu_preempt19:22:211
21886994024,12cyclictest3467602-21systemd-journal00:27:411
2188699401,4cyclictest131rcu_preempt22:57:101
21881994022,14cyclictest275-21dbus-daemon19:35:020
2189299398,5cyclictest131rcu_preempt19:29:223
2189299396,5cyclictest131rcu_preempt22:43:093
2189299394,4cyclictest131rcu_preempt23:22:463
2189299394,28cyclictest24208-21munin-run19:55:003
2189299393,4cyclictest131rcu_preempt22:24:083
2189299393,3cyclictest131rcu_preempt22:00:533
2189299392,3cyclictest131rcu_preempt19:10:023
21892993922,12cyclictest0-21swapper/319:50:013
21892993922,11cyclictest275-21dbus-daemon21:05:013
2188699393,4cyclictest131rcu_preempt20:25:011
21886993929,7cyclictest23-21ksoftirqd/119:55:021
21886993928,8cyclictest25214-21sshd20:11:211
21886993928,8cyclictest25214-21sshd20:11:201
21886993925,8cyclictest3467602-21systemd-journal19:15:011
2188699391,4cyclictest131rcu_preempt19:30:011
2188699391,3cyclictest131rcu_preempt20:40:011
2188199393,6cyclictest131rcu_preempt19:15:000
2189299383,3cyclictest131rcu_preempt22:36:103
21892993825,9cyclictest275-21dbus-daemon20:05:013
21892993823,12cyclictest37-21ksoftirqd/323:47:203
21892993823,12cyclictest37-21ksoftirqd/323:47:193
21890993822,11cyclictest21999-21latency_hist19:10:012
2188699383,3cyclictest131rcu_preempt20:10:001
2188699381,3cyclictest131rcu_preempt20:40:221
21881993829,6cyclictest19350irq/34-mmc119:52:570
21881993829,6cyclictest12-21ksoftirqd/020:50:020
21881993828,7cyclictest12-21ksoftirqd/019:40:020
2188199382,7cyclictest364-21snmpd19:46:450
21881993827,9cyclictest12-21ksoftirqd/020:55:010
2188199382,5cyclictest3467602-21systemd-journal19:42:200
2188199382,5cyclictest27867-21munin-run21:05:010
21892993728,6cyclictest23411-21cron19:40:003
21892993727,7cyclictest37-21ksoftirqd/320:35:003
21892993721,11cyclictest3467602-21systemd-journal20:45:003
21881993729,4cyclictest26549-21munin-plugin-st20:40:010
2189299364,5cyclictest131rcu_preempt20:40:033
21886993621,11cyclictest23-21ksoftirqd/119:20:011
21881993620,11cyclictest275-21dbus-daemon20:35:010
2189299352,4cyclictest131rcu_preempt19:45:003
21886993526,6cyclictest23-21ksoftirqd/121:05:011
2188699351,3cyclictest131rcu_preempt19:55:031
2188699351,3cyclictest131rcu_preempt19:55:021
21881993526,6cyclictest12-21ksoftirqd/020:05:010
21881993510,16cyclictest0-21swapper/019:10:020
2189299341,3cyclictest131rcu_preempt20:50:033
21886993423,8cyclictest23-21ksoftirqd/119:35:011
21892993326,4cyclictest210-21jbd2/mmcblk1p2-19:30:143
21892993322,8cyclictest37-21ksoftirqd/320:10:013
2188699331,3cyclictest131rcu_preempt19:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional