You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-16 - 09:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sat May 16, 2026 00:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
104614399932,3cyclictest0-21swapper/019:30:020
1046148998920,4cyclictest131rcu_preempt22:25:021
104614899862,9cyclictest131rcu_preempt20:45:011
1046148998321,32cyclictest1052321-21diskmemload22:04:521
1046143998379,2cyclictest0-21swapper/023:10:010
1046143997847,28cyclictest16950irq/224-800000000:30:020
1046143997847,28cyclictest16950irq/224-800000000:30:010
1046143997762,9cyclictest1046147-21latency_hist19:10:010
1046148997427,41cyclictest309-21in:imuxsock20:20:001
104614899692,15cyclictest3467602-21systemd-journal00:31:461
1046148996627,35cyclictest1048192-21latency_hist19:50:021
1046148996626,35cyclictest1059658-21latency_hist21:25:021
1046148996626,35cyclictest1059658-21latency_hist21:25:021
104614899662,19cyclictest1099909-21ssh22:48:311
104614899662,15cyclictest1047135-21munin-run19:30:001
104614899661,16cyclictest1114792-21ssh23:19:311
104614899652,16cyclictest1138342-21sh00:07:591
104614899652,16cyclictest1110231-21munin-run23:10:011
1046143996527,32cyclictest1098184-21latency_hist22:45:010
1046143996430,16cyclictest1134388-21ssh00:00:010
1046143996424,11cyclictest1054721-21modprobe21:15:010
1046143996424,11cyclictest1054721-21modprobe21:15:000
104614899633,7cyclictest1051773-21sshd20:59:431
104614899632,22cyclictest1142864-21ssh00:17:261
104614899632,18cyclictest1127380-21ssh23:45:051
104614899632,18cyclictest1127380-21ssh23:45:051
104614899632,17cyclictest1083960-21ssh22:15:061
104614899631,14cyclictest1083419-21ssh22:14:191
104614899631,13cyclictest1152447-21ssh00:37:111
104614899631,11cyclictest1089021-21ssh22:25:401
104614399634,33cyclictest151rcuc/023:20:010
104615799626,40cyclictest1052321-21diskmemload22:00:033
1046148996222,36cyclictest23-21ksoftirqd/122:07:001
104614899622,16cyclictest48-21kcompactd021:50:181
104614899622,12cyclictest364-21snmpd23:21:141
104614899621,9cyclictest1140425-21ssh00:12:181
104614899621,13cyclictest1123272-21ssh23:36:441
104614899621,13cyclictest1123272-21ssh23:36:431
1046148996131,26cyclictest1134052-21sh23:59:191
1046148996123,34cyclictest1144964-21sh00:21:391
1046148996122,35cyclictest23-21ksoftirqd/120:10:001
104614899612,15cyclictest1075298-21ssh21:57:281
104614899612,15cyclictest1052745-21ssh21:10:251
104614899612,15cyclictest1052745-21ssh21:10:241
104614899611,15cyclictest1126662-21ssh23:43:521
104614899611,15cyclictest1051501-21sshd20:50:431
104614899611,15cyclictest1051501-21sshd20:50:421
104614899611,12cyclictest364-21snmpd23:34:181
104614899611,11cyclictest1097471-21ssh22:43:211
104614899602,12cyclictest1117719-21ssh23:25:081
104614899601,9cyclictest364-21snmpd21:26:481
104614899601,9cyclictest1095095-21ssh22:38:301
104614899601,9cyclictest1070738-21ssh21:47:581
104614899601,8cyclictest1101933-21ssh22:52:361
104614399602,41cyclictest131rcu_preempt21:35:140
1046143996020,35cyclictest16850irq/134-800000000:11:170
1046152995944,7cyclictest1051566-21kworker/2:323:09:582
104614899592,8cyclictest1046328-21modprobe19:15:001
104614899592,11cyclictest1093343-21sh22:34:591
104614899591,11cyclictest1068876-21ssh21:44:171
1046143995947,7cyclictest1-21systemd22:00:000
1046143995925,6cyclictest1100645-21latency_hist22:50:020
104614899582,15cyclictest1062390-21ssh21:30:261
104614899582,15cyclictest1062390-21ssh21:30:261
1046148995813,40cyclictest0-21swapper/100:01:021
104614899581,15cyclictest1147822-21ssh00:27:361
104614899581,15cyclictest1147822-21ssh00:27:351
104614899581,12cyclictest1056337-21sh21:18:061
104614899581,11cyclictest1052303-21sendmail-mta21:09:391
104614899581,10cyclictest1064648-21sh21:35:111
1046143995838,14cyclictest3467602-21systemd-journal20:10:010
1046143995818,35cyclictest16850irq/134-800000022:05:020
104614899575,44cyclictest1049448-21modprobe20:15:011
104614899572,19cyclictest1107614-21ssh23:04:321
104614899572,11cyclictest48-21kcompactd023:10:341
1046143995727,4cyclictest10737271ssh21:54:230
1046143995727,15cyclictest1117356-21ssh23:24:500
1046143995710,32cyclictest151rcuc/022:30:010
1046152995647,5cyclictest1115073-21kworker/2:200:23:382
1046152995646,7cyclictest1051566-21kworker/2:321:42:102
1046152995646,6cyclictest1051566-21kworker/2:323:27:222
1046152995637,14cyclictest1-21systemd23:20:002
1046152995611,13cyclictest131rcu_preempt20:42:582
104614899562,11cyclictest1105367-21ssh22:59:521
104614899561,17cyclictest1130382-21ssh23:51:301
104614899561,16cyclictest1048433-21munin-run19:55:001
1046143995628,13cyclictest1150374-21ssh00:32:550
1046143995619,33cyclictest16850irq/134-800000022:36:090
1046143995619,32cyclictest16850irq/134-800000023:01:040
1046143995618,33cyclictest16850irq/134-800000023:30:010
104614399561,3cyclictest131rcu_preempt21:18:090
104614399561,39cyclictest1101758-21ssh22:52:120
1046152995546,6cyclictest1051566-21kworker/2:323:30:292
1046152995546,6cyclictest1051566-21kworker/2:321:23:042
1046152995546,6cyclictest1051566-21kworker/2:321:23:042
1046152995545,6cyclictest1051566-21kworker/2:321:28:002
104614899552,13cyclictest1046642-21munin-run19:20:021
104614899551,12cyclictest1052076-21systemd-run21:05:011
104614899551,11cyclictest364-21snmpd20:21:161
1046143995527,14cyclictest1145185-21ssh00:22:080
1046143995527,13cyclictest1087628-21ssh22:22:510
1046143995519,31cyclictest16850irq/134-800000023:50:360
1046143995519,31cyclictest16850irq/134-800000023:35:150
1046143995519,31cyclictest16850irq/134-800000023:35:150
1046143995519,31cyclictest16850irq/134-800000023:13:370
1046143995519,31cyclictest16850irq/134-800000022:14:580
1046143995519,31cyclictest16850irq/134-800000000:01:250
1046152995446,5cyclictest1051566-21kworker/2:322:25:482
1046152995446,5cyclictest1051566-21kworker/2:322:22:512
1046152995445,6cyclictest1051566-21kworker/2:322:08:592
1046152995428,17cyclictest30-21ksoftirqd/200:37:432
104614899543,7cyclictest1047625-21cron19:40:001
104614399545,37cyclictest16950irq/224-800000020:20:020
1046143995419,31cyclictest16850irq/134-800000000:08:450
1046143995419,30cyclictest16850irq/134-800000021:45:210
1046152995345,5cyclictest1095817-21kworker/2:122:55:242
1046152995345,5cyclictest1051566-21kworker/2:300:00:582
1046152995345,5cyclictest1037204-21kworker/2:221:12:402
1046152995345,5cyclictest1037204-21kworker/2:221:12:402
1046152995344,6cyclictest1051566-21kworker/2:322:30:152
1046152995344,6cyclictest1051566-21kworker/2:321:15:042
1046152995344,5cyclictest1051566-21kworker/2:321:36:532
1046152995338,11cyclictest1051566-21kworker/2:323:04:282
104614899536,40cyclictest0-21swapper/120:35:001
104614899532,5cyclictest3467602-21systemd-journal19:45:001
104614899532,14cyclictest364-21snmpd19:59:501
104614899532,14cyclictest364-21snmpd19:59:491
104614899531,7cyclictest1048938-21cron20:04:591
1046143995319,30cyclictest16850irq/134-800000023:45:220
1046143995319,30cyclictest16850irq/134-800000023:45:220
1046143995319,30cyclictest16850irq/134-800000023:34:070
1046143995319,30cyclictest16850irq/134-800000000:35:560
1046143995318,31cyclictest16850irq/134-800000022:55:430
1046143995318,31cyclictest16850irq/134-800000022:06:370
1046143995318,31cyclictest16850irq/134-800000021:44:350
1046143995318,31cyclictest16850irq/134-800000021:21:020
1046143995318,31cyclictest16850irq/134-800000021:21:010
1046143995316,33cyclictest16850irq/134-800000022:31:000
1046143995316,32cyclictest16850irq/134-800000022:15:050
104615799522,10cyclictest37-21ksoftirqd/300:30:003
104615799522,10cyclictest37-21ksoftirqd/300:29:593
1046152995244,5cyclictest1115073-21kworker/2:200:17:492
1046152995244,5cyclictest1060041-21kworker/2:022:36:272
1046152995244,5cyclictest1051566-21kworker/2:323:13:092
1046152995243,6cyclictest1060041-21kworker/2:022:41:072
1046152995243,6cyclictest1051566-21kworker/2:323:23:422
1046152995243,6cyclictest1051566-21kworker/2:322:12:372
1046152995243,6cyclictest1051566-21kworker/2:321:51:392
1046152995243,5cyclictest1115073-21kworker/2:200:14:472
1046152995243,5cyclictest1115073-21kworker/2:200:07:482
1046152995243,5cyclictest1037204-21kworker/2:220:51:172
1046152995243,5cyclictest1037204-21kworker/2:220:51:162
1046152995242,6cyclictest1095817-21kworker/2:122:54:362
104614899522,7cyclictest1047386-21munin-run19:35:011
1046143995218,30cyclictest16850irq/134-800000021:27:020
1046143995216,20cyclictest16850irq/134-800000023:40:110
1046157995132,14cyclictest1-21systemd22:35:023
1046152995144,4cyclictest1051566-21kworker/2:300:30:172
1046152995143,5cyclictest1095817-21kworker/2:122:48:082
1046152995143,5cyclictest1060041-21kworker/2:021:30:272
1046152995143,5cyclictest1060041-21kworker/2:021:30:262
1046152995143,5cyclictest1051566-21kworker/2:323:53:382
1046152995143,5cyclictest1051566-21kworker/2:323:40:582
1046152995143,5cyclictest1051566-21kworker/2:321:48:582
1046152995143,4cyclictest1051566-21kworker/2:322:20:042
1046152995142,6cyclictest1051566-21kworker/2:323:56:542
1046152995142,6cyclictest1051566-21kworker/2:321:56:192
1046152995138,9cyclictest30-21ksoftirqd/221:10:002
1046148995116,30cyclictest1050232-21kworker/u8:220:40:011
1046143995117,19cyclictest16850irq/134-800000021:34:120
1046143995117,19cyclictest16850irq/134-800000021:34:120
1046143995116,32cyclictest16850irq/134-800000000:15:570
1046152995041,6cyclictest1037204-21kworker/2:220:35:012
1046152995041,6cyclictest1037204-21kworker/2:219:35:002
104614899501,5cyclictest364-21snmpd19:23:071
1046157994938,8cyclictest37-21ksoftirqd/322:58:093
1046157994918,8cyclictest131rcu_preempt21:20:193
1046157994918,8cyclictest131rcu_preempt21:20:183
1046157994912,7cyclictest131rcu_preempt23:40:003
1046157994912,7cyclictest131rcu_preempt23:39:593
1046152994941,5cyclictest1051566-21kworker/2:322:00:032
1046152994941,5cyclictest1037204-21kworker/2:219:56:442
1046152994941,5cyclictest1037204-21kworker/2:219:56:432
104614899491,11cyclictest364-21snmpd20:25:051
1046157994834,9cyclictest37-21ksoftirqd/322:14:593
1046157994834,9cyclictest37-21ksoftirqd/300:05:133
1046157994834,10cyclictest1-21systemd19:35:013
1046152994840,5cyclictest1051566-21kworker/2:323:35:162
1046152994840,5cyclictest1051566-21kworker/2:323:35:162
1046152994840,5cyclictest1037204-21kworker/2:220:40:002
1046152994839,6cyclictest1051566-21kworker/2:323:45:242
1046152994839,6cyclictest1051566-21kworker/2:323:45:232
1046152994839,6cyclictest1037204-21kworker/2:219:12:422
1046152994839,5cyclictest1051566-21kworker/2:300:27:582
1046152994839,5cyclictest1051566-21kworker/2:300:27:572
1046152994838,7cyclictest1037204-21kworker/2:221:00:422
104614899481,7cyclictest3467602-21systemd-journal20:47:041
1046143994834,11cyclictest1-21systemd19:20:000
1046157994733,10cyclictest37-21ksoftirqd/322:08:063
1046157994728,14cyclictest1146459-21systemctl00:25:013
1046157994728,11cyclictest1046608-21systemctl19:20:013
1046157994714,6cyclictest131rcu_preempt00:19:223
1046152994739,5cyclictest1037204-21kworker/2:220:20:522
1046152994739,5cyclictest1037204-21kworker/2:220:15:002
1046152994739,5cyclictest1037204-21kworker/2:219:40:012
1046152994739,5cyclictest1037204-21kworker/2:219:30:022
1046152994735,9cyclictest1-21systemd20:30:002
1046143994728,15cyclictest275-21dbus-daemon20:45:000
1046143994723,19cyclictest3467602-21systemd-journal21:10:000
1046157994635,7cyclictest37-21ksoftirqd/323:57:193
1046157994633,7cyclictest1049458-21cut20:15:003
1046157994618,6cyclictest361rcuc/321:19:143
1046157994615,9cyclictest131rcu_preempt22:40:173
1046157994615,7cyclictest37-21ksoftirqd/321:10:003
1046152994637,6cyclictest1037204-21kworker/2:219:45:012
1046143994629,14cyclictest0-21swapper/019:28:440
1046157994534,7cyclictest1-21systemd22:20:013
1046157994530,11cyclictest37-21ksoftirqd/321:30:003
1046157994525,15cyclictest3467602-21systemd-journal19:25:013
104615299453,3cyclictest131rcu_preempt20:05:022
104614399452,37cyclictest1049111-21latency_hist20:05:010
104614399451,41cyclictest1050483-21sshd20:33:440
104615799441,9cyclictest0-21swapper/323:30:203
104615799441,9cyclictest0-21swapper/300:00:333
1046157994419,8cyclictest131rcu_preempt21:39:363
104615799441,8cyclictest0-21swapper/323:25:213
104615799441,8cyclictest0-21swapper/322:01:153
1046157994416,8cyclictest37-21ksoftirqd/323:05:433
1046157994411,5cyclictest131rcu_preempt00:14:583
104615299444,3cyclictest131rcu_preempt19:20:012
104615299441,3cyclictest131rcu_preempt19:25:012
104614899443,8cyclictest4109409-1kworker/1:0H19:10:031
1046143994421,16cyclictest275-21dbus-daemon19:25:000
104615799432,36cyclictest275-21dbus-daemon20:25:013
1046157994321,14cyclictest275-21dbus-daemon21:00:013
104615799431,9cyclictest0-21swapper/323:53:373
104615799431,7cyclictest0-21swapper/323:19:433
104615799431,6cyclictest0-21swapper/321:45:583
104615799425,5cyclictest131rcu_preempt22:26:033
104615799421,8cyclictest0-21swapper/323:41:443
104615799421,7cyclictest0-21swapper/323:47:053
104615799421,7cyclictest0-21swapper/323:47:043
104615799421,7cyclictest0-21swapper/300:32:083
1046157994213,6cyclictest131rcu_preempt22:24:313
104615299424,3cyclictest131rcu_preempt20:50:022
104615299423,6cyclictest131rcu_preempt20:59:592
1046152994234,5cyclictest1037204-21kworker/2:220:08:522
1046152994224,13cyclictest3467602-21systemd-journal20:15:032
1046143994229,10cyclictest12-21ksoftirqd/019:45:020
1046143994227,11cyclictest1-21systemd20:50:010
1046143994225,13cyclictest1-21systemd20:40:000
104614399422,36cyclictest1051757-21sshd20:56:220
104615799415,7cyclictest37-21ksoftirqd/323:20:243
1046157994119,14cyclictest275-21dbus-daemon21:05:013
104615799411,8cyclictest0-21swapper/323:14:073
104615799411,8cyclictest0-21swapper/321:41:283
104615799411,8cyclictest0-21swapper/321:30:283
104615799411,8cyclictest0-21swapper/321:30:273
104615799411,8cyclictest0-21swapper/321:10:163
104615799411,8cyclictest0-21swapper/321:10:153
104615799411,7cyclictest0-21swapper/322:45:473
104615799411,7cyclictest0-21swapper/321:53:163
1046157994117,6cyclictest131rcu_preempt20:40:013
104615799411,6cyclictest0-21swapper/322:53:373
104615799411,6cyclictest0-21swapper/322:38:113
104615799411,5cyclictest0-21swapper/300:37:183
104615299412,4cyclictest131rcu_preempt19:55:012
1046143994130,7cyclictest1051525-21cron20:55:010
1046143994130,7cyclictest1051525-21cron20:55:000
1046143994129,9cyclictest12-21ksoftirqd/020:00:010
1046143994129,9cyclictest12-21ksoftirqd/020:00:000
1046143994127,11cyclictest1052043-21sshd21:02:430
104614399412,3cyclictest131rcu_preempt19:15:010
1046157994027,10cyclictest1049272-21sed20:10:023
1046157994026,11cyclictest37-21ksoftirqd/323:02:133
1046157994025,11cyclictest37-21ksoftirqd/320:35:013
1046157994024,11cyclictest309-21in:imuxsock20:17:003
1046143994028,9cyclictest12-21ksoftirqd/020:25:020
104614399402,3cyclictest131rcu_preempt19:40:000
104614399401,3cyclictest131rcu_preempt20:15:010
104615799396,6cyclictest131rcu_preempt20:44:593
1046157993924,9cyclictest309-21in:imuxsock19:40:003
104614399391,3cyclictest131rcu_preempt20:28:430
1046157993822,10cyclictest0-21swapper/320:29:593
1046152993817,15cyclictest3467602-21systemd-journal19:50:002
104615799377,5cyclictest131rcu_preempt20:53:363
104615799377,5cyclictest131rcu_preempt20:53:353
104615799374,26cyclictest1048666-21modprobe20:00:013
104615799374,26cyclictest1048666-21modprobe20:00:003
104615799373,5cyclictest131rcu_preempt19:15:023
1046157993726,7cyclictest37-21ksoftirqd/319:50:033
1046157993724,9cyclictest309-21in:imuxsock20:04:593
1046157993721,11cyclictest3467602-21systemd-journal19:25:003
1046143993725,9cyclictest12-21ksoftirqd/019:50:020
1046157993624,10cyclictest37-21ksoftirqd/319:40:573
1046157993616,15cyclictest1-21systemd20:50:013
104614399351,8cyclictest1-21systemd19:50:430
1046152993427,4cyclictest1037204-21kworker/2:219:10:022
1046157993124,4cyclictest1-21systemd19:55:003
1046157993124,4cyclictest1046082-21latency_hist19:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional