You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-02 - 11:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sun Aug 02, 2026 00:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
113540999931,88cyclictest0-21swapper/021:50:020
1135423998921,33cyclictest1159484-21sh21:48:053
1135423998726,34cyclictest1176148-21sh22:23:483
1135423997913,15cyclictest131rcu_preempt21:00:013
1135423997220,48cyclictest1188352-21systemd-run22:50:013
1135423997220,48cyclictest1188352-21systemd-run22:50:013
113540999723,3cyclictest131rcu_preempt21:12:020
1135409996864,2cyclictest0-21swapper/022:35:020
1135409996864,2cyclictest0-21swapper/022:35:010
1135423996528,31cyclictest3467602-21systemd-journal23:35:013
1135423996525,35cyclictest1226451-21sh00:11:053
113542399652,15cyclictest1232423-21ssh00:24:103
1135409996533,14cyclictest0-21swapper/023:39:250
113540999651,47cyclictest12-21ksoftirqd/023:50:020
113540999651,47cyclictest12-21ksoftirqd/023:50:020
113542399643,6cyclictest3467602-21systemd-journal21:33:313
1135423996423,36cyclictest1172826-21sh22:16:403
113542399632,6cyclictest1138219-21munin-run20:05:003
113542399622,56cyclictest1135908-21munin-run19:20:013
113542399622,19cyclictest1165382-21ssh22:00:283
113542399621,8cyclictest1214010-21ssh23:44:483
113542399621,8cyclictest1198535-21ssh23:11:193
1135423996216,38cyclictest275-21dbus-daemon19:30:013
113542399621,15cyclictest1168644-21ssh22:07:373
1135409996257,3cyclictest0-21swapper/023:25:030
113542399613,52cyclictest1221165-21ssh00:00:023
113542399612,8cyclictest1224833-21ssh00:07:503
113542399611,9cyclictest1201248-21ssh23:17:143
113542399611,8cyclictest1177045-21ssh22:25:203
113542399611,8cyclictest1148047-21ssh21:23:503
113542399611,7cyclictest1237885-21ssh00:35:283
113542399611,7cyclictest1180171-21ssh22:32:253
113542399611,6cyclictest1237460-21munin-run00:35:013
113542399611,19cyclictest1145775-21ssh21:19:013
113542399603,7cyclictest1190641-21munin-run22:55:003
113542399602,6cyclictest1210643-21ssh23:37:333
113542399602,15cyclictest1196086-21ssh23:06:103
113542399601,8cyclictest1163835-21ssh21:57:263
113542399601,8cyclictest1141969-21ssh21:10:183
113542399601,14cyclictest1230477-21munin-run00:20:013
113542399591,7cyclictest1233200-21ssh00:25:313
113542399591,7cyclictest1217708-21ssh23:52:293
113542399591,7cyclictest1156746-21ssh21:42:023
113542399591,6cyclictest1202835-21ssh23:20:243
113542399591,6cyclictest1202835-21ssh23:20:243
113542399591,6cyclictest1170793-21ssh22:12:123
113542399591,5cyclictest1160658-21ssh21:50:163
113542399591,10cyclictest1136882-21sshd19:38:023
1135409995937,18cyclictest16850irq/134-800000023:55:000
113542399582,5cyclictest1139498-21sshd20:29:023
113542399582,18cyclictest1221387-21ssh00:00:053
1135423995820,33cyclictest1141588-21diskmemload22:58:593
1135423995820,33cyclictest1141588-21diskmemload22:58:593
113542399581,7cyclictest1149307-21ssh21:26:093
113542399581,6cyclictest1185516-21ssh22:43:563
113542399581,5cyclictest1214567-21ssh23:45:413
113542399581,5cyclictest1214567-21ssh23:45:403
1135409995828,25cyclictest1164973-21sendmail22:00:000
1135409995820,33cyclictest16850irq/134-800000000:35:010
1135409995817,35cyclictest16850irq/134-800000023:05:180
113540999581,3cyclictest131rcu_preempt23:15:020
1135409995810,32cyclictest1155805-21latency_hist21:40:020
113542399575,5cyclictest1140783-21sshd20:53:033
113542399572,5cyclictest1205290-21ssh23:25:453
113542399571,5cyclictest1138726-21munin-run20:15:003
113542399571,4cyclictest1181561-21ssh22:35:153
113542399571,4cyclictest1181561-21ssh22:35:153
113542399571,17cyclictest1195413-21systemctl23:05:003
113542399571,14cyclictest1154167-21ssh21:36:393
113540999573,3cyclictest131rcu_preempt00:28:440
1135409995728,13cyclictest1185276-21ssh22:43:220
113542399562,7cyclictest1137410-21sshd19:47:143
1135423995627,24cyclictest1137808-21sed19:55:023
113542399562,15cyclictest1135729-21cat19:15:023
113542399561,5cyclictest1137980-21munin-run20:00:003
113542399561,12cyclictest48-21kcompactd020:45:023
113542399561,12cyclictest120-21kswapd019:45:013
1135418995637,13cyclictest1-21systemd22:05:002
1135418995634,5cyclictest30-21ksoftirqd/200:00:022
1135409995628,12cyclictest1232519-21ssh00:24:230
113540999562,38cyclictest1200650-21ssh23:15:500
1135409995619,33cyclictest16850irq/134-800000022:58:470
1135409995619,33cyclictest16850irq/134-800000022:58:470
113540999561,40cyclictest1171772-21ssh22:14:420
1135418995547,5cyclictest1140518-21kworker/2:100:32:452
1135418995547,4cyclictest1140518-21kworker/2:100:19:262
1135418995546,5cyclictest1202455-21kworker/2:023:55:012
1135409995527,13cyclictest1150320-21ssh21:28:390
1135409995525,14cyclictest1226443-21ssh00:11:040
1135409995519,6cyclictest16850irq/134-800000021:42:590
1135409995519,32cyclictest16850irq/134-800000000:09:360
1135409995519,31cyclictest16850irq/134-800000021:17:420
1135409995518,33cyclictest16850irq/134-800000022:50:550
1135409995518,32cyclictest16850irq/134-800000021:20:360
1135409995517,33cyclictest16850irq/134-800000023:58:350
1135409995517,32cyclictest16850irq/134-800000022:21:000
1135423995425,26cyclictest3467562-21sshd20:32:213
113542399542,17cyclictest1140297-21latency_hist20:45:003
113540999542,4cyclictest131rcu_preempt20:50:010
1135409995419,31cyclictest16850irq/134-800000000:39:120
1135409995418,6cyclictest16850irq/134-800000022:18:070
1135409995418,6cyclictest16850irq/134-800000000:00:220
1135409995418,32cyclictest16850irq/134-800000023:40:440
1135409995418,32cyclictest16850irq/134-800000022:46:140
1135409995418,32cyclictest16850irq/134-800000022:46:140
1135409995417,32cyclictest16850irq/134-800000022:05:050
1135409995417,32cyclictest16850irq/134-800000021:32:270
1135409995416,33cyclictest16850irq/134-800000022:27:260
113540999541,35cyclictest131rcu_preempt22:31:280
1135409995410,39cyclictest275-21dbus-daemon21:10:010
113542399533,4cyclictest0-21swapper/319:22:373
1135418995345,5cyclictest1140518-21kworker/2:122:12:182
1135418995343,6cyclictest1202455-21kworker/2:023:46:282
1135418995343,6cyclictest1202455-21kworker/2:023:46:282
1135409995317,6cyclictest16850irq/134-800000000:15:530
1135409995317,31cyclictest16850irq/134-800000023:22:400
1135409995317,31cyclictest16850irq/134-800000023:22:400
1135409995316,32cyclictest16850irq/134-800000021:50:060
113542399523,4cyclictest364-21snmpd20:37:513
113542399523,4cyclictest364-21snmpd20:37:503
113542399522,4cyclictest0-21swapper/320:06:353
113542399522,14cyclictest1141728-21cat21:10:023
113542399521,10cyclictest48-21kcompactd020:59:283
113541899527,4cyclictest131rcu_preempt23:23:352
113541899527,4cyclictest131rcu_preempt23:23:352
113541899525,4cyclictest131rcu_preempt23:26:452
1135418995244,5cyclictest1140518-21kworker/2:122:35:262
1135418995244,5cyclictest1140518-21kworker/2:122:35:262
1135418995231,6cyclictest1135173-21kworker/2:323:05:202
1135418995210,6cyclictest131rcu_preempt22:08:382
1135414995215,7cyclictest131rcu_preempt22:40:011
1135414995215,7cyclictest131rcu_preempt22:40:001
1135409995216,31cyclictest16850irq/134-800000023:30:070
113540999521,36cyclictest1194268-21ssh23:02:200
113542399511,5cyclictest1138944-21sshd20:16:023
113542399511,4cyclictest364-21snmpd20:20:073
1135418995133,13cyclictest3467602-21systemd-journal00:25:002
1135409995117,30cyclictest16850irq/134-800000022:01:250
113542399501,5cyclictest48-21kcompactd019:30:093
1135418995042,5cyclictest1202455-21kworker/2:000:02:262
1135418995042,5cyclictest1140518-21kworker/2:122:31:332
1135418995042,5cyclictest1140518-21kworker/2:121:15:032
1135418995041,6cyclictest1140518-21kworker/2:121:37:522
1135418995036,8cyclictest1146255-21latency_hist21:20:032
1135418995035,11cyclictest3467602-21systemd-journal20:25:052
113540999505,3cyclictest131rcu_preempt19:14:590
1135418994941,5cyclictest1140518-21kworker/2:123:41:472
1135418994941,5cyclictest1140518-21kworker/2:121:32:052
1135418994941,5cyclictest1128271-21kworker/2:119:55:022
1135418994940,5cyclictest1140518-21kworker/2:121:59:342
1135418994940,5cyclictest1140518-21kworker/2:121:27:082
1135418994939,7cyclictest1140518-21kworker/2:123:14:112
1135418994939,6cyclictest1140518-21kworker/2:123:38:572
113541499491,28cyclictest0-21swapper/123:51:131
113541899484,4cyclictest131rcu_preempt22:56:312
113541899484,4cyclictest131rcu_preempt22:56:312
1135418994841,5cyclictest1140518-21kworker/2:122:18:472
1135418994841,4cyclictest1135173-21kworker/2:320:25:012
1135418994840,5cyclictest1140518-21kworker/2:122:26:312
1135418994840,5cyclictest1140518-21kworker/2:100:11:132
1135418994835,9cyclictest3467602-21systemd-journal19:30:032
1135418994823,6cyclictest30-21ksoftirqd/200:25:352
1135414994830,14cyclictest3467602-21systemd-journal20:15:001
113540999483,3cyclictest131rcu_preempt19:50:010
113541899477,3cyclictest131rcu_preempt23:02:412
1135418994739,5cyclictest1135173-21kworker/2:321:47:422
1135418994732,12cyclictest1190789-21sed22:55:012
1135414994732,11cyclictest1-21systemd00:05:001
1135414994732,10cyclictest3467602-21systemd-journal22:55:001
1135414994711,5cyclictest131rcu_preempt22:25:021
1135409994738,5cyclictest16950irq/224-800000020:40:010
1135409994738,5cyclictest16950irq/224-800000020:40:010
113540999472,3cyclictest131rcu_preempt20:25:000
113540999472,39cyclictest1136889-21cron19:40:000
1135418994640,3cyclictest1140518-21kworker/2:123:15:202
1135418994638,5cyclictest1140518-21kworker/2:100:35:392
1135418994637,6cyclictest1135173-21kworker/2:320:19:592
1135418994633,9cyclictest3467602-21systemd-journal22:25:022
113541499464,3cyclictest131rcu_preempt23:23:501
113541499464,3cyclictest131rcu_preempt23:23:501
1135414994633,10cyclictest275-21dbus-daemon23:05:011
113540999461,3cyclictest131rcu_preempt19:20:010
1135409994610,19cyclictest1135412-21grep19:10:010
1135418994537,5cyclictest1140518-21kworker/2:122:41:392
1135418994537,5cyclictest1140518-21kworker/2:100:10:022
1135414994529,12cyclictest3467602-21systemd-journal00:00:021
1135414994518,21cyclictest3467602-21systemd-journal21:05:011
113540999452,3cyclictest131rcu_preempt20:05:000
113542399447,32cyclictest1135493-21grep19:10:023
113541899445,3cyclictest131rcu_preempt23:30:452
113541899444,4cyclictest131rcu_preempt21:24:022
1135418994436,5cyclictest1135173-21kworker/2:320:07:012
1135418994436,5cyclictest1135173-21kworker/2:319:36:032
1135418994427,12cyclictest1-21systemd19:55:002
1135418994410,6cyclictest131rcu_preempt21:50:292
113541499444,3cyclictest131rcu_preempt00:36:471
1135414994428,12cyclictest23-21ksoftirqd/123:45:221
1135414994428,12cyclictest23-21ksoftirqd/123:45:221
113541499441,4cyclictest131rcu_preempt20:26:011
113541499441,3cyclictest131rcu_preempt22:30:001
1135409994436,4cyclictest1140846-21latency_hist20:55:020
113540999442,37cyclictest1140301-21munin-run20:44:590
1135418994335,5cyclictest1135173-21kworker/2:320:44:162
1135418994335,5cyclictest1135173-21kworker/2:320:11:062
1135418994327,12cyclictest3467602-21systemd-journal21:10:002
1135414994331,8cyclictest1136059-21sed19:20:031
1135414994331,7cyclictest1138990-21munin-plugin-st20:20:001
1135414994331,10cyclictest23-21ksoftirqd/121:35:021
1135414994330,9cyclictest3467602-21systemd-journal00:28:031
1135409994328,12cyclictest275-21dbus-daemon20:10:000
113540999432,3cyclictest131rcu_preempt20:20:000
1135409994317,22cyclictest16850irq/134-800000019:55:010
113540999431,3cyclictest131rcu_preempt20:35:020
1135418994234,5cyclictest1140518-21kworker/2:122:50:002
1135418994234,5cyclictest1140518-21kworker/2:122:50:002
1135418994226,12cyclictest1140007-21cron20:40:012
1135418994226,12cyclictest1140007-21cron20:40:012
1135418994226,11cyclictest309-21in:imuxsock19:17:012
113541499427,7cyclictest23-21ksoftirqd/122:56:281
113541499427,7cyclictest23-21ksoftirqd/122:56:271
113541499425,4cyclictest131rcu_preempt00:25:031
1135414994231,8cyclictest1140003-21sshd20:39:021
1135414994231,8cyclictest1140003-21sshd20:39:011
1135414994229,11cyclictest23-21ksoftirqd/122:41:331
1135414994226,12cyclictest1-21systemd20:35:011
113541499422,4cyclictest131rcu_preempt21:22:261
113541499421,4cyclictest131rcu_preempt00:35:011
113541499421,3cyclictest131rcu_preempt21:59:501
113541499421,3cyclictest131rcu_preempt19:24:021
1135409994235,4cyclictest16950irq/224-800000019:25:000
113540999421,3cyclictest131rcu_preempt20:15:020
1135418994133,5cyclictest1128271-21kworker/2:119:45:032
1135418994128,8cyclictest1-21systemd19:25:012
113541499414,3cyclictest131rcu_preempt19:40:001
113541499413,5cyclictest0-21swapper/100:10:591
1135414994130,7cyclictest1136612-21sshd19:30:031
1135414994129,9cyclictest23-21ksoftirqd/123:25:571
1135414994129,9cyclictest23-21ksoftirqd/123:07:241
1135414994128,9cyclictest3467602-21systemd-journal21:00:001
1135414994127,10cyclictest1214358-21cut23:45:031
1135414994125,11cyclictest3467602-21systemd-journal20:10:001
113541499412,4cyclictest131rcu_preempt22:20:021
113541499411,4cyclictest131rcu_preempt22:08:351
113541499411,3cyclictest150-21pr/ttyS221:38:501
1135409994125,11cyclictest275-21dbus-daemon20:30:010
1135418994032,6cyclictest1156043-21ssh21:40:232
1135418994021,13cyclictest275-21dbus-daemon19:15:002
113541499403,3cyclictest131rcu_preempt19:45:001
1135414994030,7cyclictest23-21ksoftirqd/100:06:151
1135414994025,12cyclictest23-21ksoftirqd/120:25:011
113541499402,3cyclictest131rcu_preempt21:53:191
113540999409,5cyclictest131rcu_preempt19:35:010
1135409994025,10cyclictest3467718-21lldpd19:26:230
1135409994016,16cyclictest1141047-21modprobe21:00:000
113540999401,35cyclictest1137966-21sendmail-msp20:00:010
1135418993932,5cyclictest1135173-21kworker/2:319:34:072
113541499394,6cyclictest131rcu_preempt21:11:261
113541499393,3cyclictest131rcu_preempt22:50:021
113541499393,3cyclictest131rcu_preempt22:50:011
1135414993928,8cyclictest23-21ksoftirqd/121:19:051
1135414993928,8cyclictest23-21ksoftirqd/100:17:391
1135414993927,9cyclictest23-21ksoftirqd/123:13:431
1135414993926,9cyclictest23-21ksoftirqd/123:17:281
113541499392,3cyclictest131rcu_preempt22:03:381
1135409993923,12cyclictest1141307-21sshd21:02:020
1135418993828,7cyclictest1140958-21sshd20:55:032
1135418993825,10cyclictest30-21ksoftirqd/220:55:542
1135418993821,12cyclictest1-21systemd21:05:012
113541499387,3cyclictest131rcu_preempt21:45:411
113541499385,4cyclictest131rcu_preempt21:44:481
113541499385,3cyclictest150-21pr/ttyS223:39:441
113541499383,3cyclictest131rcu_preempt21:10:001
1135414993827,9cyclictest23-21ksoftirqd/123:31:191
1135414993824,10cyclictest23-21ksoftirqd/122:13:071
1135414993823,12cyclictest1140369-21sed20:45:011
1135414993816,7cyclictest23-21ksoftirqd/122:32:501
113541499381,33cyclictest23-21ksoftirqd/121:27:501
1135414993810,6cyclictest131rcu_preempt19:14:541
1135409993828,6cyclictest12-21ksoftirqd/019:45:000
113541899374,4cyclictest131rcu_preempt20:34:592
113541899374,3cyclictest131rcu_preempt20:50:012
113541899371,3cyclictest131rcu_preempt20:05:012
1135414993725,9cyclictest23-21ksoftirqd/119:35:011
113541499372,3cyclictest131rcu_preempt20:05:001
113541899361,3cyclictest131rcu_preempt19:42:032
113541499363,3cyclictest131rcu_preempt20:50:001
1135414993528,4cyclictest1140808-21sh20:55:001
1135414993519,12cyclictest1137924-21cron19:59:591
113541499345,9cyclictest23-21ksoftirqd/119:55:001
113541499342,4cyclictest131rcu_preempt19:10:021
1135414993221,8cyclictest1137418-21sshd19:49:011
1135418992922,5cyclictest1135173-21kworker/2:319:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional