You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-11 - 10:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu Jun 11, 2026 00:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1012981999124,14cyclictest1076772-21ssh23:18:120
1012981998228,50cyclictest1113266-21ssh00:39:360
1012981997929,46cyclictest16950irq/224-800000021:05:010
1012981997929,46cyclictest16950irq/224-800000021:05:000
1012981997927,48cyclictest1066794-21ssh22:55:410
101298199792,6cyclictest1091894-21ssh23:51:340
1012981997924,11cyclictest1060999-21ssh22:43:010
1012981997916,58cyclictest16850irq/134-800000023:48:110
1012981997815,6cyclictest16850irq/134-800000000:11:010
1012981997810,7cyclictest16850irq/134-800000022:51:030
1012981997711,6cyclictest16850irq/134-800000021:39:030
1012981997625,15cyclictest1049929-21ssh22:18:250
1012981997614,5cyclictest16850irq/134-800000022:08:000
1012981997525,13cyclictest1051556-21ssh22:21:450
1012981997519,44cyclictest16850irq/134-800000022:01:070
1012981997519,44cyclictest16850irq/134-800000022:01:070
101298199751,5cyclictest1048330-21munin-run22:15:010
101298199751,5cyclictest1048330-21munin-run22:15:000
101298199748,7cyclictest16850irq/134-800000021:28:320
1012981997416,46cyclictest16850irq/134-800000023:06:430
1012981997415,6cyclictest16850irq/134-800000021:15:100
1012981997327,30cyclictest1081449-21ssh23:28:410
1012981997233,23cyclictest1094662-21ssh23:57:570
1012981997227,40cyclictest1054884-21ssh22:29:360
101298199722,65cyclictest3467602-21systemd-journal00:07:490
1012981997028,38cyclictest1034114-21ssh21:43:220
1012981996927,12cyclictest1105523-21ssh00:21:590
101298199691,64cyclictest1074474-21ssh23:12:580
1012981996910,6cyclictest16850irq/134-800000023:43:030
1012981996733,28cyclictest16850irq/134-800000022:40:010
101298199671,9cyclictest1018734-21latency_hist21:00:020
1012981996613,14cyclictest1029791-21ssh21:33:410
1012986996526,34cyclictest1036980-21sh21:49:551
1012986996522,10cyclictest275-21dbus-daemon23:59:591
1012981996528,33cyclictest16950irq/224-800000021:50:000
1012981996527,32cyclictest1103239-21cron00:17:000
1012981996522,12cyclictest1083916-21ssh23:34:110
1012981996516,44cyclictest16850irq/134-800000023:21:540
1012981996516,14cyclictest1111096-21ssh00:34:450
101298199651,60cyclictest0-21swapper/000:30:020
101298199643,57cyclictest48-21kcompactd021:55:200
101298199642,56cyclictest1039087-21ssh21:54:250
1012981996413,6cyclictest16850irq/134-800000000:03:570
101298199641,25cyclictest0-21swapper/020:07:010
101298199641,25cyclictest0-21swapper/020:07:000
101298199636,54cyclictest1016798-21sshd20:20:370
101298199636,54cyclictest1016798-21sshd20:20:370
1012981996332,26cyclictest1019147-21diskmemload23:40:020
1012981996332,26cyclictest1019147-21diskmemload23:40:020
1012981996326,16cyclictest1024714-21ssh21:22:130
101298199632,4cyclictest1069823-21ssh23:02:340
101298199632,4cyclictest1069823-21ssh23:02:330
101298199631,4cyclictest0-21swapper/022:33:490
1012986996246,11cyclictest3467602-21systemd-journal21:20:001
1012986996225,33cyclictest1105092-21sh00:20:531
1012986996222,5cyclictest1019147-21diskmemload21:52:061
1012986996220,7cyclictest1046883-1kworker/u9:022:13:131
1012986996220,7cyclictest1046883-1kworker/u9:022:13:131
101298699621,56cyclictest23-21ksoftirqd/123:36:191
101298699621,56cyclictest23-21ksoftirqd/123:36:191
101298199621,27cyclictest0-21swapper/022:47:480
1012990996135,19cyclictest1-21systemd23:20:002
1012986996128,30cyclictest1082171-21latency_hist23:30:031
1012986996124,5cyclictest1099535-21sh00:08:551
1012986996124,32cyclictest1109107-21latency_hist00:30:021
1012986996124,13cyclictest3467602-21systemd-journal00:05:011
1012986996123,4cyclictest1019147-21diskmemload23:52:411
1012986996118,6cyclictest1091076-21latency_hist23:50:011
1012986996023,4cyclictest1019147-21diskmemload23:22:531
1012986996020,4cyclictest1042627-21sh22:02:031
1012986996020,4cyclictest1042627-21sh22:02:031
1012986996016,5cyclictest0-21swapper/100:40:031
1012981996013,6cyclictest16850irq/134-800000021:12:030
1012981996013,6cyclictest16850irq/134-800000021:12:030
101298699594,34cyclictest221rcuc/122:45:031
1012986995924,30cyclictest1068510-21cron23:00:001
1012990995849,5cyclictest1016853-21kworker/2:221:54:532
1012990995827,5cyclictest1016853-21kworker/2:221:21:592
1012986995819,10cyclictest3467602-21systemd-journal20:45:011
1012986995816,8cyclictest1046883-1kworker/u9:023:03:411
1012986995816,8cyclictest1046883-1kworker/u9:023:03:411
1012990995747,7cyclictest1079786-21kworker/2:023:55:522
1012990995747,6cyclictest1028081-21kworker/2:323:21:552
1012990995739,14cyclictest1-21systemd22:05:012
1012990995739,14cyclictest1-21systemd22:05:012
101298699572,6cyclictest131rcu_preempt21:35:031
1012986995721,7cyclictest23-21ksoftirqd/122:41:071
1012981995725,2cyclictest0-21swapper/020:40:020
101298199571,27cyclictest0-21swapper/020:28:360
1012990995648,5cyclictest1079786-21kworker/2:000:36:532
1012990995648,5cyclictest1079786-21kworker/2:000:20:022
1012986995622,10cyclictest1014737-21cron19:45:011
1012986995622,10cyclictest1014737-21cron19:45:011
1012986995611,14cyclictest3467602-21systemd-journal22:30:001
1012981995622,5cyclictest1017303-21sshd20:31:100
101299099556,4cyclictest131rcu_preempt19:10:022
1012990995546,5cyclictest1028081-21kworker/2:323:05:002
1012990995546,5cyclictest1028081-21kworker/2:323:05:002
1012986995523,5cyclictest1076168-21ssh23:16:421
1012986995522,5cyclictest1052720-21ssh22:24:431
1012986995521,9cyclictest23-21ksoftirqd/123:07:141
1012986995516,9cyclictest23-21ksoftirqd/123:42:291
1012981995525,25cyclictest16950irq/224-800000021:10:000
1012990995445,6cyclictest1028081-21kworker/2:322:55:002
1012990995444,6cyclictest1006172-21kworker/2:319:15:012
1012990995444,6cyclictest1006172-21kworker/2:319:15:002
101298699544,16cyclictest1050581-21modprobe22:20:001
1012986995436,13cyclictest3467602-21systemd-journal20:50:011
1012986995424,6cyclictest23-21ksoftirqd/122:37:401
1012986995423,6cyclictest23-21ksoftirqd/121:06:341
1012986995420,9cyclictest23-21ksoftirqd/121:24:531
101298199541,26cyclictest0-21swapper/019:40:010
1012990995347,3cyclictest1016853-21kworker/2:222:11:362
1012990995347,3cyclictest1016853-21kworker/2:222:11:362
1012990995345,5cyclictest1028081-21kworker/2:323:10:012
1012990995345,5cyclictest1016853-21kworker/2:221:38:572
1012990995345,5cyclictest1016853-21kworker/2:221:14:542
1012990995345,5cyclictest1016853-21kworker/2:221:14:532
1012990995345,5cyclictest1006172-21kworker/2:320:40:022
1012990995344,5cyclictest1079786-21kworker/2:000:25:002
1012990995343,6cyclictest1079786-21kworker/2:000:27:212
1012990995327,5cyclictest1079786-21kworker/2:023:45:112
1012986995318,6cyclictest23-21ksoftirqd/121:12:361
1012986995318,6cyclictest23-21ksoftirqd/121:12:361
101298199531,23cyclictest3587003-1kworker/0:0H19:10:020
1012990995241,6cyclictest1028081-21kworker/2:322:39:262
1012990995236,11cyclictest3467602-21systemd-journal20:50:012
1012986995238,10cyclictest1-21systemd20:05:011
1012986995234,12cyclictest275-21dbus-daemon19:35:011
1012986995222,4cyclictest1034800-21latency_hist21:45:001
1012994995135,12cyclictest275-21dbus-daemon22:35:013
1012990995144,5cyclictest1028081-21kworker/2:322:49:132
1012990995143,5cyclictest1016853-21kworker/2:221:33:522
1012990995142,6cyclictest1079786-21kworker/2:023:55:012
1012990995142,6cyclictest1016853-21kworker/2:221:48:072
1012990995142,5cyclictest1079786-21kworker/2:000:03:582
1012990995141,6cyclictest1079786-21kworker/2:000:06:312
1012990995140,6cyclictest1006172-21kworker/2:320:30:012
1012990995133,13cyclictest3467602-21systemd-journal21:05:012
1012990995133,13cyclictest3467602-21systemd-journal21:05:002
101298699513,13cyclictest0-21swapper/100:20:001
1012990995042,5cyclictest1028081-21kworker/2:322:20:102
1012990995042,5cyclictest1016853-21kworker/2:221:41:332
1012990995041,6cyclictest1079786-21kworker/2:023:27:132
1012990995038,10cyclictest30-21ksoftirqd/219:35:022
1012986995021,6cyclictest23-21ksoftirqd/122:30:321
1012986995018,8cyclictest23-21ksoftirqd/121:55:231
101298199502,5cyclictest1-21systemd20:01:360
101299499494,4cyclictest131rcu_preempt21:55:543
1012994994930,15cyclictest275-21dbus-daemon00:00:003
1012990994941,5cyclictest1016853-21kworker/2:221:15:322
1012990994940,5cyclictest1028081-21kworker/2:322:42:402
1012990994939,6cyclictest1079786-21kworker/2:023:41:082
1012990994939,6cyclictest1006172-21kworker/2:319:20:372
1012990994934,11cyclictest3467602-21systemd-journal23:35:022
1012990994934,11cyclictest3467602-21systemd-journal23:35:022
1012986994923,4cyclictest1075016-21sh23:14:211
1012986994921,6cyclictest23-21ksoftirqd/121:36:001
1012981994934,10cyclictest3467602-21systemd-journal19:20:010
101298199491,28cyclictest0-21swapper/020:55:000
101299499484,3cyclictest131rcu_preempt23:05:023
101299499484,3cyclictest131rcu_preempt23:05:023
1012990994840,5cyclictest1079786-21kworker/2:000:30:342
1012990994840,5cyclictest1028081-21kworker/2:322:34:472
1012990994840,5cyclictest1016853-21kworker/2:222:10:002
1012990994839,5cyclictest1016853-21kworker/2:221:59:372
101298199481,30cyclictest0-21swapper/020:41:010
1012994994733,10cyclictest3467602-21systemd-journal23:14:383
1012990994739,5cyclictest1079786-21kworker/2:023:32:312
1012990994739,5cyclictest1028081-21kworker/2:323:11:082
1012990994739,5cyclictest1028081-21kworker/2:322:16:372
1012990994738,5cyclictest1028081-21kworker/2:322:57:542
1012986994719,5cyclictest1019147-21diskmemload22:52:501
101298199471,31cyclictest0-21swapper/020:47:230
101298199471,30cyclictest0-21swapper/020:15:270
1012994994629,13cyclictest1-21systemd23:40:013
1012994994629,13cyclictest1-21systemd23:40:013
1012994994625,14cyclictest275-21dbus-daemon00:20:013
101299099469,28cyclictest1015498-21modprobe20:00:012
1012990994638,5cyclictest1028081-21kworker/2:322:27:262
1012990994637,6cyclictest1016853-21kworker/2:221:29:092
1012986994616,4cyclictest0-21swapper/123:35:021
1012986994613,11cyclictest1-21systemd20:40:011
1012994994525,15cyclictest1-21systemd19:30:013
1012994994525,15cyclictest1-21systemd19:30:013
101299499452,39cyclictest37-21ksoftirqd/323:51:403
101299499451,3cyclictest131rcu_preempt00:14:483
1012990994537,5cyclictest1079786-21kworker/2:000:13:362
1012990994532,7cyclictest1016853-21kworker/2:220:55:002
1012986994520,5cyclictest1019147-21diskmemload00:10:411
1012994994432,7cyclictest1030308-21latency_hist21:35:013
1012994994430,11cyclictest37-21ksoftirqd/300:32:223
1012994994429,11cyclictest3467602-21systemd-journal22:55:003
1012994994427,13cyclictest275-21dbus-daemon22:05:013
1012994994427,13cyclictest275-21dbus-daemon22:05:003
101299499441,7cyclictest48-21kcompactd022:26:323
101299099444,3cyclictest131rcu_preempt21:10:012
101298699443,18cyclictest1046067-21ssh22:10:011
101298199441,26cyclictest0-21swapper/019:11:310
101298199441,26cyclictest0-21swapper/019:11:300
101299499433,4cyclictest131rcu_preempt21:50:003
1012994994332,8cyclictest1017064-21sshd20:29:353
1012994994325,13cyclictest275-21dbus-daemon21:45:003
1012990994332,7cyclictest1-21systemd19:40:012
1012990994328,7cyclictest309-21in:imuxsock20:20:002
1012990994325,12cyclictest3467602-21systemd-journal19:20:002
101298699436,9cyclictest23-21ksoftirqd/120:35:001
1012986994329,11cyclictest23-21ksoftirqd/100:33:251
101298699431,7cyclictest23-21ksoftirqd/121:30:011
1012981994325,3cyclictest0-21swapper/019:23:200
101299499423,3cyclictest131rcu_preempt00:02:103
1012994994229,9cyclictest1-21systemd21:25:013
1012994994229,10cyclictest1021488-21cut21:15:023
1012994994229,10cyclictest1021488-21cut21:15:013
101299099424,3cyclictest131rcu_preempt21:00:012
1012990994233,6cyclictest1006172-21kworker/2:320:10:022
1012990994233,6cyclictest1006172-21kworker/2:320:10:012
1012986994231,8cyclictest23-21ksoftirqd/119:20:001
1012981994236,3cyclictest16950irq/224-800000019:53:350
1012981994224,13cyclictest1-21systemd20:15:010
101298199421,6cyclictest0-21swapper/020:00:030
101299499413,3cyclictest131rcu_preempt22:13:463
101299499413,3cyclictest131rcu_preempt22:13:463
101299499412,4cyclictest131rcu_preempt22:15:033
1012994994124,13cyclictest1077503-21systemctl23:20:003
101299499412,3cyclictest131rcu_preempt23:46:283
101299499411,3cyclictest131rcu_preempt22:56:283
101299099413,29cyclictest1016828-21munin-plugin-st20:25:012
101299099413,29cyclictest1016828-21munin-plugin-st20:25:002
1012990994128,10cyclictest1015956-21sed20:05:022
1012990994126,11cyclictest0-21swapper/220:45:012
101298199411,30cyclictest0-21swapper/019:50:000
101299499404,28cyclictest1081995-21modprobe23:30:013
101299499403,4cyclictest131rcu_preempt00:40:003
1012994994031,7cyclictest37-21ksoftirqd/323:08:443
1012994994029,8cyclictest1058154-21sshd22:36:333
1012994994028,9cyclictest37-21ksoftirqd/322:40:213
101299499402,3cyclictest131rcu_preempt21:16:423
1012994994023,11cyclictest1106902-21cut00:25:023
1012994994022,11cyclictest309-21in:imuxsock20:10:033
1012994994021,14cyclictest275-21dbus-daemon21:00:013
101298699401,35cyclictest1013692-21sshd19:22:371
101299499399,6cyclictest131rcu_preempt21:05:263
1012994993928,8cyclictest37-21ksoftirqd/321:39:593
1012994993928,8cyclictest37-21ksoftirqd/321:29:133
1012994993926,9cyclictest1098698-21sshd00:06:503
1012994993926,10cyclictest37-21ksoftirqd/321:05:013
1012994993926,10cyclictest37-21ksoftirqd/321:05:013
1012994993924,10cyclictest3467602-21systemd-journal19:45:013
1012994993924,10cyclictest3467602-21systemd-journal19:45:003
101299499392,22cyclictest1015500-21modprobe20:00:003
1012994993922,13cyclictest275-21dbus-daemon20:20:003
101299499391,4cyclictest131rcu_preempt22:25:023
101298699394,5cyclictest131rcu_preempt20:21:091
101298699394,5cyclictest131rcu_preempt20:21:081
101298699391,3cyclictest131rcu_preempt20:59:351
1012994993824,10cyclictest1016022-21sshd20:09:373
1012994993824,10cyclictest1016022-21sshd20:09:373
101299499382,3cyclictest131rcu_preempt20:45:003
1012994993823,11cyclictest37-21ksoftirqd/322:10:003
1012990993831,4cyclictest1006172-21kworker/2:320:12:362
1012986993827,8cyclictest23-21ksoftirqd/119:40:011
1012986993824,10cyclictest3467602-21systemd-journal19:26:361
1012986993824,10cyclictest3467602-21systemd-journal19:26:351
101298699382,3cyclictest131rcu_preempt20:15:011
1012986993823,11cyclictest23-21ksoftirqd/121:05:011
1012986993823,11cyclictest23-21ksoftirqd/121:05:011
101299499372,3cyclictest131rcu_preempt23:40:483
101299499371,4cyclictest131rcu_preempt23:21:533
101299499371,4cyclictest131rcu_preempt20:40:003
101299499371,3cyclictest131rcu_preempt00:29:593
1012990993730,4cyclictest1006172-21kworker/2:320:31:262
1012986993727,7cyclictest23-21ksoftirqd/119:55:371
101298699371,28cyclictest0-21swapper/120:27:361
1012981993728,6cyclictest19350irq/34-mmc119:34:310
1012994993623,10cyclictest37-21ksoftirqd/319:15:023
1012994993623,10cyclictest37-21ksoftirqd/319:15:013
1012994993620,11cyclictest3467602-21systemd-journal19:45:033
101299499361,5cyclictest0-21swapper/322:47:373
101299099365,25cyclictest1014996-21modprobe19:50:002
1012986993624,9cyclictest23-21ksoftirqd/120:08:371
1012986993624,9cyclictest23-21ksoftirqd/120:08:361
1012986993624,9cyclictest23-21ksoftirqd/119:14:081
1012986993624,9cyclictest23-21ksoftirqd/119:14:081
101298199361,5cyclictest131rcu_preempt19:30:010
101298199361,5cyclictest131rcu_preempt19:30:010
101299499355,4cyclictest131rcu_preempt21:50:273
1012994993517,12cyclictest275-21dbus-daemon20:05:013
101299499351,3cyclictest131rcu_preempt23:30:133
101299499351,3cyclictest131rcu_preempt19:10:023
1012990993527,5cyclictest1006172-21kworker/2:319:40:152
1012990993527,5cyclictest1006172-21kworker/2:319:40:142
1012994993421,9cyclictest3467602-21systemd-journal20:45:033
101299499341,8cyclictest364-21snmpd20:51:333
101299499341,3cyclictest131rcu_preempt20:25:013
101299499341,3cyclictest131rcu_preempt20:25:013
101299499341,3cyclictest131rcu_preempt19:35:013
101299499341,3cyclictest131rcu_preempt19:20:013
101299099345,3cyclictest131rcu_preempt19:30:022
101299099345,3cyclictest131rcu_preempt19:30:012
101299099344,22cyclictest1015248-21modprobe19:54:592
101298699343,3cyclictest131rcu_preempt20:55:011
1012986993424,8cyclictest23-21ksoftirqd/119:48:351
1012986993423,8cyclictest1013054-21sed19:10:021
101298199342,3cyclictest131rcu_preempt19:45:020
101298199342,3cyclictest131rcu_preempt19:45:020
101299499334,4cyclictest131rcu_preempt19:55:023
1012994993319,10cyclictest275-21dbus-daemon19:40:013
1012994993318,12cyclictest1013698-21sshd19:24:363
1012986993326,4cyclictest1016585-21sh20:20:011
1012994993118,10cyclictest1-21systemd20:35:013
101298699311,4cyclictest131rcu_preempt19:55:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional