You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-18 - 08:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Jun 18, 2026 00:45:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3227772997525,11cyclictest275-21dbus-daemon22:40:013
3227758997229,14cyclictest3296603-21ssh23:29:150
3227772996926,9cyclictest1-21systemd22:35:003
3227772996925,14cyclictest3292378-21cron23:20:003
3227772996630,32cyclictest3230316-21sendmail-msp19:59:593
3227758996518,7cyclictest16850irq/134-800000000:20:590
3227772996426,9cyclictest37-21ksoftirqd/320:35:023
3227772996426,9cyclictest37-21ksoftirqd/320:35:013
3227772996325,11cyclictest37-21ksoftirqd/323:21:423
3227772996324,9cyclictest37-21ksoftirqd/322:15:013
3227758996227,3cyclictest131rcu_preempt00:28:440
322775899621,3cyclictest131rcu_preempt23:45:020
3227772996125,9cyclictest37-21ksoftirqd/323:34:203
3227772996021,11cyclictest37-21ksoftirqd/300:10:483
3227768996038,17cyclictest3467602-21systemd-journal19:59:592
3227768996024,25cyclictest275-21dbus-daemon20:20:002
322775899601,43cyclictest3259741-21ssh22:07:110
3227772995823,10cyclictest37-21ksoftirqd/322:47:393
322776899587,36cyclictest3319428-21modprobe00:20:012
322775899581,38cyclictest131rcu_preempt22:03:110
3227758995813,40cyclictest275-21dbus-daemon21:55:010
3227772995739,14cyclictest3467602-21systemd-journal19:25:013
3227772995739,14cyclictest3467602-21systemd-journal19:25:003
3227772995726,4cyclictest3275309-21sh22:41:133
3227758995729,13cyclictest3240654-21ssh21:25:000
3227758995719,6cyclictest16850irq/134-800000021:10:010
3227758995717,14cyclictest0-21swapper/023:58:080
3227758995712,40cyclictest16850irq/134-800000021:34:590
3227772995624,9cyclictest37-21ksoftirqd/322:05:393
3227772995624,11cyclictest37-21ksoftirqd/321:23:253
3227768995647,6cyclictest3276844-21kworker/2:223:00:002
3227763995637,14cyclictest1-21systemd21:45:011
322775899567,30cyclictest16950irq/224-800000000:20:020
322775899563,36cyclictest131rcu_preempt21:57:340
322775899563,36cyclictest131rcu_preempt21:57:330
3227758995614,21cyclictest16850irq/134-800000000:04:040
3227772995524,10cyclictest37-21ksoftirqd/322:20:033
3227772995524,10cyclictest37-21ksoftirqd/322:20:023
3227768995545,6cyclictest3292440-21kworker/2:100:08:422
3227768995542,8cyclictest3231389-21kworker/2:221:30:002
3227768995542,8cyclictest3231389-21kworker/2:221:29:592
3227768995534,15cyclictest30-21ksoftirqd/220:35:022
3227768995534,15cyclictest30-21ksoftirqd/220:35:012
3227758995533,4cyclictest16850irq/134-800000022:20:010
3227758995533,4cyclictest16850irq/134-800000022:20:000
3227758995526,13cyclictest3308097-21ssh23:54:450
3227758995526,13cyclictest3308097-21ssh23:54:440
3227758995515,36cyclictest16850irq/134-800000022:51:480
3227772995434,15cyclictest275-21dbus-daemon19:30:003
3227768995446,5cyclictest3231389-21kworker/2:221:34:382
3227768995445,6cyclictest3230847-21kworker/2:022:42:072
3227768995445,5cyclictest3292440-21kworker/2:100:35:102
3227768995445,5cyclictest3230847-21kworker/2:021:41:092
3227768995444,6cyclictest3292440-21kworker/2:100:32:322
3227768995444,6cyclictest3267821-21kworker/2:322:53:312
3227768995444,6cyclictest3230847-21kworker/2:022:36:522
3227768995443,6cyclictest3267821-21kworker/2:322:45:192
3227758995436,14cyclictest3467602-21systemd-journal20:49:590
3227758995426,12cyclictest3304563-21ssh23:46:330
3227758995418,18cyclictest16850irq/134-800000021:11:220
3227758995413,21cyclictest16850irq/134-800000022:44:220
3227768995345,5cyclictest3231389-21kworker/2:221:14:362
3227768995344,6cyclictest3267821-21kworker/2:300:26:302
3227768995344,6cyclictest3230847-21kworker/2:021:52:252
3227768995344,5cyclictest3230847-21kworker/2:022:03:482
3227768995344,5cyclictest3230847-21kworker/2:021:46:042
3227768995344,5cyclictest3230847-21kworker/2:021:46:032
3227768995343,6cyclictest3230847-21kworker/2:022:23:562
3227768995343,6cyclictest3230847-21kworker/2:021:55:252
3227768995343,6cyclictest3230847-21kworker/2:021:55:242
3227768995343,6cyclictest3219669-21kworker/2:319:14:592
3227768995343,5cyclictest3292440-21kworker/2:123:51:152
3227768995343,5cyclictest3292440-21kworker/2:123:51:142
3227768995343,5cyclictest3230847-21kworker/2:021:38:412
3227768995341,7cyclictest3267821-21kworker/2:323:46:212
322775899534,39cyclictest3227761-21grep19:10:020
322775899534,39cyclictest3227761-21grep19:10:010
3227758995331,17cyclictest16850irq/134-800000000:06:080
3227758995316,33cyclictest16850irq/134-800000021:47:580
3227758995316,33cyclictest16850irq/134-800000021:47:570
3227758995314,35cyclictest16850irq/134-800000022:26:000
3227758995314,19cyclictest16850irq/134-800000022:22:400
3227758995313,21cyclictest16850irq/134-800000023:20:160
3227758995313,20cyclictest16850irq/134-800000023:34:380
3227758995313,20cyclictest16850irq/134-800000022:48:520
3227768995244,5cyclictest3267821-21kworker/2:323:31:272
3227768995244,5cyclictest3230847-21kworker/2:022:18:252
3227768995244,5cyclictest3230847-21kworker/2:022:18:242
3227768995244,5cyclictest3219669-21kworker/2:319:20:032
3227768995243,6cyclictest3267821-21kworker/2:323:20:252
3227768995243,6cyclictest3230847-21kworker/2:022:12:572
3227768995243,5cyclictest3267821-21kworker/2:323:18:552
3227768995243,5cyclictest3267821-21kworker/2:300:10:132
3227768995242,6cyclictest3267821-21kworker/2:323:44:482
3227768995242,6cyclictest3267821-21kworker/2:323:28:182
3227768995241,7cyclictest3267821-21kworker/2:323:07:292
3227768995220,25cyclictest3467602-21systemd-journal22:35:012
3227758995224,13cyclictest3273183-21ssh22:36:400
3227758995218,6cyclictest16850irq/134-800000021:41:320
3227758995217,31cyclictest16850irq/134-800000021:29:270
3227758995217,31cyclictest16850irq/134-800000021:29:260
3227758995217,18cyclictest16850irq/134-800000023:01:110
3227758995216,31cyclictest16850irq/134-800000021:16:240
3227758995215,18cyclictest16850irq/134-800000023:12:260
3227758995215,18cyclictest16850irq/134-800000023:12:250
3227758995214,33cyclictest16850irq/134-800000000:38:430
3227758995213,34cyclictest16850irq/134-800000023:09:000
3227768995143,5cyclictest3292440-21kworker/2:100:01:152
3227768995143,5cyclictest3267821-21kworker/2:300:22:182
3227768995142,6cyclictest3276844-21kworker/2:223:00:072
3227768995142,6cyclictest3256947-21kworker/2:122:06:232
3227768995142,5cyclictest3267821-21kworker/2:323:37:452
3227768995142,5cyclictest3219669-21kworker/2:319:55:012
3227768995141,7cyclictest3292440-21kworker/2:123:56:282
3227763995134,12cyclictest275-21dbus-daemon00:30:011
3227758995117,6cyclictest16850irq/134-800000022:13:220
3227758995114,33cyclictest16850irq/134-800000022:57:020
3227768995042,5cyclictest3231389-21kworker/2:221:20:122
3227768995042,5cyclictest3230847-21kworker/2:022:26:182
3227768995042,4cyclictest3219669-21kworker/2:319:22:182
3227768995042,4cyclictest3219669-21kworker/2:319:22:182
3227768995041,5cyclictest3267821-21kworker/2:323:10:272
3227768995041,5cyclictest3267821-21kworker/2:323:10:262
3227768995041,5cyclictest3231389-21kworker/2:221:16:302
3227768995031,14cyclictest3467602-21systemd-journal20:15:002
3227768995030,15cyclictest3467602-21systemd-journal20:36:172
322776399503,5cyclictest32564841ssh22:00:031
322776399503,5cyclictest32564841ssh22:00:021
322775899503,42cyclictest3232380-21munin-run20:40:000
3227758995019,4cyclictest131rcu_preempt21:36:280
3227758995012,19cyclictest16850irq/134-800000023:17:470
3227758995012,19cyclictest16850irq/134-800000022:30:270
3227768994940,6cyclictest3231389-21kworker/2:220:53:202
3227768994937,9cyclictest3467602-21systemd-journal19:50:022
3227768994935,11cyclictest3467602-21systemd-journal20:09:172
3227758994937,9cyclictest3231344-21systemctl20:20:010
3227758994913,32cyclictest16850irq/134-800000023:36:250
3227772994828,15cyclictest275-21dbus-daemon20:30:003
3227768994840,5cyclictest3231389-21kworker/2:221:00:062
3227763994822,5cyclictest23-21ksoftirqd/123:20:001
3227758994827,7cyclictest3325298-21ssh00:32:580
322776899475,26cyclictest3233406-21modprobe21:00:002
3227763994731,12cyclictest275-21dbus-daemon22:15:001
3227758994731,13cyclictest275-21dbus-daemon20:10:010
3227758994710,32cyclictest16850irq/134-800000020:25:460
3227772994630,12cyclictest275-21dbus-daemon00:25:013
322777299461,3cyclictest131rcu_preempt23:45:013
3227768994638,5cyclictest3230847-21kworker/2:020:44:172
3227768994638,5cyclictest3219669-21kworker/2:319:38:282
3227768994638,5cyclictest3219669-21kworker/2:319:38:282
3227768994638,5cyclictest3219669-21kworker/2:319:30:372
322776399463,4cyclictest131rcu_preempt00:17:551
322776399463,3cyclictest131rcu_preempt22:28:501
3227763994611,5cyclictest131rcu_preempt22:43:431
322775899468,19cyclictest16850irq/134-800000000:10:140
322777299456,6cyclictest131rcu_preempt23:10:033
322777299456,6cyclictest131rcu_preempt23:10:033
3227772994528,13cyclictest3467602-21systemd-journal21:05:183
3227772994528,12cyclictest275-21dbus-daemon00:00:003
3227768994538,4cyclictest3231389-21kworker/2:221:07:142
3227768994537,5cyclictest3231389-21kworker/2:220:45:102
3227768994537,5cyclictest3219669-21kworker/2:320:01:362
322776399456,5cyclictest131rcu_preempt21:25:421
322776399456,5cyclictest131rcu_preempt21:25:411
322776399452,3cyclictest131rcu_preempt21:35:391
3227763994512,6cyclictest131rcu_preempt23:29:231
322777299444,3cyclictest131rcu_preempt20:55:023
3227768994437,4cyclictest3219669-21kworker/2:319:28:112
3227768994436,5cyclictest3230847-21kworker/2:020:22:212
322776399445,7cyclictest3283027-21sh22:58:551
3227763994437,4cyclictest0-21swapper/119:10:011
3227763994437,4cyclictest0-21swapper/119:10:011
322776399443,37cyclictest3231863-21sh20:29:591
3227763994432,9cyclictest3467602-21systemd-journal22:25:001
3227758994428,11cyclictest275-21dbus-daemon19:30:010
3227772994332,7cyclictest309-21in:imuxsock20:20:013
3227772994330,9cyclictest37-21ksoftirqd/320:15:003
3227772994329,11cyclictest275-21dbus-daemon22:25:493
3227772994329,10cyclictest3467602-21systemd-journal21:35:003
3227772994328,12cyclictest37-21ksoftirqd/322:01:433
322777299432,16cyclictest3283432-21ssh22:59:593
3227768994336,4cyclictest3219669-21kworker/2:319:42:132
3227768994335,5cyclictest3230847-21kworker/2:020:28:442
322776399435,4cyclictest364-21snmpd22:51:471
322776399433,3cyclictest131rcu_preempt22:34:441
3227763994330,10cyclictest23-21ksoftirqd/100:20:031
3227763994327,13cyclictest3293566-21sshd23:22:191
322776399432,4cyclictest131rcu_preempt21:52:121
322776399432,3cyclictest131rcu_preempt22:09:531
322776399432,38cyclictest23-21ksoftirqd/100:13:361
3227763994313,5cyclictest131rcu_preempt23:06:191
3227758994336,4cyclictest16950irq/224-800000019:15:000
3227758994324,15cyclictest3228201-21cron19:20:000
3227772994221,15cyclictest275-21dbus-daemon20:05:003
3227772994210,5cyclictest131rcu_preempt21:13:043
322776399428,9cyclictest0-21swapper/122:02:281
322776399425,32cyclictest3230339-21grep20:00:021
322776399424,9cyclictest3236651-21ssh21:15:351
322776399423,3cyclictest131rcu_preempt23:12:271
322776399423,3cyclictest131rcu_preempt23:12:261
3227763994228,11cyclictest23-21ksoftirqd/121:35:001
322776399422,3cyclictest131rcu_preempt23:49:211
3227763994221,5cyclictest23-21ksoftirqd/123:59:141
3227758994228,11cyclictest12-21ksoftirqd/019:31:170
3227758994227,11cyclictest275-21dbus-daemon19:45:000
322775899421,4cyclictest131rcu_preempt21:05:000
322777299415,3cyclictest131rcu_preempt20:10:033
3227772994132,7cyclictest37-21ksoftirqd/321:38:073
3227772994126,11cyclictest37-21ksoftirqd/321:45:193
3227772994126,11cyclictest37-21ksoftirqd/321:45:183
3227772994126,10cyclictest275-21dbus-daemon00:05:013
3227772994124,12cyclictest309-21in:imuxsock23:01:183
3227772994111,6cyclictest131rcu_preempt23:40:013
3227772994111,6cyclictest131rcu_preempt00:09:303
322776399414,3cyclictest131rcu_preempt20:15:011
3227763994129,9cyclictest3230534-21sshd20:00:161
3227763994128,9cyclictest3467602-21systemd-journal19:20:001
3227763994127,10cyclictest275-21dbus-daemon22:50:001
3227763994126,11cyclictest23-21ksoftirqd/121:14:361
3227763994126,10cyclictest311-21rs:main1
3227763994126,10cyclictest311-21rs:main1
3227763994121,4cyclictest23-21ksoftirqd/121:10:011
322776399411,8cyclictest3300850-21ssh23:38:311
322776399411,6cyclictest3273761-21ssh22:38:051
322776399411,3cyclictest131rcu_preempt00:08:311
322775899415,5cyclictest131rcu_preempt20:05:010
322775899413,4cyclictest131rcu_preempt20:45:000
3227758994121,15cyclictest275-21dbus-daemon19:25:010
3227758994121,15cyclictest275-21dbus-daemon19:25:000
322777299405,6cyclictest131rcu_preempt19:35:013
3227772994010,9cyclictest131rcu_preempt21:28:153
3227772994010,9cyclictest131rcu_preempt21:28:153
3227763994029,8cyclictest275-21dbus-daemon00:35:021
3227763994028,9cyclictest3467602-21systemd-journal20:45:011
3227763994025,11cyclictest23-21ksoftirqd/120:15:401
322776399401,8cyclictest23-21ksoftirqd/123:00:361
3227763994018,17cyclictest1-21systemd21:00:011
322776399401,3cyclictest131rcu_preempt20:25:031
322776399401,3cyclictest131rcu_preempt00:04:541
3227758994032,5cyclictest3230047-21systemd-run19:55:010
322775899401,35cyclictest3467602-21systemd-journal20:10:030
322777299392,9cyclictest3317586-21ssh00:15:253
322777299392,5cyclictest3256378-21munin-run22:00:003
322777299392,5cyclictest3256378-21munin-run22:00:003
322776399396,3cyclictest131rcu_preempt22:16:201
322776399396,3cyclictest131rcu_preempt22:16:201
322776399392,5cyclictest3239118-21ssh21:20:581
3227763993925,10cyclictest1-21systemd20:40:001
322776399391,7cyclictest3302563-21ssh23:42:111
322775899393,4cyclictest131rcu_preempt19:50:000
3227758993928,8cyclictest12-21ksoftirqd/019:40:020
3227758993928,8cyclictest12-21ksoftirqd/019:40:010
3227758993927,9cyclictest3233374-21sshd20:55:170
322777299387,3cyclictest131rcu_preempt22:55:033
322777299383,8cyclictest37-21ksoftirqd/323:45:243
3227772993827,9cyclictest37-21ksoftirqd/320:45:033
3227772993825,10cyclictest37-21ksoftirqd/300:35:023
322777299381,5cyclictest131rcu_preempt19:40:033
322777299381,5cyclictest131rcu_preempt19:40:023
3227772993812,11cyclictest0-21swapper/323:05:283
322776399382,7cyclictest3306937-21ssh23:52:031
322776399382,7cyclictest3306937-21ssh23:52:021
322776399382,7cyclictest0-21swapper/120:55:011
3227763993827,8cyclictest3230007-21sshd19:51:171
3227763993826,8cyclictest23-21ksoftirqd/120:35:031
3227763993826,8cyclictest23-21ksoftirqd/120:35:031
3227763993824,9cyclictest23-21ksoftirqd/119:40:021
3227763993824,9cyclictest23-21ksoftirqd/119:40:021
322776399382,3cyclictest131rcu_preempt23:35:001
3227763993810,23cyclictest3328453-21unin-run00:40:011
322775899382,3cyclictest131rcu_preempt19:59:590
3227772993727,8cyclictest37-21ksoftirqd/300:37:053
3227772993724,10cyclictest37-21ksoftirqd/321:55:023
3227772993722,12cyclictest37-21ksoftirqd/321:17:203
322777299372,16cyclictest3248930-21ssh21:43:133
3227758993726,8cyclictest12-21ksoftirqd/020:55:020
322775899371,3cyclictest131rcu_preempt20:35:020
322775899371,3cyclictest131rcu_preempt20:35:010
322777299362,4cyclictest131rcu_preempt20:50:013
3227772993622,9cyclictest3467602-21systemd-journal19:42:333
3227772993622,12cyclictest37-21ksoftirqd/322:22:233
322777299361,4cyclictest131rcu_preempt21:00:013
3227763993623,9cyclictest1-21systemd19:50:001
3227763993621,11cyclictest1-21systemd19:45:011
322777299352,6cyclictest131rcu_preempt19:10:023
322777299352,6cyclictest131rcu_preempt19:10:023
3227772993526,7cyclictest37-21ksoftirqd/323:27:573
3227772993522,10cyclictest37-21ksoftirqd/300:28:533
3227772993515,15cyclictest275-21dbus-daemon20:25:013
3227763993524,8cyclictest3232874-21sshd20:46:161
3227763993523,9cyclictest23-21ksoftirqd/119:15:021
3227772993422,9cyclictest3467602-21systemd-journal20:40:003
322777299341,5cyclictest3307022-21ssh23:52:153
322777299341,5cyclictest3307022-21ssh23:52:143
3227763993421,9cyclictest1-21systemd19:25:011
3227763993421,9cyclictest1-21systemd19:25:001
322775899341,31cyclictest0-21swapper/020:24:570
3227772993325,5cyclictest309-21in:imuxsock19:17:013
3227772993318,12cyclictest1-21systemd19:55:023
3227763993326,4cyclictest3228717-21sshd19:26:161
322777299322,12cyclictest3233701-21munin-run21:05:003
3227772993118,10cyclictest37-21ksoftirqd/319:10:493
3227763993123,5cyclictest3233801-21latency_hist21:05:011
3227763993122,6cyclictest23-21ksoftirqd/120:10:021
3227763993021,7cyclictest23-21ksoftirqd/119:35:021
322777299292,3cyclictest131rcu_preempt19:50:023
3227768992920,5cyclictest30-21ksoftirqd/219:10:022
3227768992920,5cyclictest30-21ksoftirqd/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional