You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-15 - 15:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Wed Apr 15, 2026 00:45:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
381712299728,4cyclictest131rcu_preempt23:20:023
381712299718,20cyclictest3834895-21ssh21:36:313
381712299718,20cyclictest3834895-21ssh21:36:313
381712299717,27cyclictest3871001-21latency_hist23:00:023
381712299717,27cyclictest3871001-21latency_hist23:00:023
3817112997127,8cyclictest3879448-21kworker/0:223:23:050
3817122997012,21cyclictest3911922-21systemd-run00:35:013
3817122996919,13cyclictest3889233-21ssh23:42:273
3817122996818,16cyclictest3849325-21latency_hist22:10:023
3817122996811,16cyclictest3903789-21ssh00:15:543
3817122996718,14cyclictest3856057-21ssh22:25:183
3817122996710,21cyclictest3829042-21ssh21:23:083
3817122996618,11cyclictest37-21ksoftirqd/300:29:403
3817122996611,17cyclictest3832460-21ssh21:30:503
3817122996510,21cyclictest3847057-21ssh22:04:553
3817122996418,11cyclictest37-21ksoftirqd/323:39:593
3817122996416,10cyclictest37-21ksoftirqd/323:28:423
3817122996317,8cyclictest37-21ksoftirqd/323:32:383
3817122996316,9cyclictest37-21ksoftirqd/300:04:283
3817122996310,16cyclictest3894758-21ssh23:55:073
3817122996219,8cyclictest3854311-21ssh22:21:253
3817122996219,8cyclictest3854311-21ssh22:21:253
3817122996213,11cyclictest37-21ksoftirqd/321:19:473
381712299618,16cyclictest3839237-21ssh21:46:283
381712299618,14cyclictest3892046-21ssh23:49:143
3817122996116,14cyclictest3877263-21ssh23:15:003
3817122996115,9cyclictest37-21ksoftirqd/322:49:393
381712299608,16cyclictest3913753-21ssh00:39:113
3817122996017,8cyclictest37-21ksoftirqd/322:39:343
3817122996016,9cyclictest37-21ksoftirqd/323:23:263
3817122996016,9cyclictest37-21ksoftirqd/300:11:223
3817122996015,10cyclictest37-21ksoftirqd/323:08:293
3817122996015,10cyclictest37-21ksoftirqd/322:17:113
3817122996014,10cyclictest37-21ksoftirqd/322:31:143
3817122996014,10cyclictest37-21ksoftirqd/322:31:143
3817122996014,10cyclictest37-21ksoftirqd/300:20:203
3817122996014,10cyclictest37-21ksoftirqd/300:20:203
3817122996010,14cyclictest3868592-21ssh22:54:513
3817112996021,33cyclictest275-21dbus-daemon00:00:000
3817122995915,8cyclictest37-21ksoftirqd/322:10:543
3817122995836,16cyclictest3467602-21systemd-journal22:00:003
3817122995836,16cyclictest3467602-21systemd-journal22:00:003
3817122995815,9cyclictest37-21ksoftirqd/322:42:023
3817122995815,9cyclictest37-21ksoftirqd/322:42:023
3817122995815,10cyclictest37-21ksoftirqd/321:50:083
3817117995840,7cyclictest3821463-21kworker/2:121:37:072
3817117995840,7cyclictest3821463-21kworker/2:121:37:072
3817122995717,7cyclictest37-21ksoftirqd/323:50:243
3817122995716,7cyclictest37-21ksoftirqd/321:42:493
3817122995715,7cyclictest37-21ksoftirqd/321:11:013
3817122995715,11cyclictest37-21ksoftirqd/321:29:043
3817122995715,11cyclictest37-21ksoftirqd/321:29:043
3817117995743,10cyclictest3864366-21kworker/2:000:05:222
3817117995737,16cyclictest1-21systemd19:25:012
381711299573,27cyclictest131rcu_preempt23:36:270
3817112995726,15cyclictest3873202-21ssh23:05:030
3817112995718,34cyclictest275-21dbus-daemon22:50:000
3817117995648,5cyclictest3821463-21kworker/2:100:31:122
3817117995635,16cyclictest1-21systemd21:50:012
381711299564,28cyclictest131rcu_preempt21:14:420
3817112995638,14cyclictest1-21systemd23:20:010
3817112995629,11cyclictest3913639-21ssh00:38:540
3817112995629,11cyclictest3850190-21ssh22:11:550
3817112995628,13cyclictest3852704-21ssh22:17:510
381711299561,27cyclictest131rcu_preempt22:32:350
381711299561,27cyclictest131rcu_preempt22:32:350
3817117995546,6cyclictest3864366-21kworker/2:023:01:522
3817117995545,8cyclictest291rcuc/222:50:012
3817114995537,14cyclictest275-21dbus-daemon23:59:591
3817112995520,17cyclictest16850irq/134-800000021:40:230
3817112995519,18cyclictest16850irq/134-800000023:54:150
3817112995518,6cyclictest16850irq/134-800000023:45:250
3817112995518,6cyclictest16850irq/134-800000023:02:150
3817112995518,19cyclictest16850irq/134-800000000:14:420
381711299551,27cyclictest131rcu_preempt00:23:430
381711299551,27cyclictest131rcu_preempt00:23:430
381712299545,14cyclictest3823274-21diskmemload23:04:033
3817117995445,6cyclictest3807074-21kworker/2:221:00:022
3817117995445,5cyclictest3821463-21kworker/2:123:22:332
3817117995443,6cyclictest3807074-21kworker/2:221:25:002
3817112995427,12cyclictest3855244-21ssh22:23:550
3817112995427,12cyclictest3855244-21ssh22:23:550
381711299542,4cyclictest131rcu_preempt23:32:550
3817112995424,15cyclictest3857384-21ssh22:28:500
3817112995419,18cyclictest16850irq/134-800000021:21:550
3817112995419,17cyclictest16850irq/134-800000021:58:270
3817112995419,17cyclictest16850irq/134-800000021:58:270
3817112995419,17cyclictest16850irq/134-800000000:02:150
3817112995418,18cyclictest16850irq/134-800000000:09:330
3817112995417,7cyclictest16850irq/134-800000000:32:310
381712299539,15cyclictest3819816-21cat20:00:033
3817122995315,7cyclictest37-21ksoftirqd/300:09:483
3817122995313,23cyclictest37-21ksoftirqd/320:35:023
3817117995345,5cyclictest3821463-21kworker/2:123:33:172
3817117995344,6cyclictest3821463-21kworker/2:123:48:572
3817117995344,6cyclictest3821463-21kworker/2:123:37:452
3817112995324,3cyclictest131rcu_preempt21:45:360
3817112995319,17cyclictest16850irq/134-800000023:14:170
3817112995319,17cyclictest16850irq/134-800000022:03:090
3817112995318,17cyclictest16850irq/134-800000022:54:030
3817112995318,17cyclictest16850irq/134-800000021:25:050
3817112995318,17cyclictest16850irq/134-800000021:25:050
3817117995244,5cyclictest3864366-21kworker/2:000:01:582
3817117995244,5cyclictest3821463-21kworker/2:122:06:172
3817117995244,5cyclictest3821463-21kworker/2:100:20:302
3817117995244,5cyclictest3821463-21kworker/2:100:20:302
3817117995244,5cyclictest3807074-21kworker/2:221:25:252
3817117995244,5cyclictest3807074-21kworker/2:221:25:252
3817117995244,5cyclictest3807074-21kworker/2:221:11:422
3817117995244,5cyclictest3807074-21kworker/2:221:05:592
3817117995243,6cyclictest3807074-21kworker/2:222:17:222
3817117995242,6cyclictest3821463-21kworker/2:122:37:432
3817117995242,6cyclictest3821463-21kworker/2:121:50:422
3817117995242,6cyclictest3821463-21kworker/2:121:44:302
3817112995224,9cyclictest131rcu_preempt21:39:350
3817112995224,9cyclictest131rcu_preempt21:39:340
3817112995224,12cyclictest3889355-21ssh23:42:450
3817112995220,6cyclictest12-21ksoftirqd/021:52:410
3817112995218,6cyclictest16850irq/134-800000023:25:390
3817112995218,17cyclictest16850irq/134-800000022:56:190
3817112995218,17cyclictest16850irq/134-800000022:56:190
3817112995217,6cyclictest16850irq/134-800000021:16:250
3817112995215,17cyclictest16850irq/134-800000021:30:330
3817122995114,7cyclictest37-21ksoftirqd/319:40:033
3817117995143,5cyclictest3864366-21kworker/2:023:09:242
3817117995143,5cyclictest3821463-21kworker/2:123:28:562
3817117995143,5cyclictest3821463-21kworker/2:121:59:132
3817117995143,5cyclictest3821463-21kworker/2:121:59:132
3817117995143,5cyclictest3821463-21kworker/2:121:31:532
3817117995142,6cyclictest3807074-21kworker/2:222:20:572
3817117995142,6cyclictest3807074-21kworker/2:222:20:572
3817117995142,5cyclictest3864366-21kworker/2:000:36:162
3817117995141,6cyclictest3821463-21kworker/2:122:56:122
3817117995141,6cyclictest3821463-21kworker/2:122:56:122
3817117995128,16cyclictest309-21in:imuxsock19:20:002
3817112995125,11cyclictest3863476-21ssh22:42:510
3817112995125,11cyclictest3863476-21ssh22:42:510
3817117995042,5cyclictest3864366-21kworker/2:023:55:262
3817117995042,5cyclictest3821463-21kworker/2:122:00:482
3817117995042,5cyclictest3821463-21kworker/2:121:17:272
3817117995042,5cyclictest3821463-21kworker/2:100:26:092
3817117995042,5cyclictest3807074-21kworker/2:222:13:062
3817117995042,5cyclictest3807074-21kworker/2:220:50:002
3817117995042,5cyclictest3807074-21kworker/2:220:12:372
3817117995042,5cyclictest3807074-21kworker/2:220:12:372
3817117995041,6cyclictest3821463-21kworker/2:123:11:452
3817117995041,6cyclictest3821463-21kworker/2:122:41:592
3817117995041,6cyclictest3821463-21kworker/2:122:41:592
3817117995041,6cyclictest3807074-21kworker/2:222:30:072
3817117995041,6cyclictest3807074-21kworker/2:222:30:072
3817117995041,5cyclictest3864366-21kworker/2:000:12:082
3817117995041,5cyclictest3821463-21kworker/2:122:54:542
381711499504,4cyclictest131rcu_preempt00:33:361
3817114995034,11cyclictest23-21ksoftirqd/121:20:001
381711299502,43cyclictest3850338-21kworker/u8:322:35:080
3817117994941,5cyclictest3864366-21kworker/2:023:16:132
3817117994941,5cyclictest3821463-21kworker/2:123:43:182
3817117994941,5cyclictest3807074-21kworker/2:222:29:242
3817117994939,6cyclictest3821463-21kworker/2:100:18:442
3817117994840,5cyclictest3807074-21kworker/2:220:17:102
3817117994840,5cyclictest3807074-21kworker/2:219:53:232
3817117994839,5cyclictest3864366-21kworker/2:023:52:142
3817117994838,7cyclictest30-21ksoftirqd/220:55:012
381711299483,4cyclictest131rcu_preempt00:15:040
381711299481,3cyclictest131rcu_preempt20:36:100
3817122994730,14cyclictest3467602-21systemd-journal19:12:223
3817122994712,9cyclictest37-21ksoftirqd/321:05:023
3817117994739,5cyclictest3807074-21kworker/2:219:57:142
3817117994739,5cyclictest3807074-21kworker/2:219:32:502
3817114994734,11cyclictest23-21ksoftirqd/120:55:011
3817114994731,12cyclictest275-21dbus-daemon22:20:001
3817112994734,10cyclictest16950irq/224-800000021:10:000
3817112994727,15cyclictest275-21dbus-daemon20:20:010
3817112994720,11cyclictest3907950-21ssh00:25:240
3817122994632,10cyclictest3822459-21sshd20:51:433
3817122994631,11cyclictest3819918-21munin-run20:05:003
3817122994630,13cyclictest3820627-21sshd20:16:103
3817122994610,8cyclictest37-21ksoftirqd/319:25:013
3817117994638,5cyclictest3807074-21kworker/2:220:30:302
3817117994638,5cyclictest3807074-21kworker/2:219:45:092
3817117994638,5cyclictest3807074-21kworker/2:219:35:112
3817117994638,5cyclictest3807074-21kworker/2:219:13:552
381711499463,3cyclictest131rcu_preempt19:20:021
381711299461,16cyclictest3819666-21munin-run20:00:010
3817122994531,11cyclictest3467602-21systemd-journal20:57:323
3817117994538,4cyclictest3807074-21kworker/2:220:27:032
3817117994537,5cyclictest3807074-21kworker/2:220:43:372
3817117994537,5cyclictest3807074-21kworker/2:220:02:422
3817117994537,5cyclictest3807074-21kworker/2:219:28:122
3817114994528,13cyclictest23-21ksoftirqd/121:51:491
3817122994429,12cyclictest3821953-21sshd20:41:113
3817122994429,11cyclictest3818070-21sshd19:29:103
3817122994429,10cyclictest48-21kcompactd019:42:493
3817122994410,6cyclictest0-21swapper/320:26:103
3817117994437,4cyclictest3807074-21kworker/2:220:25:022
3817117994436,5cyclictest3807074-21kworker/2:221:02:002
3817117994436,5cyclictest3807074-21kworker/2:219:40:092
3817114994428,12cyclictest275-21dbus-daemon20:30:011
3817112994430,11cyclictest3467602-21systemd-journal19:40:040
3817112994427,13cyclictest3847893-21ssh22:06:310
381711299442,37cyclictest3818622-21modprobe19:40:010
3817122994331,9cyclictest3819088-21sshd19:46:093
3817122994331,8cyclictest3820918-21sshd20:21:103
3817122994330,10cyclictest3823240-21sshd21:06:103
3817122994330,10cyclictest3820372-21sshd20:10:093
3817122994330,10cyclictest3820372-21sshd20:10:093
3817122994328,11cyclictest3820139-21sshd20:09:093
3817114994333,7cyclictest3817814-21sshd19:23:091
3817114994331,10cyclictest23-21ksoftirqd/123:55:001
3817114994330,9cyclictest3467602-21systemd-journal21:23:111
3817114994330,10cyclictest3467602-21systemd-journal23:00:001
3817114994330,10cyclictest3467602-21systemd-journal23:00:001
3817114994324,13cyclictest1-21systemd20:20:011
3817114994322,15cyclictest275-21dbus-daemon22:50:001
381711499431,4cyclictest131rcu_preempt23:45:251
381711299439,30cyclictest3818351-21kworker/0:120:56:580
3817122994234,4cyclictest3818340-21munin-run19:35:013
381711499424,3cyclictest131rcu_preempt21:44:431
3817114994229,10cyclictest1-21systemd23:05:001
3817114994224,13cyclictest3881544-21cron23:25:001
3817114994224,11cyclictest275-21dbus-daemon22:35:011
3817114994224,11cyclictest275-21dbus-daemon22:35:011
381711499421,4cyclictest131rcu_preempt23:32:471
381711499421,3cyclictest131rcu_preempt21:59:541
381711499421,3cyclictest131rcu_preempt21:59:541
3817112994231,8cyclictest3467602-21systemd-journal20:10:000
3817112994231,8cyclictest12-21ksoftirqd/019:15:000
3817112994227,11cyclictest275-21dbus-daemon20:50:000
3817117994133,5cyclictest3807074-21kworker/2:220:39:182
3817114994133,5cyclictest23-21ksoftirqd/123:41:271
3817114994132,6cyclictest3864334-21sh22:45:011
3817114994132,6cyclictest3864334-21sh22:45:011
3817114994131,5cyclictest3823274-21diskmemload22:55:001
3817114994128,10cyclictest23-21ksoftirqd/123:27:071
3817114994127,11cyclictest3467602-21systemd-journal19:15:001
3817114994127,11cyclictest23-21ksoftirqd/122:20:501
3817114994127,11cyclictest23-21ksoftirqd/122:20:501
381711499411,3cyclictest131rcu_preempt21:14:301
381711499411,13cyclictest36450snmpd21:46:071
3817112994134,4cyclictest16950irq/224-800000020:45:010
3817112994127,11cyclictest3821157-21sed20:25:030
381711299412,12cyclictest3817819-21modprobe19:24:090
381711299411,37cyclictest3817521-21modprobe19:16:590
3817122994031,5cyclictest3822214-21sshd20:48:103
3817122994030,7cyclictest37-21ksoftirqd/319:17:003
3817122994028,9cyclictest48-21kcompactd019:50:353
3817122994027,9cyclictest37-21ksoftirqd/320:39:283
381711499405,7cyclictest23-21ksoftirqd/123:16:251
3817114994031,5cyclictest3823274-21diskmemload23:35:161
3817114994030,6cyclictest3818594-21sendmail19:40:001
3817114994028,9cyclictest3849515-21sshd22:10:111
3817114994026,11cyclictest3914091-21systemd-run00:40:001
3817114994026,10cyclictest3820606-21latency_hist20:15:021
381711499402,5cyclictest131rcu_preempt22:28:161
381711499402,4cyclictest131rcu_preempt00:28:331
3817114994024,12cyclictest3907588-21diskmemload00:25:011
3817114994024,12cyclictest3907588-21diskmemload00:25:011
3817114994024,12cyclictest275-21dbus-daemon00:10:001
381711499402,3cyclictest131rcu_preempt00:16:281
381711499395,4cyclictest131rcu_preempt21:00:011
381711499393,4cyclictest131rcu_preempt21:31:011
3817114993925,9cyclictest23-21ksoftirqd/122:06:051
3817114993922,14cyclictest275-21dbus-daemon22:05:001
3817114993921,13cyclictest275-21dbus-daemon22:40:001
3817112993932,4cyclictest16950irq/224-800000019:48:500
381711299392,8cyclictest0-21swapper/019:54:230
3817112993924,12cyclictest3822455-21sshd20:51:110
3817117993830,5cyclictest3807074-21kworker/2:220:06:032
3817114993828,6cyclictest3903248-21latency_hist00:15:021
3817114993825,10cyclictest23-21ksoftirqd/120:35:001
3817114993823,12cyclictest3818397-21wc19:35:021
3817114993820,13cyclictest1-21systemd20:50:011
3817112993832,3cyclictest16950irq/224-800000020:25:100
381711299381,33cyclictest3467718-21lldpd20:32:120
3817114993722,11cyclictest275-21dbus-daemon20:10:011
381711499371,4cyclictest131rcu_preempt23:14:161
381711299371,32cyclictest150-21pr/ttyS221:05:000
3817114993626,8cyclictest23-21ksoftirqd/100:02:311
3817114993625,8cyclictest23-21ksoftirqd/121:25:031
3817114993625,8cyclictest23-21ksoftirqd/121:25:031
3817114993625,7cyclictest23-21ksoftirqd/121:39:031
3817114993625,7cyclictest23-21ksoftirqd/121:39:021
3817114993624,10cyclictest23-21ksoftirqd/119:55:011
3817114993621,12cyclictest23-21ksoftirqd/120:10:471
381711499361,3cyclictest131rcu_preempt23:07:211
3817112993628,5cyclictest3820403-21sh20:15:010
3817112993628,5cyclictest3820403-21sh20:15:000
3817112993627,7cyclictest12-21ksoftirqd/019:10:020
3817112993627,7cyclictest12-21ksoftirqd/019:10:010
381711299362,5cyclictest3818319-21sshd19:33:080
3817112993622,9cyclictest275-21dbus-daemon19:30:010
3817114993523,8cyclictest23-21ksoftirqd/119:50:011
381711499352,20cyclictest1-21systemd20:00:011
3817114993519,12cyclictest275-21dbus-daemon21:05:001
381711299351,4cyclictest0-21swapper/020:03:090
3817114993426,5cyclictest3822183-21latency_hist20:45:031
3817114993426,5cyclictest3821711-21latency_hist20:40:021
3817114993422,8cyclictest3467602-21systemd-journal20:00:031
3817114993322,8cyclictest3820914-21sshd20:20:101
3817117993224,6cyclictest30-21ksoftirqd/219:10:022
3817117993224,6cyclictest30-21ksoftirqd/219:10:022
3817114993224,5cyclictest3818088-21/usr/sbin/munin19:30:001
3817114993218,12cyclictest23-21ksoftirqd/119:45:011
3817114993019,6cyclictest0-21swapper/121:10:001
3817122992915,11cyclictest3817225-21latency_hist19:10:033
3817122992915,11cyclictest3817225-21latency_hist19:10:033
3817114992618,5cyclictest3817115-21latency_hist19:10:021
3817114992618,5cyclictest3817115-21latency_hist19:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional