You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-17 - 13:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Thu Sep 17, 2026 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
248251521420,3sleep33056-21PK-Backend20:35:173
249979621220,3sleep20-21swapper/221:25:172
256338421130,4sleep13056-21PK-Backend00:25:201
244988227939,15sleep20-21swapper/219:07:592
245001027336,13sleep10-21swapper/119:09:381
24739462630,4sleep33056-21PK-Backend20:15:003
244971126344,13sleep00-21swapper/019:05:430
244855926240,14sleep30-21swapper/319:05:023
24657362590,2sleep20-21swapper/219:50:162
24703932560,3sleep20-21swapper/220:05:012
25580582500,2sleep20-21swapper/200:10:222
24929532440,4sleep23056-21PK-Backend21:05:192
24592982410,3sleep00-21swapper/019:30:210
25389972370,4sleep13056-21PK-Backend23:15:231
24638902370,2sleep00-21swapper/019:45:140
2450197992912,4cyclictest34-21ksoftirqd/220:19:172
2450188992817,10cyclictest2499683-21awk21:25:150
25557332270,9sleep126-21ksoftirqd/100:05:191
2450197992717,3cyclictest34-21ksoftirqd/222:35:022
139099270,4rtkit-daemon1389-21rtkit-daemon22:01:023
25646352260,6chrt0-21swapper/300:30:153
139099260,4rtkit-daemon1389-21rtkit-daemon22:36:532
245019799259,9cyclictest34-21ksoftirqd/200:35:012
245019799255,3cyclictest34-21ksoftirqd/220:48:122
245019799253,11cyclictest34-21ksoftirqd/219:15:012
245018899252,5cyclictest13-21ksoftirqd/022:55:000
2450188992411,12cyclictest13-21ksoftirqd/020:30:020
139099240,3rtkit-daemon1389-21rtkit-daemon19:42:082
25266322230,3chrt0-21swapper/222:40:192
2450201992317,4cyclictest2457806-21turbostat19:30:013
245019799238,7cyclictest34-21ksoftirqd/221:37:242
245019799238,3cyclictest34-21ksoftirqd/200:06:102
245019799235,3cyclictest34-21ksoftirqd/219:58:482
2450197992321,1cyclictest0-21swapper/221:13:172
2450197992320,2cyclictest0-21swapper/219:34:022
2450197992310,4cyclictest34-21ksoftirqd/223:32:252
2450197992310,3cyclictest141rcu_preempt20:14:152
245019799230,21cyclictest978-21snmpd23:57:022
2450192992318,3cyclictest25656322sleep100:34:031
2450188992317,4cyclictest2539882-21cut23:20:100
25366552220,3sleep13056-21PK-Backend23:10:171
2450201992220,1cyclictest3056-21PK-Backend20:18:193
2450201992220,1cyclictest0-21swapper/319:37:093
2450201992219,2cyclictest978-21snmpd22:39:333
2450201992218,3cyclictest2499242-21apt21:25:113
2450201992218,3cyclictest1095-21runrttasks23:45:443
2450201992214,6cyclictest2520778-21interrupts22:25:163
245020199220,20cyclictest3056-21PK-Backend20:08:053
245019799229,2cyclictest141rcu_preempt23:25:002
245019799228,2cyclictest141rcu_preempt19:25:082
245019799222,9cyclictest34-21ksoftirqd/223:25:212
2450197992220,1cyclictest0-21swapper/221:02:502
2450197992220,1cyclictest0-21swapper/220:05:592
2450197992210,6cyclictest141rcu_preempt19:23:552
245018899228,5cyclictest13-21ksoftirqd/019:19:520
245018899228,2cyclictest141rcu_preempt21:15:010
245018899223,4cyclictest13-21ksoftirqd/023:55:000
245018899223,4cyclictest13-21ksoftirqd/022:05:000
2450188992220,1cyclictest13-21ksoftirqd/019:25:390
245018899221,5cyclictest13-21ksoftirqd/022:10:000
2450201992119,1cyclictest3056-21PK-Backend23:38:273
2450201992119,1cyclictest3056-21PK-Backend21:36:543
2450201992119,1cyclictest3056-21PK-Backend00:09:373
2450201992119,1cyclictest0-21swapper/323:56:533
2450201992119,1cyclictest0-21swapper/323:50:243
2450201992119,1cyclictest0-21swapper/323:08:543
2450201992119,1cyclictest0-21swapper/322:50:013
2450201992119,1cyclictest0-21swapper/321:17:283
2450201992119,1cyclictest0-21swapper/320:57:343
2450201992119,1cyclictest0-21swapper/320:57:333
2450201992119,1cyclictest0-21swapper/319:20:273
2450201992119,1cyclictest0-21swapper/300:18:433
2450201992118,2cyclictest3056-21PK-Backend20:25:033
2450201992113,6cyclictest1-21systemd23:30:003
245020199211,18cyclictest2526446-21uname22:40:163
245020199210,20cyclictest0-21swapper/322:24:203
245020199210,20cyclictest0-21swapper/300:04:103
245020199210,19cyclictest3056-21PK-Backend20:21:173
245020199210,19cyclictest3056-21PK-Backend00:11:263
245019799219,4cyclictest34-21ksoftirqd/220:25:142
245019799219,4cyclictest34-21ksoftirqd/220:20:292
2450197992119,1cyclictest0-21swapper/223:47:432
2450197992119,1cyclictest0-21swapper/223:37:352
2450197992119,1cyclictest0-21swapper/223:07:482
2450197992119,1cyclictest0-21swapper/222:48:422
2450197992119,1cyclictest0-21swapper/222:48:422
2450197992119,1cyclictest0-21swapper/222:19:352
2450197992119,1cyclictest0-21swapper/220:30:052
2450197992119,1cyclictest0-21swapper/200:19:192
2450197992119,1cyclictest0-21swapper/200:03:312
2450197992117,2cyclictest2512127-21irqstats22:00:162
2450197992116,4cyclictest34-21ksoftirqd/219:20:002
2450197992115,4cyclictest2156-21gnome-shell00:37:102
2450197992112,4cyclictest34-21ksoftirqd/222:24:552
2450197992110,3cyclictest34-21ksoftirqd/221:31:362
245019799210,20cyclictest0-21swapper/223:50:332
245019799210,20cyclictest0-21swapper/221:54:272
245019799210,20cyclictest0-21swapper/220:40:062
245019799210,20cyclictest0-21swapper/200:24:472
245019799210,10cyclictest2498974-21munin-plugin-st21:25:002
2450192992113,6cyclictest1-21systemd21:45:011
245019299210,20cyclictest3056-21PK-Backend00:19:031
245019299210,1cyclictest3056-21PK-Backend21:28:481
245018899218,4cyclictest13-21ksoftirqd/023:12:500
245018899218,4cyclictest13-21ksoftirqd/020:50:350
245018899218,4cyclictest13-21ksoftirqd/020:36:030
245018899218,4cyclictest13-21ksoftirqd/000:01:300
2450188992119,1cyclictest0-21swapper/019:11:320
2450188992116,4cyclictest13-21ksoftirqd/020:05:000
2450188992114,5cyclictest2466444-21sshd19:51:380
245018899210,20cyclictest0-21swapper/022:23:160
245018899210,20cyclictest0-21swapper/000:19:350
139099210,2rtkit-daemon1389-21rtkit-daemon23:16:373
2450201992018,1cyclictest3056-21PK-Backend23:34:503
2450201992018,1cyclictest0-21swapper/323:02:043
2450201992018,1cyclictest0-21swapper/321:10:313
2450201992018,1cyclictest0-21swapper/300:35:053
2450201992018,1cyclictest0-21swapper/300:24:103
2450201992017,2cyclictest3056-21PK-Backend20:41:313
2450201992017,2cyclictest3056-21PK-Backend19:53:123
2450201992011,7cyclictest2501700-21/usr/sbin/munin21:30:173
245020199200,7cyclictest2462657-21python319:40:193
245020199200,1cyclictest3056-21PK-Backend23:40:173
245020199200,19cyclictest3056-21PK-Backend20:31:453
245020199200,19cyclictest3056-21PK-Backend19:11:003
245020199200,19cyclictest0-21swapper/322:33:313
245020199200,19cyclictest0-21swapper/321:23:363
245020199200,18cyclictest2536954-21ntp_kernel_pll_23:10:203
245019799206,4cyclictest34-21ksoftirqd/223:00:142
245019799206,4cyclictest34-21ksoftirqd/221:50:002
2450197992018,1cyclictest0-21swapper/220:58:312
2450197992018,1cyclictest0-21swapper/220:58:302
2450197992018,1cyclictest0-21swapper/200:25:482
2450197992017,2cyclictest0-21swapper/222:57:202
245019799201,5cyclictest2529213-21/usr/sbin/munin22:50:222
245019799200,19cyclictest0-21swapper/223:44:532
245019799200,19cyclictest0-21swapper/221:19:262
245019799200,19cyclictest0-21swapper/220:54:112
245019799200,19cyclictest0-21swapper/220:36:342
2450192992018,1cyclictest3056-21PK-Backend19:34:111
2450192992018,1cyclictest0-21swapper/122:59:071
245019299200,1cyclictest0-21swapper/123:40:021
245019299200,19cyclictest3056-21PK-Backend22:36:361
245019299200,19cyclictest3056-21PK-Backend21:16:251
245019299200,19cyclictest3056-21PK-Backend20:52:181
245019299200,19cyclictest3056-21PK-Backend19:39:251
245019299200,19cyclictest3056-21PK-Backend19:15:401
245019299200,19cyclictest3056-21PK-Backend00:24:031
245019299200,19cyclictest0-21swapper/121:07:341
245019299200,19cyclictest0-21swapper/120:12:521
245019299200,19cyclictest0-21swapper/120:07:451
245019299200,19cyclictest0-21swapper/119:55:341
245019299200,19cyclictest0-21swapper/119:10:051
245018899207,4cyclictest13-21ksoftirqd/023:55:580
245018899204,4cyclictest13-21ksoftirqd/021:07:240
245018899204,3cyclictest13-21ksoftirqd/021:31:580
245018899203,3cyclictest141rcu_preempt00:36:350
2450188992018,1cyclictest0-21swapper/023:25:290
2450188992018,1cyclictest0-21swapper/022:47:570
2450188992018,1cyclictest0-21swapper/022:47:560
2450188992018,1cyclictest0-21swapper/020:41:220
2450188992018,1cyclictest0-21swapper/020:31:010
2450188992018,1cyclictest0-21swapper/019:40:050
2450188992018,1cyclictest0-21swapper/019:35:110
2450188992014,4cyclictest13-21ksoftirqd/021:05:000
2450188992013,5cyclictest436-20systemd-journal20:20:040
2450188992012,6cyclictest2558247-21sshd00:11:400
2450188992012,6cyclictest2476021-21systemd20:20:000
245018899200,1cyclictest0-21swapper/020:14:110
245018899200,19cyclictest0-21swapper/023:46:500
245018899200,19cyclictest0-21swapper/023:40:350
245018899200,19cyclictest0-21swapper/023:03:240
245018899200,19cyclictest0-21swapper/022:39:180
245018899200,19cyclictest0-21swapper/022:27:280
245018899200,19cyclictest0-21swapper/022:19:520
245018899200,19cyclictest0-21swapper/000:23:140
2450201991917,1cyclictest3056-21PK-Backend20:49:083
2450201991917,1cyclictest0-21swapper/321:55:123
2450201991916,2cyclictest2459123-21ps19:30:213
2450201991914,3cyclictest2487470-21df20:50:143
2450201991911,6cyclictest2452865-21awk19:15:113
245020199190,1cyclictest3056-21PK-Backend21:06:463
245020199190,18cyclictest3056-21PK-Backend20:03:463
245020199190,17cyclictest2507157-21/usr/sbin/munin21:45:173
245019799195,4cyclictest34-21ksoftirqd/222:10:212
2450197991917,1cyclictest0-21swapper/222:09:482
2450197991917,1cyclictest0-21swapper/219:48:412
2450197991914,3cyclictest2536380-21cpuspeed_turbos23:10:142
2450197991912,6cyclictest2521311-21sed22:25:222
2450197991911,6cyclictest2460487-21cat19:35:162
2450197991911,2cyclictest34-21ksoftirqd/221:40:122
245019799190,9cyclictest34-21ksoftirqd/222:00:012
2450192991918,1cyclictest0-21swapper/100:11:121
2450192991917,1cyclictest3056-21PK-Backend23:23:471
2450192991917,1cyclictest3056-21PK-Backend22:51:081
2450192991917,1cyclictest3056-21PK-Backend19:44:401
2450192991917,1cyclictest0-21swapper/120:18:471
2450192991917,1cyclictest0-21swapper/119:50:401
2450192991913,5cyclictest2476704-21expr20:20:121
2450192991912,5cyclictest436-20systemd-journal20:49:561
245019299190,1cyclictest0-21swapper/123:32:461
245019299190,1cyclictest0-21swapper/121:34:091
245019299190,1cyclictest0-21swapper/120:35:081
245019299190,1cyclictest0-21swapper/120:26:161
245019299190,18cyclictest3056-21PK-Backend22:23:171
245019299190,18cyclictest0-21swapper/121:37:211
245018899198,4cyclictest13-21ksoftirqd/022:41:140
245018899198,4cyclictest13-21ksoftirqd/022:11:020
245018899198,2cyclictest141rcu_preempt20:00:000
245018899197,3cyclictest13-21ksoftirqd/021:48:540
245018899196,4cyclictest13-21ksoftirqd/023:15:270
245018899196,4cyclictest13-21ksoftirqd/021:17:100
245018899193,6cyclictest1095-21runrttasks23:35:220
245018899192,4cyclictest13-21ksoftirqd/022:30:110
2450188991917,1cyclictest0-21swapper/020:06:070
2450188991912,5cyclictest2504830-21forks21:40:160
2450188991911,6cyclictest2543822-21grep23:30:160
2450188991911,5cyclictest2556796-21munin-plugin-st00:10:000
245018899190,8cyclictest141rcu_preempt21:20:150
245018899190,1cyclictest0-21swapper/019:20:250
245018899190,18cyclictest0-21swapper/021:39:360
245020199187,9cyclictest2465023-21unin-run19:50:003
245020199182,14cyclictest2156-21gnome-shell23:24:093
2450201991816,1cyclictest3056-21PK-Backend19:56:443
2450201991813,3cyclictest2528086-21memory22:45:183
2450201991813,3cyclictest2528086-21memory22:45:173
2450201991812,5cyclictest2517389-21sed22:15:213
2450201991811,6cyclictest2508269-21ls21:50:123
2450201991810,6cyclictest2530970-21awk22:55:123
245020199180,6cyclictest2514487-21taskset22:05:363
2450197991816,1cyclictest0-21swapper/223:16:272
2450192991816,1cyclictest3056-21PK-Backend23:26:101
2450192991816,1cyclictest3056-21PK-Backend20:56:241
2450192991816,1cyclictest3056-21PK-Backend20:56:241
2450192991812,5cyclictest26-21ksoftirqd/120:30:191
2450192991811,6cyclictest26-21ksoftirqd/120:00:221
2450192991811,5cyclictest2456775-21cat19:25:131
2450192991811,2cyclictest26-21ksoftirqd/120:45:011
2450192991810,6cyclictest436-20systemd-journal21:45:001
2450192991810,6cyclictest2545672-21awk23:35:201
2450192991810,6cyclictest2455487-21expr19:20:211
2450192991810,2cyclictest26-21ksoftirqd/122:49:591
2450192991810,2cyclictest26-21ksoftirqd/122:49:591
245019299180,1cyclictest3056-21PK-Backend23:51:191
245019299180,17cyclictest3056-21PK-Backend22:01:541
245018899185,4cyclictest13-21ksoftirqd/023:05:570
245018899185,4cyclictest13-21ksoftirqd/000:28:300
2450188991817,1cyclictest0-21swapper/000:30:300
2450188991816,1cyclictest0-21swapper/021:56:320
2450188991816,1cyclictest0-21swapper/020:47:030
245018899180,2cyclictest141rcu_preempt21:50:110
2450201991715,1cyclictest3056-21PK-Backend21:03:143
245020199170,16cyclictest2515690-21/usr/sbin/munin22:10:173
2450192991714,2cyclictest25110722chrt21:56:151
2450192991712,4cyclictest26-21ksoftirqd/123:58:531
2450192991711,5cyclictest2554154-21cat00:00:171
2450192991710,2cyclictest26-21ksoftirqd/123:10:001
245018899176,4cyclictest13-21ksoftirqd/022:58:100
245018899171,6cyclictest141rcu_preempt20:55:140
245018899171,6cyclictest141rcu_preempt20:55:130
2450192991614,1cyclictest3056-21PK-Backend22:28:031
2450192991614,1cyclictest3056-21PK-Backend19:46:021
2450192991613,2cyclictest3056-21PK-Backend21:20:101
245019299161,12cyclictest2490550-21/usr/sbin/munin21:00:231
245019299160,5cyclictest2495269-21chrt21:12:131
245019299160,1cyclictest3056-21PK-Backend22:19:181
2450201991511,2cyclictest2562527-21apt00:25:123
245019299151,4cyclictest26-21ksoftirqd/122:44:511
2450192991513,1cyclictest3056-21PK-Backend22:07:161
2450192991513,1cyclictest3056-21PK-Backend00:39:511
2450192991510,4cyclictest3056-21PK-Backend22:13:401
245019299150,6cyclictest2549619-21chrt23:47:221
245019299150,14cyclictest3056-21PK-Backend22:34:591
245019299143,1cyclictest141rcu_preempt23:05:011
245019299140,5cyclictest2509478-21chrt21:53:021
2450201991311,1cyclictest0-21swapper/321:40:053
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional