You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-05 - 23:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Fri Jun 05, 2026 12:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3495397999182,7cyclictest37-21ksoftirqd/310:20:023
3495397999055,25cyclictest37-21ksoftirqd/310:00:023
349538599741,40cyclictest131rcu_preempt12:11:150
3495385996916,47cyclictest12-21ksoftirqd/010:50:000
3495385996619,42cyclictest12-21ksoftirqd/009:00:000
3495393996314,27cyclictest3567446-21kworker/u8:012:00:022
3495393996314,27cyclictest3567446-21kworker/u8:012:00:022
3495385996328,29cyclictest3575193-21sh11:54:060
3495391996216,6cyclictest131rcu_preempt09:55:021
3495393996040,8cyclictest3530548-21kworker/2:110:36:402
3495393996040,8cyclictest3530548-21kworker/2:110:36:402
349538599609,4cyclictest12-21ksoftirqd/010:25:010
3495385996037,4cyclictest12-21ksoftirqd/010:41:570
3495385996028,3cyclictest12-21ksoftirqd/009:20:110
3495385996024,5cyclictest131rcu_preempt10:35:020
3495385996016,4cyclictest12-21ksoftirqd/009:40:190
3495385995916,39cyclictest12-21ksoftirqd/011:33:500
3495385995916,39cyclictest12-21ksoftirqd/011:33:490
349538599591,38cyclictest131rcu_preempt10:19:280
3495385995911,39cyclictest16850irq/134-800000012:20:000
3495393995849,5cyclictest3530548-21kworker/2:110:34:392
3495393995841,13cyclictest1-21systemd10:50:012
3495393995840,8cyclictest3489558-21kworker/2:107:50:022
3495391995740,11cyclictest1-21systemd09:45:011
3495391995715,5cyclictest131rcu_preempt10:50:011
349538599575,48cyclictest12-21ksoftirqd/011:48:340
349538599575,48cyclictest12-21ksoftirqd/011:48:330
3495385995710,42cyclictest12-21ksoftirqd/011:13:060
349538599564,28cyclictest131rcu_preempt12:08:110
349538599564,28cyclictest131rcu_preempt12:08:100
3495385995627,13cyclictest3520402-21ssh09:52:050
3495385995615,36cyclictest12-21ksoftirqd/011:30:020
3495385995615,19cyclictest131rcu_preempt09:10:010
3495393995546,5cyclictest3489558-21kworker/2:107:15:022
3495393995546,5cyclictest3489558-21kworker/2:107:15:022
3495393995522,12cyclictest30-21ksoftirqd/210:23:302
349538599558,42cyclictest12-21ksoftirqd/010:09:470
3495385995519,32cyclictest16850irq/134-800000012:25:320
3495385995518,33cyclictest16850irq/134-800000009:57:130
3495385995515,21cyclictest12-21ksoftirqd/007:40:000
349538599551,16cyclictest131rcu_preempt09:33:250
3495393995446,5cyclictest3503829-21kworker/2:210:11:282
3495393995446,5cyclictest3501338-21kworker/2:009:20:002
3495393995437,12cyclictest1-21systemd09:00:012
349538599543,5cyclictest131rcu_preempt11:43:220
3495385995427,12cyclictest3524708-21ssh10:01:400
3495385995426,3cyclictest35037931ssh09:15:000
3495385995418,18cyclictest16850irq/134-800000012:38:380
3495385995415,17cyclictest16850irq/134-800000009:40:010
3495393995345,5cyclictest3571078-21kworker/2:212:03:252
3495393995345,5cyclictest3539617-21kworker/2:311:39:442
3495393995341,7cyclictest3503829-21kworker/2:210:10:012
3495393995339,8cyclictest3503829-21kworker/2:209:50:012
3495391995317,22cyclictest131rcu_preempt09:25:451
3495385995318,18cyclictest16850irq/134-800000011:15:530
3495385995317,6cyclictest16850irq/134-800000011:07:120
349538599531,19cyclictest131rcu_preempt09:28:180
3495393995244,5cyclictest3489558-21kworker/2:108:35:022
3495393995244,5cyclictest3489558-21kworker/2:108:35:022
3495393995244,4cyclictest3501338-21kworker/2:009:40:002
3495393995243,5cyclictest3530548-21kworker/2:111:48:502
3495393995243,5cyclictest3530548-21kworker/2:111:48:492
3495393995243,5cyclictest3530548-21kworker/2:111:11:502
349539399524,22cyclictest291rcuc/212:35:022
3495391995230,13cyclictest23-21ksoftirqd/110:30:001
3495391995229,17cyclictest275-21dbus-daemon10:25:011
3495385995218,17cyclictest16850irq/134-800000010:50:060
3495385995218,17cyclictest16850irq/134-800000010:14:440
3495385995217,6cyclictest16850irq/134-800000012:02:460
3495385995217,6cyclictest16850irq/134-800000009:48:240
3495385995217,18cyclictest16850irq/134-800000012:30:230
3495385995216,6cyclictest16850irq/134-800000010:37:420
3495385995216,6cyclictest16850irq/134-800000010:37:420
3495385995216,18cyclictest16850irq/134-800000009:16:570
3495393995143,5cyclictest3530548-21kworker/2:110:30:002
3495393995143,5cyclictest3503829-21kworker/2:210:16:132
3495393995143,5cyclictest3503829-21kworker/2:209:50:362
3495393995143,5cyclictest3489558-21kworker/2:108:25:012
3495393995143,5cyclictest3489558-21kworker/2:108:00:012
3495393995142,6cyclictest3530548-21kworker/2:111:00:022
3495393995142,6cyclictest3489558-21kworker/2:108:55:012
3495391995116,24cyclictest131rcu_preempt12:17:321
3495385995129,4cyclictest12-21ksoftirqd/008:38:270
3495385995117,17cyclictest16850irq/134-800000011:37:170
3495385995115,18cyclictest16850irq/134-800000011:01:290
3495385995111,30cyclictest131rcu_preempt12:00:020
3495385995111,30cyclictest131rcu_preempt12:00:010
3495393995042,5cyclictest3530548-21kworker/2:112:26:062
3495393995042,5cyclictest3530548-21kworker/2:112:10:542
3495393995042,5cyclictest3530548-21kworker/2:111:54:392
3495393995042,5cyclictest3503829-21kworker/2:209:55:192
3495393995042,5cyclictest3501338-21kworker/2:009:41:512
3495393995042,5cyclictest3501338-21kworker/2:009:33:062
3495393995041,6cyclictest3571078-21kworker/2:212:36:542
3495393995041,5cyclictest3539617-21kworker/2:311:25:152
3495393995035,12cyclictest1-21systemd07:40:012
3495391995012,6cyclictest131rcu_preempt11:55:001
3495385995024,23cyclictest3560442-21ssh11:21:030
3495385995016,17cyclictest16850irq/134-800000012:20:450
3495385995012,33cyclictest12-21ksoftirqd/010:57:050
3495393994942,4cyclictest3539617-21kworker/2:311:31:212
3495393994942,4cyclictest3539617-21kworker/2:311:31:212
3495393994941,5cyclictest3501338-21kworker/2:009:25:202
3495393994940,5cyclictest3539617-21kworker/2:311:17:412
3495393994940,5cyclictest3489558-21kworker/2:108:10:012
3495393994939,7cyclictest3539617-21kworker/2:311:09:102
3495393994938,7cyclictest3569602-21sshd11:41:282
3495393994938,7cyclictest3489558-21kworker/2:109:11:252
349539199499,5cyclictest131rcu_preempt10:06:451
349539199497,11cyclictest23-21ksoftirqd/108:40:011
3495391994931,13cyclictest1-21systemd11:40:001
3495391994927,12cyclictest3570921-21ssh11:44:431
349538599492,14cyclictest3495852-21munin-run07:20:000
3495385994913,6cyclictest16850irq/134-800000010:29:420
349539799481,3cyclictest131rcu_preempt11:00:023
3495393994840,5cyclictest3530548-21kworker/2:112:25:022
3495393994840,5cyclictest3530548-21kworker/2:112:18:112
3495393994840,5cyclictest3503829-21kworker/2:210:00:122
3495393994840,5cyclictest3501338-21kworker/2:009:24:452
3495393994840,5cyclictest3489558-21kworker/2:107:55:012
3495393994839,5cyclictest3489558-21kworker/2:109:05:022
3495393994839,5cyclictest3489558-21kworker/2:108:05:012
3495393994839,5cyclictest3489558-21kworker/2:107:29:292
349539199488,8cyclictest23-21ksoftirqd/110:16:461
349539199484,8cyclictest23-21ksoftirqd/112:25:011
3495391994821,19cyclictest23-21ksoftirqd/111:06:541
3495385994832,12cyclictest275-21dbus-daemon08:30:010
349538599481,8cyclictest12-21ksoftirqd/008:00:010
349539799473,4cyclictest131rcu_preempt08:50:003
3495397994728,14cyclictest3467602-21systemd-journal11:39:283
349539799471,4cyclictest131rcu_preempt07:15:003
349539799471,4cyclictest131rcu_preempt07:15:003
3495393994739,5cyclictest3539617-21kworker/2:311:01:562
3495393994739,5cyclictest3530548-21kworker/2:110:54:092
3495393994718,8cyclictest291rcuc/208:40:002
3495391994732,10cyclictest3562458-21sshd11:25:281
3495391994723,19cyclictest275-21dbus-daemon08:00:001
3495391994716,20cyclictest131rcu_preempt08:30:001
3495385994724,19cyclictest12-21ksoftirqd/008:15:010
349539799466,6cyclictest131rcu_preempt12:35:013
3495397994629,13cyclictest1-21systemd11:30:013
3495397994625,13cyclictest3533255-21kworker/u8:210:45:023
3495393994638,5cyclictest3539617-21kworker/2:311:21:582
3495393994630,10cyclictest30-21ksoftirqd/208:45:002
3495393994626,14cyclictest275-21dbus-daemon08:15:012
3495391994633,8cyclictest309-21in:imuxsock07:20:001
3495391994626,13cyclictest309-21in:imuxsock07:25:021
349539799454,3cyclictest131rcu_preempt11:50:323
349539799453,3cyclictest150-21pr/ttyS212:38:373
3495397994531,9cyclictest3467602-21systemd-journal11:20:033
3495397994531,11cyclictest3467602-21systemd-journal10:20:023
349539799451,3cyclictest131rcu_preempt12:20:153
3495393994535,5cyclictest3571078-21kworker/2:212:09:132
3495393994535,5cyclictest3571078-21kworker/2:212:09:132
3495391994534,7cyclictest23-21ksoftirqd/111:20:011
3495391994520,5cyclictest23-21ksoftirqd/111:57:351
3495391994520,5cyclictest23-21ksoftirqd/111:57:351
3495391994519,22cyclictest23-21ksoftirqd/112:15:011
3495391994511,28cyclictest3562063-21unin-run11:25:001
3495385994529,13cyclictest3501326-21wc09:05:020
3495385994523,5cyclictest12-21ksoftirqd/008:25:010
349538599452,13cyclictest3498443-21munin-run08:10:010
349538599451,40cyclictest48-21kcompactd008:20:000
349539799443,3cyclictest131rcu_preempt10:54:583
3495397994433,9cyclictest37-21ksoftirqd/312:28:303
349539799441,3cyclictest131rcu_preempt12:13:313
349539799441,3cyclictest131rcu_preempt09:35:023
3495393994435,5cyclictest3530548-21kworker/2:110:40:252
3495391994426,13cyclictest275-21dbus-daemon09:35:011
3495391994425,12cyclictest3501549-21cron09:10:001
3495391994421,19cyclictest23-21ksoftirqd/111:30:381
3495391994421,19cyclictest23-21ksoftirqd/111:30:371
3495391994417,18cyclictest23-21ksoftirqd/112:05:071
3495391994417,18cyclictest23-21ksoftirqd/112:05:061
3495391994415,6cyclictest131rcu_preempt12:28:311
3495391994415,11cyclictest23-21ksoftirqd/110:31:281
3495391994414,6cyclictest131rcu_preempt12:39:351
3495385994425,5cyclictest12-21ksoftirqd/007:30:020
349538599442,3cyclictest131rcu_preempt07:11:290
349538599442,3cyclictest131rcu_preempt07:11:280
349539799438,19cyclictest37-21ksoftirqd/311:42:573
349539799434,22cyclictest0-21swapper/311:17:503
349539799433,3cyclictest131rcu_preempt09:40:273
349539799432,3cyclictest131rcu_preempt11:10:133
349539799432,3cyclictest131rcu_preempt09:53:193
349539799431,9cyclictest150-21pr/ttyS212:05:173
349539799431,9cyclictest150-21pr/ttyS212:05:173
349539799431,3cyclictest131rcu_preempt11:47:223
349539799431,3cyclictest131rcu_preempt11:47:213
3495393994335,5cyclictest3489558-21kworker/2:107:25:012
349539199439,11cyclictest23-21ksoftirqd/111:12:011
3495391994331,9cyclictest23-21ksoftirqd/108:45:001
3495391994328,11cyclictest23-21ksoftirqd/112:05:011
3495391994326,13cyclictest23-21ksoftirqd/109:14:291
349539199432,5cyclictest0-21swapper/110:40:081
3495391994323,13cyclictest3501031-21cat09:00:001
3495391994320,19cyclictest23-21ksoftirqd/110:59:081
3495391994315,7cyclictest23-21ksoftirqd/110:03:211
3495391994313,9cyclictest131rcu_preempt09:36:581
3495391994313,6cyclictest131rcu_preempt11:05:001
3495391994311,6cyclictest131rcu_preempt10:00:001
349538599437,26cyclictest3495384-21latency_hist07:10:020
349538599433,6cyclictest131rcu_preempt07:35:020
349538599431,13cyclictest3497384-21sshd07:48:290
349538599431,12cyclictest3500911-21latency_hist08:55:020
349539799428,30cyclictest364-21snmpd12:16:113
349539799425,4cyclictest131rcu_preempt09:25:033
3495397994233,5cyclictest3505894-21cron09:20:003
3495397994229,9cyclictest3530673-21date10:15:023
3495397994229,7cyclictest37-21ksoftirqd/308:15:013
349539199426,24cyclictest3497410-21cut07:50:001
349539199426,11cyclictest23-21ksoftirqd/109:46:541
349539199424,27cyclictest3496624-21modprobe07:35:001
3495391994233,6cyclictest3496881-21sendmail07:40:001
3495391994230,9cyclictest23-21ksoftirqd/109:23:291
3495391994227,11cyclictest275-21dbus-daemon08:35:011
3495391994227,11cyclictest275-21dbus-daemon08:35:011
3495391994222,15cyclictest3467602-21systemd-journal09:05:001
349539199421,4cyclictest131rcu_preempt10:13:191
3495391994214,5cyclictest131rcu_preempt07:45:011
3495391994211,6cyclictest131rcu_preempt08:50:011
3495391994210,8cyclictest23-21ksoftirqd/111:50:021
3495391994210,8cyclictest23-21ksoftirqd/111:50:021
3495391994210,6cyclictest131rcu_preempt09:20:021
3495391994210,26cyclictest3541808-21modprobe10:40:001
3495391994210,26cyclictest3541808-21modprobe10:40:001
349539799416,3cyclictest131rcu_preempt12:00:033
349539799416,3cyclictest131rcu_preempt12:00:023
349539799415,7cyclictest37-21ksoftirqd/311:35:023
349539799415,7cyclictest37-21ksoftirqd/311:35:023
349539799414,3cyclictest150-21pr/ttyS208:08:213
349539799414,3cyclictest131rcu_preempt09:38:533
349539799413,3cyclictest131rcu_preempt10:35:173
349539799413,3cyclictest131rcu_preempt10:35:173
349539799413,3cyclictest131rcu_preempt09:28:173
3495393994133,5cyclictest3489558-21kworker/2:108:20:012
3495393994132,6cyclictest3489558-21kworker/2:107:10:022
3495393994131,6cyclictest30-21ksoftirqd/208:30:002
3495393994125,12cyclictest30-21ksoftirqd/208:50:022
349539199418,7cyclictest131rcu_preempt12:35:011
349539199412,4cyclictest131rcu_preempt08:05:021
3495391994114,19cyclictest131rcu_preempt10:52:431
349539799406,7cyclictest37-21ksoftirqd/310:04:353
349539799404,3cyclictest131rcu_preempt10:07:113
349539799404,3cyclictest131rcu_preempt07:40:013
3495397994028,9cyclictest3467602-21systemd-journal08:20:003
3495397994027,10cyclictest3518321-21sshd09:47:283
3495397994027,10cyclictest3467602-21systemd-journal07:15:323
3495397994026,10cyclictest3496673-21cat07:35:013
3495397994025,11cyclictest3499280-21cut08:25:023
349539799402,3cyclictest131rcu_preempt10:26:563
349539799401,3cyclictest131rcu_preempt11:05:023
349539799401,3cyclictest131rcu_preempt10:40:423
3495397994010,6cyclictest131rcu_preempt10:30:173
3495393994013,18cyclictest30-21ksoftirqd/209:10:012
3495391994020,15cyclictest3500757-21sshd08:53:521
3495385994034,3cyclictest16950irq/224-800000007:45:030
3495385994013,24cyclictest12-21ksoftirqd/008:32:260
3495385994013,24cyclictest12-21ksoftirqd/008:32:260
349539799395,3cyclictest131rcu_preempt09:10:153
349539799391,3cyclictest131rcu_preempt07:43:443
349539799391,10cyclictest37-21ksoftirqd/311:08:583
349539199396,25cyclictest3498465-21latency_hist08:10:011
349539199395,8cyclictest23-21ksoftirqd/108:15:011
349539199394,6cyclictest131rcu_preempt08:20:001
349538599394,5cyclictest131rcu_preempt08:05:010
349538599393,6cyclictest3467602-21systemd-journal07:50:290
3495385993934,3cyclictest3500526-21cut08:50:020
349539799383,3cyclictest131rcu_preempt12:04:373
3495397993825,10cyclictest3499721-21sshd08:34:293
3495397993825,10cyclictest3499721-21sshd08:34:293
3495397993824,11cyclictest275-21dbus-daemon08:45:013
3495397993817,4cyclictest3499497-21systemd-run08:30:003
3495393993826,9cyclictest3467602-21systemd-journal07:20:002
3495391993812,4cyclictest131rcu_preempt07:23:121
349539799374,3cyclictest131rcu_preempt09:00:013
349539799373,4cyclictest131rcu_preempt07:25:003
3495397993710,5cyclictest131rcu_preempt07:26:103
349539399374,3cyclictest131rcu_preempt07:43:442
349539199374,22cyclictest3497649-21modprobe07:55:001
3495391993725,8cyclictest3495390-21sort07:10:021
349538599372,14cyclictest3496118-21latency_hist07:25:020
349539799362,3cyclictest131rcu_preempt08:55:003
349539799362,3cyclictest131rcu_preempt08:40:023
3495397993623,9cyclictest37-21ksoftirqd/309:04:163
349539799361,4cyclictest131rcu_preempt07:50:023
349539399363,3cyclictest131rcu_preempt07:35:012
3495385993623,10cyclictest3500254-21sshd08:44:290
349539799355,3cyclictest131rcu_preempt07:55:013
349539799354,4cyclictest131rcu_preempt09:10:013
349539799351,3cyclictest131rcu_preempt07:10:023
349539199356,5cyclictest131rcu_preempt07:15:021
349539199356,5cyclictest131rcu_preempt07:15:021
3495397993421,9cyclictest37-21ksoftirqd/307:57:293
349539799341,3cyclictest131rcu_preempt08:04:293
3495391993412,17cyclictest1-21systemd08:25:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional