You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-29 - 12:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Mon Jun 29, 2026 00:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25725009910410,9cyclictest131rcu_preempt20:45:011
257249599971,43cyclictest2626934-21ssh22:52:390
2572500999126,26cyclictest2585741-21latency_hist21:25:021
2572495998923,13cyclictest2667096-21ssh00:17:100
2572495998520,13cyclictest2633722-21ssh23:06:430
2572495998514,61cyclictest16850irq/134-800000021:46:340
2572495998514,61cyclictest16850irq/134-800000021:46:330
257249599851,38cyclictest2673491-21ssh00:30:260
257249599851,38cyclictest2622302-21ssh22:42:480
2572500998326,19cyclictest2617789-21sh22:33:201
257249599831,37cyclictest2615970-21ssh22:29:320
2572500998131,14cyclictest2657223-21ssh23:56:191
2572500998034,12cyclictest2639291-21ssh23:18:391
2572500997930,20cyclictest2599976-21ssh21:55:151
2572500997930,20cyclictest2599976-21ssh21:55:141
2572495997931,16cyclictest2632866-21ssh23:05:020
2572495997929,7cyclictest2662928-21ssh00:08:350
2572500997826,16cyclictest2633493-21ssh23:06:041
257249599788,58cyclictest16850irq/134-800000022:03:020
2572500997729,14cyclictest2635582-21ssh23:10:281
2572500997729,14cyclictest2635582-21ssh23:10:281
2572500997726,14cyclictest2575916-21cat20:15:021
2572495997720,13cyclictest2649958-21ssh23:40:500
2572495997720,13cyclictest2649958-21ssh23:40:500
257249599771,73cyclictest151rcuc/021:30:020
257249599771,73cyclictest151rcuc/021:30:020
2572500997635,15cyclictest2651499-21ssh23:44:231
2572500997635,15cyclictest2651499-21ssh23:44:221
2572500997624,14cyclictest2583366-21munin-run21:20:011
2572500997623,17cyclictest2622276-21ssh22:42:451
2572500997535,14cyclictest2625322-21ssh22:49:251
2572500997432,15cyclictest2664716-21ssh00:12:101
2572500997424,16cyclictest2666307-21rm00:15:211
2572500997424,13cyclictest2598003-21ssh21:51:151
2572500997422,14cyclictest2641489-21ssh23:23:151
2572495997423,11cyclictest2629550-21ssh22:58:100
2572495997417,52cyclictest16850irq/134-800000022:20:210
2572495997417,52cyclictest16850irq/134-800000022:20:200
2572495997415,48cyclictest16850irq/134-800000021:16:120
257249599741,38cyclictest2672621-21ssh00:28:560
2572500997330,15cyclictest2586367-21ssh21:26:051
2572500997330,15cyclictest2586367-21ssh21:26:051
2572500997327,9cyclictest0-21swapper/122:10:011
2572500997327,9cyclictest0-21swapper/122:10:011
2572500997326,14cyclictest2625863-21ssh22:50:031
2572500997323,15cyclictest2610336-21ssh22:17:241
2572495997325,12cyclictest2619967-21ssh22:37:550
2572495997314,54cyclictest16850irq/134-800000021:12:170
2572495997314,49cyclictest16850irq/134-800000022:48:150
2572500997229,15cyclictest2608163-21ssh22:12:521
2572500997224,14cyclictest3467602-21systemd-journal22:55:421
2572500997221,20cyclictest2632296-21ssh23:03:531
2572495997226,42cyclictest2593628-21ssh21:41:460
2572495997224,15cyclictest2665081-21ssh00:12:560
2572495997224,11cyclictest2589384-21ssh21:32:400
2572500997150,17cyclictest2663593-21cron00:09:591
2572500997131,6cyclictest2614435-21ssh22:25:521
2572500997125,14cyclictest2645333-21sh23:31:081
2572495997121,13cyclictest2638238-21ssh23:16:180
2572495997117,45cyclictest16850irq/134-800000022:15:230
2572495997117,15cyclictest2600127-21ssh21:55:330
2572495997117,15cyclictest2600127-21ssh21:55:330
2572495997114,46cyclictest16850irq/134-800000022:06:000
2572495997114,46cyclictest16850irq/134-800000022:06:000
2572495997113,19cyclictest16850irq/134-800000021:24:500
257249599704,32cyclictest16850irq/134-800000023:31:210
2572495997026,10cyclictest2659441-21ssh00:00:520
2572500996928,16cyclictest2655809-21sh23:53:281
2572500996928,16cyclictest2655809-21sh23:53:281
2572500996919,17cyclictest2643181-21ssh23:26:401
257249599696,32cyclictest16850irq/134-800000023:26:020
2572500996829,14cyclictest150-21pr/ttyS223:47:261
2572500996827,7cyclictest2580916-21ssh21:14:511
2572500996824,14cyclictest364-21snmpd20:01:441
2572495996822,11cyclictest2653216-21ssh23:47:480
2572495996812,50cyclictest16850irq/134-800000000:24:190
2572500996725,12cyclictest150-21pr/ttyS219:34:541
257249599676,32cyclictest16850irq/134-800000021:54:010
2572495996721,13cyclictest2642195-21ssh23:24:510
2572495996721,12cyclictest2616884-21ssh22:31:120
2572495996718,44cyclictest16850irq/134-800000022:15:010
2572500996653,10cyclictest2648155-21ssh23:37:151
2572500996650,12cyclictest2674106-21ssh00:31:531
2572500996646,16cyclictest2590766-21ssh21:35:341
2572500996646,16cyclictest2590766-21ssh21:35:331
2572500996628,10cyclictest2619797-21ssh22:37:341
2572500996627,8cyclictest2593105-21ssh21:40:251
2572500996621,14cyclictest2595988-21ssh21:46:531
2572500996621,14cyclictest2595988-21ssh21:46:521
2572495996636,27cyclictest16950irq/224-800000023:10:180
2572495996636,27cyclictest16950irq/224-800000023:10:170
2572495996611,15cyclictest2656941-21ssh23:55:310
2572500996447,13cyclictest2677271-21ssh00:38:461
2572500996429,12cyclictest2659914-21ssh00:01:501
2572500996427,12cyclictest2611708-21ssh22:20:091
2572500996427,12cyclictest2611708-21ssh22:20:091
2572500996425,10cyclictest0-21swapper/121:34:081
2572500996416,15cyclictest2578709-21diskmemload00:20:221
2572495996414,44cyclictest16850irq/134-800000021:38:390
2572495996414,44cyclictest16850irq/134-800000021:38:390
2572500996332,6cyclictest2576928-21sed20:35:021
257249599638,26cyclictest16850irq/134-800000023:50:190
257249599638,26cyclictest16850irq/134-800000023:50:190
2572500996243,15cyclictest3467602-21systemd-journal00:28:441
257249599627,50cyclictest16850irq/134-800000000:35:580
2572500996144,14cyclictest2602791-21ssh22:01:201
257249599611,5cyclictest2574450-21sort19:45:030
257249599611,31cyclictest2647592-21ssh23:35:570
2572500996046,10cyclictest2574822-21munin-run19:55:011
2572500996043,13cyclictest2572698-21sshd19:12:291
2572500996018,13cyclictest2576077-21modprobe20:17:001
2572500995946,10cyclictest3467602-21systemd-journal19:44:561
2572506995836,14cyclictest275-21dbus-daemon23:50:013
257250299583,39cyclictest291rcuc/223:05:032
257250299581,6cyclictest0-21swapper/221:46:252
257250299581,6cyclictest0-21swapper/221:46:252
2572500995846,9cyclictest2575558-21sshd20:05:391
2572500995843,11cyclictest364-21snmpd19:56:441
2572500995843,11cyclictest364-21snmpd19:56:431
2572500995843,11cyclictest2577152-21munin-run20:40:011
2572500995737,16cyclictest364-21snmpd20:26:281
2572500995643,10cyclictest2577640-21sshd20:45:391
2572500995643,10cyclictest2572949-21sshd19:16:391
2572500995638,14cyclictest364-21snmpd20:58:381
2572500995542,9cyclictest3467602-21systemd-journal19:25:271
2572500995541,9cyclictest48-21kcompactd020:23:411
2572500995442,9cyclictest364-21snmpd19:36:391
2572500995441,9cyclictest48-21kcompactd019:45:581
2572500995441,9cyclictest48-21kcompactd019:23:461
2572502995343,6cyclictest2576904-21kworker/2:222:21:182
2572502995343,6cyclictest2576904-21kworker/2:222:21:182
2572500995330,19cyclictest2578463-21true21:05:001
257250299524,4cyclictest131rcu_preempt23:31:212
2572502995244,5cyclictest2581040-21kworker/2:023:49:262
2572502995243,6cyclictest2628004-21kworker/2:123:40:142
2572502995243,6cyclictest2628004-21kworker/2:123:40:132
2572502995243,6cyclictest2581040-21kworker/2:022:34:442
2572502995241,7cyclictest2620891-21sendmail22:40:002
257250299522,10cyclictest2577146-21latency_hist20:40:002
2572500995239,9cyclictest48-21kcompactd020:53:391
2572500995227,22cyclictest2572447-21latency_hist19:10:011
2572506995130,15cyclictest1-21systemd23:20:013
2572506995129,16cyclictest275-21dbus-daemon20:00:013
2572506995129,16cyclictest275-21dbus-daemon20:00:013
2572502995142,6cyclictest2576904-21kworker/2:221:17:552
2572502995142,5cyclictest2576904-21kworker/2:222:08:082
2572502995142,5cyclictest2576904-21kworker/2:222:08:082
2572502995142,5cyclictest2576904-21kworker/2:222:04:552
2572502995141,6cyclictest2628004-21kworker/2:123:37:442
2572500995137,10cyclictest364-21snmpd21:06:181
2572500995137,10cyclictest364-21snmpd21:06:171
2572502995042,5cyclictest2658928-21kworker/2:200:18:392
2572502995042,5cyclictest2581040-21kworker/2:000:26:442
257250299503,3cyclictest131rcu_preempt00:00:012
2572495995039,6cyclictest16950irq/224-800000019:55:000
2572502994941,5cyclictest2581040-21kworker/2:022:44:062
2572502994941,5cyclictest2576904-21kworker/2:221:24:572
2572502994940,5cyclictest2581040-21kworker/2:022:52:362
2572502994933,11cyclictest3467602-21systemd-journal21:45:002
257250299493,11cyclictest2578711-21cron21:10:002
257250299493,11cyclictest2578711-21cron21:10:002
2572502994931,14cyclictest1-21systemd21:15:002
257249599496,9cyclictest3467602-21systemd-journal20:00:010
257249599496,9cyclictest3467602-21systemd-journal20:00:010
2572506994829,14cyclictest275-21dbus-daemon00:25:013
257250299487,5cyclictest131rcu_preempt00:35:472
2572502994843,3cyclictest2581040-21kworker/2:000:08:432
2572502994840,5cyclictest2628004-21kworker/2:123:54:472
2572502994840,5cyclictest2628004-21kworker/2:123:54:472
2572502994840,5cyclictest2581040-21kworker/2:023:29:322
2572502994840,5cyclictest2581040-21kworker/2:022:13:072
2572502994840,5cyclictest2581040-21kworker/2:000:33:352
2572502994840,5cyclictest2576904-21kworker/2:221:25:232
2572502994840,5cyclictest2576904-21kworker/2:221:25:232
2572502994840,5cyclictest2565418-21kworker/2:219:28:402
2572502994834,10cyclictest3467602-21systemd-journal19:20:012
257250299482,15cyclictest2601610-21ssh21:59:012
257250299482,15cyclictest2601610-21ssh21:59:012
257250299481,42cyclictest2576111-21cat20:20:002
2572506994728,14cyclictest275-21dbus-daemon20:50:003
2572506994727,14cyclictest275-21dbus-daemon00:30:003
2572506994713,7cyclictest131rcu_preempt00:00:023
257250299478,5cyclictest131rcu_preempt19:35:392
257250299474,3cyclictest131rcu_preempt20:50:012
2572502994739,5cyclictest2658928-21kworker/2:200:20:222
2572502994739,5cyclictest2576904-21kworker/2:222:25:512
2572502994738,6cyclictest2581040-21kworker/2:000:10:192
2572502994713,5cyclictest131rcu_preempt00:02:262
2572502994711,6cyclictest131rcu_preempt21:30:322
2572495994730,12cyclictest275-21dbus-daemon20:10:010
2572506994628,13cyclictest3467602-21systemd-journal23:33:203
2572506994627,14cyclictest275-21dbus-daemon20:40:013
257250299469,6cyclictest131rcu_preempt21:00:012
257250299464,4cyclictest131rcu_preempt23:21:062
257250299463,9cyclictest364-21snmpd20:51:542
2572502994638,5cyclictest2581040-21kworker/2:023:10:212
2572502994638,5cyclictest2581040-21kworker/2:023:10:202
2572502994638,5cyclictest2581040-21kworker/2:022:17:052
2572502994630,12cyclictest3467602-21systemd-journal21:40:002
2572502994630,12cyclictest3467602-21systemd-journal21:40:002
257250299462,6cyclictest131rcu_preempt23:05:122
257249599463,10cyclictest2576527-21grep20:25:020
2572506994535,6cyclictest2578709-21diskmemload22:18:403
2572506994534,7cyclictest2602046-21sendmail22:00:003
2572506994534,7cyclictest2602046-21sendmail22:00:003
2572506994531,8cyclictest2661277-21cut00:05:003
2572506994530,11cyclictest1-21systemd19:25:013
2572506994526,14cyclictest275-21dbus-daemon22:30:003
2572502994537,5cyclictest2581040-21kworker/2:022:47:102
257250299453,5cyclictest131rcu_preempt20:05:002
2572502994511,6cyclictest131rcu_preempt22:57:092
2572506994435,6cyclictest37-21ksoftirqd/323:37:293
2572506994434,8cyclictest37-21ksoftirqd/319:55:003
2572506994424,16cyclictest37-21ksoftirqd/321:36:493
2572506994424,16cyclictest37-21ksoftirqd/321:36:493
257250299448,6cyclictest131rcu_preempt23:18:162
257250299443,5cyclictest131rcu_preempt19:15:012
2572495994434,6cyclictest2575832-21cron20:15:000
2572495994430,10cyclictest1-21systemd20:50:000
2572495994420,20cyclictest2576108-21latency_hist20:20:020
257249599441,23cyclictest0-21swapper/021:05:020
257249599441,10cyclictest2574025-21cron19:40:000
257250699434,6cyclictest131rcu_preempt23:43:253
257250699434,6cyclictest131rcu_preempt23:43:253
2572506994331,9cyclictest2602579-21sshd22:00:393
2572506994331,10cyclictest37-21ksoftirqd/322:05:543
2572506994331,10cyclictest37-21ksoftirqd/322:05:533
2572506994327,11cyclictest3467602-21systemd-journal22:40:033
2572506994326,12cyclictest275-21dbus-daemon19:35:003
257250299434,4cyclictest131rcu_preempt20:10:012
2572502994332,8cyclictest2575044-21sshd19:59:392
2572502994332,8cyclictest2575044-21sshd19:59:392
2572495994328,9cyclictest0-21swapper/020:55:020
2572506994229,9cyclictest37-21ksoftirqd/319:45:003
2572506994229,10cyclictest37-21ksoftirqd/321:45:023
2572506994229,10cyclictest37-21ksoftirqd/321:20:403
2572506994228,10cyclictest2663457-21sshd00:09:393
2572506994227,11cyclictest2581068-21wc21:15:013
2572506994227,11cyclictest1-21systemd23:25:023
2572495994228,11cyclictest2574532-21sshd19:47:390
2572506994132,5cyclictest1-21systemd22:55:013
2572506994130,8cyclictest37-21ksoftirqd/320:27:433
2572506994129,7cyclictest309-21in:imuxsock00:19:593
2572506994124,12cyclictest37-21ksoftirqd/321:30:463
2572506994119,15cyclictest275-21dbus-daemon20:20:013
2572506994113,21cyclictest275-21dbus-daemon19:30:013
2572502994132,6cyclictest2581040-21kworker/2:021:53:482
257249599413,4cyclictest131rcu_preempt19:10:020
2572495994133,4cyclictest16950irq/224-800000021:10:010
2572495994133,4cyclictest16950irq/224-800000021:10:000
2572495994132,5cyclictest2577128-21cron20:40:000
2572495994131,6cyclictest2572980-21cron19:20:010
257250699408,5cyclictest131rcu_preempt22:47:143
2572506994030,6cyclictest2613837-21sh22:25:013
2572506994030,6cyclictest2613837-21sh22:25:013
2572506994029,9cyclictest37-21ksoftirqd/322:55:583
2572506994029,8cyclictest37-21ksoftirqd/320:15:003
2572506994027,6cyclictest309-21in:imuxsock21:20:003
2572506994027,11cyclictest37-21ksoftirqd/319:50:003
2572506994023,13cyclictest37-21ksoftirqd/321:00:013
2572506994015,21cyclictest361rcuc/322:31:303
257249599403,7cyclictest2572692-21sshd19:10:390
2572495994034,4cyclictest16950irq/224-800000021:00:020
2572495994027,9cyclictest2573239-21sshd19:21:400
2572506993930,7cyclictest37-21ksoftirqd/323:07:183
2572506993928,9cyclictest37-21ksoftirqd/323:01:173
2572506993927,9cyclictest37-21ksoftirqd/323:11:463
2572506993927,9cyclictest37-21ksoftirqd/323:11:463
257250299396,4cyclictest131rcu_preempt19:41:402
257250299393,7cyclictest30-21ksoftirqd/221:04:392
2572495993929,5cyclictest2576878-21cron20:35:000
2572495993924,11cyclictest1-21systemd20:05:000
2572495993919,15cyclictest12-21ksoftirqd/019:35:000
257250699383,6cyclictest131rcu_preempt19:20:013
2572506993827,8cyclictest2620710-21sshd22:39:393
2572506993826,10cyclictest37-21ksoftirqd/321:50:003
2572506993826,10cyclictest37-21ksoftirqd/321:50:003
2572506993825,10cyclictest37-21ksoftirqd/321:10:013
2572506993825,10cyclictest37-21ksoftirqd/321:10:013
2572506993825,10cyclictest37-21ksoftirqd/300:32:213
2572506993823,11cyclictest275-21dbus-daemon20:05:003
257250299385,5cyclictest131rcu_preempt19:46:392
2572502993825,10cyclictest1-21systemd19:35:002
2572506993727,8cyclictest37-21ksoftirqd/300:38:533
2572506993727,6cyclictest309-21in:imuxsock19:15:003
2572506993726,8cyclictest37-21ksoftirqd/321:30:003
2572506993726,8cyclictest37-21ksoftirqd/321:30:003
257250699371,8cyclictest2574153-21grep19:40:023
257250699371,33cyclictest37-21ksoftirqd/323:27:453
2572502993726,9cyclictest30-21ksoftirqd/220:15:022
2572502993723,11cyclictest1-21systemd20:25:012
257249599371,32cyclictest0-21swapper/019:25:260
2572506993628,5cyclictest2654328-21sh23:50:023
2572506993628,5cyclictest2654328-21sh23:50:023
257250699362,5cyclictest131rcu_preempt22:12:103
257250699361,3cyclictest131rcu_preempt20:10:013
257250299362,3cyclictest131rcu_preempt19:25:002
257250299361,4cyclictest131rcu_preempt19:50:242
2572506993525,7cyclictest2577897-21sshd20:52:393
2572506993523,9cyclictest37-21ksoftirqd/321:53:253
257250299354,5cyclictest131rcu_preempt20:31:502
2572502993527,5cyclictest2576904-21kworker/2:220:45:002
257249599353,3cyclictest131rcu_preempt20:45:000
257250699344,4cyclictest131rcu_preempt00:12:323
2572506993421,10cyclictest37-21ksoftirqd/321:05:023
2572502993426,5cyclictest2561114-21kworker/2:320:25:242
257249599341,29cyclictest0-21swapper/020:26:480
2572506993321,9cyclictest37-21ksoftirqd/320:41:413
2572506993319,11cyclictest37-21ksoftirqd/320:35:013
257250299323,4cyclictest131rcu_preempt19:10:022
2572506993122,6cyclictest37-21ksoftirqd/320:20:023
257250699271,5cyclictest131rcu_preempt19:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional