You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-12 - 00:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sat Jul 11, 2026 12:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2482164998631,12cyclictest2556205-21ssh11:32:500
2482164998529,13cyclictest2539790-21ssh10:58:200
2482164998529,13cyclictest2539790-21ssh10:58:200
248216499851,39cyclictest2567489-21ssh11:56:250
248216499851,39cyclictest2567489-21ssh11:56:240
2482164998427,12cyclictest2561696-21ssh11:44:330
2482164998226,14cyclictest2494450-21ssh09:22:480
2482164998141,36cyclictest16850irq/134-800000012:14:590
2482164998127,13cyclictest2520332-21ssh10:17:320
2482164997930,13cyclictest2551954-21ssh11:23:590
2482164997928,11cyclictest2565504-21ssh11:52:210
2482164997915,33cyclictest12-21ksoftirqd/010:42:040
248216499791,39cyclictest2489408-21ssh09:12:020
248216499791,39cyclictest2489408-21ssh09:12:020
248216499781,39cyclictest2582562-21ssh12:28:200
2482164997730,11cyclictest2505367-21ssh09:45:410
2482164997730,11cyclictest2505367-21ssh09:45:400
2482164997729,12cyclictest2557557-21ssh11:35:260
2482164997726,13cyclictest2537475-21ssh10:53:300
2482164997725,15cyclictest2548254-21ssh11:15:530
2482164997725,15cyclictest2548254-21ssh11:15:520
248216499772,40cyclictest2511338-21ssh09:58:350
248216499772,40cyclictest2511338-21ssh09:58:340
2482174997618,15cyclictest2533362-21latency_hist10:44:592
2482164997621,14cyclictest2586556-21ssh12:36:390
248216999753,4cyclictest131rcu_preempt12:05:011
2482164997427,8cyclictest309-21in:imuxsock11:10:010
2482177997338,31cyclictest275-21dbus-daemon08:00:023
2482174997321,12cyclictest30-21ksoftirqd/212:10:022
2482174997321,12cyclictest30-21ksoftirqd/212:10:012
248217799721,14cyclictest2534174-21sshd10:46:243
2482164997136,3cyclictest16950irq/224-800000011:45:540
248217799702,16cyclictest2538535-21sshd10:55:253
248217799702,16cyclictest2538535-21sshd10:55:243
2482164997044,21cyclictest16850irq/134-800000012:35:020
2482164997023,12cyclictest2508848-21ssh09:53:130
2482164996926,7cyclictest2498615-21ssh09:31:320
248217799682,12cyclictest2515111-21sshd10:06:253
248217799682,12cyclictest2515111-21sshd10:06:243
2482174996846,15cyclictest3467602-21systemd-journal09:20:002
2482164996818,12cyclictest2523146-21ssh10:23:300
2482177996724,39cyclictest37-21ksoftirqd/312:30:023
2482169996723,40cyclictest1-21systemd09:50:011
2482169996723,40cyclictest1-21systemd09:50:011
248216999672,13cyclictest2519075-21munin-run10:15:001
2482177996636,26cyclictest3467602-21systemd-journal07:20:033
2482174996619,12cyclictest30-21ksoftirqd/212:02:512
2482164996662,2cyclictest0-21swapper/010:05:020
2482164996662,2cyclictest0-21swapper/010:05:020
2482164996624,14cyclictest2546798-21ssh11:13:050
2482177996524,37cyclictest2557958-21sh11:36:293
248217799652,15cyclictest2533384-21munin-run10:45:013
248216499655,24cyclictest3467602-21systemd-journal10:40:000
248216499655,24cyclictest3467602-21systemd-journal10:40:000
2482164996525,10cyclictest12-21ksoftirqd/007:58:140
2482177996435,25cyclictest2560327-21sh11:41:213
248217799642,14cyclictest364-21snmpd11:12:173
248217799642,14cyclictest2491213-21ssh09:15:533
248217799642,11cyclictest2486504-21sshd08:34:143
248217799641,17cyclictest2497742-21ssh09:30:003
248217799641,14cyclictest2506228-21ssh09:47:383
248217799641,14cyclictest2506228-21ssh09:47:383
2482169996424,36cyclictest2502515-21latency_hist09:40:021
2482169996424,36cyclictest2502515-21latency_hist09:40:021
2482169996416,43cyclictest3467602-21systemd-journal11:55:241
2482169996416,43cyclictest3467602-21systemd-journal11:55:231
2482164996425,20cyclictest2552682-21ssh11:25:020
2482177996341,18cyclictest3467602-21systemd-journal10:40:003
2482177996341,18cyclictest3467602-21systemd-journal10:40:003
2482177996324,35cyclictest2522472-21sh10:21:563
248217799632,14cyclictest2486284-21munin-run08:30:013
248217799632,14cyclictest2486284-21munin-run08:30:013
2482177996318,40cyclictest2500107-21sh09:35:023
2482177996314,12cyclictest2485443-21sshd08:11:153
248217799631,15cyclictest2482656-21/usr/sbin/munin07:20:023
2482169996325,34cyclictest2493274-21sh09:20:081
248216999631,15cyclictest2484149-21sshd07:45:141
248216499635,18cyclictest2519212-21latency_hist10:15:010
2482177996218,39cyclictest275-21dbus-daemon11:20:013
2482177996218,39cyclictest275-21dbus-daemon11:20:013
2482177996217,40cyclictest2486967-21cat08:40:023
248217799621,13cyclictest2554335-21ssh11:29:003
2482177996210,4cyclictest0-21swapper/312:13:523
248216999622,14cyclictest48-21kcompactd011:30:031
248216999622,14cyclictest48-21kcompactd010:45:321
248216999622,13cyclictest2536080-21ssh10:50:151
2482164996225,33cyclictest2487821-21/usr/sbin/munin09:00:010
248216499622,11cyclictest2493089-21date09:20:020
248216499621,58cyclictest0-21swapper/007:40:020
2482177996123,32cyclictest37-21ksoftirqd/312:25:003
248217799612,14cyclictest48-21kcompactd009:55:263
248217799612,14cyclictest48-21kcompactd009:55:253
248217799612,13cyclictest48-21kcompactd009:20:193
248217799611,12cyclictest364-21snmpd12:00:253
2482169996123,33cyclictest2564981-21sh11:51:061
2482169996122,35cyclictest2561914-21sh11:45:001
248216999612,15cyclictest2486020-21munin-run08:25:001
248216999612,14cyclictest2482369-21munin-run07:15:011
248216999611,14cyclictest2579445-21ssh12:21:391
248217799602,13cyclictest2502554-21true09:40:003
248217799602,13cyclictest2502554-21true09:40:003
248217799602,12cyclictest364-21snmpd09:42:243
248217799601,13cyclictest2484936-21sshd08:00:263
2482174996014,11cyclictest30-21ksoftirqd/210:40:072
2482174996014,11cyclictest30-21ksoftirqd/210:40:062
2482174996013,9cyclictest30-21ksoftirqd/209:05:012
2482174996013,9cyclictest30-21ksoftirqd/209:05:012
2482169996028,28cyclictest1-21systemd08:10:001
248216999602,15cyclictest2584778-21ssh12:33:021
248216999602,14cyclictest364-21snmpd11:10:021
248216999601,16cyclictest2488372-21munin-run09:10:011
248216999601,14cyclictest2541732-21sshd11:02:231
248216999601,14cyclictest2490654-21munin-run09:15:011
248216999601,14cyclictest2490654-21munin-run09:15:001
2482169996011,4cyclictest0-21swapper/109:19:591
248217799591,13cyclictest2536772-21ssh10:51:513
248217499594,20cyclictest2548015-21ssh11:15:212
248217499594,20cyclictest2548015-21ssh11:15:212
2482174995913,11cyclictest30-21ksoftirqd/212:34:042
2482174995912,10cyclictest30-21ksoftirqd/212:21:542
248216999592,15cyclictest3467602-21systemd-journal10:08:261
248216999592,15cyclictest3467602-21systemd-journal10:08:261
248216999592,13cyclictest364-21snmpd11:40:021
2482169995919,35cyclictest2544286-21sh11:07:451
2482169995915,9cyclictest2503728-1kworker/u9:209:43:091
248217799582,17cyclictest2526182-21sh10:30:013
2482177995820,34cyclictest2577480-21sh12:17:343
2482177995818,35cyclictest1-21systemd11:35:023
248217799581,16cyclictest2568967-21sh11:59:493
248217799581,16cyclictest2568967-21sh11:59:483
248216999582,15cyclictest2482619-21sendmail07:20:001
248216999582,14cyclictest364-21snmpd10:59:211
248216999582,14cyclictest364-21snmpd10:59:201
248216999582,14cyclictest2497738-21munin-run09:30:001
248216999582,13cyclictest364-21snmpd07:26:071
2482169995821,34cyclictest23-21ksoftirqd/110:30:511
248216999582,12cyclictest3468525-21sendmail-mta10:39:351
248216999582,12cyclictest3468525-21sendmail-mta10:39:341
2482169995815,38cyclictest0-21swapper/109:52:441
248217799572,15cyclictest2512313-21ssh10:00:163
248217799572,15cyclictest2483696-21munin-run07:40:013
248217799572,14cyclictest364-21snmpd08:53:163
2482177995718,34cyclictest2488345-21diskmemload11:06:493
248217799571,15cyclictest2516986-21ssh10:10:123
248217799571,14cyclictest2550458-21sshd11:20:253
2482174995719,8cyclictest30-21ksoftirqd/212:28:242
2482174995712,9cyclictest30-21ksoftirqd/210:16:592
2482174995711,10cyclictest30-21ksoftirqd/209:10:152
2482174995711,10cyclictest30-21ksoftirqd/209:10:152
2482169995725,28cyclictest23-21ksoftirqd/110:22:461
248216999572,14cyclictest2563120-21ssh11:47:131
248216999572,14cyclictest2498076-21ssh09:30:151
248216999572,13cyclictest2512607-21ssh10:00:541
2482169995716,5cyclictest2488345-21diskmemload11:17:271
2482169995716,5cyclictest2488345-21diskmemload11:17:261
248216999571,15cyclictest2554731-21ssh11:29:501
248216999571,14cyclictest364-21snmpd09:05:021
248216999571,14cyclictest364-21snmpd09:05:011
2482169995711,3cyclictest0-21swapper/108:48:541
248217799563,13cyclictest2483131-21sshd07:29:153
248217799562,17cyclictest2573563-21ssh12:09:263
248217799562,17cyclictest2573563-21ssh12:09:263
248217799562,14cyclictest2588007-21munin-run12:40:013
248217499569,15cyclictest2526185-21ssh10:30:012
2482174995614,9cyclictest30-21ksoftirqd/211:43:502
2482169995621,31cyclictest310-21in:imklog12:18:591
248216999562,12cyclictest2531373-21ssh10:40:211
2482169995618,33cyclictest2484399-21sshd07:50:141
2482169995615,6cyclictest2552037-21rm11:24:111
248216999561,15cyclictest2526164-21munin-run10:30:001
2482164995626,13cyclictest2504461-21ssh09:44:020
2482164995618,33cyclictest0-21swapper/009:30:000
248217799552,15cyclictest2565339-21ssh11:51:503
248217799552,12cyclictest2483898-21sshd07:40:143
2482177995519,32cyclictest37-21ksoftirqd/311:45:283
2482177995518,33cyclictest2488345-21diskmemload09:52:153
248217799551,17cyclictest2489495-21ssh09:12:263
248217799551,17cyclictest2489495-21ssh09:12:253
248217799551,14cyclictest2584305-21ssh12:31:523
2482174995512,9cyclictest30-21ksoftirqd/209:49:062
2482174995512,9cyclictest30-21ksoftirqd/209:49:052
248216999556,15cyclictest2485739-21latency_hist08:20:001
248216999552,16cyclictest2509899-21ssh09:55:141
248216999552,16cyclictest2509899-21ssh09:55:141
248216999552,15cyclictest2574148-21ssh12:10:221
248216999552,11cyclictest2588009-21systemd-tmpfile12:40:021
248216999551,14cyclictest2581404-21ssh12:25:321
248216999551,13cyclictest364-21snmpd07:58:051
248217799542,12cyclictest2541850-21ssh11:02:383
2482177995417,33cyclictest2520195-21sh10:17:063
2482177995414,9cyclictest37-21ksoftirqd/310:30:593
248217499542,14cyclictest2515791-21ssh10:08:012
248217499542,14cyclictest2515791-21ssh10:08:002
2482174995410,9cyclictest2512654-21ssh10:00:592
2482169995425,24cyclictest2483656-21sendmail07:40:001
248216999542,12cyclictest2483415-21munin-run07:35:011
248216999541,17cyclictest2519706-21ssh10:16:021
2482164995444,6cyclictest16850irq/134-800000012:20:010
248217799532,8cyclictest2487798-21cron09:00:003
2482177995314,34cyclictest0-21swapper/307:55:013
248217499536,21cyclictest2553820-21ssh11:27:472
2482174995336,13cyclictest1-21systemd10:35:002
2482174995313,7cyclictest30-21ksoftirqd/211:46:152
2482174995311,9cyclictest30-21ksoftirqd/211:09:152
248216999532,15cyclictest2486760-21cron08:40:011
248216999531,15cyclictest2483905-21sshd07:41:141
248216999531,13cyclictest2572294-21ssh12:06:391
248216999531,13cyclictest2572294-21ssh12:06:381
248216499531,36cyclictest2525547-21ssh10:28:380
248216499531,24cyclictest131rcu_preempt07:35:020
248217799522,16cyclictest2485746-21munin-plugin-st08:20:003
248217799522,13cyclictest364-21snmpd07:45:403
248217799522,12cyclictest2487369-21grep08:50:023
248216999523,11cyclictest2486582-21latency_hist08:35:011
2482164995225,12cyclictest2527813-21ssh10:33:180
248216499522,36cyclictest2569746-21ssh12:01:030
248217799513,15cyclictest2488152-21latency_hist09:05:033
248217799513,15cyclictest2488152-21latency_hist09:05:023
248217799512,12cyclictest2486033-21unin-run08:25:003
248217799511,8cyclictest3467602-21systemd-journal08:45:003
248217499515,27cyclictest2538902-21kworker/u8:311:00:022
248217499515,27cyclictest2538902-21kworker/u8:311:00:022
248216999513,9cyclictest2487800-21cron09:00:001
2482164995123,13cyclictest2572785-21ssh12:07:400
2482164995123,13cyclictest2572785-21ssh12:07:390
2482164995115,10cyclictest131rcu_preempt08:15:000
248217799501,15cyclictest2482282-21cat07:10:023
2482174995032,12cyclictest3467602-21systemd-journal08:30:002
2482174995032,12cyclictest3467602-21systemd-journal08:30:002
2482174995011,8cyclictest30-21ksoftirqd/211:52:062
2482174995010,8cyclictest2557727-21ssh11:35:482
248216999505,8cyclictest2484958-21modprobe08:05:001
248216999502,12cyclictest275-21dbus-daemon08:15:021
2482164995041,5cyclictest16950irq/224-800000009:10:010
2482164995032,11cyclictest12-21ksoftirqd/010:50:010
248216499503,11cyclictest364-21snmpd12:25:020
2482174994941,5cyclictest2495413-21kworker/2:211:23:452
248216999492,11cyclictest2487561-21munin-run08:55:001
2482169994915,10cyclictest23-21ksoftirqd/107:25:011
2482164994925,20cyclictest2501454-21ssh09:37:310
2482164994925,20cyclictest2501454-21ssh09:37:300
248216499492,41cyclictest1-21systemd08:55:010
2482174994833,12cyclictest2521777-21ssh10:20:172
2482174994832,11cyclictest3467602-21systemd-journal09:10:002
2482174994829,13cyclictest3467602-21systemd-journal08:40:002
2482174994811,6cyclictest0-21swapper/211:58:122
2482174994811,6cyclictest0-21swapper/211:58:112
248216999482,13cyclictest2482272-21latency_hist07:10:021
248216999481,14cyclictest2487118-21wc08:45:021
2482164994834,10cyclictest1-21systemd07:25:000
2482177994729,14cyclictest275-21dbus-daemon08:10:003
248217499479,23cyclictest2475452-21kworker/u8:307:15:022
2482174994732,12cyclictest2518608-21ssh10:13:532
248217499473,16cyclictest2530960-21kworker/u8:411:02:332
2482174994712,11cyclictest30-21ksoftirqd/209:33:582
2482164994733,11cyclictest1-21systemd08:35:010
2482174994630,12cyclictest3467602-21systemd-journal07:20:202
2482174994612,8cyclictest2576387-21ssh12:15:052
248217499461,16cyclictest0-21swapper/211:34:142
248216999461,15cyclictest364-21snmpd08:28:121
248216999461,15cyclictest364-21snmpd08:28:111
248216499465,36cyclictest12-21ksoftirqd/010:00:030
248216499463,37cyclictest2482496-21latency_hist07:15:020
248217799452,12cyclictest3467602-21systemd-journal09:10:003
2482174994537,5cyclictest2486827-21kworker/2:108:55:362
2482174994527,13cyclictest2534410-21ssh10:46:542
2482174994523,17cyclictest2501347-21ssh09:37:172
2482174994523,17cyclictest2501347-21ssh09:37:172
2482164994527,14cyclictest2541526-21ssh11:01:540
2482164994527,14cyclictest2488069-21sshd09:03:140
2482164994527,14cyclictest2488069-21sshd09:03:140
248217799443,9cyclictest275-21dbus-daemon07:35:013
2482174994432,9cyclictest2507782-21ssh09:50:462
2482174994429,12cyclictest2510102-21ssh09:55:402
2482174994429,12cyclictest2510102-21ssh09:55:392
248217499441,13cyclictest0-21swapper/209:30:022
248216499445,9cyclictest131rcu_preempt07:50:000
2482164994430,10cyclictest3467602-21systemd-journal07:20:000
248217499439,5cyclictest0-21swapper/212:37:322
2482174994332,9cyclictest30-21ksoftirqd/208:15:012
2482174994330,9cyclictest275-21dbus-daemon08:45:012
2482174994326,13cyclictest2575781-21ssh12:14:082
2482174994325,14cyclictest2546793-21ssh11:13:052
248217499432,15cyclictest2502785-21ssh09:40:032
2482177994211,5cyclictest2482493-21latency_hist07:15:023
2482174994231,9cyclictest2494767-21ssh09:23:402
2482174994231,7cyclictest2483501-21cat07:35:022
2482174994226,13cyclictest2536187-21ssh10:50:292
2482174994126,11cyclictest2482652-21systemd-run07:20:012
2482174994126,10cyclictest2487329-21grep08:50:012
248216499412,13cyclictest364-21snmpd08:25:200
248216499412,13cyclictest364-21snmpd08:25:200
2482174994030,7cyclictest30-21ksoftirqd/208:05:002
2482174994023,12cyclictest275-21dbus-daemon07:40:002
2482174994017,15cyclictest30-21ksoftirqd/208:00:002
2482164994028,8cyclictest12-21ksoftirqd/008:40:000
2482164994022,14cyclictest275-21dbus-daemon08:20:010
248216499401,36cyclictest0-21swapper/007:45:010
248217499395,7cyclictest30-21ksoftirqd/207:40:022
2482174993924,11cyclictest2484184-21munin-run07:50:012
2482174993828,7cyclictest30-21ksoftirqd/208:55:022
2482174993825,10cyclictest2485996-21sshd08:23:142
248216499385,21cyclictest3467602-21systemd-journal08:05:000
2482164993831,4cyclictest0-21swapper/007:10:010
2482174993728,5cyclictest2484435-21munin-plugin-st07:55:002
2482164993733,2cyclictest0-21swapper/008:20:040
2482164993728,6cyclictest2487036-21modprobe08:43:140
2482164993728,6cyclictest12-21ksoftirqd/008:10:020
2482174993622,11cyclictest2485700-21cron08:17:002
248216499365,23cyclictest2487308-21munin-run08:50:000
2482164993629,4cyclictest16950irq/224-800000007:55:000
2482174993523,9cyclictest30-21ksoftirqd/208:10:012
2482174993424,7cyclictest2483148-21munin-run07:30:002
2482174993420,11cyclictest2486531-21systemd-run08:35:002
248216499331,29cyclictest0-21swapper/007:25:290
2482174993021,7cyclictest30-21ksoftirqd/207:10:032
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional