You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-29 - 14:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sun Mar 29, 2026 00:45:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2835667998581,2cyclictest0-21swapper/020:20:020
2835674996922,41cyclictest275-21dbus-daemon23:45:002
2835667996932,3cyclictest0-21swapper/023:28:180
2835667996927,37cyclictest2850510-21modprobe21:29:590
2835667996926,38cyclictest2919174-21sh00:10:000
2835679996859,6cyclictest37-21ksoftirqd/319:20:013
2835679996827,37cyclictest2866538-21sh22:07:003
2835667996827,11cyclictest2893375-21munin-run23:10:000
2835674996725,38cyclictest2894836-21sh23:13:112
2835679996626,36cyclictest2913858-21sh23:57:223
283567999661,10cyclictest2930264-21sshd00:35:233
2835674996524,37cyclictest2855222-21sh21:40:282
2835667996552,6cyclictest0-21swapper/022:35:020
2835679996423,36cyclictest2924032-21sh00:20:533
2835679996423,36cyclictest2924032-21sh00:20:523
283567499644,21cyclictest2838907-21cat20:10:032
2835667996415,42cyclictest2910795-21ls23:50:020
283567999632,56cyclictest2880642-21ssh22:40:013
283567999632,15cyclictest2836405-21munin-run19:25:003
2835674996325,34cyclictest30-21ksoftirqd/221:00:012
2835674996321,38cyclictest2851181-21sh21:31:142
2835679996230,28cyclictest3467602-21systemd-journal21:35:003
283567999621,9cyclictest2903375-21ssh23:33:013
283567999621,9cyclictest2885492-21ssh22:51:013
283567999621,8cyclictest2878126-21ssh22:34:143
283567999621,6cyclictest2929152-21ssh00:32:583
283567999621,6cyclictest2929152-21ssh00:32:573
283567999621,12cyclictest2847301-21ssh21:22:123
2835674996228,30cyclictest30-21ksoftirqd/221:56:292
2835674996214,10cyclictest3467602-21systemd-journal21:25:002
2835667996213,18cyclictest2868191-21sh22:10:420
2835667996210,47cyclictest2891277-21ls23:05:010
283567999612,5cyclictest2888893-21ssh22:59:313
283567999612,5cyclictest2888893-21ssh22:59:303
283567999611,9cyclictest2872702-21ssh22:21:143
283567999611,8cyclictest2916423-21ssh00:03:243
283567999611,6cyclictest2910254-21ssh23:49:073
283567999611,4cyclictest2891961-21sshd23:06:213
2835674996127,31cyclictest30-21ksoftirqd/222:40:022
2835674996126,30cyclictest30-21ksoftirqd/223:21:082
2835674996125,32cyclictest2914896-40mandb00:00:022
2835674996124,31cyclictest1-21systemd20:35:002
2835674996120,9cyclictest30-21ksoftirqd/223:35:092
2835667996116,38cyclictest2852616-21modprobe21:35:000
2835667996114,13cyclictest2896418-21sh23:16:440
2835679996027,29cyclictest2876069-21sh22:29:323
283567999602,4cyclictest3467602-21systemd-journal21:59:173
2835679996021,36cyclictest37-21ksoftirqd/300:12:243
2835679996021,36cyclictest37-21ksoftirqd/300:12:233
283567999601,9cyclictest2883562-21ssh22:46:393
283567999601,9cyclictest2883562-21ssh22:46:383
283567999601,8cyclictest2889853-21ssh23:01:223
283567999601,7cyclictest2898208-21ssh23:20:403
283567999601,7cyclictest2860925-21ssh21:54:163
283567999601,5cyclictest3467602-21systemd-journal21:35:573
2835674996025,31cyclictest2900293-21sh23:25:352
2835674996023,34cyclictest2870123-21sh22:15:142
2835674996023,33cyclictest30-21ksoftirqd/200:20:382
2835674996023,33cyclictest30-21ksoftirqd/200:20:382
2835674996023,32cyclictest30-21ksoftirqd/200:38:482
283567499602,13cyclictest2841830-21cron21:10:002
2835674996017,10cyclictest309-21in:imuxsock22:48:082
2835674996017,10cyclictest309-21in:imuxsock22:48:072
2835674996015,11cyclictest1-21systemd19:50:002
283566799609,9cyclictest2863350-21modprobe22:00:000
2835667996028,15cyclictest2882540-21ssh22:44:340
2835667996019,35cyclictest2914893-40ogrotate23:59:590
283567999592,6cyclictest2868110-21ssh22:10:293
283567999592,5cyclictest2905026-21ssh23:36:463
283567999592,17cyclictest2895847-21ssh23:15:133
283567999591,8cyclictest2926159-21ssh00:25:423
283567999591,7cyclictest2917507-21ssh00:05:343
283567999591,7cyclictest2900337-21ssh23:25:423
283567999591,7cyclictest2849765-21ssh21:28:083
283567999591,6cyclictest2907924-21ssh23:43:433
283567999591,6cyclictest2894120-21ssh23:11:183
283567999591,6cyclictest2855414-21ssh21:40:563
283567999591,5cyclictest2922173-21ssh00:16:373
283567999591,5cyclictest2882649-21ssh22:44:493
283567999591,15cyclictest2841862-21systemd-run21:10:013
2835674995922,11cyclictest30-21ksoftirqd/222:00:452
2835674995921,8cyclictest30-21ksoftirqd/223:30:162
283567499592,18cyclictest2889290-21cat23:00:032
283567499592,18cyclictest2889290-21cat23:00:022
2835674995917,6cyclictest0-21swapper/220:20:002
2835667995933,21cyclictest2859125-21ssh21:50:010
2835667995928,15cyclictest2845808-21ssh21:19:060
2835667995916,11cyclictest2927795-21latency_hist00:30:000
2835667995912,11cyclictest0-21swapper/022:08:330
283567999582,6cyclictest2839497-21sshd20:22:363
283567999582,4cyclictest2911356-21ssh23:51:283
283567999582,4cyclictest2870719-21ssh22:16:503
283567999582,4cyclictest2863874-21ssh22:00:403
283567999582,4cyclictest2842643-21ssh21:11:213
283567999581,6cyclictest2857350-21ssh21:45:293
2835674995822,32cyclictest2845528-21sh21:18:152
2835674995819,4cyclictest2897154-21sh23:18:372
2835674995817,4cyclictest2893565-21latency_hist23:10:032
2835667995811,16cyclictest2838294-21sed20:00:030
2835679995739,14cyclictest3467602-21systemd-journal20:50:013
283567999572,17cyclictest2837009-21wc19:35:013
283567999572,16cyclictest2839251-21munin-run20:20:013
283567999571,14cyclictest2845131-21ssh21:17:183
2835674995749,5cyclictest2838756-21kworker/2:021:38:562
2835674995724,30cyclictest309-21in:imuxsock22:10:002
2835674995717,10cyclictest30-21ksoftirqd/200:26:002
283566799579,18cyclictest2854310-21rm21:38:470
2835667995713,40cyclictest2872381-21ssh22:20:260
283567999561,9cyclictest2841023-21sshd20:50:203
283567999561,9cyclictest2841023-21sshd20:50:193
283567999561,5cyclictest2839758-21sshd20:29:203
283567999561,12cyclictest2840025-21munin-run20:35:003
2835674995617,11cyclictest30-21ksoftirqd/222:31:022
283567999553,5cyclictest2836637-21sshd19:28:193
283567999552,6cyclictest2837946-21munin-run19:55:003
283567499558,13cyclictest1-21systemd21:50:012
2835674995523,28cyclictest30-21ksoftirqd/221:12:232
283567499552,17cyclictest2910703-21ssh23:50:022
2835674995521,11cyclictest30-21ksoftirqd/221:05:012
2835674995518,5cyclictest2922659-21sh00:17:552
283566799557,24cyclictest284836650rm21:25:010
2835667995527,13cyclictest2916488-21ssh00:03:320
2835667995519,31cyclictest16850irq/134-800000022:29:390
283567999542,4cyclictest2838960-21sshd20:13:193
283567999541,13cyclictest2841321-21munin-run21:00:013
283567499544,10cyclictest2837947-21grep19:55:002
2835674995428,22cyclictest30-21ksoftirqd/221:53:002
2835667995426,12cyclictest2870122-21ssh22:15:150
2835667995426,12cyclictest2860442-21ssh21:52:580
2835667995424,15cyclictest2885670-21ssh22:51:410
2835667995415,6cyclictest12-21ksoftirqd/022:48:430
2835667995415,6cyclictest12-21ksoftirqd/022:48:420
2835667995411,38cyclictest2879302-21ssh22:36:390
283567999532,5cyclictest2838170-21sshd19:57:193
283567999532,14cyclictest2835867-21modprobe19:15:013
283567999531,4cyclictest2840538-21sshd20:44:183
283567499535,20cyclictest2838205-21munin-run20:00:012
2835674995326,23cyclictest30-21ksoftirqd/200:32:282
2835674995326,23cyclictest30-21ksoftirqd/200:32:272
2835674995323,25cyclictest30-21ksoftirqd/200:00:292
2835674995316,5cyclictest2891401-21sort23:05:012
283566799536,14cyclictest2905046-21ssh23:36:480
2835667995344,6cyclictest12-21ksoftirqd/023:35:010
2835667995327,10cyclictest2864969-21ssh22:03:410
2835667995326,12cyclictest2921061-21ssh00:14:200
2835667995326,12cyclictest2921061-21ssh00:14:190
2835667995325,13cyclictest2899198-21ssh23:23:190
2835667995321,16cyclictest2844057-21ssh21:15:000
283567999522,6cyclictest275-21dbus-daemon20:40:023
283567999522,6cyclictest275-21dbus-daemon20:40:023
283567999521,4cyclictest2841605-21munin-run21:05:003
2835674995226,22cyclictest30-21ksoftirqd/200:09:462
2835674995219,6cyclictest2876270-21latency_hist22:30:032
2835670995210,29cyclictest3467602-21systemd-journal00:20:001
283567999511,5cyclictest364-21snmpd19:46:133
283567999511,5cyclictest2837124-21sshd19:35:203
283567999511,5cyclictest2837124-21sshd19:35:193
283567499518,14cyclictest30-21ksoftirqd/222:52:422
2835674995125,23cyclictest30-21ksoftirqd/223:54:432
2835674995112,7cyclictest2862004-1kworker/u9:222:15:012
2835670995130,12cyclictest2871997-21cron22:20:011
2835667995122,14cyclictest2906713-21ssh23:40:280
2835667995116,31cyclictest1-21systemd23:54:220
283567999502,4cyclictest364-21snmpd20:00:573
283567999502,4cyclictest2838734-21systemd-run20:10:003
2835674995023,23cyclictest2839168-21latency_hist20:15:012
283567499502,17cyclictest2921327-21cron00:15:012
283567499502,17cyclictest2921327-21cron00:15:012
2835674995020,6cyclictest2836954-21latency_hist19:35:022
2835674995016,9cyclictest2874253-21date22:25:022
2835674995013,9cyclictest3467602-21systemd-journal20:20:032
2835670995040,6cyclictest2836406-21grep19:25:001
2835670995036,10cyclictest23-21ksoftirqd/100:00:021
283567099503,4cyclictest131rcu_preempt21:23:481
2835667995030,17cyclictest16850irq/134-800000000:24:020
2835667995030,17cyclictest16850irq/134-800000000:24:020
2835667995023,11cyclictest2931375-21ssh00:38:130
2835667995016,4cyclictest2922312-21ssh00:17:050
2835667995015,4cyclictest2894090-21ssh23:11:140
2835667995011,10cyclictest0-21swapper/000:30:590
2835667995011,10cyclictest0-21swapper/000:30:590
283567999492,13cyclictest2837442-21munin-plugin-st19:44:593
283567499493,16cyclictest2836136-21munin-run19:20:002
283567499493,12cyclictest2839910-21latency_hist20:30:012
2835674994911,33cyclictest0-21swapper/221:30:002
2835670994930,15cyclictest0-21swapper/119:15:021
2835670994929,14cyclictest0-21swapper/123:40:001
283567099491,4cyclictest131rcu_preempt22:28:301
283566799499,36cyclictest0-21swapper/022:56:230
283566799499,36cyclictest0-21swapper/022:56:220
2835667994930,14cyclictest1-21systemd19:35:000
2835674994819,5cyclictest2836550-21latency_hist19:25:032
2835674994814,7cyclictest30-21ksoftirqd/220:50:002
2835670994832,12cyclictest275-21dbus-daemon20:00:011
2835670994832,11cyclictest275-21dbus-daemon23:55:011
2835667994830,13cyclictest3467602-21systemd-journal20:35:000
2835667994818,19cyclictest16850irq/134-800000021:41:320
2835674994739,5cyclictest2838756-21kworker/2:020:40:022
2835674994739,5cyclictest2838756-21kworker/2:020:40:012
283567499472,17cyclictest364-21snmpd22:43:122
283567499471,15cyclictest2837467-21ls19:45:012
2835670994730,13cyclictest3467602-21systemd-journal23:25:001
2835670994729,14cyclictest3467602-21systemd-journal20:25:001
2835670994728,14cyclictest275-21dbus-daemon22:35:011
2835670994722,19cyclictest3467602-21systemd-journal00:25:001
2835670994722,19cyclictest3467602-21systemd-journal00:25:001
2835667994740,4cyclictest16950irq/224-800000020:15:010
2835674994632,11cyclictest3467602-21systemd-journal19:36:182
2835674994632,11cyclictest3467602-21systemd-journal19:36:182
283567499463,14cyclictest1-21systemd19:30:012
283567099464,3cyclictest131rcu_preempt23:16:231
2835670994629,13cyclictest3467602-21systemd-journal21:50:001
2835670994624,14cyclictest2863381-21ssh22:00:001
2835667994639,4cyclictest16950irq/224-800000020:44:590
283567499452,14cyclictest2838576-21cat20:05:012
2835670994536,7cyclictest23-21ksoftirqd/122:54:091
2835670994534,8cyclictest23-21ksoftirqd/100:27:161
2835670994531,11cyclictest23-21ksoftirqd/119:30:021
2835670994530,9cyclictest2836117-21cron19:20:001
2835670994530,12cyclictest23-21ksoftirqd/121:12:481
2835670994529,13cyclictest23-21ksoftirqd/122:43:191
2835670994529,13cyclictest1-21systemd23:35:011
283567099451,35cyclictest221rcuc/121:55:021
2835674994431,9cyclictest1-21systemd20:45:012
2835670994428,12cyclictest275-21dbus-daemon22:15:001
2835670994427,13cyclictest275-21dbus-daemon20:50:001
2835670994419,18cyclictest275-21dbus-daemon23:05:011
283567099441,3cyclictest131rcu_preempt23:29:541
2835667994417,23cyclictest2836159-21modprobe19:19:590
283567099434,30cyclictest2921341-21munin-run00:15:011
283567099434,30cyclictest2921341-21munin-run00:15:001
2835670994332,8cyclictest23-21ksoftirqd/121:40:031
2835670994326,13cyclictest275-21dbus-daemon00:40:011
2835667994335,5cyclictest1895574-1kworker/0:1H19:14:050
2835667994326,11cyclictest16950irq/224-800000021:10:000
283567999423,17cyclictest2835760-21sort19:10:013
2835670994226,11cyclictest3467602-21systemd-journal20:15:001
2835670994226,11cyclictest23-21ksoftirqd/122:01:391
2835670994224,12cyclictest3467602-21systemd-journal21:00:001
2835670994221,14cyclictest2919195-21systemd-run00:10:011
2835667994235,4cyclictest16950irq/224-800000019:55:010
2835667994230,10cyclictest12-21ksoftirqd/020:25:020
2835667994228,10cyclictest275-21dbus-daemon19:40:020
2835667994228,10cyclictest275-21dbus-daemon19:40:010
2835667994224,13cyclictest2841048-21cron20:55:010
2835667994224,13cyclictest2841048-21cron20:55:000
2835667994224,13cyclictest2840318-21munin-run20:40:020
2835667994224,13cyclictest2840318-21munin-run20:40:010
2835667994216,23cyclictest2835610-21latency_hist19:10:010
2835667994215,22cyclictest2839799-21latency_hist20:30:010
2835670994129,7cyclictest2893454-21latency_hist23:10:021
2835670994125,10cyclictest309-21in:imuxsock22:40:011
2835667994134,4cyclictest16950irq/224-800000020:57:180
2835667994134,4cyclictest16950irq/224-800000019:45:010
2835667994127,11cyclictest3467602-21systemd-journal20:48:330
2835674994029,8cyclictest2841120-21cat20:55:022
2835674994029,8cyclictest2841120-21cat20:55:012
283567099406,25cyclictest1-21systemd21:20:001
2835670994031,6cyclictest2915192-21ssh00:00:061
2835670994031,6cyclictest23-21ksoftirqd/120:15:021
2835670994028,8cyclictest3467602-21systemd-journal21:34:591
2835670994027,8cyclictest2840553-21modprobe20:44:591
2835670994027,7cyclictest23-21ksoftirqd/119:45:001
2835670994027,11cyclictest23-21ksoftirqd/121:41:361
2835670994025,11cyclictest275-21dbus-daemon19:50:011
283566799401,12cyclictest2836643-21sshd19:29:200
283567099393,7cyclictest23-21ksoftirqd/122:08:391
2835670993927,7cyclictest2895535-21sh23:14:591
2835670993926,11cyclictest23-21ksoftirqd/123:44:351
2835670993926,10cyclictest23-21ksoftirqd/122:20:341
2835670993925,10cyclictest3467602-21systemd-journal20:32:031
2835670993924,11cyclictest23-21ksoftirqd/122:45:271
2835670993924,11cyclictest23-21ksoftirqd/122:45:261
2835670993923,10cyclictest310-21in:imklog21:25:491
2835670993916,16cyclictest275-21dbus-daemon21:05:011
283566799398,7cyclictest0-21swapper/019:46:130
2835670993829,5cyclictest2910787-21latency_hist23:50:011
283566799387,10cyclictest364-21snmpd20:03:060
283566799382,32cyclictest2841557-21sshd21:00:190
2835670993727,7cyclictest2840241-21sshd20:35:191
2835670993727,7cyclictest2840241-21sshd20:35:181
2835670993726,7cyclictest23-21ksoftirqd/122:59:531
2835670993726,7cyclictest23-21ksoftirqd/122:59:521
2835670993716,15cyclictest275-21dbus-daemon20:55:011
2835670993716,15cyclictest275-21dbus-daemon20:55:001
283567099364,3cyclictest131rcu_preempt20:10:001
2835670993626,6cyclictest2837150-21cron19:40:011
2835670993626,6cyclictest2837150-21cron19:40:001
2835670993624,7cyclictest2837007-21cat19:35:021
283567099355,21cyclictest2841855-21cut21:09:591
2835670993523,10cyclictest23-21ksoftirqd/100:33:221
2835670993523,10cyclictest23-21ksoftirqd/100:33:211
283566799353,28cyclictest0-21swapper/020:05:570
2835674993421,9cyclictest1-21systemd19:15:002
2835670993426,6cyclictest23-21ksoftirqd/120:27:191
2835667993421,10cyclictest2836384-21sshd19:23:200
2835670993323,7cyclictest23-21ksoftirqd/119:10:011
2835670993221,8cyclictest2837922-21sshd19:52:341
283567099312,3cyclictest131rcu_preempt20:05:021
2835674992517,5cyclictest2835812-21cat19:10:032
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional