You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-11 - 05:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sat Apr 11, 2026 00:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2630162999934,24cyclictest131rcu_preempt19:15:033
2630158997232,37cyclictest2716311-21sh00:14:482
2630158996926,39cyclictest30-21ksoftirqd/223:55:412
2630158996926,39cyclictest30-21ksoftirqd/223:55:402
2630158996832,32cyclictest2706335-21sh23:51:142
2630158996830,32cyclictest2704583-21sh23:47:172
2630158996830,32cyclictest2704583-21sh23:47:162
2630158996822,41cyclictest2636326-21diskmemload23:06:122
2630158996727,36cyclictest2695425-21sh23:25:572
2630158996727,36cyclictest2695425-21sh23:25:572
263015899662,18cyclictest2672320-21ssh22:32:472
263015899661,14cyclictest2722511-21ssh00:29:042
263015899661,14cyclictest2722511-21ssh00:29:042
263015899651,14cyclictest2678837-21ssh22:47:432
263015899651,14cyclictest2678837-21ssh22:47:432
2630158996428,31cyclictest3467602-21systemd-journal19:40:002
2630158996426,34cyclictest2634578-21sed20:35:012
2630158996424,36cyclictest2676638-21sh22:42:432
263015899642,13cyclictest364-21snmpd21:56:172
263015899641,14cyclictest2713928-21ssh00:09:142
263015899641,12cyclictest364-21snmpd20:25:452
2630152996427,32cyclictest12-21ksoftirqd/022:13:330
2630158996324,35cyclictest2636326-21diskmemload22:15:262
263015899632,19cyclictest2636917-21ssh21:11:002
263015899632,13cyclictest2662455-21munin-run22:10:012
2630158996321,39cyclictest30-21ksoftirqd/200:31:212
263015899631,9cyclictest2653143-21ssh21:48:352
263015899631,7cyclictest2718409-21ssh00:19:342
263015299632,10cyclictest2635813-21tr21:00:000
2630158996223,34cyclictest2636326-21diskmemload21:16:432
263015899622,17cyclictest2648486-21ssh21:37:502
263015899622,14cyclictest2643625-21sh21:26:272
263015899621,9cyclictest2726543-21ssh00:38:132
263015899621,9cyclictest2726543-21ssh00:38:132
263015899621,9cyclictest2685692-21ssh23:03:432
263015899621,8cyclictest2683161-21ssh22:57:522
263015899621,8cyclictest2650193-21ssh21:41:352
263015899621,8cyclictest2645555-21ssh21:30:522
263015899621,8cyclictest2645555-21ssh21:30:512
263015899621,7cyclictest2719254-21ssh00:21:092
263015899621,7cyclictest2669639-21ssh22:26:222
2630158996217,41cyclictest1-21systemd21:00:002
263015899621,13cyclictest2691791-21sshd23:17:462
2630158996130,28cyclictest30-21ksoftirqd/219:35:022
263015899612,8cyclictest2659431-21ssh22:02:572
263015899612,15cyclictest2635518-21sshd20:53:302
263015899612,15cyclictest2635518-21sshd20:53:302
263015899611,9cyclictest2693975-21ssh23:22:482
263015899611,8cyclictest2688733-21ssh23:10:312
263015899611,8cyclictest2642432-21ssh21:23:592
263015899611,14cyclictest2674479-21sh22:37:442
263015899611,10cyclictest2654829-21ssh21:52:192
263015899611,10cyclictest2654829-21ssh21:52:192
263015899601,9cyclictest2698946-21ssh23:34:312
263015899601,7cyclictest2701946-21ssh23:41:052
263015899601,7cyclictest2699633-21ssh23:35:392
263015899601,7cyclictest2680652-21ssh22:51:552
263015899601,7cyclictest2664380-21ssh22:14:372
263015899601,6cyclictest2710251-21ssh00:00:132
263015899582,15cyclictest2632956-21munin-run20:05:002
263015899581,8cyclictest2667146-21ssh22:20:352
2630152995827,15cyclictest2694033-21ssh23:22:530
2630152995824,4cyclictest27055611ssh23:49:560
2630152995824,4cyclictest27055611ssh23:49:560
263016299577,6cyclictest131rcu_preempt00:08:143
2630162995738,14cyclictest37-21ksoftirqd/320:15:013
263015899572,6cyclictest2631932-21unin-run19:45:012
263015899571,11cyclictest2630355-21munin-run19:15:012
263015899571,11cyclictest2630355-21munin-run19:15:012
2630162995638,14cyclictest1-21systemd00:00:013
2630162995638,14cyclictest1-21systemd00:00:013
2630162995624,8cyclictest37-21ksoftirqd/321:20:003
263015899562,4cyclictest2635266-21sshd20:48:302
263015899561,12cyclictest364-21snmpd19:53:222
263015899561,11cyclictest2633432-21sshd20:12:302
2630152995626,24cyclictest12-21ksoftirqd/021:27:540
2630152995546,5cyclictest16950irq/224-800000020:20:010
2630152995539,12cyclictest1-21systemd23:15:000
2630152995527,13cyclictest2684307-21ssh23:00:060
263015899542,7cyclictest2634999-21sshd20:40:302
263015899541,9cyclictest2632139-21sshd19:47:042
2630152995430,19cyclictest2636326-21diskmemload21:10:000
2630152995427,12cyclictest2671855-21ssh22:31:320
2630152995427,12cyclictest2656597-21ssh21:56:170
2630152995421,29cyclictest0-21swapper/022:38:310
2630152995420,4cyclictest2710054-21latency_hist00:00:030
2630152995420,4cyclictest2710054-21latency_hist00:00:020
2630152995420,28cyclictest275-21dbus-daemon21:20:010
263016299536,6cyclictest131rcu_preempt22:22:223
2630162995326,22cyclictest37-21ksoftirqd/322:00:023
263015899532,6cyclictest364-21snmpd20:16:382
2630158995325,25cyclictest30-21ksoftirqd/220:25:012
2630158995325,25cyclictest30-21ksoftirqd/220:25:012
263015899532,46cyclictest2630590-21cron19:20:002
263015899531,9cyclictest2630857-21sshd19:22:282
263015899531,6cyclictest364-21snmpd20:35:302
263015899531,6cyclictest364-21snmpd20:35:302
2630158995315,34cyclictest3467562-21sshd19:57:062
2630152995327,11cyclictest2717781-21ssh00:18:050
2630152995325,13cyclictest2645812-21ssh21:31:340
2630152995325,13cyclictest2645812-21ssh21:31:340
2630152995324,3cyclictest131rcu_preempt22:24:250
2630152995324,25cyclictest2641432-21ssh21:21:190
2630152995324,14cyclictest2713677-21ssh00:08:390
2630152995314,34cyclictest2727462-21cut00:40:020
2630152995314,34cyclictest2727462-21cut00:40:020
263016299528,5cyclictest131rcu_preempt23:35:263
2630162995238,7cyclictest2718524-21cron00:20:003
263015899522,5cyclictest364-21snmpd19:25:462
263015699527,23cyclictest2624044-21kworker/1:019:20:021
2630156995237,12cyclictest2714561-21sshd00:10:231
2630152995243,5cyclictest16950irq/224-800000021:40:000
2630152995219,28cyclictest12-21ksoftirqd/021:43:420
263016299514,7cyclictest131rcu_preempt00:30:013
263016299514,7cyclictest131rcu_preempt00:30:013
2630162995115,6cyclictest131rcu_preempt00:25:003
263015899512,12cyclictest364-21snmpd21:05:542
2630156995131,14cyclictest1-21systemd22:55:011
263015299514,30cyclictest131rcu_preempt22:06:110
263015299512,42cyclictest2682080-21cat22:55:020
2630162995032,14cyclictest3467602-21systemd-journal20:10:013
263016299501,8cyclictest131rcu_preempt20:55:023
263016299501,8cyclictest131rcu_preempt20:55:013
2630162995015,12cyclictest37-21ksoftirqd/321:25:003
2630156995039,7cyclictest23-21ksoftirqd/123:52:531
2630152995045,3cyclictest0-21swapper/019:50:020
263015299504,13cyclictest2666670-21cron22:20:000
2630152995034,13cyclictest1-21systemd20:45:000
2630152995034,12cyclictest3467602-21systemd-journal23:00:000
2630152995031,14cyclictest275-21dbus-daemon19:20:010
2630152995023,3cyclictest131rcu_preempt23:07:200
2630152995023,24cyclictest12-21ksoftirqd/019:30:020
2630152995022,12cyclictest2700398-21ssh23:37:390
2630152995020,15cyclictest2710517-21ssh00:01:030
2630162994936,8cyclictest1-21systemd20:25:013
2630162994936,8cyclictest1-21systemd20:25:013
2630162994930,15cyclictest2638531-21sed21:15:013
263015899491,11cyclictest364-21snmpd21:01:372
2630152994942,3cyclictest12-21ksoftirqd/019:40:000
2630152994933,12cyclictest1-21systemd21:05:000
2630152994928,4cyclictest27067111ssh23:52:190
2630152994915,6cyclictest12-21ksoftirqd/021:52:150
2630152994915,6cyclictest12-21ksoftirqd/021:52:150
2630162994834,11cyclictest2631650-21systemctl19:40:003
2630162994833,11cyclictest1-21systemd23:05:003
2630162994827,16cyclictest1-21systemd23:55:013
263015899481,5cyclictest364-21snmpd20:06:292
2630152994832,12cyclictest275-21dbus-daemon20:05:010
263015299482,5cyclictest3467602-21systemd-journal23:26:310
263015299482,5cyclictest3467602-21systemd-journal23:26:310
2630152994823,10cyclictest2715163-21ssh00:11:490
2630152994822,11cyclictest2692375-21ssh23:19:130
2630152994819,13cyclictest2724938-21ssh00:34:370
263016299478,6cyclictest131rcu_preempt22:41:483
2630162994740,5cyclictest37-21ksoftirqd/321:31:223
2630162994740,5cyclictest37-21ksoftirqd/321:31:223
2630162994737,6cyclictest37-21ksoftirqd/319:10:023
2630162994711,6cyclictest131rcu_preempt22:50:013
2630162994711,6cyclictest131rcu_preempt22:50:013
2630162994711,5cyclictest131rcu_preempt23:00:023
263015699472,3cyclictest131rcu_preempt00:03:461
2630152994739,5cyclictest16950irq/224-800000020:40:010
2630152994739,5cyclictest16950irq/224-800000020:40:000
263015299473,12cyclictest2720757-21ssh00:25:000
2630152994715,28cyclictest12-21ksoftirqd/022:49:560
2630152994715,28cyclictest12-21ksoftirqd/022:49:550
263016299468,6cyclictest131rcu_preempt22:17:113
2630162994634,7cyclictest37-21ksoftirqd/320:45:003
2630162994626,13cyclictest1-21systemd20:40:013
2630162994626,13cyclictest1-21systemd20:40:013
2630162994619,18cyclictest0-21swapper/321:10:003
2630162994613,11cyclictest37-21ksoftirqd/321:41:013
263015699464,3cyclictest131rcu_preempt22:38:061
263015699464,3cyclictest131rcu_preempt21:10:021
2630156994622,17cyclictest275-21dbus-daemon21:20:011
263015699461,4cyclictest131rcu_preempt21:33:311
263015699461,4cyclictest131rcu_preempt21:33:301
263015299462,39cyclictest2636570-21ssh21:10:040
2630152994617,25cyclictest12-21ksoftirqd/023:42:100
2630152994616,14cyclictest2697392-21ssh23:30:250
2630152994611,4cyclictest2634232-21sshd20:28:310
263016299455,8cyclictest37-21ksoftirqd/322:11:403
263016299455,6cyclictest131rcu_preempt22:33:593
2630162994536,6cyclictest37-21ksoftirqd/323:45:183
2630162994536,6cyclictest37-21ksoftirqd/323:45:183
2630162994530,11cyclictest309-21in:imuxsock21:45:303
2630162994529,11cyclictest1-21systemd23:35:003
263016299452,5cyclictest131rcu_preempt22:08:403
2630162994515,8cyclictest131rcu_preempt22:51:273
2630156994531,9cyclictest275-21dbus-daemon22:05:011
2630152994537,5cyclictest16950irq/224-800000020:25:010
2630152994537,5cyclictest16950irq/224-800000020:25:010
263015299452,38cyclictest1-21systemd00:30:010
263015299452,38cyclictest1-21systemd00:30:010
2630152994517,13cyclictest2675742-21ssh22:40:170
263015299451,40cyclictest2652333-21ssh21:46:300
263016299449,6cyclictest131rcu_preempt19:30:023
2630162994434,6cyclictest37-21ksoftirqd/319:35:003
2630162994429,8cyclictest309-21in:imuxsock22:40:003
263016299442,14cyclictest37-21ksoftirqd/321:55:023
263016299442,14cyclictest37-21ksoftirqd/321:55:013
2630162994413,6cyclictest131rcu_preempt23:20:023
2630162994410,6cyclictest131rcu_preempt21:00:003
2630156994435,5cyclictest2636326-21diskmemload23:05:011
263015699443,3cyclictest131rcu_preempt21:39:521
2630156994430,10cyclictest2727228-21modprobe00:40:001
2630156994430,10cyclictest2727228-21modprobe00:40:001
2630156994430,10cyclictest23-21ksoftirqd/122:34:431
2630156994428,11cyclictest23-21ksoftirqd/123:15:121
263015699442,11cyclictest2714373-21cat00:10:021
263015299449,31cyclictest0-21swapper/022:29:240
2630152994415,8cyclictest2658598-21ssh22:00:420
263016299438,5cyclictest131rcu_preempt00:15:003
263016299436,6cyclictest131rcu_preempt22:30:033
263016299435,7cyclictest131rcu_preempt23:44:163
2630162994333,7cyclictest37-21ksoftirqd/323:06:483
2630162994332,6cyclictest2633768-21latency_hist20:20:013
2630162994331,9cyclictest2724443-21sshd00:33:283
2630162994330,10cyclictest37-21ksoftirqd/321:35:043
2630162994325,12cyclictest275-21dbus-daemon21:05:023
2630162994319,17cyclictest2712057-21munin-plugin-st00:05:013
263015899431,11cyclictest2630180-21latency_hist19:10:022
2630156994326,14cyclictest23-21ksoftirqd/120:00:011
2630156994324,15cyclictest23-21ksoftirqd/123:26:451
2630156994324,15cyclictest23-21ksoftirqd/123:26:451
2630156994324,13cyclictest275-21dbus-daemon23:59:591
2630156994324,13cyclictest275-21dbus-daemon23:59:591
2630156994311,10cyclictest131rcu_preempt21:43:251
263015299431,6cyclictest131rcu_preempt19:35:000
2630162994230,8cyclictest1-21systemd19:55:013
2630162994214,5cyclictest131rcu_preempt00:38:153
2630162994214,5cyclictest131rcu_preempt00:38:143
2630156994226,11cyclictest23-21ksoftirqd/123:05:221
263015699421,5cyclictest131rcu_preempt22:18:251
263015699421,3cyclictest131rcu_preempt23:47:561
263015699421,3cyclictest131rcu_preempt23:47:561
2630152994226,13cyclictest275-21dbus-daemon19:55:010
263016299411,9cyclictest0-21swapper/323:29:593
263016299411,9cyclictest0-21swapper/323:29:593
2630162994111,9cyclictest37-21ksoftirqd/321:28:263
2630162994111,14cyclictest275-21dbus-daemon19:15:013
2630162994111,14cyclictest275-21dbus-daemon19:15:013
263015699418,4cyclictest131rcu_preempt21:45:501
263015699416,7cyclictest23-21ksoftirqd/122:58:001
263015699414,10cyclictest23-21ksoftirqd/123:20:191
2630156994129,7cyclictest23-21ksoftirqd/121:23:561
2630156994127,11cyclictest23-21ksoftirqd/120:50:011
2630156994127,10cyclictest3467602-21systemd-journal20:54:301
2630156994127,10cyclictest3467602-21systemd-journal20:54:301
263015699411,4cyclictest131rcu_preempt22:45:131
263015699411,4cyclictest131rcu_preempt22:45:131
2630152994126,12cyclictest2632650-21sshd19:57:290
2630152994123,14cyclictest2635513-21sshd20:52:300
2630152994123,14cyclictest2635513-21sshd20:52:300
2630152994122,13cyclictest275-21dbus-daemon19:15:010
2630152994122,13cyclictest275-21dbus-daemon19:15:010
2630152994114,21cyclictest12-21ksoftirqd/020:15:020
263016299409,6cyclictest131rcu_preempt19:45:293
263016299408,6cyclictest131rcu_preempt20:35:013
263016299405,8cyclictest37-21ksoftirqd/323:24:553
2630162994030,7cyclictest37-21ksoftirqd/323:11:173
2630162994024,12cyclictest2632687-21systemctl20:00:013
2630162994023,13cyclictest2631885-21sshd19:40:293
263015699406,5cyclictest131rcu_preempt00:15:111
263015699403,3cyclictest131rcu_preempt20:45:001
2630156994029,8cyclictest23-21ksoftirqd/122:43:531
2630156994027,9cyclictest23-21ksoftirqd/123:31:221
263015699402,3cyclictest131rcu_preempt22:08:551
263015699401,3cyclictest131rcu_preempt21:51:031
263015699401,3cyclictest131rcu_preempt21:51:031
263015299408,29cyclictest12-21ksoftirqd/019:20:070
2630162993928,8cyclictest37-21ksoftirqd/320:49:143
2630162993911,5cyclictest131rcu_preempt22:05:013
2630156993929,7cyclictest23-21ksoftirqd/121:14:531
2630156993925,11cyclictest23-21ksoftirqd/122:27:401
2630156993925,10cyclictest23-21ksoftirqd/100:28:281
2630156993925,10cyclictest23-21ksoftirqd/100:28:281
2630156993923,10cyclictest23-21ksoftirqd/120:15:021
263015699392,12cyclictest2667059-21sh22:20:221
2630156993827,8cyclictest3467602-21systemd-journal19:48:291
2630156993825,9cyclictest23-21ksoftirqd/123:35:401
2630156993825,10cyclictest23-21ksoftirqd/121:02:301
2630156993822,11cyclictest3467602-21systemd-journal20:40:001
2630156993822,11cyclictest3467602-21systemd-journal20:40:001
263015699381,3cyclictest131rcu_preempt23:42:401
2630156993729,6cyclictest23-21ksoftirqd/123:12:211
2630156993727,8cyclictest23-21ksoftirqd/122:14:001
2630156993727,8cyclictest23-21ksoftirqd/121:26:181
2630156993727,7cyclictest23-21ksoftirqd/120:25:011
2630156993727,7cyclictest23-21ksoftirqd/120:25:011
2630156993726,6cyclictest2658124-21latency_hist22:00:011
2630156993724,9cyclictest309-21in:imuxsock19:30:031
2630156993724,9cyclictest23-21ksoftirqd/100:21:461
2630156993724,6cyclictest309-21in:imuxsock20:05:011
2630156993724,10cyclictest23-21ksoftirqd/100:34:011
2630156993722,11cyclictest2631144-21systemd-run19:30:011
263015299373,29cyclictest2635263-21modprobe20:47:300
2630152993726,8cyclictest2631893-21sshd19:42:290
2630152993724,10cyclictest12-21ksoftirqd/019:10:020
2630156993625,7cyclictest3467602-21systemd-journal19:20:291
2630156993620,12cyclictest23-21ksoftirqd/120:17:001
2630152993622,8cyclictest309-21in:imuxsock20:35:000
263015299361,32cyclictest0-21swapper/020:10:020
263016299357,5cyclictest131rcu_preempt19:25:003
263016299355,8cyclictest37-21ksoftirqd/320:28:303
2630156993526,5cyclictest2634640-21cat20:35:021
263016299348,5cyclictest131rcu_preempt20:05:003
263015699343,3cyclictest131rcu_preempt21:00:011
2630156993427,4cyclictest2632447-21latency_hist19:55:011
2630156993422,9cyclictest23-21ksoftirqd/120:30:021
263015699332,5cyclictest131rcu_preempt19:40:001
2630156993225,4cyclictest2630309-21sshd19:11:291
2630156993225,4cyclictest2630309-21sshd19:11:291
263015699322,24cyclictest2633207-21munin-run20:10:001
2630156993115,13cyclictest1-21systemd19:45:011
2630156992215,5cyclictest23-21ksoftirqd/119:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional