You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-24 - 20:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sun May 24, 2026 12:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
385497399971,91cyclictest0-21swapper/011:00:020
3854973998831,52cyclictest3915349-21ssh11:04:480
3854973998117,59cyclictest3922392-21ssh11:19:440
3854973998024,23cyclictest3861146-21diskmemload10:01:500
3854973998024,23cyclictest3861146-21diskmemload10:01:500
3854973997931,30cyclictest3893841-21ssh10:19:050
3854973997719,53cyclictest3895587-21ssh10:22:380
3854973997719,53cyclictest3895587-21ssh10:22:380
385497399763,69cyclictest3884766-21sendmail-msp10:00:010
3854973997621,37cyclictest3940685-21rm11:58:220
385497399753,8cyclictest1-21systemd07:20:010
3854973997537,34cyclictest16950irq/224-800000011:10:010
3854973997529,29cyclictest3947669-21ssh12:13:130
3854973997526,33cyclictest0-21swapper/012:29:250
385497399752,20cyclictest3901306-21/usr/sbin/munin10:35:010
385497399752,20cyclictest3901306-21/usr/sbin/munin10:35:010
385497399751,53cyclictest0-21swapper/011:22:260
3854973997429,41cyclictest3925637-21ssh11:26:190
385497399741,5cyclictest3904694-21ssh10:41:490
385497399741,56cyclictest150-21pr/ttyS210:08:560
3854973997330,26cyclictest3944338-21ssh12:05:520
385497399722,4cyclictest0-21swapper/008:40:000
385497399722,14cyclictest3880224-21latency_hist09:50:020
385497399722,14cyclictest3880224-21latency_hist09:50:020
3854973997131,22cyclictest3902193-21ssh10:36:370
385497399711,2cyclictest0-21swapper/012:36:100
385497399702,6cyclictest3862668-21ssh09:13:110
3854973996932,31cyclictest16850irq/134-800000011:40:030
3854973996932,31cyclictest16850irq/134-800000011:40:020
3854973996920,44cyclictest3910717-21ssh10:55:000
3854973996828,35cyclictest3951015-21date12:20:020
385497399681,4cyclictest0-21swapper/011:51:570
3854973996729,34cyclictest3881460-21ssh09:52:550
3854973996727,10cyclictest3953227-21ssh12:25:010
3854973996726,36cyclictest3870537-21sh09:29:550
3854973996622,39cyclictest3927484-21ssh11:30:080
3854973996621,29cyclictest3956647-21ssh12:32:110
385497399653,6cyclictest3868380-21latency_hist09:25:030
3854973996525,23cyclictest3943014-21ssh12:03:100
385497399652,12cyclictest48-21kcompactd011:40:290
385497399651,4cyclictest0-21swapper/011:47:490
385497399651,4cyclictest0-21swapper/011:47:480
385497399651,4cyclictest0-21swapper/010:48:520
385497399651,17cyclictest3873457-21ssh09:35:540
385497399642,5cyclictest3875959-21sshd09:41:070
385497399642,5cyclictest3858579-21munin-run08:20:010
385497399633,15cyclictest3856992-21munin-run07:50:000
385497399632,8cyclictest3865886-21munin-run09:20:010
385497399632,11cyclictest3467602-21systemd-journal10:11:360
385498699624,30cyclictest0-21swapper/309:45:033
385497399622,19cyclictest3918339-21ssh11:10:430
3854973996213,44cyclictest3861355-21latency_hist09:10:020
3854986996112,20cyclictest3932026-21ssh11:40:023
3854986996112,20cyclictest3932026-21ssh11:40:023
385497399613,16cyclictest3860121-21unin-run08:50:010
3854973996124,31cyclictest3857756-21sshd08:04:360
3854973996027,29cyclictest3897568-21ssh10:26:420
385497399602,11cyclictest3859070-21munin-run08:30:010
385497399591,4cyclictest3855141-21sshd07:14:070
385497399582,17cyclictest3860616-21latency_hist09:00:000
385497399582,17cyclictest3858832-21munin-run08:25:010
385497399581,7cyclictest3856432-21sshd07:38:060
385497399581,53cyclictest0-21swapper/008:34:420
385497399581,4cyclictest0-21swapper/009:33:280
385497399581,4cyclictest0-21swapper/009:33:280
3854981995739,13cyclictest3467602-21systemd-journal09:50:002
3854981995739,13cyclictest3467602-21systemd-journal09:50:002
385497399572,4cyclictest3857992-21sshd08:06:070
3854986995638,14cyclictest275-21dbus-daemon07:30:013
3854986995632,18cyclictest3467602-21systemd-journal10:30:003
385497399561,12cyclictest364-21snmpd08:52:460
385497399561,12cyclictest364-21snmpd07:54:270
3854981995545,6cyclictest3936724-21kworker/2:312:39:412
3854978995537,14cyclictest3467602-21systemd-journal11:20:001
385497399553,6cyclictest364-21snmpd07:26:590
385497399552,5cyclictest364-21snmpd09:04:260
385497399552,5cyclictest364-21snmpd09:04:260
385497399551,10cyclictest3858260-21sshd08:14:390
385497399551,10cyclictest3858260-21sshd08:14:390
3854986995434,15cyclictest275-21dbus-daemon12:20:003
385498699541,6cyclictest131rcu_preempt12:40:033
385497399542,6cyclictest364-21snmpd07:41:200
385497399542,6cyclictest364-21snmpd07:41:200
3854986995337,13cyclictest37-21ksoftirqd/309:20:013
3854978995338,12cyclictest1-21systemd10:25:011
3854978995338,12cyclictest1-21systemd10:25:001
3854978995313,6cyclictest131rcu_preempt09:54:251
385497399531,6cyclictest364-21snmpd08:40:190
3854986995227,18cyclictest275-21dbus-daemon10:20:003
3854981995244,5cyclictest3936724-21kworker/2:312:24:332
3854981995244,5cyclictest3843225-21kworker/2:010:36:102
385497399521,13cyclictest3855691-21cron07:25:010
3854986995133,12cyclictest275-21dbus-daemon10:00:013
3854981995143,5cyclictest3859540-21kworker/2:110:19:062
3854981995141,6cyclictest3936724-21kworker/2:312:31:192
385497399511,6cyclictest364-21snmpd07:58:540
385497399511,5cyclictest364-21snmpd07:31:440
385497399511,5cyclictest364-21snmpd07:31:440
3854986995033,12cyclictest3467602-21systemd-journal12:31:073
3854986995020,10cyclictest3955570-21ssh12:30:033
3854981995043,4cyclictest3843225-21kworker/2:010:05:012
3854981995043,4cyclictest3843225-21kworker/2:010:05:012
3854981995041,6cyclictest3843225-21kworker/2:010:34:162
3854981995041,6cyclictest3843225-21kworker/2:010:34:162
3854981995041,5cyclictest3859540-21kworker/2:109:25:442
3854978995036,11cyclictest23-21ksoftirqd/109:05:011
3854978995036,11cyclictest23-21ksoftirqd/109:05:011
3854986994944,3cyclictest371ksoftirqd/310:25:023
3854986994944,3cyclictest371ksoftirqd/310:25:023
3854986994931,12cyclictest309-21in:imuxsock07:35:003
3854986994931,12cyclictest309-21in:imuxsock07:34:593
3854981994941,5cyclictest3936724-21kworker/2:312:25:322
3854981994941,5cyclictest3859540-21kworker/2:111:21:322
3854981994941,5cyclictest3859540-21kworker/2:111:06:102
3854981994941,5cyclictest3859540-21kworker/2:109:23:382
3854981994941,5cyclictest3859540-21kworker/2:109:11:242
3854981994941,5cyclictest3843225-21kworker/2:012:02:512
3854981994941,5cyclictest3843225-21kworker/2:010:55:302
3854981994939,5cyclictest3843225-21kworker/2:011:30:302
385497899492,9cyclictest3905802-21ssh10:44:211
3854986994833,8cyclictest37-21ksoftirqd/311:35:003
3854986994828,14cyclictest309-21in:imuxsock08:55:003
3854981994840,5cyclictest3859540-21kworker/2:111:26:052
3854981994840,5cyclictest3859540-21kworker/2:110:06:452
3854981994840,5cyclictest3843225-21kworker/2:011:46:042
3854981994840,5cyclictest3843225-21kworker/2:011:46:042
3854981994840,5cyclictest3843225-21kworker/2:010:53:212
3854981994839,5cyclictest3859540-21kworker/2:109:35:092
3854981994830,13cyclictest275-21dbus-daemon11:20:002
3854978994834,10cyclictest3875271-21systemctl09:40:001
3854986994724,16cyclictest3881579-21kworker/u8:110:33:073
3854986994724,16cyclictest3881579-21kworker/u8:110:33:073
3854986994722,20cyclictest37-21ksoftirqd/309:13:533
3854986994712,5cyclictest131rcu_preempt11:20:003
3854981994740,4cyclictest3843225-21kworker/2:011:59:102
3854981994739,5cyclictest3859540-21kworker/2:110:26:082
3854981994739,5cyclictest3843225-21kworker/2:011:43:302
385497899473,3cyclictest131rcu_preempt09:26:011
3854978994731,12cyclictest1-21systemd07:55:011
3854978994731,10cyclictest0-21swapper/107:40:001
385497899472,13cyclictest3934163-21ssh11:44:421
3854986994632,10cyclictest37-21ksoftirqd/312:02:543
3854986994631,10cyclictest1-21systemd07:25:023
3854986994629,12cyclictest1-21systemd10:10:003
3854986994626,13cyclictest37-21ksoftirqd/310:40:003
3854981994638,5cyclictest3936724-21kworker/2:312:11:322
3854981994637,6cyclictest3859540-21kworker/2:109:30:302
3854981994637,6cyclictest3859540-21kworker/2:109:30:302
3854981994637,5cyclictest3859540-21kworker/2:110:11:292
3854981994637,5cyclictest3843225-21kworker/2:011:37:512
3854981994637,5cyclictest3843225-21kworker/2:011:37:512
3854981994630,12cyclictest3467602-21systemd-journal07:35:002
3854981994630,12cyclictest3467602-21systemd-journal07:35:002
385497899469,6cyclictest131rcu_preempt10:10:161
3854978994632,10cyclictest275-21dbus-daemon10:40:001
385497899461,13cyclictest3892722-21ssh10:16:341
385498699453,4cyclictest131rcu_preempt11:48:133
385498699453,4cyclictest131rcu_preempt11:48:133
3854986994528,5cyclictest37-21ksoftirqd/312:09:153
3854986994528,12cyclictest37-21ksoftirqd/312:00:003
3854986994528,11cyclictest309-21in:imuxsock08:20:013
3854986994528,10cyclictest3857496-21sh08:00:003
3854986994524,15cyclictest3880067-21ssh09:50:013
3854986994524,15cyclictest3880067-21ssh09:50:013
3854981994537,5cyclictest3859540-21kworker/2:111:13:192
3854981994537,5cyclictest3859540-21kworker/2:109:58:002
3854981994537,5cyclictest3843225-21kworker/2:011:02:452
3854981994537,5cyclictest3843225-21kworker/2:008:21:472
3854981994534,8cyclictest1-21systemd07:40:002
3854981994528,14cyclictest3467602-21systemd-journal07:20:002
385497899454,3cyclictest131rcu_preempt10:30:511
385497899454,3cyclictest131rcu_preempt10:30:511
385497899453,4cyclictest131rcu_preempt12:32:541
385497899451,27cyclictest3953303-21ssh12:25:021
3854978994511,6cyclictest131rcu_preempt11:37:291
3854978994511,6cyclictest131rcu_preempt11:37:291
385497399451,13cyclictest3855013-21ls07:10:020
3854986994427,12cyclictest37-21ksoftirqd/311:05:013
385498699442,30cyclictest3860892-21munin-run09:05:003
385498699442,30cyclictest3860892-21munin-run09:05:003
3854986994420,5cyclictest37-21ksoftirqd/311:05:363
3854981994431,7cyclictest3467602-21systemd-journal07:49:592
3854981994428,13cyclictest3861280-21sed09:10:022
3854981994428,12cyclictest275-21dbus-daemon07:45:012
3854981994428,12cyclictest275-21dbus-daemon07:45:012
385497899446,5cyclictest131rcu_preempt11:50:021
385497899446,5cyclictest131rcu_preempt11:50:021
3854978994437,4cyclictest3854984-21latency_hist07:10:011
3854978994428,12cyclictest1-21systemd12:10:011
3854978994420,17cyclictest275-21dbus-daemon09:20:011
385498699433,35cyclictest150-21pr/ttyS211:22:443
3854986994328,11cyclictest3908424-21cron10:50:003
3854986994328,11cyclictest37-21ksoftirqd/309:20:443
3854981994336,4cyclictest3843225-21kworker/2:009:51:062
3854981994335,5cyclictest3843225-21kworker/2:008:51:472
385497899439,6cyclictest131rcu_preempt12:00:441
385497899439,5cyclictest131rcu_preempt12:37:501
385497899433,4cyclictest131rcu_preempt09:57:081
3854978994330,11cyclictest23-21ksoftirqd/110:00:431
3854978994330,11cyclictest23-21ksoftirqd/110:00:431
3854978994330,10cyclictest3868409-21sed09:25:031
385497899431,3cyclictest131rcu_preempt08:50:021
385498699429,13cyclictest37-21ksoftirqd/307:35:033
385498699428,7cyclictest131rcu_preempt10:15:023
3854986994230,10cyclictest37-21ksoftirqd/312:12:303
3854986994228,11cyclictest37-21ksoftirqd/312:20:063
3854986994228,10cyclictest1-21systemd08:35:013
385498699422,6cyclictest131rcu_preempt11:00:023
3854986994225,13cyclictest3857152-21sed07:50:023
385498699422,3cyclictest131rcu_preempt09:54:133
3854986994215,5cyclictest131rcu_preempt08:40:003
3854986994214,24cyclictest131rcu_preempt11:53:443
3854981994234,5cyclictest3859540-21kworker/2:109:18:482
3854981994229,10cyclictest3939041-21systemd-run11:55:002
3854981994226,12cyclictest1-21systemd10:25:012
3854981994226,12cyclictest1-21systemd10:25:012
3854981994223,15cyclictest3467602-21systemd-journal08:20:002
3854978994228,9cyclictest3467602-21systemd-journal10:25:111
3854978994220,16cyclictest275-21dbus-daemon09:35:011
3854978994220,16cyclictest275-21dbus-daemon09:35:011
385498699414,24cyclictest3860603-21modprobe09:00:003
3854986994129,8cyclictest3861146-21diskmemload09:40:003
3854986994126,11cyclictest3873135-21sed09:35:033
3854986994124,13cyclictest37-21ksoftirqd/311:27:303
3854981994133,5cyclictest3936724-21kworker/2:312:07:522
3854981994130,9cyclictest30-21ksoftirqd/210:45:022
385498199412,3cyclictest131rcu_preempt09:45:002
3854978994131,8cyclictest23-21ksoftirqd/107:34:321
3854978994131,8cyclictest23-21ksoftirqd/107:34:321
3854978994127,11cyclictest23-21ksoftirqd/110:58:461
3854978994127,11cyclictest1-21systemd08:30:011
3854978994127,10cyclictest23-21ksoftirqd/110:09:071
3854978994125,11cyclictest1-21systemd08:20:001
3854978994124,12cyclictest275-21dbus-daemon11:15:011
385497899411,3cyclictest131rcu_preempt11:03:471
385497899411,3cyclictest131rcu_preempt08:25:021
385497899411,37cyclictest23-21ksoftirqd/107:25:011
3854986994029,8cyclictest37-21ksoftirqd/310:40:203
3854986994027,10cyclictest37-21ksoftirqd/310:00:593
3854986994027,10cyclictest37-21ksoftirqd/310:00:593
3854986994027,10cyclictest37-21ksoftirqd/309:32:443
3854986994025,11cyclictest37-21ksoftirqd/311:43:353
3854986994024,12cyclictest275-21dbus-daemon07:55:003
3854986994024,11cyclictest275-21dbus-daemon08:15:013
3854986994024,11cyclictest275-21dbus-daemon08:15:013
385498699402,3cyclictest131rcu_preempt09:25:513
3854981994024,11cyclictest275-21dbus-daemon09:05:012
3854981994024,11cyclictest275-21dbus-daemon09:05:012
3854978994028,9cyclictest23-21ksoftirqd/111:08:221
3854978994027,8cyclictest23-21ksoftirqd/108:35:011
3854978994022,13cyclictest275-21dbus-daemon12:20:011
385498699393,10cyclictest0-21swapper/310:55:013
3854986993927,7cyclictest309-21in:imuxsock08:25:013
3854986993926,9cyclictest3467602-21systemd-journal07:44:343
3854986993926,9cyclictest3467602-21systemd-journal07:44:343
385498699392,5cyclictest131rcu_preempt09:10:003
3854986993925,10cyclictest37-21ksoftirqd/311:13:163
385498199395,10cyclictest30-21ksoftirqd/208:10:012
3854981993927,9cyclictest3860577-21sshd08:56:082
3854981993926,9cyclictest30-21ksoftirqd/212:20:002
3854981993925,11cyclictest30-21ksoftirqd/207:15:012
3854981993922,12cyclictest3467602-21systemd-journal08:40:002
3854978993929,7cyclictest3877717-21cat09:45:021
3854978993928,9cyclictest23-21ksoftirqd/112:10:521
3854978993927,9cyclictest23-21ksoftirqd/110:46:441
3854978993924,11cyclictest275-21dbus-daemon11:25:011
3854978993924,11cyclictest275-21dbus-daemon09:10:011
385498199381,6cyclictest131rcu_preempt08:05:002
3854978993828,8cyclictest23-21ksoftirqd/112:25:121
3854978993826,8cyclictest3925160-21sshd11:25:071
3854978993824,10cyclictest3467602-21systemd-journal07:44:021
3854978993824,10cyclictest3467602-21systemd-journal07:44:011
3854978993824,10cyclictest23-21ksoftirqd/109:49:461
3854978993824,10cyclictest23-21ksoftirqd/109:49:461
3854978993822,11cyclictest3467718-21lldpd10:50:021
3854986993727,5cyclictest37-21ksoftirqd/307:20:003
3854986993721,11cyclictest309-21in:imuxsock08:30:013
3854986993714,17cyclictest275-21dbus-daemon07:15:013
385498199376,4cyclictest131rcu_preempt08:00:002
385498199371,4cyclictest131rcu_preempt07:55:012
3854978993727,7cyclictest23-21ksoftirqd/111:50:261
3854978993725,8cyclictest3855125-21sshd07:10:071
3854978993724,9cyclictest3855393-21sshd07:17:071
3854978993724,10cyclictest23-21ksoftirqd/111:31:381
3854978993719,13cyclictest275-21dbus-daemon07:50:001
3854986993620,9cyclictest3859858-21munin-plugin-st08:45:003
385498199366,7cyclictest30-21ksoftirqd/207:27:342
3854981993620,11cyclictest275-21dbus-daemon08:35:002
385498199361,3cyclictest131rcu_preempt10:49:262
385498199361,3cyclictest131rcu_preempt08:50:002
3854978993627,7cyclictest23-21ksoftirqd/111:56:211
3854986993523,5cyclictest37-21ksoftirqd/308:05:003
3854981993527,5cyclictest3859997-21latency_hist08:45:042
385497899358,5cyclictest131rcu_preempt08:54:471
3854978993526,7cyclictest23-21ksoftirqd/108:05:021
3854978993524,8cyclictest23-21ksoftirqd/109:13:151
3854978993524,8cyclictest23-21ksoftirqd/108:15:001
3854978993524,8cyclictest23-21ksoftirqd/108:15:001
3854978993523,9cyclictest3855993-21sed07:30:011
385498699342,27cyclictest3860094-21cron08:50:003
3854981993423,9cyclictest30-21ksoftirqd/207:24:322
3854978993421,10cyclictest23-21ksoftirqd/108:45:021
3854978993420,8cyclictest3858026-21grep08:10:001
385498699331,3cyclictest131rcu_preempt07:10:023
3854978993323,7cyclictest23-21ksoftirqd/107:59:361
3854978993318,12cyclictest3859684-21cat08:40:021
385498199323,3cyclictest131rcu_preempt07:10:022
385498199323,21cyclictest0-21swapper/208:11:172
385498199323,21cyclictest0-21swapper/208:11:172
3854981993221,9cyclictest30-21ksoftirqd/208:30:012
3854978993223,6cyclictest23-21ksoftirqd/108:59:541
3854986993123,5cyclictest3858044-21latency_hist08:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional