You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-19 - 23:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sat Sep 19, 2026 12:45:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2996643991021,99cyclictest0-21swapper/010:25:020
2996643999924,53cyclictest3075273-21ssh11:47:030
2996643999732,20cyclictest3028583-21ssh10:05:070
299664399891,53cyclictest0-21swapper/009:35:090
299664399891,53cyclictest0-21swapper/009:35:090
299664399871,82cyclictest3038930-21sshd10:28:220
299664399862,46cyclictest3096311-21ssh12:32:340
2996643998624,33cyclictest3071577-21sh11:39:220
2996643998624,33cyclictest3071577-21sh11:39:220
2996643998320,33cyclictest3057238-21rm11:07:570
2996643998227,51cyclictest3023657-21sh09:54:590
299664399821,77cyclictest3050637-21ssh10:53:440
2996643998217,28cyclictest1-21systemd12:15:020
2996643998130,34cyclictest3059769-21ssh11:13:260
299664399801,48cyclictest0-21swapper/011:21:020
2996657997834,38cyclictest275-21dbus-daemon09:15:013
2996643997726,46cyclictest3061241-21ssh11:16:250
2996643997726,46cyclictest3061241-21ssh11:16:250
2996643997632,27cyclictest3005056-21ssh09:14:470
2996643997625,27cyclictest3002818-21diskmemload09:49:080
299664399761,51cyclictest0-21swapper/010:35:590
299664399761,51cyclictest0-21swapper/010:35:590
2996653997466,6cyclictest30-21ksoftirqd/212:00:022
2996653997466,6cyclictest30-21ksoftirqd/212:00:022
2996643997430,40cyclictest16950irq/224-800000011:30:010
299664399741,69cyclictest3016881-21sed09:40:010
2996643997330,27cyclictest3005987-21ssh09:16:260
299664399731,67cyclictest0-21swapper/010:32:220
299664399731,54cyclictest3090702-21ssh12:20:050
2996643997232,36cyclictest3079324-21ssh11:55:410
2996643997232,36cyclictest3079324-21ssh11:55:410
299664399721,53cyclictest3083455-21munin-run12:05:000
2996657997128,39cyclictest3002284-21cron09:00:003
299665799711,46cyclictest0-21swapper/308:05:023
2996643997127,26cyclictest3089318-21ssh12:17:260
299664399711,65cyclictest364-21snmpd09:08:390
299664399711,46cyclictest364-21snmpd10:48:570
2996643997031,35cyclictest3008733-21ssh09:22:320
299664399702,41cyclictest3062387-21kworker/u8:111:34:310
2996643996929,37cyclictest3011578-21ssh09:28:380
299664399692,62cyclictest0-21swapper/012:05:180
299664399692,62cyclictest0-21swapper/012:05:180
2996657996826,6cyclictest2997330-21latency_hist07:20:023
2996657996821,5cyclictest3067300-21latency_hist11:30:023
299664399682,34cyclictest3094971-21latency_hist12:30:030
299664399682,34cyclictest3094971-21latency_hist12:30:020
2996643996820,43cyclictest16850irq/134-800000011:54:290
2996643996719,31cyclictest16850irq/134-800000010:02:490
2996657996628,30cyclictest37-21ksoftirqd/312:20:023
2996643996631,31cyclictest3073175-21ssh11:42:390
2996643996614,48cyclictest2997363-21sshd07:24:560
2996649996528,33cyclictest3053543-21pluginstate11:00:011
2996649996521,40cyclictest23-21ksoftirqd/111:58:261
2996649996521,40cyclictest23-21ksoftirqd/111:58:261
299664399653,28cyclictest3034936-21ssh10:19:360
299664399652,59cyclictest3019147-21munin-run09:45:000
299664399652,59cyclictest3019147-21munin-run09:45:000
299664399652,58cyclictest3467602-21systemd-journal09:59:530
299664399651,34cyclictest0-21swapper/010:43:340
2996657996422,11cyclictest37-21ksoftirqd/310:53:003
2996649996432,28cyclictest3039379-21sh10:29:201
2996649996431,29cyclictest275-21dbus-daemon10:05:011
2996649996424,35cyclictest3031208-21sh10:11:081
299664999642,19cyclictest3097315-21latency_hist12:35:011
299664999642,16cyclictest3041934-21ssh10:35:011
299664399647,53cyclictest305516850ssh11:03:320
299664399641,59cyclictest2996638-21cyclictest12:35:100
299664399641,59cyclictest2996638-21cyclictest12:35:100
2996657996323,4cyclictest3070949-21sh11:37:533
2996657996323,4cyclictest3070949-21sh11:37:533
299664999632,17cyclictest364-21snmpd08:45:011
299664999632,15cyclictest3009186-21ssh09:23:301
2996649996320,38cyclictest3467602-21systemd-journal07:19:211
2996657996225,5cyclictest3046761-21latency_hist10:45:023
2996657996217,10cyclictest3097335-21munin-run12:35:003
2996649996228,29cyclictest3006480-21sh09:17:401
299664999622,18cyclictest3069590-21munin-run11:35:011
2996649996216,9cyclictest3051296-21wc10:55:021
2996649996215,6cyclictest3467602-21systemd-journal11:30:011
2996649996212,7cyclictest3001760-21sshd08:48:211
299664399622,26cyclictest2998663-21munin-run07:50:000
299664399621,57cyclictest0-21swapper/010:13:080
299664999612,15cyclictest3467602-21systemd-journal09:00:011
299664999611,18cyclictest3088940-21ssh12:16:281
2996643996131,26cyclictest3051662-21ssh10:55:440
2996657996021,11cyclictest37-21ksoftirqd/309:45:053
2996649996031,26cyclictest3000964-21sshd08:30:211
299664999602,15cyclictest3076071-21ssh11:49:001
299664999601,18cyclictest3033648-21ssh10:16:271
2996657995921,9cyclictest37-21ksoftirqd/312:05:153
2996657995921,9cyclictest37-21ksoftirqd/312:05:143
2996649995921,34cyclictest275-21dbus-daemon09:15:011
2996649995918,6cyclictest3094840-21rm12:29:351
2996649995918,6cyclictest3094840-21rm12:29:341
2996649995918,6cyclictest3087206-21rm12:12:541
2996649995916,9cyclictest3018737-1kworker/u9:109:49:021
299664999591,19cyclictest3091110-21ssh12:21:031
299664399592,53cyclictest2998916-21munin-run07:55:010
2996643995921,34cyclictest12-21ksoftirqd/007:15:020
299664399591,54cyclictest2999952-21munin-run08:15:000
299664399591,54cyclictest2999952-21munin-run08:15:000
2996657995824,4cyclictest3086274-21sh12:10:343
2996657995817,8cyclictest3074372-21cat11:45:023
2996649995814,6cyclictest23-21ksoftirqd/111:07:411
299664399581,53cyclictest3001750-21sshd08:46:210
2996657995730,25cyclictest37-21ksoftirqd/311:47:433
2996657995722,10cyclictest37-21ksoftirqd/310:35:023
2996657995722,10cyclictest3467602-21systemd-journal12:27:323
2996657995722,10cyclictest3467602-21systemd-journal12:27:323
2996657995720,10cyclictest37-21ksoftirqd/309:20:013
299664399573,50cyclictest2997897-21munin-run07:35:010
299664399571,50cyclictest48-21kcompactd008:01:450
299664399571,50cyclictest48-21kcompactd007:36:140
299664399571,50cyclictest48-21kcompactd007:19:020
2996657995630,23cyclictest37-21ksoftirqd/311:10:023
2996657995624,4cyclictest3027476-21sh10:03:023
2996657995623,4cyclictest3048820-21sh10:50:003
2996657995611,28cyclictest3002573-21kworker/u8:210:10:023
299664999562,17cyclictest3467602-21systemd-journal11:25:001
2996649995621,6cyclictest23-21ksoftirqd/111:12:461
299664999562,15cyclictest3045823-21ssh10:43:161
2996649995621,31cyclictest3074076-21sh11:44:431
2996649995619,31cyclictest2999603-21cut08:05:011
299664999561,15cyclictest3014792-21ssh09:35:171
299664399562,50cyclictest3000745-21munin-run08:30:000
299664399561,50cyclictest48-21kcompactd008:51:070
299664399561,50cyclictest48-21kcompactd008:43:200
2996657995523,6cyclictest37-21ksoftirqd/311:24:213
2996657995521,9cyclictest37-21ksoftirqd/309:20:073
299665799552,16cyclictest3018589-21ssh09:43:423
299665799552,16cyclictest3018589-21ssh09:43:423
2996649995526,26cyclictest23-21ksoftirqd/109:43:451
2996649995526,26cyclictest23-21ksoftirqd/109:43:451
299664999552,19cyclictest3036283-21ssh10:22:221
299664999552,15cyclictest3014684-21latency_hist09:35:091
299664999552,15cyclictest3014684-21latency_hist09:35:091
2996649995512,10cyclictest275-21dbus-daemon08:15:011
2996649995512,10cyclictest275-21dbus-daemon08:15:001
299664399553,48cyclictest2999136-21sshd07:56:200
299664399551,50cyclictest3000969-21sshd08:31:210
2996657995424,5cyclictest3082346-21sh12:02:183
2996657995423,5cyclictest3053105-21sh10:59:133
2996657995421,10cyclictest37-21ksoftirqd/311:53:553
2996657995414,9cyclictest0-21swapper/309:35:003
2996657995414,9cyclictest0-21swapper/309:35:003
2996653995436,14cyclictest1-21systemd10:20:012
2996653995431,18cyclictest3023763-21sort09:55:022
2996649995430,21cyclictest23-21ksoftirqd/111:20:021
2996649995430,21cyclictest23-21ksoftirqd/111:20:021
299664999542,16cyclictest3044087-21ssh10:39:341
299664999542,16cyclictest3044087-21ssh10:39:341
299664999541,17cyclictest3048871-21wc10:50:021
299664399547,32cyclictest2996645-21cat07:10:020
299664399542,47cyclictest3002566-21munin-run09:05:010
299664399542,47cyclictest3001259-21munin-run08:40:010
299664399541,48cyclictest48-21kcompactd007:28:280
299664399541,48cyclictest3002250-21sshd08:55:240
2996653995344,5cyclictest3046572-21kworker/2:011:30:302
2996653995344,5cyclictest3030604-21kworker/2:110:30:022
2996653995343,7cyclictest3062767-21kworker/2:211:51:032
2996653995336,12cyclictest1-21systemd09:25:002
299664999539,10cyclictest3467602-21systemd-journal07:50:001
2996649995320,5cyclictest3002861-21latency_hist09:10:011
2996649995316,4cyclictest3002818-21diskmemload11:35:501
2996649995316,4cyclictest3002818-21diskmemload11:35:491
2996649995315,8cyclictest3022951-1kworker/u9:209:55:041
2996649995313,6cyclictest0-21swapper/112:38:001
2996649995313,6cyclictest0-21swapper/112:37:591
2996649995313,4cyclictest0-21swapper/110:06:281
299664999531,17cyclictest3084545-21ssh12:07:011
299664999531,17cyclictest3084545-21ssh12:07:011
2996643995344,7cyclictest12-21ksoftirqd/008:20:020
2996657995226,4cyclictest3061169-21sh11:16:163
2996657995226,4cyclictest3061169-21sh11:16:163
2996657995225,4cyclictest3059758-21sh11:13:253
2996653995212,21cyclictest3023893-21ssh09:55:032
299664999522,14cyclictest364-21snmpd09:03:111
299664399522,46cyclictest364-21snmpd08:07:160
299665799516,38cyclictest3037436-21grep10:25:023
2996657995121,7cyclictest37-21ksoftirqd/309:59:243
2996657995121,10cyclictest37-21ksoftirqd/309:50:413
2996657995120,6cyclictest2998658-21sh07:49:593
2996653995143,5cyclictest3062767-21kworker/2:212:24:482
2996653995143,5cyclictest3002577-21kworker/2:310:13:132
2996653995143,5cyclictest3002577-21kworker/2:310:02:102
2996653995142,6cyclictest3062767-21kworker/2:212:05:032
2996653995142,6cyclictest3062767-21kworker/2:212:05:032
2996653995142,6cyclictest2999962-21kworker/2:108:35:022
2996653995142,5cyclictest3002577-21kworker/2:309:20:012
2996653995130,16cyclictest275-21dbus-daemon09:35:012
2996653995130,16cyclictest275-21dbus-daemon09:35:012
299664999512,18cyclictest3053982-21ssh11:00:411
2996649995119,13cyclictest275-21dbus-daemon08:55:011
299664999511,16cyclictest2997851-21sshd07:31:211
299664399513,44cyclictest364-21snmpd08:23:580
299664399513,44cyclictest364-21snmpd08:23:580
2996657995020,10cyclictest37-21ksoftirqd/312:38:503
2996657995020,10cyclictest37-21ksoftirqd/312:38:503
2996657995012,6cyclictest131rcu_preempt10:18:453
2996653995042,5cyclictest3062767-21kworker/2:212:00:322
2996653995042,5cyclictest3062767-21kworker/2:211:44:332
2996653995042,5cyclictest3030604-21kworker/2:110:43:422
2996653995042,5cyclictest3005182-21kworker/2:209:29:522
2996653995041,6cyclictest3002577-21kworker/2:309:45:462
2996653995041,6cyclictest3002577-21kworker/2:309:39:112
2996653995041,5cyclictest3046572-21kworker/2:011:17:092
2996653995041,5cyclictest3046572-21kworker/2:011:17:092
2996653995040,6cyclictest3062767-21kworker/2:212:15:002
2996653995039,6cyclictest3062767-21kworker/2:212:27:402
2996653995039,6cyclictest3062767-21kworker/2:212:27:402
2996653995031,15cyclictest1-21systemd11:00:002
299664999502,17cyclictest3010829-21ssh09:26:551
299664999502,16cyclictest3076873-21ssh11:50:231
2996657994921,8cyclictest37-21ksoftirqd/309:30:013
2996653994942,5cyclictest3030604-21kworker/2:110:30:212
2996653994941,6cyclictest291rcuc/210:50:022
2996653994941,5cyclictest3074256-21kworker/2:112:35:332
2996653994941,5cyclictest3074256-21kworker/2:112:35:332
2996653994933,12cyclictest3467602-21systemd-journal08:05:002
2996653994931,13cyclictest3030582-21cron10:10:002
2996653994929,15cyclictest1-21systemd09:09:592
299664999492,17cyclictest3021725-21ssh09:50:201
2996649994915,5cyclictest3001263-21sendmail08:40:011
299664399492,42cyclictest2998392-21modprobe07:43:210
2996653994840,5cyclictest3074256-21kworker/2:112:30:292
2996653994840,5cyclictest3062767-21kworker/2:212:15:502
2996653994840,5cyclictest3030604-21kworker/2:110:51:592
2996653994839,5cyclictest2989328-21kworker/2:007:25:012
2996653994831,13cyclictest3058137-21systemctl11:10:012
299665799478,5cyclictest131rcu_preempt11:55:293
299665799478,5cyclictest131rcu_preempt11:55:283
2996657994734,9cyclictest275-21dbus-daemon08:20:003
2996657994710,6cyclictest131rcu_preempt11:01:363
2996653994739,5cyclictest3062767-21kworker/2:211:46:092
2996653994739,5cyclictest2989328-21kworker/2:007:18:062
2996653994738,7cyclictest30-21ksoftirqd/211:39:592
2996653994738,7cyclictest30-21ksoftirqd/211:39:592
2996653994731,13cyclictest30-21ksoftirqd/210:40:012
2996653994731,13cyclictest30-21ksoftirqd/210:40:012
2996653994730,11cyclictest3467602-21systemd-journal11:10:022
2996653994728,15cyclictest1-21systemd09:05:012
299664999472,17cyclictest2999181-21readlink08:00:001
2996649994712,4cyclictest0-21swapper/112:01:221
2996653994638,5cyclictest3002577-21kworker/2:310:24:382
2996653994638,5cyclictest3002577-21kworker/2:309:10:232
2996653994638,5cyclictest2758822-21kworker/2:308:15:362
2996653994636,5cyclictest3001249-21sendmail08:40:012
2996653994625,17cyclictest1-21systemd07:35:012
299665799455,5cyclictest131rcu_preempt09:37:373
2996653994537,5cyclictest3046572-21kworker/2:011:22:192
2996653994531,11cyclictest30-21ksoftirqd/209:43:432
2996653994531,11cyclictest30-21ksoftirqd/209:43:432
2996653994527,11cyclictest30-21ksoftirqd/211:05:022
299664999452,14cyclictest3000507-21munin-run08:25:011
299664999452,14cyclictest3000507-21munin-run08:25:011
299665799441,7cyclictest3039464-21ssh10:29:293
2996653994428,12cyclictest1-21systemd11:30:012
2996653994427,12cyclictest3002302-21tr09:00:002
2996649994414,4cyclictest2998140-21run-parts07:40:001
299664999441,14cyclictest2998421-21systemctl07:45:011
2996657994327,13cyclictest275-21dbus-daemon08:00:003
2996653994327,12cyclictest30-21ksoftirqd/208:14:212
2996653994327,12cyclictest30-21ksoftirqd/208:14:212
2996653994324,14cyclictest275-21dbus-daemon08:45:012
299665799425,5cyclictest131rcu_preempt12:21:073
299665799425,3cyclictest131rcu_preempt11:35:013
2996657994229,10cyclictest1-21systemd10:15:023
2996657994227,10cyclictest1-21systemd08:55:013
299665399424,31cyclictest2998411-21munin-run07:45:002
2996653994234,5cyclictest2998436-21kworker/2:207:48:432
2996649994216,6cyclictest23-21ksoftirqd/108:15:201
2996649994118,6cyclictest23-21ksoftirqd/108:10:021
299665799402,9cyclictest3467602-21systemd-journal07:36:223
2996657994021,14cyclictest1-21systemd07:25:013
2996653994032,5cyclictest2998436-21kworker/2:207:55:022
2996653994013,6cyclictest131rcu_preempt07:36:222
299664999402,5cyclictest131rcu_preempt07:30:011
299665799391,15cyclictest346771850lldpd08:25:153
2996653993930,6cyclictest2758822-21kworker/2:308:26:242
2996653993928,8cyclictest30-21ksoftirqd/207:30:002
2996653993913,17cyclictest3467602-21systemd-journal08:00:002
2996649993924,11cyclictest3467602-21systemd-journal07:51:321
2996657993825,8cyclictest37-21ksoftirqd/310:35:113
2996657993825,8cyclictest37-21ksoftirqd/310:35:113
2996657993824,11cyclictest275-21dbus-daemon08:35:003
2996657993724,11cyclictest37-21ksoftirqd/308:50:023
2996657993724,10cyclictest3467602-21systemd-journal08:15:003
2996657993724,10cyclictest3467602-21systemd-journal08:15:003
2996657993721,12cyclictest3467602-21systemd-journal09:10:003
2996649993727,7cyclictest3000711-21sshd08:25:221
2996657993624,9cyclictest275-21dbus-daemon09:05:013
2996657993624,8cyclictest3467602-21systemd-journal07:50:023
299665399365,9cyclictest30-21ksoftirqd/207:11:592
2996653993625,9cyclictest30-21ksoftirqd/207:10:022
2996657993525,6cyclictest2996839-21munin-run07:15:003
2996657993519,11cyclictest1-21systemd07:45:013
2996653993524,8cyclictest3001792-21pluginstate08:50:012
2996653993522,10cyclictest30-21ksoftirqd/208:25:022
2996653993522,10cyclictest30-21ksoftirqd/208:25:022
299665799342,5cyclictest131rcu_preempt08:40:023
2996657993423,7cyclictest309-21in:imuxsock08:25:003
2996657993423,7cyclictest309-21in:imuxsock08:24:593
2996657993418,13cyclictest1-21systemd07:30:013
2996653993425,7cyclictest30-21ksoftirqd/208:55:022
2996649993420,10cyclictest3467602-21systemd-journal07:22:201
2996649993418,12cyclictest275-21dbus-daemon07:15:011
2996657993323,7cyclictest37-21ksoftirqd/307:10:023
299665399334,20cyclictest2999699-21munin-plugin-st08:10:002
2996657993215,13cyclictest1-21systemd08:45:013
299665799321,28cyclictest48-21kcompactd008:05:053
2996657993122,6cyclictest37-21ksoftirqd/307:34:313
2996649992821,5cyclictest2996654-21latency_hist07:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional