You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-25 - 21:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #4, slot #1

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot1s.osadl.org (updated Mon May 25, 2026 12:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1885621030,2chrt18857-21/usr/sbin/munin08:02:540
1885621030,2chrt18857-21/usr/sbin/munin08:02:540
1885621030,2chrt18857-21/usr/sbin/munin08:02:540
2388121000,2chrt23884-21unixbench_singl11:07:591
2388121000,2chrt23884-21unixbench_singl11:07:591
94482800,3chrt0-21swapper/309:02:583
94482800,3chrt0-21swapper/309:02:583
15972760,3chrt0-21swapper/208:42:552
15972760,3chrt0-21swapper/208:42:552
220092690,4chrt0-21swapper/209:37:512
220092690,4chrt0-21swapper/209:37:512
220092690,4chrt0-21swapper/209:37:512
224902670,1chrt0-21swapper/009:37:570
224902670,1chrt0-21swapper/009:37:570
224902670,1chrt0-21swapper/009:37:570
139822650,3sleep30-21swapper/310:42:523
139822650,3sleep30-21swapper/310:42:523
139822650,3sleep30-21swapper/310:42:523
281342610,2chrt0-21swapper/308:27:533
281342610,2chrt0-21swapper/308:27:533
281342610,2chrt0-21swapper/308:27:533
91152600,2sleep09288-21gltestperf11:57:480
91152600,2sleep09288-21gltestperf11:57:480
91152600,2sleep09288-21gltestperf11:57:480
155782590,3sleep30-21swapper/307:53:003
155782590,3sleep30-21swapper/307:53:003
235532570,3sleep30-21swapper/311:07:573
235532570,3sleep30-21swapper/311:07:573
326252560,3chrt0-21swapper/310:07:473
282242560,2chrt0-21swapper/208:27:562
282242560,2chrt0-21swapper/208:27:562
282242560,2chrt0-21swapper/208:27:562
145382560,3sleep20-21swapper/209:17:532
145382560,3sleep20-21swapper/209:17:532
83022470,4sleep08305-21fschecks_count09:02:470
83022470,4sleep08305-21fschecks_count09:02:470
226452460,1sleep022646-21/usr/sbin/munin11:07:470
226452460,1sleep022646-21/usr/sbin/munin11:07:470
255252450,1sleep10-21swapper/108:22:401
255252450,1sleep10-21swapper/108:22:401
187522440,3sleep20-21swapper/210:57:392
187522440,3sleep20-21swapper/210:57:392
187522440,3sleep20-21swapper/210:57:392
169482440,3sleep30-21swapper/307:57:503
169482440,3sleep30-21swapper/307:57:503
46232430,2sleep14624-21ntp_states10:17:541
300092430,3sleep20-21swapper/211:27:422
300092430,3sleep20-21swapper/211:27:422
39552410,2chrt0-21swapper/110:17:461
39552410,2chrt0-21swapper/110:17:461
25192410,2sleep02523-21kernelversion10:12:500
25192410,2sleep02523-21kernelversion10:12:500
215532410,2sleep021654-21grep11:02:560
215532410,2sleep021654-21grep11:02:560
171762390,3sleep10-21swapper/112:17:541
171762390,3sleep10-21swapper/112:17:541
32291993832,4cyclictest19771-21gltestperf09:32:470
32291993832,4cyclictest19771-21gltestperf09:32:470
32291993832,4cyclictest19771-21gltestperf09:32:470
32291993829,7cyclictest20692-21kernelversion12:27:500
32291993829,7cyclictest20692-21kernelversion12:27:500
255802370,3sleep10-21swapper/109:47:511
255802370,3sleep10-21swapper/109:47:511
221092360,8sleep1281ktimers/109:37:541
221092360,8sleep1281ktimers/109:37:541
221092360,8sleep1281ktimers/109:37:541
32296993527,6cyclictest17252-21threads07:57:541
32296993527,6cyclictest17252-21threads07:57:541
161162350,2sleep016142-21cat07:57:400
161162350,2sleep016142-21cat07:57:400
32296993425,7cyclictest23938-21netstat09:42:521
32296993425,7cyclictest23938-21netstat09:42:521
32291993428,4cyclictest20537-21cron11:02:350
32291993428,4cyclictest20537-21cron11:02:350
32291993427,5cyclictest550-21cron10:47:490
32291993427,5cyclictest550-21cron10:47:490
32291993427,5cyclictest550-21cron10:47:490
32291993425,6cyclictest749-21runrttasks07:27:570
32291993327,4cyclictest23911-21fschecks_count08:17:480
32291993327,4cyclictest23911-21fschecks_count08:17:480
32291993325,6cyclictest25229-21tr11:12:530
32291993325,6cyclictest25229-21tr11:12:530
251362330,3chrt0-21swapper/012:37:590
251362330,3chrt0-21swapper/012:37:590
155872330,2sleep20-21swapper/212:12:562
155872330,2sleep20-21swapper/212:12:562
32296993227,4cyclictest31014-21gltestperf10:02:491
32296993227,4cyclictest31014-21gltestperf10:02:491
32296993227,4cyclictest13066-21latency_hist10:42:351
32296993227,4cyclictest13066-21latency_hist10:42:351
32296993226,5cyclictest28433-21fschecks_count11:22:481
32296993226,5cyclictest28433-21fschecks_count11:22:481
32296993225,5cyclictest376-21sendmail08:42:351
32291993227,4cyclictest21134-21latency_hist09:37:350
32291993227,4cyclictest21134-21latency_hist09:37:350
32291993227,4cyclictest21134-21latency_hist09:37:350
32291993223,7cyclictest26108-21threads09:47:580
32291993223,7cyclictest26108-21threads09:47:580
32297993125,3cyclictest351rcuc/211:56:212
32297993125,3cyclictest351rcuc/211:56:212
32296993127,3cyclictest10392-21latency_hist07:42:351
32296993127,3cyclictest10392-21latency_hist07:42:351
32296993122,7cyclictest2882-21latency_hist07:22:351
32296993122,7cyclictest2882-21latency_hist07:22:351
32296993122,7cyclictest2882-21latency_hist07:22:351
3229199316,10cyclictest4491-21df_inode08:52:480
3229199316,10cyclictest4491-21df_inode08:52:480
32291993125,4cyclictest12993-21gltestperf12:07:480
32291993024,5cyclictest30564-21unixbench_singl08:32:580
32291993024,5cyclictest30564-21unixbench_singl08:32:580
32291993024,4cyclictest32229-21cyclictest09:46:500
32291993024,4cyclictest32229-21cyclictest09:46:500
32291993023,5cyclictest6847-21idleruntime-cro11:52:350
32291993023,5cyclictest6847-21idleruntime-cro11:52:350
32291993023,5cyclictest30864-21cut10:02:480
32291993023,5cyclictest30864-21cut10:02:480
32291993023,5cyclictest14818-21sshd10:45:530
32291993023,5cyclictest14818-21sshd10:45:530
32291993023,5cyclictest14818-21sshd10:45:530
224212300,3sleep00-21swapper/012:32:480
224212300,3sleep00-21swapper/012:32:480
32296992924,4cyclictest32229-21cyclictest09:22:501
32296992924,4cyclictest32229-21cyclictest09:22:501
32296992924,4cyclictest32229-21cyclictest09:22:501
32296992923,4cyclictest8145-21kernelversion10:27:501
32296992923,4cyclictest8145-21kernelversion10:27:501
32296992922,5cyclictest907-21fschecks_time08:42:471
32296992922,5cyclictest907-21fschecks_time08:42:471
32296992921,6cyclictest28629-21cat09:57:361
152662290,2sleep115267-21systemctl07:52:581
152662290,2sleep115267-21systemctl07:52:581
32296992822,4cyclictest14259-21idleruntime09:17:501
32296992822,4cyclictest14259-21idleruntime09:17:501
32296992821,5cyclictest7738-21needreboot11:52:521
32296992821,5cyclictest7738-21needreboot11:52:521
32296992821,5cyclictest23415-21sshd12:33:531
32296992821,5cyclictest23415-21sshd12:33:531
32296992821,5cyclictest21626-21sshd12:29:521
32296992821,5cyclictest21626-21sshd12:29:521
32296992821,5cyclictest20769-21sshd09:32:581
32296992821,5cyclictest20769-21sshd09:32:581
32296992821,5cyclictest20769-21sshd09:32:581
3229199284,9cyclictest16200-21/usr/sbin/munin07:57:460
3229199284,9cyclictest16200-21/usr/sbin/munin07:57:460
32291992824,2cyclictest0-21swapper/008:27:570
32291992824,2cyclictest0-21swapper/008:27:570
32291992824,2cyclictest0-21swapper/008:27:570
32291992821,5cyclictest7367-21sshd10:26:520
32291992821,5cyclictest7367-21sshd10:26:520
32291992821,5cyclictest7312-21fschecks_time11:52:480
32291992821,5cyclictest7312-21fschecks_time11:52:480
67402270,3sleep20-21swapper/210:22:572
67402270,3sleep20-21swapper/210:22:572
32296992721,4cyclictest1045-21sendmail_mailtr10:07:571
32296992721,4cyclictest1045-21sendmail_mailtr10:07:571
32296992717,7cyclictest10662-21latency_hist12:02:361
32296992717,7cyclictest10662-21latency_hist12:02:361
32296992717,7cyclictest10662-21latency_hist12:02:361
3229199274,10cyclictest20441-21if_wlan008:07:490
3229199274,10cyclictest20441-21if_wlan008:07:490
32291992722,3cyclictest0-21swapper/010:07:480
32291992722,3cyclictest0-21swapper/010:07:480
32291992722,3cyclictest0-21swapper/009:17:350
32291992722,3cyclictest0-21swapper/009:17:350
32291992721,4cyclictest32229-21cyclictest10:27:560
32291992721,4cyclictest32229-21cyclictest10:27:560
32291992720,5cyclictest22126-21fschecks_time08:12:480
32291992720,5cyclictest22126-21fschecks_time08:12:480
32291992720,5cyclictest12754-21sshd10:38:520
32291992720,5cyclictest12754-21sshd10:38:520
32291992719,6cyclictest13681-21apache208:27:450
32291992719,6cyclictest13681-21apache208:27:450
188852270,2sleep10-21swapper/108:02:551
188852270,2sleep10-21swapper/108:02:551
188852270,2sleep10-21swapper/108:02:551
32296992622,2cyclictest0-21swapper/108:57:481
32296992622,2cyclictest0-21swapper/108:57:481
32296992621,4cyclictest572-21cups-browsed08:27:571
32296992621,4cyclictest572-21cups-browsed08:27:571
32296992621,4cyclictest572-21cups-browsed08:27:571
32296992621,3cyclictest0-21swapper/109:12:511
32296992621,3cyclictest0-21swapper/109:12:511
32296992621,3cyclictest0-21swapper/108:12:461
32296992621,3cyclictest0-21swapper/108:12:461
32296992620,5cyclictest6031-21cut10:22:491
32296992620,5cyclictest6031-21cut10:22:491
32296992620,4cyclictest547-21rs:main1
32296992620,4cyclictest547-21rs:main1
32296992619,5cyclictest29149-21cut09:57:491
32296992619,5cyclictest29149-21cut09:57:491
32296992618,5cyclictest16603-21sshd10:49:531
32296992618,5cyclictest16603-21sshd10:49:531
32296992618,5cyclictest16603-21sshd10:49:531
32296992617,7cyclictest26324-21ntp_kernel_pll_08:22:521
32296992617,7cyclictest26324-21ntp_kernel_pll_08:22:521
32296992617,7cyclictest26324-21ntp_kernel_pll_08:22:521
3229199266,5cyclictest29011-21ntp_states11:22:550
3229199266,5cyclictest29011-21ntp_states11:22:550
3229199265,5cyclictest6959-21df07:32:460
3229199265,5cyclictest6959-21df07:32:460
32291992622,3cyclictest4958-21latency_hist11:47:350
32291992622,3cyclictest4958-21latency_hist11:47:350
32291992622,3cyclictest4958-21latency_hist11:47:350
3229199262,10cyclictest27020-21needreboot11:17:510
3229199262,10cyclictest27020-21needreboot11:17:510
32291992620,5cyclictest18839-21kernelversion12:22:510
32291992620,5cyclictest18839-21kernelversion12:22:510
32291992620,4cyclictest32229-21cyclictest07:51:440
32291992620,4cyclictest32229-21cyclictest07:51:440
32291992620,4cyclictest16944-21kernelversion12:17:500
32291992620,4cyclictest16944-21kernelversion12:17:500
32291992619,5cyclictest32229-21cyclictest09:52:560
32291992619,5cyclictest29025-21fschecks_time09:57:480
32291992619,5cyclictest29025-21fschecks_time09:57:480
32291992619,4cyclictest0-21swapper/011:37:480
32291992616,8cyclictest0-21swapper/009:12:160
32291992616,8cyclictest0-21swapper/009:12:160
32291992616,8cyclictest0-21swapper/009:12:160
32291992613,3cyclictest181rcu_preempt08:47:120
32291992613,3cyclictest181rcu_preempt08:47:120
39132250,3sleep20-21swapper/207:22:562
39132250,3sleep20-21swapper/207:22:562
32296992521,3cyclictest32229-21cyclictest12:41:531
32296992521,3cyclictest32229-21cyclictest12:41:531
32296992521,2cyclictest0-21swapper/107:42:481
32296992521,2cyclictest0-21swapper/107:42:481
32296992520,3cyclictest0-21swapper/112:07:551
32296992520,3cyclictest0-21swapper/112:07:551
32296992518,5cyclictest22505-21needreboot08:12:521
32296992518,5cyclictest22505-21needreboot08:12:521
32296992518,5cyclictest18982-21latency_hist10:57:461
32296992518,5cyclictest18982-21latency_hist10:57:461
32296992518,5cyclictest18982-21latency_hist10:57:461
32291992521,3cyclictest6511-21cut08:57:470
32291992521,3cyclictest6511-21cut08:57:470
32291992519,4cyclictest0-21swapper/010:37:350
32291992519,4cyclictest0-21swapper/010:37:350
32291992518,6cyclictest421-21timerwakeupswit07:12:570
32291992518,5cyclictest5552-21sendmail10:22:350
32291992518,5cyclictest11460-21missed_timers12:02:510
32291992518,5cyclictest11460-21missed_timers12:02:510
32291992518,5cyclictest0-21swapper/008:37:490
32291992512,4cyclictest90-1pr/ttyS012:07:570
32291992512,4cyclictest90-1pr/ttyS012:07:570
32291992512,2cyclictest181rcu_preempt09:17:580
32291992512,2cyclictest181rcu_preempt09:17:580
31992250,2chrt0-21swapper/311:42:423
31992250,2chrt0-21swapper/311:42:423
31992250,2chrt0-21swapper/311:42:423
156012250,2sleep115602-21/usr/sbin/munin10:47:491
32296992419,3cyclictest0-21swapper/107:12:561
32296992419,2cyclictest0-21swapper/110:42:471
32296992419,2cyclictest0-21swapper/110:42:471
32296992417,5cyclictest0-21swapper/107:27:591
32296992416,6cyclictest7560-21netstat07:32:531
32296992416,6cyclictest7560-21netstat07:32:531
32296992414,8cyclictest14155-21cron07:52:351
32296992414,8cyclictest14155-21cron07:52:351
3229199249,2cyclictest181rcu_preempt07:38:510
3229199249,2cyclictest181rcu_preempt07:38:510
3229199249,11cyclictest31068-21awk11:27:550
3229199249,11cyclictest31068-21awk11:27:550
3229199249,11cyclictest31068-21awk11:27:550
32291992420,2cyclictest0-21swapper/007:42:550
32291992420,2cyclictest0-21swapper/007:42:550
32291992419,3cyclictest32229-21cyclictest12:16:410
32291992419,3cyclictest32229-21cyclictest12:16:410
32291992419,3cyclictest171ktimers/007:26:130
32291992419,3cyclictest171ktimers/007:26:130
32291992418,4cyclictest14678-21systemd-journal10:53:050
32291992418,4cyclictest14678-21systemd-journal10:53:050
32291992418,4cyclictest14678-21systemd-journal10:53:050
32291992417,5cyclictest26198-21missed_timers08:22:500
32291992416,2cyclictest181rcu_preempt07:17:580
32291992416,2cyclictest181rcu_preempt07:17:580
32291992416,2cyclictest181rcu_preempt07:17:580
32291992410,2cyclictest181rcu_preempt09:24:150
32291992410,2cyclictest181rcu_preempt09:24:150
32291992410,2cyclictest181rcu_preempt09:24:150
225162240,3chrt0-21swapper/111:07:391
225162240,3chrt0-21swapper/111:07:391
3229699234,3cyclictest318312latency_hist11:32:381
3229699234,3cyclictest318312latency_hist11:32:381
3229699234,3cyclictest318312latency_hist11:32:381
32296992319,3cyclictest4678-21tail08:52:491
32296992319,3cyclictest4678-21tail08:52:491
32296992317,4cyclictest18123-21kernelversion09:27:491
32296992317,4cyclictest18123-21kernelversion09:27:491
32296992317,4cyclictest18123-21kernelversion09:27:491
32296992317,3cyclictest0-21swapper/109:07:591
32296992317,3cyclictest0-21swapper/109:07:591
32296992317,3cyclictest0-21swapper/109:07:591
32296992316,5cyclictest749-21runrttasks10:32:481
32296992316,5cyclictest749-21runrttasks10:32:481
32296992315,6cyclictest12575-21latency_hist12:07:411
32296992315,6cyclictest12575-21latency_hist12:07:411
32296992315,6cyclictest12575-21latency_hist12:07:411
32296992314,7cyclictest26809-21cat11:17:491
32296992314,7cyclictest26809-21cat11:17:491
32296992313,8cyclictest25179-21needreboot11:12:521
32296992313,8cyclictest25179-21needreboot11:12:521
3229199239,2cyclictest181rcu_preempt11:36:300
3229199239,2cyclictest181rcu_preempt11:36:300
32291992316,5cyclictest0-21swapper/011:37:520
32291992316,5cyclictest0-21swapper/011:37:520
32291992316,5cyclictest0-21swapper/011:37:520
32291992314,6cyclictest0-21swapper/008:51:460
32291992314,6cyclictest0-21swapper/008:51:460
305832230,4chrt0-21swapper/210:02:392
305832230,4chrt0-21swapper/210:02:392
12632230,3sleep20-21swapper/210:07:582
12632230,3sleep20-21swapper/210:07:582
116182230,1sleep2351rcuc/212:02:552
116182230,1sleep2351rcuc/212:02:552
116182230,1sleep2351rcuc/212:02:552
78732220,3sleep20-21swapper/207:32:572
78732220,3sleep20-21swapper/207:32:572
32296992218,3cyclictest32229-21cyclictest12:13:521
32296992218,3cyclictest32229-21cyclictest12:13:521
32296992218,2cyclictest0-21swapper/108:32:571
32296992218,2cyclictest0-21swapper/108:32:571
32296992216,4cyclictest400-21sendmail_mailst11:32:551
32296992216,4cyclictest400-21sendmail_mailst11:32:551
32296992216,4cyclictest400-21sendmail_mailst11:32:551
32297992116,3cyclictest101872chrt09:07:482
32297992116,3cyclictest101872chrt09:07:482
32297992116,3cyclictest101872chrt09:07:482
32296992117,2cyclictest0-21swapper/111:47:481
32296992117,2cyclictest0-21swapper/111:47:481
32296992116,2cyclictest0-21swapper/111:42:501
32296992116,2cyclictest0-21swapper/111:42:501
32296992116,2cyclictest0-21swapper/111:42:501
32296992114,4cyclictest0-21swapper/112:22:591
32296992114,4cyclictest0-21swapper/112:22:591
32296992114,4cyclictest0-21swapper/111:37:501
32296992114,4cyclictest0-21swapper/111:37:501
32296992114,4cyclictest0-21swapper/111:37:501
32296992111,8cyclictest14678-21systemd-journal07:27:221
32296992111,8cyclictest14678-21systemd-journal07:27:221
32296992110,9cyclictest729-21dbus-daemon10:57:551
32296992110,9cyclictest729-21dbus-daemon10:57:551
32298992013,4cyclictest24792sleep307:17:583
32298992013,4cyclictest24792sleep307:17:583
32298992013,4cyclictest24792sleep307:17:583
280712190,3sleep20-21swapper/209:52:582
32298991811,4cyclictest173082sleep312:17:563
32298991811,4cyclictest173082sleep312:17:563
3229899181,12cyclictest18530-21chrt09:27:553
3229899181,12cyclictest18530-21chrt09:27:553
3229899181,12cyclictest18530-21chrt09:27:553
32296991814,2cyclictest0-21swapper/108:47:531
32296991814,2cyclictest0-21swapper/108:47:531
125802180,4sleep30-21swapper/309:12:513
125802180,4sleep30-21swapper/309:12:513
3229899172,9cyclictest0-21swapper/311:32:463
3229899172,9cyclictest0-21swapper/311:32:463
3229899172,9cyclictest0-21swapper/311:32:463
3229899171,12cyclictest28453-21chrt11:22:483
3229899171,12cyclictest28453-21chrt11:22:483
3229899171,11cyclictest30947-21chrt10:02:483
3229899171,11cyclictest25835-21taskset08:22:463
3229899171,11cyclictest25835-21taskset08:22:463
3229899171,11cyclictest25835-21taskset08:22:463
3229899171,11cyclictest23695-21taskset08:17:443
3229899171,11cyclictest23695-21taskset08:17:443
3229899171,10cyclictest5746-21taskset08:53:463
3229899171,10cyclictest5746-21taskset08:53:463
3229899163,9cyclictest0-21swapper/310:13:013
3229899163,9cyclictest0-21swapper/310:13:013
3229899161,9cyclictest0-21swapper/309:47:473
3229899161,11cyclictest8530-21taskset07:37:363
3229899161,11cyclictest8530-21taskset07:37:363
3229899161,11cyclictest7196-21chrt11:52:473
3229899161,11cyclictest7196-21chrt11:52:473
3229899161,11cyclictest12083-21chrt09:12:473
3229899161,11cyclictest12083-21chrt09:12:473
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional