You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-16 - 13:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #4, slot #1

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack4slot1s.osadl.org (updated Tue Dec 16, 2025 01:37:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
311721090,2chrt3118-21/usr/sbin/munin23:06:581
3213021050,3sleep1281ktimers/122:57:011
243042800,2sleep30-21swapper/319:41:583
243042800,2sleep30-21swapper/319:41:583
64832680,2chrt6486-21grep21:47:021
28052660,8sleep02808-21grep21:37:040
158812580,1sleep115984-21memory20:46:571
213192530,4sleep021323-21fschecks_count23:56:530
213192530,4sleep021323-21fschecks_count23:56:530
77002430,3sleep00-21swapper/020:25:050
12360994127,12cyclictest583-21needreboot00:27:000
12360994127,12cyclictest583-21needreboot00:27:000
114472390,1sleep011508-21latency_hist20:35:060
114472390,1sleep011508-21latency_hist20:35:060
143952380,3sleep30-21swapper/323:36:553
143952380,3sleep30-21swapper/323:36:553
12361993830,6cyclictest24575-21latency_hist00:05:041
191812370,3sleep10-21swapper/123:50:071
191812370,3sleep10-21swapper/123:50:071
12361993630,4cyclictest9364-21gltestperf21:56:541
12361993630,4cyclictest9364-21gltestperf21:56:541
12361993527,6cyclictest5795-21sshd23:12:441
12361993527,6cyclictest5795-21sshd23:12:441
12361993525,8cyclictest25433-21latency_hist22:40:041
12360993527,6cyclictest3890-21fschecks_time00:36:540
12360993527,6cyclictest18141-21needreboot23:46:590
12360993527,6cyclictest18141-21needreboot23:46:590
12360993527,6cyclictest18141-21needreboot23:46:590
12360993525,8cyclictest4954-21latency_hist21:45:030
12360993525,8cyclictest4954-21latency_hist21:45:030
12361993429,4cyclictest12359-21cyclictest23:57:011
12361993429,4cyclictest12359-21cyclictest23:57:011
12361993429,3cyclictest26568-21rt-features22:42:011
12361993427,5cyclictest5553-21sshd20:17:401
12361993427,5cyclictest5553-21sshd20:17:401
12361993427,5cyclictest29620-21threads00:17:021
12361993427,5cyclictest29620-21threads00:17:021
12361993427,5cyclictest11239-21sshd20:34:081
12361993427,5cyclictest11239-21sshd20:34:081
12360993426,6cyclictest27470-21fschecks_count19:51:530
12360993426,6cyclictest12359-21cyclictest00:15:080
12360993426,6cyclictest12359-21cyclictest00:15:080
12361993327,4cyclictest11924-21fschecks_time20:36:531
12361993326,5cyclictest18998-21sshd20:55:391
12361993326,5cyclictest18998-21sshd20:55:391
12361993325,6cyclictest32568-21idleruntime-cro20:05:031
12360993327,4cyclictest12359-21cyclictest22:54:400
12361993228,3cyclictest25852-21cut19:46:541
12361993226,4cyclictest12359-21cyclictest22:21:561
12361993226,4cyclictest12359-21cyclictest22:21:561
12360993225,5cyclictest12359-21cyclictest00:06:590
12360993225,5cyclictest12359-21cyclictest00:06:590
12360993225,5cyclictest1030-21sshd21:32:400
12360993225,5cyclictest1030-21sshd21:32:400
12360993224,6cyclictest14970-21sshd20:44:080
12360993224,6cyclictest14970-21sshd20:44:080
12360993223,6cyclictest14001-21fschecks_time23:36:510
12360993223,6cyclictest14001-21fschecks_time23:36:510
12361993127,3cyclictest29788-21head22:51:541
12361993124,5cyclictest11387-21kernelversion22:01:571
12360993125,5cyclictest7967-21/usr/sbin/munin20:27:010
12360993125,5cyclictest7967-21/usr/sbin/munin20:27:010
12360993124,5cyclictest7782-21sshd23:19:250
12360993123,6cyclictest2902-21kernelversion20:11:550
12360993123,6cyclictest0-21swapper/020:02:010
12361993026,3cyclictest32703-21head00:26:561
12361993026,3cyclictest32703-21head00:26:561
12361993023,5cyclictest5301-21latency_hist00:40:051
12361993023,5cyclictest22261-21kernelversion19:36:561
12361993023,5cyclictest22261-21kernelversion19:36:561
12360993026,3cyclictest842-21gltestperf20:06:540
12360993026,3cyclictest842-21gltestperf20:06:540
12360993025,4cyclictest20523-21cut22:26:540
12360993025,4cyclictest20523-21cut22:26:540
12360993024,5cyclictest16917-21kernelversion22:16:550
12360993024,4cyclictest9071-21sendmail_mailst23:22:010
12360993024,4cyclictest12359-21cyclictest22:01:340
12360993023,5cyclictest5156-21needreboot23:11:590
12360993023,5cyclictest5156-21needreboot23:11:590
12361992925,3cyclictest23168-21head21:06:541
12361992925,3cyclictest23168-21head21:06:541
12361992921,6cyclictest16553-21grep23:42:011
12361992921,6cyclictest16553-21grep23:42:011
12360992923,4cyclictest749-21runrttasks21:56:550
12360992923,4cyclictest749-21runrttasks21:56:550
12360992921,5cyclictest0-21swapper/000:31:540
287972280,5sleep128799-21grep22:47:041
287972280,5sleep128799-21grep22:47:041
12361992821,5cyclictest30187-21timerandwakeup19:57:021
12361992821,5cyclictest14933-21sshd20:43:401
12361992821,5cyclictest14933-21sshd20:43:401
12361992821,5cyclictest10432-21idleruntime23:26:561
12361992821,5cyclictest10432-21idleruntime23:26:561
12361992820,6cyclictest24764-21grep19:42:031
12361992820,6cyclictest24764-21grep19:42:031
12360992822,5cyclictest24410-21latency22:36:560
12360992822,5cyclictest15208-21missed_timers22:11:580
12360992822,5cyclictest15208-21missed_timers22:11:580
12360992822,4cyclictest28242-21head22:46:590
12360992822,4cyclictest28242-21head22:46:590
1236099282,10cyclictest166392timerandwakeup23:42:020
1236099282,10cyclictest166392timerandwakeup23:42:020
12360992819,6cyclictest0-21swapper/022:31:530
12360992819,6cyclictest0-21swapper/022:31:530
12361992723,3cyclictest12359-21cyclictest22:26:541
12361992723,3cyclictest12359-21cyclictest22:26:541
12361992722,3cyclictest0-21swapper/121:51:551
12361992721,5cyclictest1567-21timerandwakeup20:07:021
12361992721,5cyclictest1567-21timerandwakeup20:07:021
12360992721,5cyclictest27191-21sshd22:44:390
12360992721,4cyclictest0-21swapper/021:17:030
12360992721,4cyclictest0-21swapper/021:17:030
12360992721,4cyclictest0-21swapper/021:02:020
12360992720,5cyclictest1751-21switchtime23:02:030
12360992720,5cyclictest1751-21switchtime23:02:030
12361992621,4cyclictest12359-21cyclictest22:31:531
12361992621,4cyclictest12359-21cyclictest22:31:531
12361992621,3cyclictest27124-21gltestperf00:11:541
12361992621,3cyclictest0-21swapper/123:25:031
12361992620,4cyclictest0-21swapper/100:22:031
12361992619,5cyclictest7967-21/usr/sbin/munin20:27:021
12361992619,5cyclictest7967-21/usr/sbin/munin20:27:021
12361992619,5cyclictest3463-21latency_hist00:35:051
12361992619,5cyclictest0-21swapper/120:12:021
12361992613,11cyclictest142-21systemd-journal21:38:031
12361992612,12cyclictest729-21dbus-daemon23:17:021
12360992621,3cyclictest0-21swapper/021:11:550
12360992620,4cyclictest0-21swapper/021:25:030
12360992620,4cyclictest0-21swapper/019:31:550
12360992619,5cyclictest24393-21sshd00:02:470
12360992619,5cyclictest19251-21sshd23:50:450
12360992619,5cyclictest19251-21sshd23:50:450
12360992618,6cyclictest22812-21/usr/sbin/munin21:07:020
12360992618,6cyclictest22812-21/usr/sbin/munin21:07:020
12360992616,8cyclictest18217-21ls20:52:020
12360992616,8cyclictest18217-21ls20:52:020
300402250,2sleep20-21swapper/219:57:012
12361992519,4cyclictest0-21swapper/123:37:001
12361992519,4cyclictest0-21swapper/123:37:001
12361992519,4cyclictest0-21swapper/119:32:041
12361992519,3cyclictest0-21swapper/121:27:011
12361992519,3cyclictest0-21swapper/121:27:011
12361992518,5cyclictest17727-21cut23:46:541
12361992517,6cyclictest18344-21ls20:52:031
12361992517,6cyclictest18344-21ls20:52:031
12361992511,11cyclictest142-21systemd-journal21:43:041
12361992511,11cyclictest142-21systemd-journal21:43:041
12360992520,4cyclictest12359-21cyclictest20:57:020
12360992520,4cyclictest12359-21cyclictest20:57:020
12360992520,3cyclictest0-21swapper/023:06:560
12360992519,4cyclictest30701-21systemctl20:00:040
12360992519,3cyclictest0-21swapper/022:57:020
12360992518,5cyclictest8459-21sshd21:52:400
12360992518,5cyclictest773-21apache221:26:550
12360992518,5cyclictest773-21apache221:26:550
12360992518,5cyclictest4444-21fschecks_time20:16:550
12360992518,5cyclictest4444-21fschecks_time20:16:550
12360992518,5cyclictest21670-21sshd19:35:390
12360992518,5cyclictest21670-21sshd19:35:390
12360992518,5cyclictest13462-21ntp_offset22:06:590
12360992518,5cyclictest0-21swapper/019:42:040
12360992518,5cyclictest0-21swapper/019:42:040
12361992419,3cyclictest0-21swapper/122:12:011
12361992419,3cyclictest0-21swapper/122:12:011
12361992417,5cyclictest27568-21fschecks_time19:51:531
12361992417,5cyclictest12979-21timerwakeupswit23:32:021
12361992417,5cyclictest0-21swapper/122:07:001
12361992416,6cyclictest0-21swapper/123:52:051
12361992416,6cyclictest0-21swapper/123:52:051
12361992415,7cyclictest32195-21ls21:31:511
12361992415,7cyclictest32195-21ls21:31:511
1236099249,2cyclictest181rcu_preempt21:47:010
1236099247,3cyclictest181rcu_preempt00:11:530
12360992419,3cyclictest0-21swapper/023:32:010
12360992418,4cyclictest0-21swapper/020:47:040
12360992415,6cyclictest0-21swapper/022:22:060
12360992415,6cyclictest0-21swapper/022:22:060
1236199232,18cyclictest21621-21bingrep21:01:581
12361992318,3cyclictest0-21swapper/123:01:581
12361992318,3cyclictest0-21swapper/123:01:581
12361992318,3cyclictest0-21swapper/120:21:581
12361992316,5cyclictest25137-21fschecks_time00:06:531
12361992316,5cyclictest25137-21fschecks_time00:06:531
12360992318,3cyclictest0-21swapper/000:22:020
12360992317,5cyclictest12268-21missed_timers20:36:560
12360992314,6cyclictest25678-21fschecks_count19:46:540
12361992217,3cyclictest0-21swapper/122:17:021
12361992216,4cyclictest0-21swapper/121:16:571
12361992216,4cyclictest0-21swapper/121:16:571
12361992214,6cyclictest28714-21gltestperf21:21:541
12361992117,2cyclictest0-21swapper/121:12:031
12360992115,3cyclictest0-21swapper/023:26:360
12360992115,3cyclictest0-21swapper/023:26:360
72022180,2sleep20-21swapper/220:22:052
314042180,3sleep30-21swapper/321:27:023
314042180,3sleep30-21swapper/321:27:023
227862181,6sleep20-21swapper/200:00:062
227862181,6sleep20-21swapper/200:00:062
1236399181,9cyclictest0-21swapper/323:05:403
1236399181,12cyclictest22828-21taskset00:00:063
1236399181,12cyclictest22828-21taskset00:00:063
1236399181,12cyclictest21088-21chrt21:01:543
252792161,2sleep20-21swapper/222:39:102
1236399162,10cyclictest0-21swapper/322:35:043
1236399162,10cyclictest0-21swapper/322:35:043
1236399161,8cyclictest0-21swapper/300:34:323
1236399161,12cyclictest0-21swapper/322:50:063
1236399161,12cyclictest0-21swapper/322:50:063
1236399161,12cyclictest0-21swapper/319:50:053
1236399161,11cyclictest7475-21chrt21:51:543
1236399161,11cyclictest28402-21chrt21:21:523
1236399161,11cyclictest12455-21taskset23:31:563
1236399161,10cyclictest0-21swapper/322:16:533
1236299167,6cyclictest107802sleep220:32:032
1236299167,6cyclictest107802sleep220:32:032
1236299167,5cyclictest0-21swapper/220:45:042
1236299167,5cyclictest0-21swapper/220:45:042
1236299162,10cyclictest0-21swapper/220:01:522
12362991611,3cyclictest210632sleep221:01:532
1236299161,11cyclictest0-21swapper/222:51:512
12362991610,3cyclictest34322chrt21:41:362
12362991610,3cyclictest34322chrt21:41:362
190302151,2sleep30-21swapper/320:56:043
190302151,2sleep30-21swapper/320:56:043
1236399158,4cyclictest0-21swapper/300:06:533
1236399158,4cyclictest0-21swapper/300:06:533
1236399152,10cyclictest0-21swapper/300:30:053
1236399152,10cyclictest0-21swapper/300:30:053
1236399151,9cyclictest8032-21taskset23:20:063
1236399151,8cyclictest0-21swapper/322:20:393
1236399151,8cyclictest0-21swapper/322:20:393
1236399151,8cyclictest0-21swapper/320:13:393
1236399151,8cyclictest0-21swapper/319:30:403
1236399151,12cyclictest0-21swapper/322:51:513
1236399151,12cyclictest0-21swapper/300:01:523
12363991510,3cyclictest258272sleep321:12:033
12363991510,2cyclictest0-21swapper/323:50:053
12363991510,2cyclictest0-21swapper/323:50:053
12363991510,2cyclictest0-21swapper/323:50:053
1236299159,3cyclictest0-21swapper/200:16:572
1236299159,3cyclictest0-21swapper/200:16:572
1236299158,4cyclictest0-21swapper/223:41:562
1236299158,4cyclictest0-21swapper/223:41:562
1236299152,10cyclictest0-21swapper/222:50:062
1236299152,10cyclictest0-21swapper/222:50:062
1236299152,10cyclictest0-21swapper/219:50:052
1236299152,10cyclictest0-21swapper/200:01:522
1236299151,9cyclictest17366-21chrt23:45:182
1236299151,9cyclictest17366-21chrt23:45:182
1236299151,9cyclictest17366-21chrt23:45:182
1236299151,11cyclictest9841-21chrt21:57:002
1236299151,11cyclictest9841-21chrt21:57:002
1236299151,10cyclictest7887-21chrt20:25:512
1236299151,10cyclictest7887-21chrt20:25:512
1236299151,10cyclictest30450-21taskset21:26:532
1236299151,10cyclictest30450-21taskset21:26:532
12362991510,3cyclictest70562sleep223:17:002
272702141,2sleep20-21swapper/219:51:032
12882140,2sleep20-21swapper/223:01:562
12882140,2sleep20-21swapper/223:01:562
1236399149,3cyclictest96002chrt20:30:063
1236399149,3cyclictest96002chrt20:30:063
1236399149,3cyclictest286102chrt00:16:423
1236399149,3cyclictest286102chrt00:16:423
1236399148,4cyclictest222512sleep319:36:563
1236399148,4cyclictest222512sleep319:36:563
1236399147,4cyclictest26132sleep321:37:043
1236399142,8cyclictest27027-21taskset22:42:363
1236399142,5cyclictest0-21swapper/322:36:543
1236399141,9cyclictest564-21chrt23:00:083
1236399141,9cyclictest564-21chrt23:00:083
1236399141,9cyclictest29080-21chrt19:55:333
1236399141,9cyclictest12485-21chrt22:05:053
1236399141,9cyclictest0-21swapper/322:27:393
1236399141,9cyclictest0-21swapper/322:27:393
1236399141,9cyclictest0-21swapper/322:14:183
1236399141,9cyclictest0-21swapper/322:14:183
1236399141,9cyclictest0-21swapper/320:52:393
1236399141,9cyclictest0-21swapper/320:52:393
1236399141,9cyclictest0-21swapper/320:07:393
1236399141,9cyclictest0-21swapper/320:07:393
1236399141,10cyclictest27275-21taskset21:17:013
1236399141,10cyclictest27275-21taskset21:17:013
1236399141,10cyclictest13852-21chrt22:07:023
1236399141,10cyclictest0-21swapper/323:41:543
1236399141,10cyclictest0-21swapper/323:41:543
12363991410,2cyclictest0-21swapper/323:11:583
12363991410,2cyclictest0-21swapper/323:11:583
1236299149,3cyclictest144972chrt22:10:472
1236299149,3cyclictest144972chrt22:10:472
1236299149,3cyclictest0-21swapper/222:26:592
1236299149,3cyclictest0-21swapper/222:26:592
1236299149,2cyclictest0-21swapper/221:15:192
1236299149,2cyclictest0-21swapper/221:15:192
1236299147,4cyclictest0-21swapper/222:06:532
1236299142,9cyclictest0-21swapper/221:46:522
1236299141,9cyclictest6170-21chrt23:15:072
1236299141,9cyclictest6170-21chrt23:15:072
1236299141,9cyclictest18961-21chrt20:55:092
1236299141,9cyclictest18961-21chrt20:55:092
1236299141,9cyclictest17129-21chrt20:50:232
1236299141,9cyclictest17129-21chrt20:50:232
1236299141,9cyclictest11593-21chrt20:35:562
1236299141,10cyclictest4933-21chrt00:37:032
1236299141,10cyclictest27261-21chrt00:11:552
1236299141,10cyclictest25574-21chrt00:06:562
1236299141,10cyclictest25574-21chrt00:06:562
1236299141,10cyclictest24060-21chrt21:07:032
1236299141,10cyclictest24060-21chrt21:07:032
1236299141,10cyclictest20587-21chrt23:52:052
1236299141,10cyclictest20587-21chrt23:52:052
1236299141,10cyclictest1501-21taskset00:30:042
1236299141,10cyclictest1501-21taskset00:30:042
1236299141,10cyclictest10545-21chrt23:26:572
1236299141,10cyclictest10545-21chrt23:26:572
1236399138,3cyclictest96952chrt20:30:183
1236399138,3cyclictest96952chrt20:30:183
1236399138,3cyclictest82092chrt23:21:513
1236399138,3cyclictest60152chrt20:20:333
1236399138,3cyclictest34522chrt21:41:493
1236399138,3cyclictest34522chrt21:41:493
1236399138,3cyclictest319762chrt00:24:463
1236399138,3cyclictest287262chrt19:53:443
1236399138,3cyclictest282612chrt00:14:573
1236399138,3cyclictest116042chrt20:36:053
1236399131,9cyclictest9930-21chrt23:25:133
1236399131,9cyclictest9930-21chrt23:25:133
1236399131,9cyclictest22702-21chrt21:05:273
1236399131,9cyclictest22702-21chrt21:05:273
1236399131,9cyclictest0-21swapper/320:18:403
1236399131,9cyclictest0-21swapper/320:18:403
1236399131,8cyclictest8882-21chrt21:55:203
1236399131,8cyclictest8882-21chrt21:55:203
1236399131,8cyclictest5169-21chrt21:45:073
1236399131,8cyclictest3560-21chrt00:35:163
1236399131,8cyclictest15278-21chrt20:45:223
1236399131,10cyclictest32403-21taskset22:57:033
1236399131,10cyclictest32247-21chrt21:31:533
1236399131,10cyclictest32247-21chrt21:31:533
1236399131,10cyclictest13657-21taskset20:41:523
1236399131,10cyclictest13657-21taskset20:41:523
1236399131,10cyclictest0-21swapper/320:01:523
1236299139,2cyclictest0-21swapper/223:06:392
1236299138,3cyclictest311472chrt22:55:132
1236299138,3cyclictest257182chrt22:41:462
1236299138,3cyclictest231192chrt19:38:152
1236299138,3cyclictest231192chrt19:38:152
1236299138,3cyclictest163172chrt22:15:252
1236299138,3cyclictest136952chrt23:35:572
1236299138,3cyclictest136952chrt23:35:572
1236299138,3cyclictest0-21swapper/220:18:512
1236299138,3cyclictest0-21swapper/220:18:512
1236299131,9cyclictest7020-21chrt21:50:092
1236299131,9cyclictest420-21chrt20:06:372
1236299131,9cyclictest420-21chrt20:06:372
1236299131,9cyclictest22112-21chrt22:31:532
1236299131,9cyclictest22112-21chrt22:31:532
1236299131,9cyclictest15985-21chrt20:46:572
1236299131,8cyclictest8089-21chrt23:20:172
1236299131,8cyclictest23504-21chrt19:40:082
1236299131,8cyclictest23504-21chrt19:40:082
1236299131,8cyclictest1441-21chrt21:35:092
1236299131,8cyclictest0-21swapper/222:20:392
1236299131,8cyclictest0-21swapper/222:20:392
1236299131,8cyclictest0-21swapper/220:13:392
1236299131,8cyclictest0-21swapper/219:30:402
1236299131,7cyclictest0-21swapper/221:25:032
115242130,3sleep20-21swapper/222:01:582
1236399121,8cyclictest19216-21taskset23:50:173
1236399121,8cyclictest19216-21taskset23:50:173
1236299121,9cyclictest24838-21chrt21:11:542
1236299121,9cyclictest11813-21chrt23:30:382
1236299121,8cyclictest31968-21chrt21:30:112
1236299121,8cyclictest31968-21chrt21:30:112
1236299121,8cyclictest30377-21chrt00:20:152
1236299121,8cyclictest1699-21chrt00:30:082
plot09:55:358
setyrange[0.8:*]5
setylabel"Numberoflatencysamples"6
setxrange[0:400]4
setxlabel"Latencyus-Maximumusplottedon2
settitle"Latencyrack4slot1s"0
setterminalpbmcolor1
setoutput"r4s1s.ppm"7
setlogscaley3
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional