You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-30 - 09:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #4, slot #1

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot1s.osadl.org (updated Wed Apr 30, 2025 00:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
313321230,2sleep10-21swapper/119:35:421
313321230,2sleep10-21swapper/119:35:421
274902790,5sleep127489-21ntp_kernel_pll_22:05:411
274902790,5sleep127489-21ntp_kernel_pll_22:05:411
157362760,3chrt0-21swapper/021:35:370
157362760,3chrt0-21swapper/021:35:370
113142660,4chrt11308-21fschecks_count00:15:371
113142660,4chrt11308-21fschecks_count00:15:371
237742630,1chrt0-21swapper/123:20:461
237742630,1chrt0-21swapper/123:20:461
262652600,2sleep30-21swapper/320:35:503
180062550,2sleep018112-21/usr/sbin/munin21:40:440
236252540,3sleep30-21swapper/320:30:403
236252540,3sleep30-21swapper/320:30:403
168822530,3sleep20-21swapper/223:05:232
168822530,3sleep20-21swapper/223:05:232
150112530,1sleep10-21swapper/123:00:241
126672530,3sleep20-21swapper/221:25:462
126672530,3sleep20-21swapper/221:25:462
215722500,1sleep10-21swapper/121:50:371
215722500,1sleep10-21swapper/121:50:371
293992490,2chrt0-21swapper/119:20:391
293992490,2chrt0-21swapper/119:20:391
201802490,4chrt0-21swapper/323:10:473
201802490,4chrt0-21swapper/323:10:473
81882470,3chrt0-21swapper/200:05:452
81882470,3chrt0-21swapper/200:05:452
287722470,2sleep20-21swapper/220:45:222
3502460,3chrt0-21swapper/223:45:392
137482460,3chrt0-21swapper/321:30:373
137482460,3chrt0-21swapper/321:30:373
128772460,3chrt0-21swapper/300:20:213
128772460,3chrt0-21swapper/300:20:213
107342460,3sleep30-21swapper/319:55:423
107342460,3sleep30-21swapper/319:55:423
63642450,3chrt0-21swapper/319:45:373
25487994430,9cyclictest91-1pr/ttyS021:05:360
25487994430,9cyclictest91-1pr/ttyS021:05:360
24392430,3sleep30-21swapper/323:50:433
24392430,3sleep30-21swapper/323:50:433
203942430,2sleep20-21swapper/221:45:482
203942430,2sleep20-21swapper/221:45:482
2548999422,17cyclictest32733-21chrt20:55:392
2548999422,17cyclictest32733-21chrt20:55:392
176842420,3sleep30-21swapper/321:40:383
149372410,4sleep10-21swapper/120:05:491
149372410,4sleep10-21swapper/120:05:491
143572410,3chrt0-21swapper/200:20:492
143572410,3chrt0-21swapper/200:20:492
21302400,2sleep00-21swapper/021:00:270
21302400,2sleep00-21swapper/021:00:270
25487993919,15cyclictest171ktimers/000:25:330
25487993919,15cyclictest171ktimers/000:25:330
2549099377,12cyclictest0-21swapper/320:00:333
2549099377,12cyclictest0-21swapper/320:00:333
2549099364,24cyclictest0-21swapper/323:30:383
2549099364,24cyclictest0-21swapper/319:25:343
2549099363,25cyclictest0-21swapper/300:10:353
2549099363,25cyclictest0-21swapper/300:10:353
64422350,2sleep06480-21ntp_states22:35:440
64422350,2sleep06480-21ntp_states22:35:440
2549099356,24cyclictest0-21swapper/322:55:323
2549099354,25cyclictest0-21swapper/320:05:373
2549099354,25cyclictest0-21swapper/320:05:373
2548999358,16cyclictest118912sleep220:00:332
2548999358,16cyclictest118912sleep220:00:332
2549099341,28cyclictest0-21swapper/323:45:333
25488993427,5cyclictest3190-21threads21:00:481
25488993427,5cyclictest3190-21threads21:00:481
25488993426,6cyclictest3876-21cut23:55:371
25488993426,6cyclictest3876-21cut23:55:371
2549099334,23cyclictest0-21swapper/321:35:323
2549099334,23cyclictest0-21swapper/321:35:323
2549099333,23cyclictest0-21swapper/319:40:353
25488993326,5cyclictest281ktimers/120:50:341
25488993325,6cyclictest281ktimers/119:40:351
25487993328,3cyclictest32300-21grep22:20:210
25487993328,3cyclictest32300-21grep22:20:210
296642320,3sleep30-21swapper/323:35:473
2549099325,23cyclictest0-21swapper/300:30:333
2549099325,23cyclictest0-21swapper/300:30:333
2549099323,22cyclictest0-21swapper/323:05:333
2549099323,22cyclictest0-21swapper/323:05:333
2549099323,22cyclictest0-21swapper/319:20:383
2549099323,22cyclictest0-21swapper/319:20:383
2549099323,20cyclictest0-21swapper/320:55:383
2549099323,20cyclictest0-21swapper/320:55:383
2548999321,11cyclictest6955-21taskset21:10:462
2548999321,11cyclictest6955-21taskset21:10:462
25488993225,5cyclictest6383-21if_eth021:10:381
25488993225,5cyclictest6383-21if_eth021:10:381
25488993224,6cyclictest383-21cron22:50:221
25488993224,6cyclictest383-21cron22:50:221
25488993224,6cyclictest281ktimers/121:55:331
25488993224,6cyclictest281ktimers/121:55:331
25488993224,5cyclictest281ktimers/122:15:351
25488993224,5cyclictest281ktimers/122:15:351
2548899321,28cyclictest281ktimers/120:15:331
25487993226,4cyclictest16346-21unixbench_multi23:00:470
25487993226,4cyclictest16346-21unixbench_multi23:00:470
25487993225,5cyclictest3657-21latency_hist22:30:210
25487993225,5cyclictest3657-21latency_hist22:30:210
25487993218,10cyclictest91-1pr/ttyS023:30:380
291512310,2sleep20-21swapper/223:35:422
2549099313,19cyclictest0-21swapper/300:05:333
2549099313,19cyclictest0-21swapper/300:05:333
25488993123,6cyclictest281ktimers/121:20:341
25488993123,6cyclictest281ktimers/121:20:341
25488993123,5cyclictest281ktimers/120:45:381
25488993123,5cyclictest281ktimers/120:45:381
25488993123,5cyclictest271rcuc/121:35:341
25488993123,5cyclictest271rcuc/121:35:341
25488993122,7cyclictest1-21systemd19:25:471
2548799318,19cyclictest171ktimers/021:45:340
2548799318,19cyclictest171ktimers/021:45:340
25487993126,3cyclictest5116-21ntp_offset19:40:450
25487993122,3cyclictest0-21swapper/019:40:200
25487993122,3cyclictest0-21swapper/019:40:200
25487993119,10cyclictest0-21swapper/021:15:360
25487993119,10cyclictest0-21swapper/021:15:360
25487993118,10cyclictest0-21swapper/000:10:350
25487993118,10cyclictest0-21swapper/000:10:350
2549099302,2cyclictest0-21swapper/319:30:383
2549099302,2cyclictest0-21swapper/319:30:383
2549099302,21cyclictest0-21swapper/320:45:383
2549099302,21cyclictest0-21swapper/320:45:383
25488993023,5cyclictest31887-21sshd23:42:481
25488993023,5cyclictest31887-21sshd23:42:481
25488993023,5cyclictest281ktimers/121:40:331
2548799305,12cyclictest16-21ksoftirqd/019:25:360
25487993024,4cyclictest24632-21cron22:00:210
25487993024,4cyclictest24632-21cron22:00:210
25487993020,7cyclictest0-21swapper/023:30:300
2549099292,21cyclictest0-21swapper/321:05:363
2549099292,21cyclictest0-21swapper/321:05:363
2549099291,3cyclictest0-21swapper/320:40:363
2549099291,3cyclictest0-21swapper/320:25:333
2549099291,3cyclictest0-21swapper/320:25:333
25488992924,4cyclictest24778-21fschecks_time23:25:371
25488992924,4cyclictest24778-21fschecks_time23:25:371
25488992924,4cyclictest24320-21awk20:30:491
25488992924,4cyclictest24320-21awk20:30:491
25488992924,2cyclictest0-21swapper/122:05:201
25488992924,2cyclictest0-21swapper/122:05:201
25488992923,4cyclictest281ktimers/123:05:331
25488992923,4cyclictest281ktimers/123:05:331
25488992922,5cyclictest2266-21kernelversion23:50:411
25488992922,5cyclictest2266-21kernelversion23:50:411
25487992924,3cyclictest2240-21cron19:35:210
25487992924,3cyclictest2240-21cron19:35:210
25487992924,3cyclictest0-21swapper/000:00:210
25487992924,3cyclictest0-21swapper/000:00:210
25487992923,4cyclictest20722-21unixbench_multi20:20:460
25487992923,4cyclictest20722-21unixbench_multi20:20:460
25487992922,5cyclictest383-21cron20:25:290
25487992922,5cyclictest383-21cron20:25:290
25487992922,5cyclictest32636-21tail23:45:380
25487992922,5cyclictest25528-21missed_timers22:00:420
25487992922,5cyclictest25528-21missed_timers22:00:420
25487992922,5cyclictest14068-21processes22:55:430
25487992922,4cyclictest0-21swapper/021:25:200
25487992922,4cyclictest0-21swapper/021:25:200
2549099283,5cyclictest0-21swapper/323:15:343
2549099282,3cyclictest0-21swapper/321:25:333
2549099282,3cyclictest0-21swapper/321:25:333
2549099282,3cyclictest0-21swapper/320:15:333
2549099282,2cyclictest0-21swapper/319:35:363
2549099282,2cyclictest0-21swapper/319:35:363
2549099282,22cyclictest0-21swapper/323:00:343
2549099282,22cyclictest0-21swapper/323:00:343
2549099281,4cyclictest0-21swapper/321:10:343
2549099281,4cyclictest0-21swapper/321:10:343
2549099281,3cyclictest0-21swapper/322:05:323
2549099281,3cyclictest0-21swapper/322:05:323
2549099281,3cyclictest0-21swapper/320:50:343
2549099281,3cyclictest0-21swapper/319:50:363
2549099281,3cyclictest0-21swapper/319:50:363
25488992824,3cyclictest19733-21kernelversion21:45:401
25488992824,3cyclictest19733-21kernelversion21:45:401
25488992824,2cyclictest0-21swapper/121:10:201
25488992824,2cyclictest0-21swapper/121:10:201
25488992823,3cyclictest25486-21cyclictest00:25:131
25488992823,3cyclictest25486-21cyclictest00:25:131
25488992822,4cyclictest464-21fschecks_count20:55:411
25488992822,4cyclictest464-21fschecks_count20:55:411
25488992821,5cyclictest15069-21sshd21:34:481
25488992821,5cyclictest15069-21sshd21:34:481
2548799288,12cyclictest171ktimers/023:50:350
2548799288,12cyclictest171ktimers/023:50:350
25487992823,4cyclictest2855-21needreboot21:00:440
25487992823,4cyclictest2855-21needreboot21:00:440
25487992823,3cyclictest5504-21idleruntime-cro22:35:200
25487992823,3cyclictest5504-21idleruntime-cro22:35:200
25487992822,4cyclictest20363-21idleruntime-cro00:40:200
25487992822,4cyclictest20363-21idleruntime-cro00:40:200
25487992822,4cyclictest12344-21tail00:15:470
25487992822,4cyclictest12344-21tail00:15:470
25487992822,4cyclictest0-21swapper/019:55:430
25487992822,4cyclictest0-21swapper/019:55:430
25487992819,7cyclictest21829-21tr21:50:410
25487992819,7cyclictest21829-21tr21:50:410
2548799281,22cyclictest13863-21sed20:05:300
2549099272,3cyclictest0-21swapper/322:30:333
2549099272,3cyclictest0-21swapper/322:30:333
2549099271,3cyclictest0-21swapper/321:50:323
2549099271,3cyclictest0-21swapper/321:50:323
2549099271,3cyclictest0-21swapper/320:10:323
2549099271,3cyclictest0-21swapper/320:10:323
2549099271,3cyclictest0-21swapper/300:25:333
2549099271,3cyclictest0-21swapper/300:25:333
25488992723,3cyclictest25486-21cyclictest22:20:341
25488992723,3cyclictest25486-21cyclictest22:20:341
25488992721,4cyclictest28349-21latency_hist23:35:281
25488992721,4cyclictest146-21systemd-journal23:39:481
25488992720,5cyclictest8035-21latency_hist19:50:271
25488992720,5cyclictest364-21kernelversion23:45:391
25488992720,5cyclictest21283-21kernelversion23:15:401
25488992720,5cyclictest19203-21kernelversion00:35:401
25488992720,5cyclictest19203-21kernelversion00:35:401
25487992722,3cyclictest17957-21gltestperf20:15:370
25487992722,3cyclictest13815-21gltestperf21:30:380
25487992722,3cyclictest13815-21gltestperf21:30:380
25487992722,3cyclictest12117-21ntp_kernel_pll_22:50:420
25487992722,3cyclictest12117-21ntp_kernel_pll_22:50:420
25487992722,3cyclictest0-21swapper/000:05:210
25487992722,3cyclictest0-21swapper/000:05:210
25487992721,4cyclictest6505-21fschecks_time19:45:390
25487992721,4cyclictest14910-21latency_hist00:25:300
25487992721,4cyclictest14910-21latency_hist00:25:300
82902260,2sleep08289-21needreboot22:40:410
82902260,2sleep08289-21needreboot22:40:410
2549099261,3cyclictest0-21swapper/323:55:343
2549099261,3cyclictest0-21swapper/323:55:343
2549099261,3cyclictest0-21swapper/322:25:373
2549099261,3cyclictest0-21swapper/322:25:373
2549099261,2cyclictest0-21swapper/322:40:333
2549099261,2cyclictest0-21swapper/322:40:333
25488992621,3cyclictest7768-21gltestperf00:05:371
25488992621,3cyclictest7768-21gltestperf00:05:371
25488992620,4cyclictest9854-21latency_hist19:55:241
25488992620,4cyclictest9854-21latency_hist19:55:241
25488992620,4cyclictest7401-21latency_hist00:05:301
2548799266,16cyclictest171ktimers/000:30:340
2548799266,16cyclictest171ktimers/000:30:340
25487992621,3cyclictest29915-21unixbench-2d22:10:470
25487992621,3cyclictest0-21swapper/022:05:350
25487992621,3cyclictest0-21swapper/022:05:350
25487992620,4cyclictest28729-21gltestperf23:35:370
25487992620,4cyclictest0-21swapper/019:50:410
25487992620,4cyclictest0-21swapper/019:50:410
25487992619,5cyclictest0-21swapper/021:29:480
25487992619,5cyclictest0-21swapper/021:29:480
25487992618,6cyclictest32673-21fschecks_count22:20:330
25487992618,6cyclictest32673-21fschecks_count22:20:330
25487992616,7cyclictest0-21swapper/023:40:390
25487992616,7cyclictest0-21swapper/023:40:390
234342260,3chrt0-21swapper/220:30:392
234342260,3chrt0-21swapper/220:30:392
181092260,1sleep1281ktimers/100:30:491
181092260,1sleep1281ktimers/100:30:491
2549099252,3cyclictest0-21swapper/321:20:343
2549099252,3cyclictest0-21swapper/321:20:343
2549099252,3cyclictest0-21swapper/321:15:333
2549099252,3cyclictest0-21swapper/321:15:333
2549099251,6cyclictest0-21swapper/323:40:393
2549099251,6cyclictest0-21swapper/323:40:393
2549099251,3cyclictest0-21swapper/323:25:333
2549099251,3cyclictest0-21swapper/323:25:333
2549099251,3cyclictest0-21swapper/322:50:343
2549099251,3cyclictest0-21swapper/322:50:343
2549099251,3cyclictest0-21swapper/321:55:333
2549099251,3cyclictest0-21swapper/321:55:333
2549099251,3cyclictest0-21swapper/300:00:333
2549099251,3cyclictest0-21swapper/300:00:333
25488992521,2cyclictest0-21swapper/120:00:201
25488992521,2cyclictest0-21swapper/120:00:201
25488992519,4cyclictest9317-21latency_hist22:45:211
25488992519,4cyclictest9317-21latency_hist22:45:211
25487992521,2cyclictest0-21swapper/023:20:210
25487992520,3cyclictest6623-21needreboot21:10:420
25487992520,3cyclictest6623-21needreboot21:10:420
25487992520,3cyclictest31120-21gltestperf20:50:380
25487992520,3cyclictest0-21swapper/023:30:200
25487992518,4cyclictest0-21swapper/020:45:410
25487992518,4cyclictest0-21swapper/020:45:410
25487992511,4cyclictest16-21ksoftirqd/020:05:370
25487992511,4cyclictest16-21ksoftirqd/020:05:370
2549099242,3cyclictest0-21swapper/300:35:343
2549099242,3cyclictest0-21swapper/300:35:343
2549099242,2cyclictest0-21swapper/322:35:343
2549099242,2cyclictest0-21swapper/322:35:343
2549099241,9cyclictest32139-21chrt22:18:333
2549099241,9cyclictest32139-21chrt22:18:333
2549099241,3cyclictest110412sleep322:49:003
2549099241,3cyclictest110412sleep322:49:003
2549099241,3cyclictest0-21swapper/322:00:353
2549099241,3cyclictest0-21swapper/322:00:353
25488992420,2cyclictest0-21swapper/122:35:481
25488992420,2cyclictest0-21swapper/122:35:481
25488992420,2cyclictest0-21swapper/122:10:421
25488992419,3cyclictest0-21swapper/120:00:371
25488992419,3cyclictest0-21swapper/120:00:371
25488992418,3cyclictest0-21swapper/121:25:401
25488992418,3cyclictest0-21swapper/121:25:401
25488992417,5cyclictest9790-21kernelversion00:10:411
25488992417,5cyclictest9790-21kernelversion00:10:411
25488992417,5cyclictest5751-21fschecks_time00:00:371
25488992417,5cyclictest5643-21latency_hist22:35:271
25488992417,5cyclictest5643-21latency_hist22:35:271
25487992419,4cyclictest18176-21unixbench-2d23:05:450
25487992419,4cyclictest18176-21unixbench-2d23:05:450
25487992419,3cyclictest0-21swapper/020:40:420
25487992418,5cyclictest10184-21needreboot22:45:420
25487992418,5cyclictest10184-21needreboot22:45:420
25487992418,4cyclictest8223-21proc_pri00:05:460
25487992418,4cyclictest8223-21proc_pri00:05:460
25487992418,4cyclictest19283-21gltestperf23:10:370
25487992418,4cyclictest19283-21gltestperf23:10:370
25487992418,4cyclictest16441-21needreboot20:10:410
25487992418,4cyclictest16441-21needreboot20:10:410
202992240,3sleep30-21swapper/320:20:433
202992240,3sleep30-21swapper/320:20:433
2549099232,5cyclictest0-21swapper/322:10:423
2549099231,8cyclictest23975-21chrt23:20:483
2549099231,8cyclictest23975-21chrt23:20:483
2549099231,7cyclictest387-21taskset22:20:343
2549099231,7cyclictest387-21taskset22:20:343
2549099231,7cyclictest14687-21chrt00:25:003
2549099231,7cyclictest14687-21chrt00:25:003
2549099231,6cyclictest3618-21chrt21:02:513
2549099231,6cyclictest3618-21chrt21:02:513
25488992319,2cyclictest0-21swapper/121:15:401
25488992319,2cyclictest0-21swapper/121:15:401
25488992316,5cyclictest25248-21cpuspeed20:35:381
25487992318,4cyclictest12231-21gltestperf20:00:370
25487992317,3cyclictest0-21swapper/019:20:400
25487992317,3cyclictest0-21swapper/019:20:400
25487992316,5cyclictest0-21swapper/023:20:460
25487992316,5cyclictest0-21swapper/023:20:460
25487992316,4cyclictest0-21swapper/020:30:470
25487992316,4cyclictest0-21swapper/020:30:470
25487992315,6cyclictest1-21systemd20:38:490
2549099221,3cyclictest0-21swapper/321:45:343
2549099221,3cyclictest0-21swapper/321:45:343
25488992218,2cyclictest0-21swapper/120:10:451
25488992218,2cyclictest0-21swapper/120:10:451
25488992217,4cyclictest12556-21unixbench_multi22:50:481
25488992217,4cyclictest12556-21unixbench_multi22:50:481
25488992216,3cyclictest0-21swapper/120:40:491
25488992117,2cyclictest0-21swapper/123:10:341
25488992117,2cyclictest0-21swapper/123:10:341
25488992117,2cyclictest0-21swapper/120:20:451
25488992117,2cyclictest0-21swapper/120:20:451
25488992117,2cyclictest0-21swapper/100:25:401
25488992117,2cyclictest0-21swapper/100:25:401
25488992116,3cyclictest0-21swapper/122:25:491
25488992116,3cyclictest0-21swapper/122:25:491
25488992111,6cyclictest281ktimers/123:00:341
25488992111,6cyclictest281ktimers/123:00:341
2548999207,8cyclictest0-21swapper/221:45:292
2548999204,9cyclictest0-21swapper/223:30:312
2548999203,9cyclictest0-21swapper/222:50:342
2548999203,9cyclictest0-21swapper/222:50:342
2548999203,9cyclictest0-21swapper/200:15:352
2548999203,9cyclictest0-21swapper/200:15:352
25488992016,3cyclictest22339-21mailstats20:25:451
25488992016,3cyclictest22339-21mailstats20:25:451
25488992015,3cyclictest0-21swapper/119:30:421
25488992015,3cyclictest0-21swapper/119:30:421
2548999195,4cyclictest0-21swapper/219:30:362
2548999195,4cyclictest0-21swapper/219:30:362
2548999193,9cyclictest0-21swapper/222:30:352
2548999193,9cyclictest0-21swapper/222:30:352
2548999193,9cyclictest0-21swapper/222:00:372
2548999193,9cyclictest0-21swapper/222:00:372
2548999193,9cyclictest0-21swapper/221:56:482
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional