You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-27 - 09:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #4, slot #1

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot1s.osadl.org (updated Wed Aug 27, 2025 00:44:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2932521140,3sleep20-21swapper/222:02:352
2932521140,3sleep20-21swapper/222:02:352
1578021090,3chrt0-21swapper/300:17:393
1578021090,3chrt0-21swapper/300:17:393
107182850,3chrt0-21swapper/221:12:382
107182850,3chrt0-21swapper/221:12:382
90252770,3sleep30-21swapper/300:02:153
90252770,3sleep30-21swapper/300:02:153
172082700,5sleep10-21swapper/121:32:241
172082700,5sleep10-21swapper/121:32:241
102022690,2sleep20-21swapper/200:02:382
102022690,2sleep20-21swapper/200:02:382
4962670,3sleep30-21swapper/319:22:293
32602670,2chrt0-21swapper/120:52:391
29010996220,29cyclictest0-21swapper/022:07:270
29010996220,29cyclictest0-21swapper/022:07:270
242292610,3chrt0-21swapper/200:42:142
29010996025,25cyclictest0-21swapper/022:32:290
29010996025,25cyclictest0-21swapper/022:32:290
17772590,3sleep2351rcuc/223:42:322
2901499581,6cyclictest181rcu_preempt20:22:271
256742580,3sleep30-21swapper/323:17:413
234622580,3chrt0-21swapper/323:12:383
234622580,3chrt0-21swapper/323:12:383
29010995620,27cyclictest0-21swapper/021:12:250
29010995620,27cyclictest0-21swapper/021:12:250
5022550,3chrt0-21swapper/123:37:371
5022550,3chrt0-21swapper/123:37:371
29010995523,25cyclictest394-21snmpd23:57:260
29010995523,25cyclictest394-21snmpd23:57:260
29010995417,23cyclictest0-21swapper/019:22:250
2901099534,41cyclictest394-21snmpd21:47:250
2901099534,41cyclictest394-21snmpd21:47:250
29010995316,23cyclictest0-21swapper/000:37:250
2901499523,8cyclictest181rcu_preempt22:57:261
2901499523,8cyclictest181rcu_preempt22:57:261
164662520,2sleep20-21swapper/222:52:402
164662520,2sleep20-21swapper/222:52:402
135312520,3sleep20-21swapper/222:47:292
135312520,3sleep20-21swapper/222:47:292
29010995118,26cyclictest0-21swapper/000:12:260
29010995118,26cyclictest0-21swapper/000:12:260
2901499493,9cyclictest181rcu_preempt22:52:271
2901499493,9cyclictest181rcu_preempt22:52:271
170152490,2chrt0-21swapper/300:22:293
170152490,2chrt0-21swapper/300:22:293
29010994815,20cyclictest0-21swapper/000:32:270
29010994815,20cyclictest0-21swapper/000:32:270
29010994734,10cyclictest11800-21apt-get19:52:260
29010994734,10cyclictest11800-21apt-get19:52:260
29010994632,10cyclictest21225-21apt-get20:17:280
29010994632,10cyclictest21225-21apt-get20:17:280
29010994630,10cyclictest26773-21apt-get21:57:280
29010994630,10cyclictest26773-21apt-get21:57:280
29010994628,11cyclictest22767-21apt-get23:12:320
29010994628,11cyclictest22767-21apt-get23:12:320
29010994619,8cyclictest21018-21apt-get21:42:290
29010994619,8cyclictest21018-21apt-get21:42:290
29010994525,16cyclictest171ktimers/023:52:260
29010994525,16cyclictest171ktimers/023:52:260
29010994433,8cyclictest9599-21192.168.115.40-manager19:46:200
29010994433,8cyclictest9599-21192.168.115.40-manager19:46:200
29010994432,9cyclictest32119-21192.168.115.40-manager20:43:210
29010994432,10cyclictest3191-21192.168.115.40-manager23:45:450
2901099434,32cyclictest663-21runrttasks22:02:290
2901099434,32cyclictest663-21runrttasks22:02:290
29010994332,9cyclictest1403-21192.168.115.40-manager22:14:130
29010994332,9cyclictest1403-21192.168.115.40-manager22:14:130
29010994329,10cyclictest5797-21apt-get22:27:290
29010994329,10cyclictest5797-21apt-get22:27:290
29010994328,10cyclictest28437-21apt-get23:27:290
29010994328,10cyclictest28437-21apt-get23:27:290
209792430,2chrt20980-21/usr/sbin/munin23:07:301
193682430,2chrt19369-21fschecks_time21:37:330
193682430,2chrt19369-21fschecks_time21:37:330
101972430,3sleep30-21swapper/322:37:333
101972430,3sleep30-21swapper/322:37:333
2901499421,7cyclictest181rcu_preempt23:27:291
2901499421,7cyclictest181rcu_preempt23:27:291
2901099426,28cyclictest55150irq/47-eth000:07:260
2901099426,28cyclictest55150irq/47-eth000:07:260
29010994232,8cyclictest14645-21192.168.115.40-manager22:48:100
29010994232,8cyclictest14645-21192.168.115.40-manager22:48:100
29010994231,9cyclictest5221-21192.168.115.40-manager22:23:080
29010994231,9cyclictest5221-21192.168.115.40-manager22:23:080
29010994231,8cyclictest26373-21192.168.115.40-manager21:55:210
29010994231,8cyclictest26373-21192.168.115.40-manager21:55:210
29010994214,12cyclictest11558-21apt21:17:270
29010994214,12cyclictest11558-21apt21:17:270
29010994214,11cyclictest18425-21taskset22:58:260
29010994214,11cyclictest18425-21taskset22:58:260
29010994213,12cyclictest9447-21ssh21:12:140
29010994213,12cyclictest9447-21ssh21:12:140
183002420,3sleep30-21swapper/321:32:373
183002420,3sleep30-21swapper/321:32:373
122762420,3chrt0-21swapper/319:52:323
122762420,3chrt0-21swapper/319:52:323
59502410,3sleep10-21swapper/122:27:311
59502410,3sleep10-21swapper/122:27:311
29016994123,11cyclictest0-21swapper/320:17:293
29016994123,11cyclictest0-21swapper/320:17:293
2901499413,34cyclictest394-21snmpd21:02:271
2901499413,34cyclictest394-21snmpd21:02:271
29010994131,8cyclictest5072-21192.168.115.40-manager23:50:460
29010994131,8cyclictest5072-21192.168.115.40-manager23:50:460
29010994131,7cyclictest3470-21192.168.115.40-manager20:53:410
29010994131,5cyclictest30322-21apt-get23:32:250
29010994131,5cyclictest30322-21apt-get23:32:250
29010994130,9cyclictest7305-21192.168.115.40-manager21:04:210
29010994130,9cyclictest7305-21192.168.115.40-manager21:04:210
29010994128,10cyclictest11466-21apt-get22:42:320
29010994128,10cyclictest11466-21apt-get22:42:320
29010994117,13cyclictest24994-21taskset20:27:250
29010994117,13cyclictest24994-21taskset20:27:250
29010994115,16cyclictest5340-21seq20:58:330
29010994115,16cyclictest5340-21seq20:58:330
2901099411,27cyclictest26563-21taskset23:22:310
29010994031,6cyclictest28567-21192.168.115.40-manager20:36:210
29010994028,9cyclictest23135-21apt-get20:22:270
29010994017,7cyclictest21143-21chrt20:17:210
29010994017,7cyclictest21143-21chrt20:17:210
29010994016,9cyclictest10777-21chrt00:06:180
29010994016,9cyclictest10777-21chrt00:06:180
29010994015,15cyclictest10911-21chrt22:38:470
29010994015,15cyclictest10911-21chrt22:38:470
2901499391,8cyclictest181rcu_preempt20:17:281
2901499391,8cyclictest181rcu_preempt20:17:281
29014993914,20cyclictest281ktimers/120:57:261
29014993914,20cyclictest281ktimers/120:57:261
29010993929,8cyclictest20563-21192.168.115.40-manager23:06:210
29010993929,8cyclictest20563-21192.168.115.40-manager23:06:210
29010993929,8cyclictest20249-21192.168.115.40-manager00:30:200
29010993928,8cyclictest1769-21192.168.115.40-manager20:50:510
29010993928,8cyclictest1769-21192.168.115.40-manager20:50:510
29010993925,11cyclictest20884-21apt-get23:07:280
29010993924,11cyclictest9922-21apt-get19:47:270
29010993924,11cyclictest9922-21apt-get19:47:270
29010993923,10cyclictest16772-21apt-get00:22:250
29010993923,10cyclictest16772-21apt-get00:22:250
29010993916,17cyclictest171ktimers/020:37:320
29010993916,17cyclictest171ktimers/020:37:320
29010993916,10cyclictest3366-21chrt22:20:060
29010993916,10cyclictest3366-21chrt22:20:060
29010993915,15cyclictest16893-21taskset20:03:130
29010993915,15cyclictest16893-21taskset20:03:130
29010993914,9cyclictest16282-21seq00:19:330
29010993914,9cyclictest16282-21seq00:19:330
29015993810,16cyclictest0-21swapper/220:17:292
29015993810,16cyclictest0-21swapper/220:17:292
29010993829,6cyclictest16737-21192.168.115.40-manager22:55:500
29010993829,6cyclictest16737-21192.168.115.40-manager22:55:500
29010993828,8cyclictest0-21swapper/020:07:510
29010993827,8cyclictest32207-21apt-get23:37:280
29010993827,8cyclictest32207-21apt-get23:37:280
29010993815,9cyclictest15014-21chrt21:26:070
29010993815,9cyclictest15014-21chrt21:26:070
29010993814,7cyclictest25983-21taskset23:18:350
29010993814,15cyclictest15081-21chrt19:59:200
29010993814,15cyclictest15081-21chrt19:59:200
29010993814,10cyclictest7521-21taskset19:39:100
29010993814,10cyclictest7521-21taskset19:39:100
2901499371,5cyclictest181rcu_preempt19:47:271
2901499371,5cyclictest181rcu_preempt19:47:271
29010993730,4cyclictest4256-21apt-get19:32:280
29010993722,13cyclictest0-21swapper/019:27:260
29010993722,13cyclictest0-21swapper/019:27:260
2901599365,18cyclictest321932sleep223:37:272
2901599365,18cyclictest321932sleep223:37:272
29014993625,6cyclictest0-21swapper/122:17:281
29014993625,6cyclictest0-21swapper/122:17:281
109942360,2sleep110854-21latency_hist00:07:211
109942360,2sleep110854-21latency_hist00:07:211
29014993530,3cyclictest13525-21fschecks_time22:47:291
29014993530,3cyclictest13525-21fschecks_time22:47:291
29010993513,10cyclictest18569-21taskset21:33:360
29010993513,10cyclictest18569-21taskset21:33:360
287692350,10sleep1281ktimers/122:02:301
287692350,10sleep1281ktimers/122:02:301
47362340,3chrt0-21swapper/220:57:352
47362340,3chrt0-21swapper/220:57:352
29010993425,7cyclictest0-21swapper/021:32:130
29010993425,7cyclictest0-21swapper/021:32:130
279532340,2chrt27956-21sort20:32:371
7422330,2sleep129-21ksoftirqd/122:12:351
7422330,2sleep129-21ksoftirqd/122:12:351
160202330,3sleep10-21swapper/120:02:311
160202330,3sleep10-21swapper/120:02:311
29014993225,5cyclictest19481-21kernelversion23:02:321
29014993225,5cyclictest19481-21kernelversion23:02:321
29014993211,16cyclictest281ktimers/121:42:291
29014993211,16cyclictest281ktimers/121:42:291
293612310,2sleep10-21swapper/120:37:371
293612310,2sleep10-21swapper/120:37:371
29014993125,4cyclictest20998-21cut00:32:311
29014993125,4cyclictest20998-21cut00:32:311
29014993123,6cyclictest11471-21latency_hist21:17:201
29014993123,6cyclictest11471-21latency_hist21:17:201
116212310,2sleep30-21swapper/319:52:153
116212310,2sleep30-21swapper/319:52:153
2901499303,8cyclictest181rcu_preempt21:52:311
2901499303,8cyclictest181rcu_preempt21:52:311
29014993026,3cyclictest29003-21cyclictest22:46:511
29014993026,3cyclictest29003-21cyclictest22:46:511
272162300,3sleep20-21swapper/223:22:382
155052300,8sleep115508-21needreboot00:17:331
155052300,8sleep115508-21needreboot00:17:331
46932290,1sleep14795-21latency19:32:331
2901599296,14cyclictest231162sleep220:22:262
176422290,3sleep30-21swapper/322:57:313
176422290,3sleep30-21swapper/322:57:313
29014992823,4cyclictest704-21gltestperf19:22:301
29014992821,5cyclictest14405-21ntp_offset19:57:361
29014992821,5cyclictest14405-21ntp_offset19:57:361
29014992820,6cyclictest15664-21gltestperf21:27:311
29014992820,6cyclictest15664-21gltestperf21:27:311
29014992819,7cyclictest8695-21sort22:32:411
29014992819,7cyclictest8695-21sort22:32:411
29014992819,7cyclictest2356-21/usr/sbin/munin19:27:361
29014992819,7cyclictest2356-21/usr/sbin/munin19:27:361
29014992817,8cyclictest0-21swapper/123:12:311
29014992817,8cyclictest0-21swapper/123:12:311
2901499281,21cyclictest26979-21latency_hist20:32:231
29014992721,3cyclictest0-21swapper/100:07:391
29014992721,3cyclictest0-21swapper/100:07:391
29014992720,5cyclictest9735-21fschecks_count22:37:301
29014992720,5cyclictest9735-21fschecks_count22:37:301
29014992720,5cyclictest9384-21sshd21:11:511
29014992720,5cyclictest9384-21sshd21:11:511
29014992720,5cyclictest25809-21processes20:27:371
29014992619,5cyclictest902-21needreboot20:47:381
29014992619,5cyclictest902-21needreboot20:47:381
29014992521,2cyclictest0-21swapper/121:22:301
29014992521,2cyclictest0-21swapper/121:22:301
29014992519,3cyclictest0-21swapper/121:47:341
29014992519,3cyclictest0-21swapper/121:47:341
29016992413,6cyclictest324472sleep322:12:263
29016992413,6cyclictest324472sleep322:12:263
2901699241,16cyclictest1899-21taskset22:17:273
2901699241,16cyclictest1899-21taskset22:17:273
29014992418,5cyclictest26748-21fschecks_time23:22:341
29014992418,5cyclictest25310-21needreboot23:17:371
29014992418,5cyclictest20232-21sshd00:30:091
29014992418,4cyclictest8706-21needreboot19:42:341
29014992418,4cyclictest8706-21needreboot19:42:341
2901699231,9cyclictest0-21swapper/323:07:283
2901599232,12cyclictest2337-21taskset19:27:222
2901599231,15cyclictest0-21swapper/221:47:232
2901599231,15cyclictest0-21swapper/221:47:232
29014992317,5cyclictest19858-21latency20:12:341
29014992317,5cyclictest19858-21latency20:12:341
29014992317,3cyclictest0-21swapper/123:52:311
29014992317,3cyclictest0-21swapper/123:52:311
29014992317,3cyclictest0-21swapper/123:32:271
29014992317,3cyclictest0-21swapper/123:32:271
29014992316,5cyclictest4148-21missed_timers23:47:381
29014992316,5cyclictest4148-21missed_timers23:47:381
29014992316,5cyclictest4038-21forks22:22:321
29014992316,5cyclictest4038-21forks22:22:321
29014992316,5cyclictest23231-21ntp_kernel_pll_00:37:341
29014992312,9cyclictest146-21systemd-journal20:46:511
9302220,4sleep20-21swapper/220:47:382
9302220,4sleep20-21swapper/220:47:382
2901699224,6cyclictest0-21swapper/319:27:283
2901699224,6cyclictest0-21swapper/319:27:283
2901699223,9cyclictest0-21swapper/322:32:303
2901699223,9cyclictest0-21swapper/322:32:303
2901699222,9cyclictest0-21swapper/323:02:273
2901699222,9cyclictest0-21swapper/323:02:273
2901699222,11cyclictest431rcuc/320:22:263
29014992218,3cyclictest31127-21latency22:07:331
29014992218,3cyclictest31127-21latency22:07:331
29014992217,3cyclictest0-21swapper/123:57:321
29014992217,3cyclictest0-21swapper/123:57:321
29014992216,5cyclictest11911-21head21:17:321
29014992216,5cyclictest11911-21head21:17:321
29014992216,4cyclictest17835-21idleruntime20:07:321
29014992214,6cyclictest27429-21needreboot21:57:351
29014992214,6cyclictest27429-21needreboot21:57:351
2901499221,15cyclictest16858-21ls00:22:271
2901499221,15cyclictest16858-21ls00:22:271
306642210,4sleep20-21swapper/222:07:292
306642210,4sleep20-21swapper/222:07:292
2901699213,9cyclictest0-21swapper/323:52:283
2901699213,9cyclictest0-21swapper/323:52:283
2901699213,9cyclictest0-21swapper/319:32:283
2901499217,9cyclictest0-21swapper/119:52:271
2901499217,9cyclictest0-21swapper/119:52:271
29014992115,3cyclictest0-21swapper/123:42:441
29014992114,5cyclictest19354-21fschecks_time21:37:331
29014992114,5cyclictest19354-21fschecks_time21:37:331
2901699203,9cyclictest0-21swapper/321:22:283
2901699203,9cyclictest0-21swapper/321:22:283
2901699203,9cyclictest0-21swapper/321:07:283
2901699203,9cyclictest0-21swapper/321:07:283
2901699203,9cyclictest0-21swapper/300:42:233
2901699202,8cyclictest0-21swapper/321:57:283
2901699202,8cyclictest0-21swapper/321:57:283
2901599203,9cyclictest0-21swapper/200:32:292
2901599203,9cyclictest0-21swapper/200:32:292
29014992015,3cyclictest0-21swapper/100:12:311
29014992015,3cyclictest0-21swapper/100:12:311
2901699195,3cyclictest0-21swapper/323:32:253
2901699195,3cyclictest0-21swapper/323:32:253
2901699193,9cyclictest0-21swapper/322:27:293
2901699193,9cyclictest0-21swapper/322:27:293
2901699193,9cyclictest0-21swapper/321:47:263
2901699193,9cyclictest0-21swapper/321:47:263
2901699193,9cyclictest0-21swapper/321:42:313
2901699193,9cyclictest0-21swapper/321:42:313
2901699193,9cyclictest0-21swapper/320:02:273
2901699193,9cyclictest0-21swapper/320:02:273
2901699193,9cyclictest0-21swapper/300:02:283
2901699193,9cyclictest0-21swapper/300:02:283
2901699192,9cyclictest0-21swapper/300:12:283
2901699192,9cyclictest0-21swapper/300:12:283
141652190,2sleep20-21swapper/221:22:352
141652190,2sleep20-21swapper/221:22:352
2901699183,9cyclictest0-21swapper/320:57:273
2901699183,9cyclictest0-21swapper/320:57:273
2901699183,9cyclictest0-21swapper/320:27:273
2901699183,9cyclictest0-21swapper/320:27:273
2901699183,9cyclictest0-21swapper/319:37:283
2901699183,9cyclictest0-21swapper/319:37:283
2901699183,8cyclictest0-21swapper/322:07:283
2901699183,8cyclictest0-21swapper/322:07:283
2901699183,8cyclictest0-21swapper/320:52:283
2901699183,8cyclictest0-21swapper/320:47:313
2901699183,8cyclictest0-21swapper/320:47:313
2901699183,8cyclictest0-21swapper/320:37:323
2901699183,8cyclictest0-21swapper/320:37:323
2901699182,8cyclictest0-21swapper/321:17:293
2901699182,8cyclictest0-21swapper/321:17:293
2901699182,7cyclictest0-21swapper/322:52:273
2901699182,7cyclictest0-21swapper/322:52:273
29016991813,3cyclictest302122sleep323:32:203
29016991813,3cyclictest302122sleep323:32:203
2901699181,12cyclictest8043-21chrt19:42:273
2901699181,12cyclictest8043-21chrt19:42:273
2901699181,12cyclictest19156-21taskset00:27:323
29014991812,3cyclictest0-21swapper/119:37:301
29014991812,3cyclictest0-21swapper/119:37:301
2901699174,7cyclictest0-21swapper/320:42:263
2901699173,8cyclictest0-21swapper/323:42:323
2901699173,8cyclictest0-21swapper/322:02:293
2901699173,8cyclictest0-21swapper/322:02:293
2901699173,8cyclictest0-21swapper/321:37:303
2901699173,8cyclictest0-21swapper/321:37:303
2901699173,8cyclictest0-21swapper/319:57:283
2901699173,8cyclictest0-21swapper/319:57:283
2901699173,8cyclictest0-21swapper/300:32:293
2901699173,8cyclictest0-21swapper/300:32:293
2901699173,7cyclictest0-21swapper/323:22:313
2901699173,6cyclictest0-21swapper/300:07:283
2901699173,6cyclictest0-21swapper/300:07:283
2901699171,12cyclictest3924-21chrt22:22:303
2901699171,12cyclictest3924-21chrt22:22:303
2901699171,12cyclictest17466-21chrt20:07:273
2901599173,9cyclictest0-21swapper/200:27:202
2901599173,9cyclictest0-21swapper/200:27:202
2901599173,10cyclictest0-21swapper/222:22:512
2901599173,10cyclictest0-21swapper/222:22:512
2901599173,10cyclictest0-21swapper/221:07:282
2901599173,10cyclictest0-21swapper/221:07:282
29015991711,4cyclictest229892sleep221:47:242
29015991711,4cyclictest229892sleep221:47:242
29015991710,5cyclictest304072sleep223:32:272
29015991710,5cyclictest304072sleep223:32:272
2901699163,9cyclictest0-21swapper/323:47:333
2901699163,9cyclictest0-21swapper/323:47:333
2901699163,9cyclictest0-21swapper/321:52:313
2901699163,9cyclictest0-21swapper/321:52:313
2901699161,11cyclictest32210-21taskset23:37:273
2901699161,11cyclictest32210-21taskset23:37:273
2901699161,10cyclictest20490-21chrt20:12:423
2901699161,10cyclictest20490-21chrt20:12:423
2901599167,7cyclictest0-21swapper/200:27:232
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional