You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-15 - 21:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Sat Nov 15, 2025 12:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3990992725,1cyclictest34-21ksoftirqd/312:15:003
399099240,23cyclictest0-21swapper/309:01:353
398499240,23cyclictest0-21swapper/209:01:352
398499240,1cyclictest11588-21apt-get10:46:352
398099240,23cyclictest21530-21awk10:00:001
3990992321,1cyclictest313-21systemd-journal10:32:223
3990992321,1cyclictest29721-21awk12:29:593
399099230,23cyclictest0-21swapper/309:56:353
398499230,1cyclictest30805-21idleruntime-cro11:25:002
398099230,22cyclictest313-21systemd-journal12:14:591
398099230,1cyclictest3539-21apt-get10:30:001
3976992322,0cyclictest0-21swapper/010:06:330
3976992320,2cyclictest25025-21idleruntime-cro09:00:000
397699230,22cyclictest0-21swapper/008:01:350
3990992220,1cyclictest8810-21sendmail-msp07:20:003
3990992220,1cyclictest22251-21apt-get08:55:003
3990992220,1cyclictest0-21swapper/308:54:593
399099220,21cyclictest8166-21apt-get08:26:353
399099220,21cyclictest0-21swapper/311:21:363
399099220,21cyclictest0-21swapper/310:16:353
399099220,1cyclictest8106-21munin-run11:44:593
3984992220,1cyclictest8814-21basename07:20:002
3984992220,1cyclictest22255-21perf08:55:002
3984992219,2cyclictest22861-21apt-get08:55:012
398499220,21cyclictest23939-21latency_hist10:05:002
398499220,21cyclictest0-21swapper/210:16:352
3980992222,0cyclictest0-21swapper/108:01:351
3980992220,1cyclictest30811-21idleruntime-cro11:24:591
3980992220,1cyclictest20090-21turbostat.cron12:10:001
3980992220,1cyclictest0-21swapper/110:15:011
398099220,21cyclictest9894-21apt-get09:36:361
398099220,21cyclictest25436-21apt-get09:01:351
398099220,21cyclictest0-21swapper/110:41:341
398099220,21cyclictest0-21swapper/109:46:371
398099220,21cyclictest0-21swapper/108:55:011
398099220,21cyclictest0-21swapper/108:11:361
398099220,21cyclictest0-21swapper/107:16:341
398099220,1cyclictest6772-21apt-get10:36:361
398099220,1cyclictest11198-21latency_hist07:25:001
3976992221,1cyclictest0-21swapper/010:41:340
3976992220,2cyclictest598-21rs:main0
397699220,21cyclictest8095-21cat11:45:000
397699220,21cyclictest4314-21apt-get07:11:340
397699220,21cyclictest0-21swapper/012:36:340
397699220,21cyclictest0-21swapper/010:01:340
397699220,21cyclictest0-21swapper/009:11:340
397699220,21cyclictest0-21swapper/009:06:340
397699220,1cyclictest3225-21grep11:35:000
397699220,1cyclictest31110-21cron10:20:000
397699220,1cyclictest0-21swapper/007:46:330
3990992121,0cyclictest30088-21apt-get12:31:353
3990992120,0cyclictest0-21swapper/311:26:363
3990992120,0cyclictest0-21swapper/310:06:353
399099210,21cyclictest15266-21sendmail12:00:003
399099210,21cyclictest0-21swapper/309:37:133
399099210,20cyclictest2637-21apt-get09:21:343
399099210,20cyclictest0-21swapper/311:06:363
399099210,0cyclictest1506-21apt-get10:26:363
399099210,0cyclictest0-21swapper/308:21:353
3984992120,1cyclictest24318-21apt-get10:06:352
3984992120,1cyclictest22481-21awk12:15:002
3984992120,1cyclictest0-21swapper/210:41:332
3984992120,0cyclictest3288-21apt-get08:16:352
3984992119,1cyclictest15430-21apt-get08:41:332
398499210,20cyclictest14712-21apt-get09:46:372
398499210,20cyclictest1183-21apt-get11:31:372
398499210,20cyclictest0-21swapper/209:21:342
398499210,20cyclictest0-21swapper/208:26:352
398499210,1cyclictest0-21swapper/211:45:002
398499210,0cyclictest0-21swapper/208:21:352
3980992121,0cyclictest0-21swapper/109:06:341
3980992121,0cyclictest0-21swapper/108:26:341
3980992120,0cyclictest0-21swapper/111:16:361
3980992119,1cyclictest3539-21apt-get10:29:591
398099210,21cyclictest0-21swapper/108:31:331
398099210,20cyclictest0-21swapper/111:11:371
398099210,20cyclictest0-21swapper/107:31:351
398099210,0cyclictest17106-21apt-get09:51:361
3976992121,0cyclictest0-21swapper/010:11:360
397699210,20cyclictest0-21swapper/012:11:350
397699210,20cyclictest0-21swapper/009:56:340
397699210,20cyclictest0-21swapper/009:16:350
397699210,20cyclictest0-21swapper/007:36:340
397699210,0cyclictest17827-21apt-get08:46:350
397699210,0cyclictest0-21swapper/011:36:350
397699210,0cyclictest0-21swapper/010:36:360
397699210,0cyclictest0-21swapper/008:41:340
397699210,0cyclictest0-21swapper/008:26:350
397699210,0cyclictest0-21swapper/008:11:390
397699210,0cyclictest0-21swapper/007:16:340
3990992020,0cyclictest0-21swapper/312:06:373
3990992020,0cyclictest0-21swapper/311:31:383
3990992020,0cyclictest0-21swapper/309:51:353
3990992020,0cyclictest0-21swapper/309:41:353
3990992020,0cyclictest0-21swapper/308:36:353
3990992020,0cyclictest0-21swapper/308:31:333
3990992020,0cyclictest0-21swapper/308:16:353
3990992018,1cyclictest29857-21perf09:10:003
399099200,20cyclictest0-21swapper/312:21:383
399099200,0cyclictest0-21swapper/311:46:353
399099200,0cyclictest0-21swapper/310:41:343
399099200,0cyclictest0-21swapper/308:41:343
3984992020,0cyclictest0-21swapper/212:31:362
3984992020,0cyclictest0-21swapper/212:30:002
3984992020,0cyclictest0-21swapper/211:26:362
3984992020,0cyclictest0-21swapper/210:26:362
3984992020,0cyclictest0-21swapper/208:31:322
3984992018,1cyclictest0-21swapper/209:10:002
398499201,18cyclictest6161-21sshd10:32:222
398499200,0cyclictest21200-21apt-get11:06:362
398499200,0cyclictest12962-21apt-get08:36:342
3980992020,0cyclictest0-21swapper/111:46:361
3980992020,0cyclictest0-21swapper/110:06:351
3980992020,0cyclictest0-21swapper/108:41:341
3980992020,0cyclictest0-21swapper/108:36:341
3980992020,0cyclictest0-21swapper/107:46:331
3980992019,1cyclictest26087-21apt-get07:56:341
3980992019,1cyclictest16374-21apt-get07:36:331
3980992019,0cyclictest32483-21apt-get12:36:341
398099200,20cyclictest0-21swapper/109:11:351
398099200,19cyclictest0-21swapper/107:11:341
398099200,0cyclictest0-21swapper/111:51:341
398099200,0cyclictest0-21swapper/110:46:351
3976992020,0cyclictest0-21swapper/012:31:360
3976992020,0cyclictest0-21swapper/012:21:380
3976992020,0cyclictest0-21swapper/011:56:350
3976992020,0cyclictest0-21swapper/011:20:020
3976992020,0cyclictest0-21swapper/011:01:360
3976992020,0cyclictest0-21swapper/009:51:360
3976992020,0cyclictest0-21swapper/009:46:370
3976992020,0cyclictest0-21swapper/009:36:360
3976992020,0cyclictest0-21swapper/009:05:000
3976992020,0cyclictest0-21swapper/008:36:340
3976992020,0cyclictest0-21swapper/007:51:360
3976992020,0cyclictest0-21swapper/007:41:330
3976992019,0cyclictest0-21swapper/011:51:340
3976992018,1cyclictest11205-21apt07:24:590
397699200,19cyclictest14342-21wc09:45:000
397699200,0cyclictest0-21swapper/011:16:360
3990991919,0cyclictest0-21swapper/312:36:343
3990991919,0cyclictest0-21swapper/312:01:403
3990991919,0cyclictest0-21swapper/310:56:363
3990991919,0cyclictest0-21swapper/309:11:353
3990991918,0cyclictest26717-21apt-get10:11:373
3990991917,1cyclictest2909-21turbostat.cron08:15:003
3990991917,1cyclictest16009-21munin-run07:35:003
3990991917,1cyclictest0-21swapper/310:25:003
3984991919,0cyclictest30231-21apt-get09:11:352
3984991919,0cyclictest18083-21apt-get12:06:372
3984991919,0cyclictest16386-21apt-get10:56:372
3984991919,0cyclictest0-21swapper/212:01:402
3984991919,0cyclictest0-21swapper/209:51:362
3984991919,0cyclictest0-21swapper/209:41:352
3984991919,0cyclictest0-21swapper/209:36:342
3984991918,1cyclictest19506-21apt-get09:56:342
3984991917,1cyclictest1125-21latency_hist10:25:002
3984991917,1cyclictest0-21swapper/208:15:002
3980991919,0cyclictest0-21swapper/111:26:351
3980991919,0cyclictest0-21swapper/109:21:331
3980991918,1cyclictest32628-21apt-get09:16:361
3980991918,1cyclictest3229-21munin-run11:35:001
3980991918,1cyclictest18898-21apt-get07:41:331
3980991918,1cyclictest0-21swapper/111:56:361
3976991919,0cyclictest0-21swapper/012:01:390
3976991919,0cyclictest0-21swapper/011:46:350
3976991919,0cyclictest0-21swapper/007:56:340
3976991918,0cyclictest0-21swapper/012:16:350
3976991918,0cyclictest0-21swapper/007:31:360
397699190,0cyclictest10566-21apt-get08:31:330
397699190,0cyclictest0-21swapper/009:31:350
3990991818,0cyclictest10867-21apt-get11:51:343
3990991818,0cyclictest0-21swapper/309:46:363
3990991817,1cyclictest32628-21apt-get09:16:363
3984991818,0cyclictest0-21swapper/212:36:342
3984991816,1cyclictest11205-21apt07:25:002
3980991818,0cyclictest0-21swapper/112:16:351
3980991818,0cyclictest0-21swapper/111:36:351
3980991818,0cyclictest0-21swapper/110:51:371
3980991818,0cyclictest0-21swapper/110:21:341
3980991818,0cyclictest0-21swapper/107:51:351
3980991817,1cyclictest30901-21apt-get08:06:351
3980991817,1cyclictest29110-21apt-get10:16:351
3976991818,0cyclictest27688-21apt-get12:26:350
3976991818,0cyclictest0-21swapper/008:21:340
3976991818,0cyclictest0-21swapper/008:16:340
3976991816,1cyclictest23194-21sh11:09:590
3990991717,0cyclictest25991-21apt-get11:16:363
3990991717,0cyclictest0-21swapper/309:26:353
3990991716,1cyclictest0-21swapper/307:25:003
3990991716,0cyclictest18898-21apt-get07:41:333
3984991717,0cyclictest5097-21apt-get09:26:352
3984991717,0cyclictest0-21swapper/211:51:342
3984991717,0cyclictest0-21swapper/211:16:362
3984991717,0cyclictest0-21swapper/210:11:372
3984991716,1cyclictest593-21dbus-daemon11:50:012
398499170,16cyclictest0-21swapper/208:46:352
398499170,0cyclictest0-21swapper/207:36:342
3980991717,0cyclictest0-21swapper/110:56:371
3980991717,0cyclictest0-21swapper/110:01:351
3980991716,1cyclictest11584-21apt-get07:26:351
3980991714,2cyclictest27045-21runrttasks12:21:591
398099170,0cyclictest0-21swapper/109:26:341
3976991717,0cyclictest0-21swapper/011:26:360
3976991717,0cyclictest0-21swapper/011:11:360
3976991717,0cyclictest0-21swapper/008:06:340
3976991716,0cyclictest0-21swapper/009:26:350
3990991616,0cyclictest0-21swapper/309:31:363
3990991616,0cyclictest0-21swapper/308:01:343
399099160,2cyclictest0-21swapper/307:46:343
399099160,0cyclictest0-21swapper/307:36:343
3984991616,0cyclictest28507-21apt-get08:01:352
3984991616,0cyclictest16008-21perf07:35:002
3984991616,0cyclictest0-21swapper/209:31:352
3984991616,0cyclictest0-21swapper/209:16:362
3984991614,1cyclictest23597-21apt-get11:11:372
3980991616,0cyclictest0-21swapper/111:06:361
3980991616,0cyclictest0-21swapper/111:01:351
3980991616,0cyclictest0-21swapper/108:51:361
3980991614,1cyclictest0-21swapper/108:46:411
3976991616,0cyclictest0-21swapper/012:06:380
397699160,9cyclictest13-21ksoftirqd/007:26:450
397699160,0cyclictest0-21swapper/010:24:210
3990991513,1cyclictest0-21swapper/310:51:423
399099150,1cyclictest32226-21mailstats08:06:433
399099150,0cyclictest0-21swapper/311:38:583
398499153,2cyclictest151rcu_preempt10:55:012
3984991515,0cyclictest0-21swapper/207:41:322
398499150,2cyclictest151rcu_preempt07:50:002
398099153,8cyclictest0-21swapper/112:26:351
398099152,12cyclictest5772-21apt-get08:21:351
3980991515,0cyclictest12312-21apt-get09:41:341
3976991513,1cyclictest22240-21latency_hist08:55:000
397699150,7cyclictest0-21swapper/009:21:340
3990991414,0cyclictest0-21swapper/312:16:203
399099140,14cyclictest0-21swapper/311:11:083
399099140,14cyclictest0-21swapper/310:47:163
399099140,14cyclictest0-21swapper/310:35:313
399099140,14cyclictest0-21swapper/308:47:203
399099140,14cyclictest0-21swapper/307:11:203
399099140,0cyclictest0-21swapper/311:01:183
399099140,0cyclictest0-21swapper/307:58:183
399099140,0cyclictest0-21swapper/307:50:133
3984991414,0cyclictest0-21swapper/207:13:162
398499140,14cyclictest0-21swapper/212:20:002
398499140,14cyclictest0-21swapper/211:37:262
398499140,14cyclictest0-21swapper/211:01:182
398499140,14cyclictest0-21swapper/208:06:122
398499140,14cyclictest0-21swapper/207:50:132
398499140,0cyclictest0-21swapper/212:16:152
398499140,0cyclictest0-21swapper/211:56:092
398499140,0cyclictest0-21swapper/210:35:132
398499140,0cyclictest0-21swapper/207:56:072
3980991414,0cyclictest0-21swapper/109:31:251
398099140,14cyclictest0-21swapper/112:31:071
398099140,14cyclictest0-21swapper/111:40:141
398099140,14cyclictest0-21swapper/108:17:231
397699141,11cyclictest20850irq/140-nvme0q110:50:110
397699140,14cyclictest0-21swapper/010:45:150
397699140,14cyclictest0-21swapper/010:25:270
397699140,0cyclictest0-21swapper/010:31:190
3990991313,0cyclictest0-21swapper/307:27:473
398499130,0cyclictest0-21swapper/207:25:352
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional