You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-24 - 13:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Mar 24, 2026 00:45:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1310315997732,13cyclictest275-21dbus-daemon00:25:011
1310315997732,13cyclictest275-21dbus-daemon00:25:011
1310319997232,12cyclictest3467602-21systemd-journal23:55:032
131031299729,37cyclictest16950irq/224-800000023:25:030
1310322997133,33cyclictest1-21systemd22:30:003
1310312997167,2cyclictest0-21swapper/000:05:030
1310312997158,9cyclictest0-21swapper/020:40:020
1310319997030,11cyclictest1-21systemd00:05:012
131032299662,14cyclictest1317466-21ssh21:12:023
1310322996527,34cyclictest1325191-21cat21:30:023
1310315996528,8cyclictest1375982-21sshd23:28:161
1310315996528,12cyclictest23-21ksoftirqd/122:52:221
1310312996526,36cyclictest1333644-21latency_hist21:50:030
131032299643,16cyclictest1402401-21munin-run00:30:003
1310322996428,32cyclictest1329267-21sh21:39:503
131032299641,12cyclictest48-21kcompactd019:40:073
131032299641,12cyclictest1313854-21sshd20:18:153
1310315996426,8cyclictest1354521-21sshd22:38:161
131032299631,8cyclictest1399293-21ssh00:22:323
131032299631,8cyclictest1399293-21ssh00:22:323
131032299631,16cyclictest1332059-21ssh21:45:463
131032299631,14cyclictest1343145-21ssh22:11:413
131032299622,7cyclictest1331260-21sshd21:44:163
131032299622,5cyclictest1406003-21sshd00:38:163
1310322996222,36cyclictest1373785-21sh23:23:083
131032299622,17cyclictest1347255-21ssh22:21:143
131032299622,15cyclictest1390039-21ssh00:00:403
131032299621,9cyclictest1326376-21ssh21:32:443
131032299621,8cyclictest1377317-21ssh23:31:063
1310315996223,7cyclictest1372491-21ls23:20:021
131032299612,7cyclictest1382816-21ssh23:44:123
131032299612,5cyclictest364-21snmpd20:26:263
131032299612,4cyclictest1361662-21munin-run22:55:003
131032299612,12cyclictest1380319-21sshd23:38:253
131032299611,6cyclictest1403649-21ssh00:32:393
131032299611,14cyclictest1354725-21sh22:38:453
131031299617,30cyclictest16950irq/224-800000023:35:030
131032299602,10cyclictest150-21pr/ttyS222:34:393
131032299601,8cyclictest1318900-21ssh21:15:103
131032299601,7cyclictest1387861-21ssh23:55:443
131032299601,7cyclictest1370035-21ssh23:14:303
131032299601,7cyclictest1322278-21ssh21:23:143
131032299601,6cyclictest1384177-21ssh23:47:143
1310315996027,10cyclictest23-21ksoftirqd/123:11:031
1310315996022,12cyclictest275-21dbus-daemon21:35:011
131031299606,5cyclictest12-21ksoftirqd/000:28:300
131032299592,6cyclictest1336910-21ssh21:57:213
131032299592,15cyclictest1310474-21sshd19:12:163
131032299591,8cyclictest1393811-21ssh00:09:593
131032299591,7cyclictest1357681-21ssh22:45:193
131032299591,6cyclictest1394488-21ssh00:11:113
131032299591,6cyclictest1394488-21ssh00:11:113
131032299591,6cyclictest1370858-21ssh23:16:063
131032299591,6cyclictest1365557-21ssh23:04:023
131032299591,6cyclictest1365557-21ssh23:04:013
131032299591,5cyclictest48-21kcompactd022:55:173
1310315995925,8cyclictest23-21ksoftirqd/122:57:541
1310312995931,12cyclictest1321837-21ssh21:22:120
1310322995824,30cyclictest1316227-21latency_hist21:05:023
131032299582,14cyclictest48-21kcompactd023:50:263
131032299581,7cyclictest1396222-21ssh00:15:053
131032299581,6cyclictest1376557-21ssh23:29:373
131032299581,6cyclictest1340674-21ssh22:05:503
131032299581,5cyclictest1367047-21sshd23:07:173
131032299581,5cyclictest1346041-21ssh22:18:383
131032299581,5cyclictest1311055-21munin-run19:25:013
131032299581,13cyclictest364-21snmpd22:01:153
1310319995842,13cyclictest275-21dbus-daemon23:19:592
1310312995820,8cyclictest1313166-21latency_hist20:05:010
131032299572,6cyclictest1315162-21sshd20:41:163
131032299572,12cyclictest1334209-21ssh21:50:503
131032299571,7cyclictest1356766-21ssh22:43:313
131032299571,7cyclictest1356766-21ssh22:43:303
131032299571,12cyclictest364-21snmpd20:20:103
1310315995726,8cyclictest23-21ksoftirqd/121:25:141
131031299578,10cyclictest12-21ksoftirqd/023:01:170
131031299578,10cyclictest12-21ksoftirqd/023:01:170
1310312995714,8cyclictest12-21ksoftirqd/022:18:260
1310322995626,27cyclictest3467562-21sshd20:09:173
131031299562,39cyclictest1397292-21ssh00:18:030
1310312995613,39cyclictest0-21swapper/022:30:090
131032299553,5cyclictest1315409-21sshd20:45:173
131032299553,5cyclictest1315409-21sshd20:45:163
131032299552,7cyclictest364-21snmpd19:37:203
131032299551,8cyclictest1315926-21sshd20:58:023
131032299551,7cyclictest1314897-21sshd20:39:163
131032299551,16cyclictest1313124-21munin-run20:05:013
1310315995541,10cyclictest23-21ksoftirqd/100:10:021
131031299558,13cyclictest1383013-21sh23:44:500
1310312995527,13cyclictest1357254-21ssh22:44:400
1310312995527,13cyclictest1357254-21ssh22:44:400
1310312995526,13cyclictest1359911-21ssh22:50:290
1310312995516,7cyclictest12-21ksoftirqd/023:47:330
1310312995512,8cyclictest12-21ksoftirqd/022:47:220
131032299542,14cyclictest364-21snmpd19:45:023
1310319995438,12cyclictest1365658-21sshd23:04:172
1310319995438,12cyclictest1365658-21sshd23:04:162
1310315995437,13cyclictest275-21dbus-daemon23:45:011
1310312995442,9cyclictest12-21ksoftirqd/022:10:010
131031299541,38cyclictest1405256-21ssh00:36:170
1310312995410,10cyclictest12-21ksoftirqd/022:55:410
131032299532,8cyclictest311-21rs:main3
1310319995345,5cyclictest1289283-21kworker/2:321:11:152
1310315995337,11cyclictest1-21systemd21:55:011
1310315995316,22cyclictest0-21swapper/121:10:021
131031299539,6cyclictest12-21ksoftirqd/023:20:010
131031299535,32cyclictest0-21swapper/022:26:310
1310312995321,28cyclictest16850irq/134-800000022:01:520
1310312995319,10cyclictest12-21ksoftirqd/000:35:000
1310312995312,7cyclictest12-21ksoftirqd/021:30:010
131032299521,6cyclictest1310770-21munin-run19:20:013
131032299521,13cyclictest364-21snmpd20:31:403
1310319995230,16cyclictest275-21dbus-daemon19:45:002
1310312995243,7cyclictest12-21ksoftirqd/000:22:550
1310312995243,7cyclictest12-21ksoftirqd/000:22:550
1310312995243,5cyclictest16950irq/224-800000023:06:210
1310312995234,14cyclictest1-21systemd00:00:010
1310312995229,7cyclictest1330433-21ssh21:42:100
1310312995224,13cyclictest1392397-21ssh00:06:130
1310312995223,14cyclictest1369096-21ssh23:12:060
1310319995143,5cyclictest1289283-21kworker/2:322:10:192
1310319995142,6cyclictest1340160-21kworker/2:000:24:532
1310319995142,6cyclictest1340160-21kworker/2:000:24:522
1310319995142,5cyclictest1340160-21kworker/2:022:53:252
131031299517,10cyclictest12-21ksoftirqd/021:50:260
1310312995128,9cyclictest1328498-21ssh21:37:500
1310312995111,30cyclictest16850irq/134-800000022:10:590
131032299502,4cyclictest364-21snmpd20:50:313
131032299501,13cyclictest364-21snmpd21:08:413
1310319995042,5cyclictest1353132-21kworker/2:123:52:152
1310319995042,5cyclictest1340160-21kworker/2:022:40:382
1310319995042,5cyclictest1340160-21kworker/2:022:40:382
1310319995042,5cyclictest1311317-21kworker/2:019:50:032
1310319995042,5cyclictest1289283-21kworker/2:322:28:572
1310319995042,5cyclictest1289283-21kworker/2:321:31:492
1310319995042,5cyclictest1289283-21kworker/2:321:17:132
1310319995041,6cyclictest1353132-21kworker/2:123:23:152
1310315995033,13cyclictest1-21systemd19:50:011
131031299507,8cyclictest12-21ksoftirqd/020:45:020
1310312995041,6cyclictest1316484-21latency_hist21:10:010
1310312995035,9cyclictest1355195-21modprobe22:39:590
1310312995026,8cyclictest1347313-21ssh22:21:210
131032299492,6cyclictest1312549-21modprobe19:51:153
131032299492,14cyclictest1311520-21sshd19:32:553
131032299492,14cyclictest1311520-21sshd19:32:553
1310322994913,31cyclictest275-21dbus-daemon19:30:003
131032299491,10cyclictest1310328-21latency_hist19:10:023
1310319994941,5cyclictest1391689-21kworker/2:200:19:422
1310319994941,5cyclictest1353132-21kworker/2:123:44:052
1310319994941,5cyclictest1289283-21kworker/2:322:16:162
1310319994941,5cyclictest1289283-21kworker/2:321:35:482
1310319994939,6cyclictest1289283-21kworker/2:322:05:362
1310319994935,10cyclictest1384620-21sshd23:48:262
1310319994931,15cyclictest1-21systemd23:15:012
1310319994929,16cyclictest3467602-21systemd-journal00:32:022
1310312994935,10cyclictest1-21systemd20:35:010
1310312994920,25cyclictest0-21swapper/021:11:240
1310312994915,29cyclictest16850irq/134-800000023:20:500
131031999487,6cyclictest131rcu_preempt20:30:002
1310319994839,5cyclictest1289283-21kworker/2:321:28:462
1310319994838,6cyclictest1289283-21kworker/2:321:05:012
1310319994830,13cyclictest1406663-21cron00:40:002
1310319994830,12cyclictest3467602-21systemd-journal00:30:002
1310319994829,15cyclictest30-21ksoftirqd/221:24:282
1310319994810,6cyclictest131rcu_preempt23:35:002
1310319994740,4cyclictest1353132-21kworker/2:123:25:402
1310319994739,5cyclictest1340160-21kworker/2:022:58:072
1310319994739,5cyclictest1340160-21kworker/2:022:33:412
1310319994739,5cyclictest1340160-21kworker/2:000:14:562
1310319994739,5cyclictest1340160-21kworker/2:000:14:552
1310319994739,5cyclictest1289283-21kworker/2:322:20:142
1310319994738,5cyclictest1353132-21kworker/2:123:10:012
1310319994738,5cyclictest1314167-21kworker/2:220:40:002
1310315994728,14cyclictest275-21dbus-daemon20:45:011
1310312994733,10cyclictest16950irq/224-800000023:34:590
1310312994719,13cyclictest1320561-21ssh21:19:240
1310312994713,6cyclictest0-21swapper/000:12:140
1310312994713,6cyclictest0-21swapper/000:12:140
1310319994638,5cyclictest1320858-21kworker/2:221:45:272
1310319994633,7cyclictest1331507-21latency_hist21:44:592
1310319994623,19cyclictest30-21ksoftirqd/223:40:022
1310315994632,10cyclictest23-21ksoftirqd/122:19:591
131031599462,5cyclictest131rcu_preempt19:15:031
1310312994631,12cyclictest275-21dbus-daemon21:00:010
1310312994628,15cyclictest1325790-21ssh21:31:130
1310312994628,14cyclictest1314386-21sshd20:27:590
131031299462,12cyclictest3467602-21systemd-journal19:55:020
131031299462,12cyclictest12-21ksoftirqd/021:57:330
131032299452,5cyclictest364-21snmpd19:59:523
1310319994531,11cyclictest3467602-21systemd-journal19:29:022
131031599457,7cyclictest0-21swapper/119:25:021
1310312994528,13cyclictest1313594-21sshd20:13:160
1310312994518,13cyclictest1386388-21ssh23:52:270
1310312994515,8cyclictest12-21ksoftirqd/020:23:340
1310312994512,5cyclictest0-21swapper/019:30:010
1310319994436,5cyclictest1353132-21kworker/2:122:46:082
1310319994436,5cyclictest1340160-21kworker/2:022:36:142
1310319994436,5cyclictest1320858-21kworker/2:222:02:142
1310319994430,11cyclictest1337308-21sshd21:58:152
1310312994427,13cyclictest1312058-21sshd19:44:170
1310319994335,5cyclictest1311317-21kworker/2:019:55:012
1310319994331,9cyclictest1315710-21systemd-run20:55:012
131031599433,4cyclictest131rcu_preempt00:40:031
1310315994333,8cyclictest23-21ksoftirqd/121:14:001
1310315994329,11cyclictest1374712-21sed23:25:031
1310315994326,13cyclictest275-21dbus-daemon20:00:011
1310315994316,17cyclictest1310304-21cyclictest23:10:021
1310312994327,11cyclictest1-21systemd19:55:020
131031999426,6cyclictest131rcu_preempt20:05:022
131031999423,7cyclictest131rcu_preempt00:08:312
1310319994228,9cyclictest1313885-21run-parts20:20:002
1310319994211,6cyclictest131rcu_preempt21:50:292
131031599424,6cyclictest131rcu_preempt00:00:301
1310315994231,8cyclictest23-21ksoftirqd/121:58:141
1310315994229,8cyclictest1380952-21systemctl23:39:591
1310315994228,11cyclictest23-21ksoftirqd/122:47:551
1310315994227,11cyclictest1-21systemd23:50:021
1310315994226,13cyclictest275-21dbus-daemon22:25:021
1310315994226,11cyclictest23-21ksoftirqd/100:30:261
131031599421,10cyclictest1322966-21rm21:25:001
131031299422,9cyclictest12-21ksoftirqd/021:05:010
131031999418,5cyclictest131rcu_preempt19:34:302
131031999418,5cyclictest131rcu_preempt19:34:292
131031599415,4cyclictest131rcu_preempt00:15:141
131031599414,3cyclictest131rcu_preempt19:39:591
1310315994128,10cyclictest23-21ksoftirqd/123:55:461
1310315994127,11cyclictest1344501-21cat22:15:021
131031599412,6cyclictest131rcu_preempt21:50:011
131031599412,4cyclictest131rcu_preempt00:30:011
1310312994126,11cyclictest12-21ksoftirqd/020:10:020
131031999409,4cyclictest131rcu_preempt19:25:002
131031599404,5cyclictest131rcu_preempt22:00:271
1310315994029,9cyclictest23-21ksoftirqd/120:15:011
1310315994028,9cyclictest23-21ksoftirqd/123:51:121
1310315994027,11cyclictest23-21ksoftirqd/121:44:101
1310315994025,11cyclictest23-21ksoftirqd/123:35:011
131031299409,5cyclictest0-21swapper/019:24:590
1310312994029,8cyclictest12-21ksoftirqd/020:50:020
1310312994029,8cyclictest12-21ksoftirqd/020:50:020
131031299402,33cyclictest1310502-21munin-run19:14:590
1310319993923,12cyclictest3467602-21systemd-journal20:35:012
1310319993923,12cyclictest1313416-21sed20:10:022
1310315993927,9cyclictest23-21ksoftirqd/123:04:451
1310315993927,9cyclictest23-21ksoftirqd/123:04:451
1310315993926,10cyclictest23-21ksoftirqd/121:35:161
1310315993923,12cyclictest23-21ksoftirqd/119:35:011
1310315993923,12cyclictest23-21ksoftirqd/119:35:011
1310315993918,15cyclictest275-21dbus-daemon19:20:011
1310312993932,4cyclictest16950irq/224-800000019:40:000
1310319993830,5cyclictest1311317-21kworker/2:020:00:012
1310319993829,6cyclictest1289283-21kworker/2:321:10:012
1310319993829,5cyclictest1289283-21kworker/2:320:50:012
1310319993829,5cyclictest1289283-21kworker/2:320:50:002
1310319993824,11cyclictest30-21ksoftirqd/220:41:162
1310319993821,12cyclictest3467602-21systemd-journal20:25:012
131031599384,3cyclictest131rcu_preempt20:19:591
131031599382,4cyclictest131rcu_preempt21:19:251
1310315993824,10cyclictest23-21ksoftirqd/120:58:361
131031599381,4cyclictest131rcu_preempt22:31:221
131031599381,4cyclictest131rcu_preempt20:55:011
131031599381,4cyclictest131rcu_preempt20:30:161
131031599381,4cyclictest131rcu_preempt00:14:021
131031599381,4cyclictest131rcu_preempt00:14:011
1310312993829,6cyclictest12-21ksoftirqd/019:50:000
1310312993827,7cyclictest12-21ksoftirqd/019:35:030
1310312993827,7cyclictest12-21ksoftirqd/019:35:020
1310312993824,10cyclictest12-21ksoftirqd/020:19:590
131031999373,9cyclictest30-21ksoftirqd/220:59:572
131031599374,3cyclictest131rcu_preempt20:05:021
131031599373,3cyclictest131rcu_preempt22:30:021
131031599372,4cyclictest131rcu_preempt20:48:011
131031599372,4cyclictest131rcu_preempt20:48:001
131031599372,3cyclictest131rcu_preempt22:41:001
131031599372,3cyclictest131rcu_preempt22:40:591
131031599372,3cyclictest131rcu_preempt19:10:021
1310315993723,10cyclictest23-21ksoftirqd/122:05:471
1310315993723,10cyclictest23-21ksoftirqd/119:43:521
1310315993720,11cyclictest0-21swapper/120:40:011
131031599371,16cyclictest0-21swapper/119:55:001
1310319993625,9cyclictest30-21ksoftirqd/219:20:032
131031599363,4cyclictest131rcu_preempt20:30:011
131031599362,4cyclictest131rcu_preempt21:00:031
1310315993624,8cyclictest23-21ksoftirqd/120:25:021
1310315993621,10cyclictest275-21dbus-daemon20:10:001
1310319993522,9cyclictest30-21ksoftirqd/220:10:242
131031299351,30cyclictest0-21swapper/019:17:160
131031999344,19cyclictest1310503-21grep19:15:002
131031299343,8cyclictest0-21swapper/020:52:580
131031999333,26cyclictest1311807-21munin-run19:40:002
1310315993122,7cyclictest23-21ksoftirqd/119:28:321
131031299318,16cyclictest1310311-21sort19:10:020
131031999303,19cyclictest1310349-21cat19:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional