You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-08 - 15:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Wed Jul 08, 2026 00:45:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
133097799871,66cyclictest1361956-21ssh22:01:440
133097799871,66cyclictest1361956-21ssh22:01:440
1330977998626,56cyclictest1382674-21ssh22:45:030
1330977998331,48cyclictest1435207-21ssh00:35:310
133097799831,3cyclictest0-21swapper/020:30:010
1330977998230,47cyclictest3467602-21systemd-journal00:00:000
1330977998222,56cyclictest1429467-21ssh00:23:440
1330977998222,56cyclictest1429467-21ssh00:23:440
1330977998221,33cyclictest0-21swapper/021:14:530
1330977998220,56cyclictest16850irq/134-800000021:47:350
1330977998220,56cyclictest16850irq/134-800000021:47:350
1330977998128,36cyclictest1414745-21ssh23:52:510
133097799801,75cyclictest0-21swapper/022:36:530
1330977997727,33cyclictest1403413-21ssh23:29:100
133097799762,8cyclictest1401447-21munin-run23:25:010
133097799751,57cyclictest0-21swapper/022:41:140
1330977997428,17cyclictest0-21swapper/023:16:480
1330977997428,17cyclictest0-21swapper/023:16:470
133097799741,69cyclictest1343693-21ssh21:23:420
133097799732,64cyclictest1346800-21latency_hist21:30:020
133097799732,64cyclictest1346800-21latency_hist21:30:010
1330977997312,56cyclictest16850irq/134-800000000:05:520
1330983997224,10cyclictest1396735-21latency_hist23:15:032
133097799723,41cyclictest1370510-21latency_hist22:20:020
133097799722,53cyclictest1347987-21ssh21:32:360
133097799714,60cyclictest1336492-21wc20:55:020
133097799711,55cyclictest0-21swapper/022:30:520
133097799711,4cyclictest1337123-21cron21:10:000
1330977997018,36cyclictest1426969-21ssh00:18:280
133097799701,64cyclictest0-21swapper/021:56:180
133097799691,7cyclictest1351503-21latency_hist21:40:020
133097799691,7cyclictest1351503-21latency_hist21:40:020
133097799691,2cyclictest0-21swapper/000:13:030
133097799691,2cyclictest0-21swapper/000:13:030
133097799683,60cyclictest1335317-21munin-run20:35:010
1330977996833,30cyclictest19350irq/34-mmc123:45:010
1330977996820,29cyclictest16850irq/134-800000021:54:200
133097799681,5cyclictest3467602-21systemd-journal22:58:020
133097799681,4cyclictest0-21swapper/022:22:390
1330977996812,41cyclictest16850irq/134-800000023:32:300
1330977996721,28cyclictest1395076-21ssh23:11:240
133097799671,63cyclictest1385259-21sshd22:50:400
133097799671,48cyclictest0-21swapper/023:05:490
1330977996714,36cyclictest1366112-21ssh22:10:270
133097799671,14cyclictest1432644-21ssh00:30:070
133097799662,39cyclictest1432519-21latency_hist00:30:020
133097799661,4cyclictest0-21swapper/021:42:390
133097799661,2cyclictest0-21swapper/021:16:190
133097799661,2cyclictest0-21swapper/021:16:180
1330987996543,17cyclictest1-21systemd00:00:003
133097799652,59cyclictest1420521-21munin-run00:05:010
133097799652,59cyclictest1420521-21munin-run00:05:000
1330977996525,24cyclictest1390335-21ssh23:01:250
1330977996511,49cyclictest16850irq/134-800000023:48:410
1330977996511,49cyclictest16850irq/134-800000023:48:400
133097799632,56cyclictest1332731-21munin-run19:45:000
133098199622,7cyclictest1370488-21ssh22:20:001
133097799622,6cyclictest1333496-21munin-run20:00:000
133097799622,55cyclictest0-21swapper/023:37:460
133097799622,55cyclictest0-21swapper/023:37:460
1330977996220,25cyclictest1374446-21ssh22:28:030
133097799621,8cyclictest1365109-21sshd22:08:380
133097799611,9cyclictest1332518-21latency_hist19:40:010
133097799611,7cyclictest1331138-21sshd19:13:390
133098199598,10cyclictest23-21ksoftirqd/123:37:411
133098199598,10cyclictest23-21ksoftirqd/123:37:401
1330981995913,30cyclictest1379988-21modprobe22:40:001
133097799591,9cyclictest1331377-21sshd19:15:390
133097799591,15cyclictest364-21snmpd20:14:350
133097799591,13cyclictest1336565-21modprobe20:55:020
133097799582,52cyclictest1333223-21cron19:54:590
133097799581,7cyclictest1332959-21sshd19:47:390
133097799571,4cyclictest1332180-21sshd19:33:400
1330987995640,11cyclictest1-21systemd19:30:013
1330987995640,11cyclictest1-21systemd19:30:003
1330987995638,14cyclictest1-21systemd20:50:013
133097799561,6cyclictest1331922-21sshd19:26:390
133097799561,6cyclictest1331922-21sshd19:26:380
133097799561,3cyclictest0-21swapper/020:01:010
1330977995611,40cyclictest0-21swapper/020:49:520
1330981995539,11cyclictest3467602-21systemd-journal22:00:011
133097799552,48cyclictest1334016-21latency_hist20:10:010
133097799552,48cyclictest1334016-21latency_hist20:10:010
133097799551,2cyclictest0-21swapper/021:02:000
133097799543,47cyclictest1335556-21cron20:40:010
1330987995336,12cyclictest275-21dbus-daemon19:55:003
1330983995343,7cyclictest1318016-21kworker/2:019:15:022
1330981995336,11cyclictest1-21systemd19:25:011
1330981995333,15cyclictest275-21dbus-daemon22:30:001
133097799531,47cyclictest364-21snmpd20:23:360
133097799531,13cyclictest1331705-21true19:25:010
133097799531,12cyclictest364-21snmpd20:16:450
1330987995230,17cyclictest1437172-21unin-run00:40:013
133098799522,3cyclictest131rcu_preempt21:20:013
133098799522,3cyclictest131rcu_preempt21:20:013
1330983995244,5cyclictest1337158-21kworker/2:023:36:592
1330983995244,5cyclictest1337158-21kworker/2:023:36:582
1330983995244,5cyclictest1337158-21kworker/2:022:06:452
1330983995244,5cyclictest1330358-21kworker/2:122:05:032
1330983995244,5cyclictest1330358-21kworker/2:122:05:032
1330981995230,18cyclictest1427638-21cron00:20:001
133097799521,5cyclictest1335815-21sshd20:40:390
1330983995143,5cyclictest1330358-21kworker/2:123:49:572
1330983995143,5cyclictest1330358-21kworker/2:123:49:572
1330983995142,6cyclictest1410970-21kworker/2:300:05:012
1330983995142,6cyclictest1410970-21kworker/2:300:05:012
1330983995142,5cyclictest1410970-21kworker/2:300:09:472
1330983995142,5cyclictest1330358-21kworker/2:123:29:252
1330983995142,5cyclictest1330358-21kworker/2:100:12:132
1330983995142,5cyclictest1330358-21kworker/2:100:12:132
1330983995141,6cyclictest1410970-21kworker/2:300:36:192
1330981995134,12cyclictest275-21dbus-daemon00:15:011
1330981995134,12cyclictest275-21dbus-daemon00:15:001
1330981995112,28cyclictest221rcuc/123:10:041
1330983995042,5cyclictest1337158-21kworker/2:021:52:192
1330983995042,5cyclictest1337158-21kworker/2:021:46:262
1330983995042,5cyclictest1337158-21kworker/2:021:46:252
1330983995042,5cyclictest1337158-21kworker/2:021:42:062
1330983995042,5cyclictest1337158-21kworker/2:021:25:492
1330983995042,5cyclictest1337158-21kworker/2:021:25:492
1330983995042,5cyclictest1330358-21kworker/2:120:20:012
1330983995041,6cyclictest1330358-21kworker/2:123:00:492
1330983995041,5cyclictest1330358-21kworker/2:123:18:112
1330983995041,5cyclictest1330358-21kworker/2:123:18:112
1330981995036,10cyclictest275-21dbus-daemon22:55:011
1330987994934,10cyclictest3467602-21systemd-journal21:30:003
1330987994934,10cyclictest3467602-21systemd-journal21:30:003
1330983994942,4cyclictest1337158-21kworker/2:021:59:332
1330983994941,5cyclictest1410970-21kworker/2:300:34:492
1330983994941,5cyclictest1330358-21kworker/2:120:35:012
1330983994941,5cyclictest1330358-21kworker/2:120:00:012
1330983994940,5cyclictest1330358-21kworker/2:122:40:022
1330983994939,7cyclictest1410970-21kworker/2:323:58:312
1330983994932,13cyclictest275-21dbus-daemon19:45:012
1330981994934,12cyclictest1-21systemd19:45:001
1330981994934,11cyclictest1419068-21sshd00:01:401
1330981994934,11cyclictest1419068-21sshd00:01:391
1330981994926,16cyclictest3467602-21systemd-journal00:00:001
1330981994922,12cyclictest131rcu_preempt23:35:021
133098199491,5cyclictest131rcu_preempt00:25:001
133098199491,5cyclictest131rcu_preempt00:25:001
1330987994814,6cyclictest131rcu_preempt21:44:063
1330983994840,5cyclictest1410970-21kworker/2:300:25:072
1330983994840,5cyclictest1337158-21kworker/2:022:51:312
1330983994839,5cyclictest1337158-21kworker/2:021:12:392
1330983994838,6cyclictest1337158-21kworker/2:022:26:182
1330983994817,10cyclictest131rcu_preempt23:52:562
1330981994836,8cyclictest1351364-21systemctl21:40:011
1330981994836,8cyclictest1351364-21systemctl21:40:011
1330981994812,8cyclictest131rcu_preempt22:07:471
1330981994810,6cyclictest131rcu_preempt00:10:021
133097799482,40cyclictest1331029-21latency_hist19:10:020
1330987994731,12cyclictest37-21ksoftirqd/320:20:003
1330987994730,14cyclictest37-21ksoftirqd/300:20:003
133098799471,3cyclictest131rcu_preempt21:56:523
133098399474,3cyclictest131rcu_preempt22:35:022
1330983994740,5cyclictest1330358-21kworker/2:122:40:062
1330983994739,5cyclictest1337158-21kworker/2:023:08:142
1330983994739,5cyclictest1337158-21kworker/2:022:57:032
1330983994739,5cyclictest1330358-21kworker/2:122:45:432
1330983994738,6cyclictest1337158-21kworker/2:023:42:132
1330983994738,5cyclictest1337158-21kworker/2:021:38:082
1330983994738,5cyclictest1337158-21kworker/2:021:38:082
1330987994631,12cyclictest1-21systemd23:05:023
1330987994624,16cyclictest1-21systemd00:15:013
1330987994624,16cyclictest1-21systemd00:15:013
133098799461,3cyclictest131rcu_preempt19:50:023
1330983994639,5cyclictest1330358-21kworker/2:123:24:382
1330983994639,4cyclictest1337158-21kworker/2:022:24:042
1330983994638,5cyclictest1330358-21kworker/2:100:21:252
1330983994638,5cyclictest1330358-21kworker/2:100:21:252
1330983994631,12cyclictest1332456-21cat19:40:002
1330983994629,12cyclictest30-21ksoftirqd/222:15:022
1330983994628,15cyclictest3467602-21systemd-journal20:20:032
1330981994633,9cyclictest23-21ksoftirqd/122:22:151
1330981994631,12cyclictest1332647-21sed19:40:031
1330981994628,12cyclictest3467602-21systemd-journal21:00:001
1330981994627,14cyclictest1-21systemd19:55:011
133098199462,6cyclictest131rcu_preempt23:28:491
1330981994613,9cyclictest131rcu_preempt00:30:301
133098799455,28cyclictest361rcuc/321:55:023
133098799453,3cyclictest131rcu_preempt22:17:023
133098799451,3cyclictest131rcu_preempt23:47:323
133098799451,3cyclictest131rcu_preempt23:47:323
1330987994510,5cyclictest131rcu_preempt21:35:243
1330987994510,5cyclictest131rcu_preempt21:35:243
1330983994536,5cyclictest1337158-21kworker/2:022:18:362
133098199457,7cyclictest131rcu_preempt19:59:591
133098199456,7cyclictest23-21ksoftirqd/122:12:321
1330981994533,7cyclictest309-21in:imuxsock20:15:001
1330981994530,11cyclictest23-21ksoftirqd/121:25:011
1330981994515,6cyclictest131rcu_preempt23:24:301
1330981994513,5cyclictest131rcu_preempt00:30:001
1330981994513,10cyclictest131rcu_preempt21:51:251
1330981994511,5cyclictest131rcu_preempt22:46:201
1330987994434,7cyclictest1433759-21sshd00:32:443
133098799443,3cyclictest131rcu_preempt22:04:413
133098799443,3cyclictest131rcu_preempt22:04:403
1330987994433,7cyclictest1333477-21cron20:00:003
1330987994430,11cyclictest37-21ksoftirqd/323:22:113
1330987994427,12cyclictest1336337-21sshd20:54:403
133098799442,3cyclictest131rcu_preempt23:10:373
1330983994436,5cyclictest1330358-21kworker/2:121:23:442
1330983994426,13cyclictest275-21dbus-daemon19:30:012
1330983994426,13cyclictest275-21dbus-daemon19:30:012
1330983994421,15cyclictest275-21dbus-daemon20:40:002
1330981994434,7cyclictest23-21ksoftirqd/120:15:381
1330981994430,9cyclictest275-21dbus-daemon21:50:011
1330981994430,9cyclictest275-21dbus-daemon21:50:011
1330981994430,9cyclictest1335304-21cron20:34:591
1330981994427,12cyclictest309-21in:imuxsock21:05:001
1330981994426,13cyclictest275-21dbus-daemon19:15:011
1330981994426,12cyclictest275-21dbus-daemon20:10:001
1330981994426,12cyclictest275-21dbus-daemon20:10:001
133098199442,5cyclictest131rcu_preempt23:55:001
1330981994411,5cyclictest131rcu_preempt19:35:001
133098799434,3cyclictest131rcu_preempt22:05:273
1330987994328,10cyclictest275-21dbus-daemon22:25:013
133098799432,10cyclictest1379783-21ssh22:39:343
1330983994335,5cyclictest1333246-21kworker/2:220:55:012
1330983994335,5cyclictest1330358-21kworker/2:121:17:542
1330983994335,5cyclictest1330358-21kworker/2:121:17:532
133098199437,23cyclictest3467602-21systemd-journal20:30:001
133098199436,8cyclictest23-21ksoftirqd/123:20:011
133098199436,8cyclictest23-21ksoftirqd/123:20:001
1330981994330,11cyclictest23-21ksoftirqd/121:10:121
1330981994330,10cyclictest1396563-21cron23:15:011
1330981994327,12cyclictest1-21systemd00:40:011
133098199432,4cyclictest131rcu_preempt22:45:011
1330981994324,14cyclictest1-21systemd23:45:011
1330981994315,6cyclictest131rcu_preempt21:20:021
1330981994315,6cyclictest131rcu_preempt21:20:011
133098799426,3cyclictest131rcu_preempt22:59:403
133098799423,3cyclictest131rcu_preempt23:07:213
133098799423,3cyclictest131rcu_preempt00:01:233
133098799423,3cyclictest131rcu_preempt00:01:233
1330987994226,11cyclictest3467602-21systemd-journal19:30:033
133098799422,3cyclictest131rcu_preempt22:14:153
133098799421,3cyclictest131rcu_preempt23:42:133
133098799421,3cyclictest131rcu_preempt22:44:513
1330983994233,5cyclictest1330358-21kworker/2:100:20:012
133098199425,5cyclictest131rcu_preempt21:25:301
133098199425,5cyclictest131rcu_preempt21:25:301
1330981994227,11cyclictest23-21ksoftirqd/123:48:291
1330981994227,11cyclictest23-21ksoftirqd/123:48:291
1330981994213,7cyclictest23-21ksoftirqd/120:05:001
1330981994213,5cyclictest131rcu_preempt23:05:031
1330987994133,6cyclictest37-21ksoftirqd/321:11:503
133098799412,3cyclictest131rcu_preempt22:27:353
133098799411,4cyclictest131rcu_preempt23:31:363
1330983994128,10cyclictest3467602-21systemd-journal20:42:402
133098199417,6cyclictest131rcu_preempt19:30:021
133098199417,6cyclictest131rcu_preempt19:30:011
1330981994127,10cyclictest3467602-21systemd-journal22:58:401
133098199411,4cyclictest131rcu_preempt21:41:441
1330981994111,13cyclictest23-21ksoftirqd/122:31:331
133098799406,4cyclictest150-21pr/ttyS220:59:593
133098799403,3cyclictest131rcu_preempt21:31:353
133098799403,3cyclictest131rcu_preempt21:20:443
1330987994029,8cyclictest37-21ksoftirqd/323:25:193
1330987994027,9cyclictest37-21ksoftirqd/300:28:263
1330987994027,10cyclictest1332422-21sshd19:36:313
1330987994024,11cyclictest275-21dbus-daemon22:55:013
133098799401,3cyclictest131rcu_preempt21:45:403
133098799401,3cyclictest131rcu_preempt21:45:393
133098799401,34cyclictest37-21ksoftirqd/300:25:033
133098799401,34cyclictest37-21ksoftirqd/300:25:033
1330983994032,5cyclictest1337158-21kworker/2:021:31:482
1330981994030,6cyclictest1335573-21sendmail20:40:011
1330981994026,11cyclictest23-21ksoftirqd/122:00:491
1330981994026,11cyclictest23-21ksoftirqd/122:00:491
133098799397,5cyclictest131rcu_preempt22:48:543
1330987993925,9cyclictest3467602-21systemd-journal23:51:313
1330987993922,12cyclictest1-21systemd20:30:013
1330983993929,6cyclictest1333230-21sh19:55:012
1330983993928,7cyclictest1331425-21cat19:20:012
133098199396,4cyclictest131rcu_preempt19:50:021
1330981993927,9cyclictest3467602-21systemd-journal21:30:311
133098799384,3cyclictest131rcu_preempt21:10:003
1330987993824,9cyclictest3467602-21systemd-journal23:40:013
1330987993824,9cyclictest3467602-21systemd-journal23:40:013
1330983993829,6cyclictest1330358-21kworker/2:121:00:022
1330983993826,8cyclictest30-21ksoftirqd/219:49:592
133098199389,6cyclictest131rcu_preempt20:25:021
1330981993828,7cyclictest1331377-21sshd19:15:381
1330987993726,9cyclictest37-21ksoftirqd/323:15:353
1330987993726,9cyclictest37-21ksoftirqd/323:15:353
133098399376,9cyclictest30-21ksoftirqd/219:32:192
1330983993725,9cyclictest3467602-21systemd-journal20:10:022
1330983993725,9cyclictest1405641-21ssh23:33:502
1330981993712,6cyclictest131rcu_preempt21:10:011
133098799364,4cyclictest131rcu_preempt22:33:323
133098799364,4cyclictest131rcu_preempt20:10:013
133098799364,4cyclictest131rcu_preempt20:10:003
133098799363,23cyclictest1335320-21cut20:35:003
1330987993627,6cyclictest1334778-21sshd20:20:403
1330987993623,7cyclictest37-21ksoftirqd/320:05:003
133098399361,5cyclictest131rcu_preempt21:10:012
133098799352,5cyclictest131rcu_preempt20:45:003
133098799352,4cyclictest131rcu_preempt00:06:393
1330987993523,9cyclictest275-21dbus-daemon20:40:013
1330983993519,12cyclictest30-21ksoftirqd/220:48:132
133098199354,4cyclictest131rcu_preempt19:10:021
133098799345,24cyclictest1334411-21wc20:15:023
1330987993426,5cyclictest1331331-21latency_hist19:15:023
133098799341,3cyclictest131rcu_preempt19:45:003
1330983993427,5cyclictest1333765-21latency_hist20:04:592
133098399341,29cyclictest30-21ksoftirqd/220:30:022
133098199342,5cyclictest131rcu_preempt20:45:021
1330981993418,12cyclictest23-21ksoftirqd/120:46:051
1330987993319,11cyclictest37-21ksoftirqd/319:20:013
1330987993315,14cyclictest37-21ksoftirqd/321:04:403
1330983993325,5cyclictest1334024-21sh20:10:012
1330983993325,5cyclictest1334024-21sh20:10:012
1330983993322,8cyclictest30-21ksoftirqd/221:05:022
1330981993319,10cyclictest3467602-21systemd-journal20:51:421
1330987993225,4cyclictest1330900-21latency_hist19:10:023
1330983993222,7cyclictest1331710-21systemctl19:25:002
1330983993219,9cyclictest1331100-21sed19:10:022
1330987993119,7cyclictest1331877-21wc19:25:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional