You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-17 - 11:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Mon Aug 17, 2026 00:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15453779910339,4cyclictest23-21ksoftirqd/120:10:011
1545373998339,35cyclictest12-21ksoftirqd/000:10:020
154538399819,4cyclictest131rcu_preempt22:55:023
1545373997530,20cyclictest1569500-21ssh21:48:030
1545373997430,19cyclictest1602706-21ssh22:59:200
1545383997021,14cyclictest275-21dbus-daemon21:50:003
1545373997037,25cyclictest0-21swapper/022:17:290
154537399685,19cyclictest1621805-21modprobe23:39:590
1545373996828,18cyclictest1646184-21ssh00:31:160
1545383996719,12cyclictest275-21dbus-daemon22:35:013
1545380996745,8cyclictest1633855-21kworker/2:100:24:322
154537399674,3cyclictest131rcu_preempt23:56:290
154537399674,3cyclictest131rcu_preempt23:56:280
1545373996725,20cyclictest1577886-21ssh22:05:460
1545373996725,20cyclictest1577886-21ssh22:05:450
1545373996627,20cyclictest0-21swapper/021:42:590
154538399654,3cyclictest131rcu_preempt23:25:023
1545383996522,4cyclictest1558653-21latency_hist21:25:023
1545373996526,19cyclictest1587393-21ssh22:25:550
1545373996524,36cyclictest12-21ksoftirqd/022:35:010
1545373996429,18cyclictest0-21swapper/021:20:340
1545373996420,34cyclictest16850irq/134-800000021:50:060
1545373996330,14cyclictest12-21ksoftirqd/023:30:000
1545373996320,34cyclictest16850irq/134-800000022:00:030
1545383996222,4cyclictest1577411-21latency_hist22:05:023
1545383996221,10cyclictest37-21ksoftirqd/319:50:023
1545383996121,4cyclictest1575018-21latency_hist22:00:023
1545383996119,13cyclictest275-21dbus-daemon00:40:003
1545373996033,18cyclictest0-21swapper/022:10:500
1545373996020,3cyclictest131rcu_preempt23:12:310
1545373996020,31cyclictest16850irq/134-800000022:20:280
1545373996014,37cyclictest16850irq/134-800000022:50:020
154537399601,39cyclictest131rcu_preempt23:31:300
154537399601,39cyclictest131rcu_preempt23:31:300
1545383995918,4cyclictest1551591-21diskmemload00:11:113
154537399594,3cyclictest131rcu_preempt21:28:220
154537399594,36cyclictest131rcu_preempt22:50:510
1545383995828,26cyclictest1618572-21sh23:33:003
1545383995828,26cyclictest1618572-21sh23:32:593
1545383995826,28cyclictest3467562-21sshd23:51:283
1545383995820,5cyclictest0-21swapper/321:20:003
1545383995818,6cyclictest37-21ksoftirqd/321:39:303
154538399581,18cyclictest1551679-21latency_hist21:10:033
1545380995840,8cyclictest1545152-21kworker/2:121:11:482
1545373995828,14cyclictest1604164-21ssh23:02:020
1545373995820,6cyclictest16850irq/134-800000023:49:400
1545383995724,5cyclictest1560705-21sh21:29:233
1545383995721,4cyclictest1561569-21sh21:30:533
1545383995721,4cyclictest1561569-21sh21:30:533
1545383995717,6cyclictest1588451-21sshd22:28:283
1545383995717,11cyclictest1636095-21latency_hist00:10:013
1545383995717,11cyclictest1636095-21latency_hist00:10:013
154537399572,4cyclictest131rcu_preempt00:29:410
154537399572,4cyclictest131rcu_preempt00:29:400
1545373995724,28cyclictest1605630-21date23:05:020
154537399571,4cyclictest131rcu_preempt21:36:110
1545373995714,34cyclictest16850irq/134-800000021:15:220
1545383995623,29cyclictest1547169-21cron19:44:593
1545383995622,4cyclictest1551591-21diskmemload00:01:443
1545383995616,9cyclictest1605376-1kworker/u9:123:10:003
1545383995614,10cyclictest37-21ksoftirqd/322:50:023
154537399562,3cyclictest131rcu_preempt23:24:460
154537399562,3cyclictest131rcu_preempt23:24:450
1545373995621,32cyclictest1596086-21latency_hist22:45:030
1545373995619,17cyclictest16850irq/134-800000023:18:260
1545373995619,17cyclictest16850irq/134-800000000:35:530
1545373995618,18cyclictest16850irq/134-800000021:31:090
1545373995618,18cyclictest16850irq/134-800000021:31:080
1545373995617,6cyclictest16850irq/134-800000022:04:010
1545373995616,18cyclictest16850irq/134-800000000:05:430
1545373995616,18cyclictest16850irq/134-800000000:05:420
1545373995614,34cyclictest16850irq/134-800000021:12:070
1545383995523,12cyclictest37-21ksoftirqd/322:06:083
1545383995523,12cyclictest37-21ksoftirqd/322:06:073
1545383995520,11cyclictest37-21ksoftirqd/323:19:013
1545383995519,7cyclictest37-21ksoftirqd/323:45:103
1545380995529,19cyclictest0-21swapper/221:27:462
1545373995518,18cyclictest16850irq/134-800000023:54:210
1545383995425,10cyclictest37-21ksoftirqd/322:14:553
1545383995420,9cyclictest37-21ksoftirqd/323:44:543
1545383995419,5cyclictest1596199-21latency_hist22:45:013
1545383995419,11cyclictest309-21in:imuxsock20:45:003
1545383995417,7cyclictest37-21ksoftirqd/322:39:473
1545380995445,6cyclictest1633855-21kworker/2:100:14:592
1545380995444,6cyclictest1551633-21kworker/2:323:51:492
1545380995432,8cyclictest1551633-21kworker/2:323:50:012
1545373995417,18cyclictest16850irq/134-800000000:04:320
1545373995416,18cyclictest16850irq/134-800000000:16:500
1545383995323,5cyclictest1644608-21sh00:28:063
1545383995323,5cyclictest1644608-21sh00:28:053
1545383995317,5cyclictest1584625-21sh22:20:033
1545383995315,10cyclictest275-21dbus-daemon19:40:013
1545383995315,10cyclictest275-21dbus-daemon19:40:003
154538099535,4cyclictest131rcu_preempt23:35:022
154538099535,4cyclictest131rcu_preempt23:35:012
1545380995345,5cyclictest1551633-21kworker/2:322:11:282
1545380995345,5cyclictest1551633-21kworker/2:300:00:262
154537399533,16cyclictest1550049-21modprobe20:40:000
1545383995216,10cyclictest37-21ksoftirqd/323:37:593
1545380995243,5cyclictest1545152-21kworker/2:121:45:012
1545380995242,6cyclictest1545152-21kworker/2:121:15:372
1545380995231,8cyclictest1551633-21kworker/2:322:51:152
1545380995224,7cyclictest30-21ksoftirqd/222:00:592
1545380995215,6cyclictest131rcu_preempt19:50:022
1545373995245,5cyclictest12-21ksoftirqd/020:50:030
1545373995227,17cyclictest0-21swapper/023:44:030
1545373995214,33cyclictest16850irq/134-800000000:24:270
1545383995144,3cyclictest37-21ksoftirqd/320:05:023
1545383995134,12cyclictest1-21systemd00:20:003
1545383995121,4cyclictest1545613-21sh19:15:003
1545383995118,4cyclictest1551591-21diskmemload00:20:063
1545380995143,5cyclictest1633855-21kworker/2:100:16:472
1545380995143,5cyclictest1551633-21kworker/2:322:23:342
1545380995143,5cyclictest1545152-21kworker/2:121:48:362
1545380995142,6cyclictest1551633-21kworker/2:322:20:022
1545380995142,5cyclictest1551633-21kworker/2:322:37:562
1545380995142,5cyclictest1551633-21kworker/2:321:40:022
1545380995141,5cyclictest1633855-21kworker/2:100:10:012
1545380995141,5cyclictest1633855-21kworker/2:100:10:012
1545377995134,13cyclictest1575025-21systemctl22:00:011
1545373995116,17cyclictest16850irq/134-800000022:38:450
1545383995020,8cyclictest37-21ksoftirqd/321:43:553
1545383995017,8cyclictest37-21ksoftirqd/323:24:013
1545383995017,8cyclictest37-21ksoftirqd/323:24:003
1545383995015,8cyclictest1549590-21sed20:30:023
1545380995042,5cyclictest1633855-21kworker/2:100:32:072
1545380995042,5cyclictest1633855-21kworker/2:100:27:262
1545380995042,5cyclictest1633855-21kworker/2:100:27:262
1545380995042,5cyclictest1551633-21kworker/2:323:40:242
1545380995042,5cyclictest1551633-21kworker/2:323:25:252
1545380995042,5cyclictest1551633-21kworker/2:322:47:342
1545380995042,5cyclictest1551633-21kworker/2:322:28:502
1545380995042,5cyclictest1551633-21kworker/2:321:34:502
1545380995042,5cyclictest1551633-21kworker/2:321:34:502
1545380995042,5cyclictest1545152-21kworker/2:122:06:012
1545380995042,5cyclictest1545152-21kworker/2:122:06:002
1545380995041,5cyclictest1633855-21kworker/2:100:39:552
1545380995041,5cyclictest1551633-21kworker/2:323:36:112
1545380995041,5cyclictest1545152-21kworker/2:121:59:082
1545377995034,12cyclictest3467602-21systemd-journal19:30:281
154537399505,3cyclictest131rcu_preempt20:10:000
1545383994931,14cyclictest275-21dbus-daemon19:30:013
154538399492,13cyclictest1552436-21ssh21:11:283
1545383994921,11cyclictest37-21ksoftirqd/323:00:273
1545380994941,5cyclictest1551633-21kworker/2:323:21:102
1545380994941,5cyclictest1551633-21kworker/2:323:21:092
1545380994941,5cyclictest1551633-21kworker/2:323:05:252
1545380994941,5cyclictest1551633-21kworker/2:322:42:362
1545380994941,5cyclictest1551633-21kworker/2:300:00:032
1545380994941,5cyclictest1551633-21kworker/2:300:00:022
1545380994934,11cyclictest275-21dbus-daemon20:55:012
1545380994924,9cyclictest1551633-21kworker/2:323:10:062
154537799491,3cyclictest131rcu_preempt23:58:101
154537799491,3cyclictest131rcu_preempt23:58:091
1545383994819,8cyclictest37-21ksoftirqd/322:54:343
1545377994823,16cyclictest23-21ksoftirqd/123:05:021
154537399484,4cyclictest131rcu_preempt20:25:010
1545373994832,12cyclictest1546889-21cron19:40:000
1545373994832,12cyclictest1546889-21cron19:40:000
154537399481,9cyclictest275-21dbus-daemon20:05:010
154537399481,3cyclictest131rcu_preempt19:25:000
1545380994739,5cyclictest1551633-21kworker/2:323:01:432
1545380994739,5cyclictest1551633-21kworker/2:322:56:062
1545380994739,5cyclictest1545152-21kworker/2:120:55:552
1545380994738,6cyclictest1551633-21kworker/2:323:15:362
1545380994731,11cyclictest275-21dbus-daemon20:45:002
1545377994731,11cyclictest275-21dbus-daemon22:10:011
1545377994731,11cyclictest275-21dbus-daemon22:10:001
1545373994730,13cyclictest1-21systemd20:19:590
154537399472,3cyclictest131rcu_preempt19:50:000
1545383994631,11cyclictest1-21systemd23:15:013
1545383994627,14cyclictest275-21dbus-daemon20:25:013
1545383994619,7cyclictest37-21ksoftirqd/323:57:403
1545383994619,7cyclictest37-21ksoftirqd/323:57:403
1545383994617,7cyclictest1548161-21cut20:00:023
1545380994638,5cyclictest1551633-21kworker/2:321:24:182
1545383994534,9cyclictest37-21ksoftirqd/320:34:593
154538099454,36cyclictest1-21systemd20:10:012
1545380994537,5cyclictest1545152-21kworker/2:119:35:402
1545380994537,5cyclictest1545152-21kworker/2:119:35:402
1545380994536,5cyclictest1551633-21kworker/2:322:31:442
1545377994532,10cyclictest23-21ksoftirqd/123:10:011
154537799452,3cyclictest131rcu_preempt21:16:201
1545377994516,20cyclictest131rcu_preempt22:20:001
1545377994515,8cyclictest23-21ksoftirqd/100:20:001
154537799451,3cyclictest131rcu_preempt00:30:031
154537799451,3cyclictest131rcu_preempt00:30:021
154537399454,3cyclictest131rcu_preempt20:00:010
1545373994538,4cyclictest16950irq/224-800000019:30:000
154537399451,4cyclictest131rcu_preempt19:35:010
1545380994436,5cyclictest1545152-21kworker/2:121:51:402
1545380994433,6cyclictest309-21in:imuxsock20:15:002
1545380994433,6cyclictest309-21in:imuxsock20:15:002
1545380994423,16cyclictest3467602-21systemd-journal20:35:012
1545380994413,23cyclictest1551583-21modprobe21:10:002
154537799447,5cyclictest131rcu_preempt22:31:381
154537799443,3cyclictest131rcu_preempt21:10:471
1545377994432,9cyclictest23-21ksoftirqd/123:21:161
1545377994432,9cyclictest23-21ksoftirqd/123:21:151
1545377994430,11cyclictest23-21ksoftirqd/100:23:011
1545377994426,13cyclictest275-21dbus-daemon21:40:001
1545377994425,14cyclictest275-21dbus-daemon20:40:011
1545373994426,14cyclictest1545568-21sshd19:10:280
1545380994331,9cyclictest3467602-21systemd-journal19:50:032
1545377994330,9cyclictest3467602-21systemd-journal00:03:291
1545377994327,11cyclictest1-21systemd21:00:011
154537399432,3cyclictest131rcu_preempt20:55:030
154537399431,3cyclictest131rcu_preempt19:45:010
154537399431,3cyclictest131rcu_preempt19:20:000
1545380994234,5cyclictest1545152-21kworker/2:119:20:022
1545380994225,13cyclictest3467602-21systemd-journal21:05:012
154537799423,3cyclictest131rcu_preempt21:51:591
1545377994230,8cyclictest1-21systemd21:35:011
1545377994230,8cyclictest1-21systemd21:35:001
1545377994228,11cyclictest23-21ksoftirqd/120:50:001
154537799422,3cyclictest131rcu_preempt21:10:031
1545377994219,7cyclictest131rcu_preempt21:50:031
154537799421,3cyclictest131rcu_preempt23:27:321
1545377994210,4cyclictest131rcu_preempt23:20:001
154537399422,3cyclictest131rcu_preempt20:45:000
1545383994130,7cyclictest1551043-21cron20:59:593
1545380994133,5cyclictest1545152-21kworker/2:120:25:002
154537799413,6cyclictest1551331-21sh21:05:001
1545377994129,10cyclictest23-21ksoftirqd/100:09:371
1545377994129,10cyclictest23-21ksoftirqd/100:09:361
1545377994126,10cyclictest3467602-21systemd-journal19:50:011
1545377994124,12cyclictest1-21systemd23:40:011
154537399414,3cyclictest131rcu_preempt21:05:010
154537399411,11cyclictest1551681-21cut21:10:020
1545383994030,8cyclictest37-21ksoftirqd/319:20:023
1545380994027,10cyclictest30-21ksoftirqd/219:45:032
1545380994022,13cyclictest30-21ksoftirqd/220:05:002
154538099401,4cyclictest131rcu_preempt19:15:012
154538099401,3cyclictest131rcu_preempt20:00:012
1545377994030,8cyclictest23-21ksoftirqd/100:40:021
1545377994029,8cyclictest23-21ksoftirqd/121:29:591
1545377994027,10cyclictest23-21ksoftirqd/121:20:181
1545377994026,11cyclictest1567658-21ssh21:44:061
1545377994025,11cyclictest3467602-21systemd-journal20:10:031
1545377994025,11cyclictest3467602-21systemd-journal20:10:021
154537799402,3cyclictest131rcu_preempt19:20:001
154537799401,4cyclictest131rcu_preempt22:40:011
154537399401,4cyclictest131rcu_preempt19:55:000
154537399401,3cyclictest131rcu_preempt20:59:590
154537399401,3cyclictest131rcu_preempt20:30:010
1545383993929,8cyclictest37-21ksoftirqd/322:23:053
1545383993928,7cyclictest1646415-21sshd00:31:483
1545383993919,14cyclictest275-21dbus-daemon20:20:003
1545380993931,5cyclictest1545152-21kworker/2:119:21:442
1545380993923,10cyclictest0-21swapper/220:30:002
154537799398,4cyclictest131rcu_preempt22:59:001
154537799395,5cyclictest131rcu_preempt22:04:491
154537799394,3cyclictest131rcu_preempt22:10:351
1545377993931,6cyclictest23-21ksoftirqd/123:51:431
1545377993931,5cyclictest1551591-21diskmemload23:45:111
1545377993929,7cyclictest23-21ksoftirqd/123:12:171
154537799392,3cyclictest131rcu_preempt20:45:011
1545373993924,11cyclictest12-21ksoftirqd/020:11:060
1545373993924,11cyclictest12-21ksoftirqd/020:11:050
1545383993822,11cyclictest275-21dbus-daemon19:55:013
154538099385,19cyclictest1546363-21kworker/u8:219:30:022
154537799386,6cyclictest131rcu_preempt22:51:261
154537799385,3cyclictest131rcu_preempt23:35:031
154537799385,3cyclictest131rcu_preempt23:35:021
154537799383,3cyclictest131rcu_preempt00:11:121
1545377993829,7cyclictest23-21ksoftirqd/120:35:011
1545377993828,6cyclictest1589132-21sh22:30:011
1545377993824,10cyclictest3467602-21systemd-journal19:22:281
154537799382,3cyclictest131rcu_preempt22:46:531
1545377993823,11cyclictest1-21systemd19:40:011
1545377993823,11cyclictest1-21systemd19:40:001
1545377993810,5cyclictest131rcu_preempt19:58:281
1545383993726,9cyclictest37-21ksoftirqd/320:50:003
1545383993724,11cyclictest37-21ksoftirqd/320:55:013
154538099375,22cyclictest1550031-21grep20:40:002
154538099374,3cyclictest131rcu_preempt20:20:012
1545380993729,5cyclictest1545152-21kworker/2:119:30:582
154537799374,3cyclictest131rcu_preempt23:45:021
1545377993726,8cyclictest23-21ksoftirqd/122:44:561
1545377993726,8cyclictest1545378-21cat19:10:011
1545377993725,9cyclictest23-21ksoftirqd/100:35:011
1545377993724,10cyclictest1-21systemd20:30:021
1545377993715,12cyclictest1586795-21cat22:25:021
154537799371,4cyclictest131rcu_preempt19:44:591
1545377993710,5cyclictest131rcu_preempt19:28:271
154537399372,3cyclictest131rcu_preempt20:35:020
1545383993625,8cyclictest1546320-21wc19:25:033
1545383993621,12cyclictest37-21ksoftirqd/321:52:553
1545377993623,10cyclictest1547857-21sed19:55:021
154537799361,3cyclictest131rcu_preempt20:25:001
1545373993619,11cyclictest0-21swapper/019:10:010
1545383993523,9cyclictest37-21ksoftirqd/321:05:013
1545380993516,4cyclictest30-21ksoftirqd/220:48:292
1545377993523,9cyclictest275-21dbus-daemon20:20:011
1545377993523,8cyclictest23-21ksoftirqd/120:52:191
1545377993523,7cyclictest23-21ksoftirqd/120:05:031
154537799351,6cyclictest0-21swapper/119:15:021
1545380993427,5cyclictest1545152-21kworker/2:119:10:012
1545383993323,7cyclictest37-21ksoftirqd/320:10:013
154538399314,4cyclictest131rcu_preempt19:35:023
154538399311,4cyclictest131rcu_preempt20:40:013
154538399301,3cyclictest131rcu_preempt19:10:013
154538399291,4cyclictest131rcu_preempt20:15:013
154538399291,4cyclictest131rcu_preempt20:15:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional