You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-20 - 08:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Fri Mar 20, 2026 00:45:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
86840999384,3cyclictest0-21swapper/020:40:030
86840999073,10cyclictest179093-21sh00:30:030
8684099702,48cyclictest105985-21systemd-run21:40:010
8684599691,63cyclictest48-21kcompactd020:00:271
86840996928,18cyclictest0-21swapper/000:08:410
86840996913,37cyclictest16950irq/224-800000023:45:020
8684599681,14cyclictest3467602-21systemd-journal23:21:211
86845996729,35cyclictest23-21ksoftirqd/122:50:041
86845996726,38cyclictest183287-21sh00:40:021
8684599671,8cyclictest178115-21ssh00:28:061
8684599671,11cyclictest128264-21sshd22:31:361
86845996626,36cyclictest108484-21sh21:45:271
86840996610,35cyclictest16950irq/224-800000021:30:020
86840996610,35cyclictest16950irq/224-800000021:30:020
8684599652,9cyclictest150-21pr/ttyS222:55:311
8684599652,18cyclictest174515-21ssh00:19:501
8684599652,10cyclictest3467602-21systemd-journal23:35:141
8684599651,11cyclictest165374-21ssh23:58:181
8684599641,9cyclictest161248-21ssh23:48:401
8684599641,9cyclictest154099-21ssh23:31:511
8684599641,9cyclictest114152-21ssh21:59:021
8684599641,15cyclictest163579-21ssh23:54:091
8684599641,10cyclictest132028-21ssh22:40:181
8684099646,35cyclictest86843-21latency_hist19:10:010
8685499633,9cyclictest37-21ksoftirqd/321:55:003
8684599632,9cyclictest130187-21ssh22:36:101
86845996323,36cyclictest158760-21sh23:42:481
86845996322,35cyclictest275-21dbus-daemon19:45:001
8684599631,9cyclictest97270-21ssh21:19:501
8684599631,9cyclictest171528-21ssh00:12:391
8684599631,9cyclictest147849-21ssh23:17:221
8684599631,8cyclictest170084-21ssh00:09:341
8684599631,12cyclictest118724-21ssh22:09:471
8684599631,10cyclictest106563-21ssh21:41:061
86849996240,16cyclictest0-21swapper/200:00:012
8684599622,8cyclictest95052-21ssh21:14:381
8684599622,8cyclictest175126-21ssh00:20:511
8684599622,7cyclictest91206-21munin-run20:35:011
8684599622,6cyclictest102220-21ssh21:30:551
8684599622,10cyclictest121633-21ssh22:16:101
8684599621,9cyclictest167541-21ssh00:03:241
8684599621,9cyclictest126553-21ssh22:27:411
8684599621,9cyclictest119528-21ssh22:11:231
8684599621,8cyclictest180033-21ssh00:32:231
8684599621,8cyclictest100013-21ssh21:25:421
8684599621,8cyclictest100013-21ssh21:25:421
8684599621,6cyclictest145389-21ssh23:11:361
8684599621,5cyclictest153056-21ssh23:29:471
8684599621,14cyclictest97817-21ssh21:20:311
8684599612,5cyclictest90953-21munin-run20:30:011
8684599612,5cyclictest87004-21sshd19:11:351
86845996122,33cyclictest104067-21sh21:35:111
8684599612,14cyclictest91486-21munin-run20:40:011
8684599611,9cyclictest134323-21ssh22:45:451
8684599611,9cyclictest116279-21ssh22:04:001
8684599611,9cyclictest110801-21ssh21:50:571
8684599611,7cyclictest143194-21ssh23:06:281
8684599611,7cyclictest123973-21ssh22:21:381
8684599611,6cyclictest140812-21ssh23:00:511
8684099616,25cyclictest88331-21latency_hist19:40:000
8684599602,11cyclictest88860-21/usr/sbin/munin19:50:011
8684599601,6cyclictest90099-21cat20:10:031
8684599601,6cyclictest90099-21cat20:10:021
86840996018,7cyclictest12-21ksoftirqd/000:17:120
86840996016,8cyclictest12-21ksoftirqd/022:03:390
8684599592,7cyclictest150-21pr/ttyS220:41:331
8684599591,6cyclictest89080-21sshd19:51:361
8684599591,4cyclictest92734-21sshd21:02:421
86840995928,16cyclictest129976-21ssh22:35:390
86840995927,16cyclictest144222-21ssh23:09:120
86840995912,10cyclictest12-21ksoftirqd/021:46:330
8684599581,5cyclictest87810-21sshd19:29:361
8684099586,16cyclictest147484-21ssh23:16:310
86840995833,19cyclictest16850irq/134-800000000:05:010
86840995814,8cyclictest12-21ksoftirqd/023:35:570
8684599572,14cyclictest87306-21munin-run19:20:011
8684599572,14cyclictest87306-21munin-run19:20:011
86840995747,6cyclictest16850irq/134-800000000:00:030
86840995725,28cyclictest12-21ksoftirqd/021:15:330
86840995720,32cyclictest16850irq/134-800000022:22:210
86840995719,33cyclictest16850irq/134-800000021:34:500
86840995716,11cyclictest12-21ksoftirqd/019:35:020
86840995716,11cyclictest12-21ksoftirqd/019:35:020
86840995715,9cyclictest12-21ksoftirqd/023:49:020
86840995714,9cyclictest12-21ksoftirqd/020:40:010
86840995714,32cyclictest16850irq/134-800000022:55:110
86840995713,14cyclictest155008-21ssh23:34:240
8684599563,5cyclictest90132-21sshd20:14:361
8684599561,4cyclictest92203-21sshd20:52:411
8684599561,4cyclictest87547-21sshd19:22:211
8684599561,11cyclictest364-21snmpd19:58:021
86840995627,14cyclictest171091-21ssh00:11:350
86840995612,12cyclictest182844-21ssh00:39:140
86849995547,5cyclictest95722-21kworker/2:122:06:322
86849995538,12cyclictest3467602-21systemd-journal20:05:002
8684599553,7cyclictest88092-21munin-run19:35:011
8684599553,7cyclictest88092-21munin-run19:35:001
8684599552,5cyclictest364-21snmpd20:21:231
8684099556,15cyclictest133669-21ssh22:44:330
86840995529,11cyclictest127236-21ssh22:29:290
86840995518,33cyclictest16850irq/134-800000022:11:230
86840995518,32cyclictest16850irq/134-800000023:21:020
8685499547,8cyclictest37-21ksoftirqd/323:45:203
86854995435,14cyclictest3467602-21systemd-journal19:15:013
86849995445,6cyclictest118853-21kworker/2:322:20:452
86849995444,6cyclictest95722-21kworker/2:121:25:362
86849995444,6cyclictest95722-21kworker/2:121:25:362
8684599542,11cyclictest364-21snmpd20:15:561
8684599541,15cyclictest92449-21sshd20:56:361
86840995427,12cyclictest113832-21ssh21:58:170
8684099542,37cyclictest129581-21munin-run22:35:020
86840995423,16cyclictest107001-21ssh21:42:080
86840995418,18cyclictest16850irq/134-800000022:05:550
86840995413,7cyclictest12-21ksoftirqd/000:21:250
86840995411,8cyclictest12-21ksoftirqd/022:19:500
8685499538,8cyclictest37-21ksoftirqd/320:30:003
8685499535,9cyclictest37-21ksoftirqd/321:14:243
86854995344,7cyclictest37-21ksoftirqd/320:25:023
86849995344,6cyclictest95722-21kworker/2:122:41:182
8684599532,13cyclictest93044-21ssh21:10:011
8684099539,12cyclictest95415-21latency_hist21:15:020
86840995319,31cyclictest16850irq/134-800000022:49:410
86840995318,31cyclictest16850irq/134-800000021:20:170
86840995317,31cyclictest16850irq/134-800000023:11:480
8684099531,21cyclictest1-21systemd20:20:000
8685499527,7cyclictest37-21ksoftirqd/321:16:253
8685499526,7cyclictest37-21ksoftirqd/300:34:023
86854995234,14cyclictest90418-21systemctl20:20:003
86854995211,11cyclictest37-21ksoftirqd/322:40:013
8684999525,34cyclictest172747-21kworker/u8:000:35:012
86849995243,6cyclictest118853-21kworker/2:323:43:432
86849995243,5cyclictest118853-21kworker/2:323:11:332
86849995243,5cyclictest118853-21kworker/2:322:46:362
8684599521,14cyclictest86940-21latency_hist19:10:031
8684099526,15cyclictest137382-21ssh22:53:140
86840995220,16cyclictest16850irq/134-800000023:02:270
86840995219,17cyclictest16850irq/134-800000023:26:290
86840995218,5cyclictest16850irq/134-800000000:26:500
86840995217,17cyclictest16850irq/134-800000023:50:050
86840995217,17cyclictest16850irq/134-800000021:50:230
8685499519,11cyclictest37-21ksoftirqd/320:05:023
86854995112,10cyclictest37-21ksoftirqd/323:40:063
8684999516,6cyclictest131rcu_preempt23:36:352
86849995143,5cyclictest95722-21kworker/2:121:54:322
86849995142,5cyclictest118853-21kworker/2:322:31:112
86849995141,7cyclictest118853-21kworker/2:323:19:262
86849995133,13cyclictest30-21ksoftirqd/220:35:002
8684599514,41cyclictest0-21swapper/119:36:441
8685499507,8cyclictest37-21ksoftirqd/322:44:443
8685499507,8cyclictest37-21ksoftirqd/300:03:293
8685499506,10cyclictest37-21ksoftirqd/323:10:013
8685499505,11cyclictest37-21ksoftirqd/300:22:323
8685499504,9cyclictest37-21ksoftirqd/322:47:193
8685499504,9cyclictest37-21ksoftirqd/322:06:333
8685499504,8cyclictest37-21ksoftirqd/300:38:593
8685499504,7cyclictest37-21ksoftirqd/322:33:163
8685499503,8cyclictest37-21ksoftirqd/323:22:323
8685499503,10cyclictest37-21ksoftirqd/300:08:463
8685499501,11cyclictest37-21ksoftirqd/300:29:383
86849995042,5cyclictest80315-21kworker/2:221:18:002
86849995042,5cyclictest118853-21kworker/2:323:04:052
86849995041,6cyclictest95722-21kworker/2:122:26:502
86849995041,6cyclictest80315-21kworker/2:220:58:322
86849995040,6cyclictest71872-21kworker/2:119:20:012
86849995040,6cyclictest71872-21kworker/2:119:20:002
86849995040,6cyclictest170318-21kworker/2:200:25:002
86849995025,19cyclictest275-21dbus-daemon19:50:002
8684599502,4cyclictest0-21swapper/120:49:551
8685499497,9cyclictest37-21ksoftirqd/322:52:193
8685499496,11cyclictest37-21ksoftirqd/323:57:063
8685499495,8cyclictest37-21ksoftirqd/323:03:063
8685499495,8cyclictest37-21ksoftirqd/321:20:433
8685499495,14cyclictest37-21ksoftirqd/323:50:283
8685499494,7cyclictest37-21ksoftirqd/322:04:023
86849994941,5cyclictest95722-21kworker/2:122:36:552
86849994941,5cyclictest95722-21kworker/2:121:37:572
86849994941,5cyclictest89327-21kworker/2:020:30:022
86849994941,5cyclictest118853-21kworker/2:323:21:062
86849994941,5cyclictest118853-21kworker/2:300:15:352
86849994940,6cyclictest95722-21kworker/2:100:01:512
86849994940,6cyclictest80315-21kworker/2:220:55:022
86849994940,5cyclictest95722-21kworker/2:123:25:272
86849994939,7cyclictest80315-21kworker/2:222:00:472
86849994929,14cyclictest1-21systemd22:00:012
8684999491,3cyclictest131rcu_preempt00:09:362
86840994920,16cyclictest275-21dbus-daemon20:55:000
8685499488,8cyclictest37-21ksoftirqd/322:18:463
8685499488,6cyclictest131rcu_preempt21:39:313
8685499486,7cyclictest37-21ksoftirqd/321:46:023
8685499486,4cyclictest131rcu_preempt19:55:013
8685499485,9cyclictest37-21ksoftirqd/323:38:063
8685499484,11cyclictest37-21ksoftirqd/321:28:393
8685499484,11cyclictest37-21ksoftirqd/321:28:393
86854994840,5cyclictest361rcuc/322:21:103
86854994838,7cyclictest361rcuc/321:33:563
86854994832,10cyclictest114483-21cron21:59:593
8685499483,11cyclictest37-21ksoftirqd/320:35:013
8685499482,8cyclictest37-21ksoftirqd/323:30:573
8684999484,28cyclictest183177-21modprobe00:40:002
86849994840,5cyclictest95722-21kworker/2:100:12:572
86849994840,5cyclictest118853-21kworker/2:323:51:482
86849994839,6cyclictest170318-21kworker/2:200:26:032
86849994838,5cyclictest118853-21kworker/2:323:05:582
86849994834,9cyclictest309-21in:imuxsock19:30:012
86849994831,10cyclictest89351-21cron20:00:002
86849994827,16cyclictest3467602-21systemd-journal20:20:002
8685499477,10cyclictest37-21ksoftirqd/300:11:303
8685499476,11cyclictest37-21ksoftirqd/321:42:343
8685499476,10cyclictest37-21ksoftirqd/322:56:233
86854994710,5cyclictest131rcu_preempt23:29:573
86849994739,5cyclictest95722-21kworker/2:122:13:192
86849994739,5cyclictest95722-21kworker/2:121:32:402
86849994737,6cyclictest118853-21kworker/2:323:47:282
86849994727,15cyclictest1-21systemd21:10:012
8685499466,14cyclictest37-21ksoftirqd/323:15:053
8685499465,9cyclictest37-21ksoftirqd/300:15:053
8685499464,7cyclictest37-21ksoftirqd/322:29:213
8685499462,7cyclictest37-21ksoftirqd/322:12:173
8685499461,13cyclictest37-21ksoftirqd/323:12:583
8685499459,6cyclictest131rcu_preempt21:00:013
86849994537,5cyclictest118853-21kworker/2:322:53:322
86849994536,6cyclictest118853-21kworker/2:323:30:312
86849994530,11cyclictest30-21ksoftirqd/221:43:372
8685499443,7cyclictest37-21ksoftirqd/319:20:013
8685499443,7cyclictest37-21ksoftirqd/319:20:003
86854994435,6cyclictest37-21ksoftirqd/319:39:593
8685499441,5cyclictest150-21pr/ttyS220:37:003
8684999448,11cyclictest30-21ksoftirqd/219:50:362
86849994435,5cyclictest95722-21kworker/2:122:17:552
86849994432,7cyclictest71872-21kworker/2:119:35:192
8684999442,16cyclictest97617-21ssh21:20:032
86840994426,13cyclictest275-21dbus-daemon20:35:010
86849994335,5cyclictest80315-21kworker/2:221:46:242
86840994330,9cyclictest1-21systemd20:50:010
8684999426,11cyclictest131rcu_preempt20:10:002
8684999426,11cyclictest131rcu_preempt20:10:002
86849994227,8cyclictest309-21in:imuxsock21:05:002
86849994216,17cyclictest275-21dbus-daemon19:25:002
86840994235,4cyclictest16950irq/224-800000021:10:020
86840994235,4cyclictest16950irq/224-800000020:07:360
86840994235,4cyclictest16950irq/224-800000020:07:360
86840994224,13cyclictest92491-21munin-plugin-st21:00:010
8685499414,10cyclictest37-21ksoftirqd/319:25:003
86854994134,5cyclictest37-21ksoftirqd/320:45:013
86849994119,13cyclictest275-21dbus-daemon20:25:002
8684099415,4cyclictest90163-21munin-run20:15:000
86854994026,9cyclictest37-21ksoftirqd/320:50:013
86854994023,12cyclictest275-21dbus-daemon20:15:013
86849994032,5cyclictest80315-21kworker/2:220:10:372
8684099404,31cyclictest0-21swapper/019:46:400
86840994026,10cyclictest90674-21cron20:24:590
86840994023,13cyclictest89670-21latency_hist20:05:010
8684099401,23cyclictest131rcu_preempt19:51:460
8685499396,4cyclictest131rcu_preempt19:27:223
8685499391,7cyclictest0-21swapper/320:50:043
86854993911,7cyclictest37-21ksoftirqd/319:35:013
86854993911,7cyclictest37-21ksoftirqd/319:35:003
86849993929,7cyclictest140244-21sendmail-msp23:00:002
86840993926,10cyclictest3467718-21lldpd19:23:510
86840993925,10cyclictest3467718-21lldpd19:57:210
8684099392,33cyclictest88589-21modprobe19:43:360
86849993830,5cyclictest80315-21kworker/2:221:11:042
86849993825,9cyclictest309-21in:imuxsock19:43:362
86840993825,10cyclictest3467718-21lldpd19:13:510
8685499378,4cyclictest131rcu_preempt19:50:013
86854993725,9cyclictest93112-21cut21:10:013
8684999372,4cyclictest131rcu_preempt20:49:592
8684099372,6cyclictest275-21dbus-daemon19:20:010
8684099372,6cyclictest275-21dbus-daemon19:20:010
86854993627,6cyclictest37-21ksoftirqd/320:10:023
86854993627,6cyclictest37-21ksoftirqd/320:10:013
8684999366,23cyclictest3467602-21systemd-journal20:40:002
8684999352,5cyclictest131rcu_preempt20:45:012
8684999352,4cyclictest131rcu_preempt19:35:022
8684999352,4cyclictest131rcu_preempt19:35:012
86840993526,6cyclictest12-21ksoftirqd/019:30:010
86840993523,8cyclictest12-21ksoftirqd/020:30:030
8684999343,9cyclictest30-21ksoftirqd/219:11:462
86849993419,12cyclictest30-21ksoftirqd/219:10:032
86840993423,8cyclictest12-21ksoftirqd/021:05:020
86854993123,6cyclictest37-21ksoftirqd/321:05:013
86854993121,7cyclictest361rcuc/319:42:003
86854993118,10cyclictest37-21ksoftirqd/320:00:013
86854992617,7cyclictest37-21ksoftirqd/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional