You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-22 - 03:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Wed Apr 22, 2026 00:45:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17633249911289,15cyclictest30-21ksoftirqd/223:00:012
17633269910921,4cyclictest131rcu_preempt21:50:023
17633249910254,3cyclictest30-21ksoftirqd/221:50:012
176331299866,46cyclictest1832733-21rm23:36:220
1763312998321,47cyclictest16850irq/134-800000021:23:040
1763312998230,31cyclictest1830061-21ssh23:30:080
1763312998226,39cyclictest1843819-21ssh00:02:050
1763324998136,10cyclictest275-21dbus-daemon19:59:592
1763312998118,59cyclictest16850irq/134-800000021:53:440
176331299802,5cyclictest1806077-21/usr/sbin/munin22:35:010
176331299802,5cyclictest1806077-21/usr/sbin/munin22:35:010
176331299801,11cyclictest1859343-21sshd00:38:130
176331299773,40cyclictest1857589-21sshd00:34:150
1763312997733,39cyclictest3467602-21systemd-journal20:30:000
1763312997722,50cyclictest1818414-21ssh23:03:280
1763312997631,28cyclictest1802374-21ssh22:26:010
1763312997622,50cyclictest0-21swapper/022:18:470
1763312997514,56cyclictest16850irq/134-800000023:26:090
1763312997410,54cyclictest16850irq/134-800000021:28:420
176331299731,54cyclictest1820183-21ssh23:07:320
176331299731,3cyclictest48-21kcompactd022:39:070
1763326997225,42cyclictest3467602-21systemd-journal21:44:143
1763326997225,42cyclictest3467602-21systemd-journal21:44:133
176331299721,4cyclictest150-21pr/ttyS221:43:010
176331299721,4cyclictest150-21pr/ttyS221:43:010
176331299713,40cyclictest1851258-21ssh00:19:370
176331299712,5cyclictest1-21systemd19:28:140
1763326997027,37cyclictest275-21dbus-daemon20:45:003
176331299702,61cyclictest3467602-21systemd-journal22:10:330
176331299702,4cyclictest1791045-21/usr/sbin/munin22:00:010
1763312997023,43cyclictest275-21dbus-daemon22:25:000
1763312997022,43cyclictest16850irq/134-800000021:19:540
1763312997021,45cyclictest16850irq/134-800000021:46:240
176331299701,8cyclictest1780242-21/usr/sbin/munin21:35:000
176331299701,8cyclictest1780242-21/usr/sbin/munin21:35:000
176331299693,62cyclictest1823803-21ssh23:15:400
176331299693,33cyclictest1792126-21sh22:02:180
1763312996921,44cyclictest150-21pr/ttyS200:20:170
176331299691,64cyclictest0-21swapper/022:06:550
176331299691,62cyclictest0-21swapper/023:55:020
176331299691,4cyclictest0-21swapper/021:35:140
176331299683,32cyclictest1768792-21sed20:55:010
176331299682,61cyclictest1770351-21ssh21:11:440
176331299682,61cyclictest1770351-21ssh21:11:440
1763312996724,39cyclictest12-21ksoftirqd/019:45:150
1763312996722,13cyclictest16850irq/134-800000023:59:460
1763312996722,13cyclictest16850irq/134-800000023:59:460
176331299671,62cyclictest0-21swapper/023:48:130
176331299662,7cyclictest1821695-21ssh23:10:510
1763312996627,23cyclictest1853818-21ssh00:25:060
176331299662,59cyclictest1-21systemd20:38:130
176331299661,60cyclictest0-21swapper/022:43:050
176331299652,5cyclictest1825796-21ssh23:20:150
176331299652,59cyclictest1814346-21ssh22:53:590
176331299652,17cyclictest1845831-21ssh00:06:370
176331299651,2cyclictest0-21swapper/023:00:010
176331299632,6cyclictest275-21dbus-daemon21:10:010
1763326996225,33cyclictest1829959-21sh23:30:023
176331299629,48cyclictest1834574-21ssh23:40:260
1763326996123,34cyclictest1847212-21latency_hist00:10:023
1763326996123,31cyclictest309-21in:imuxsock23:55:003
1763326996117,11cyclictest275-21dbus-daemon19:35:003
176331299612,5cyclictest1763520-21munin-run19:15:010
176331299612,55cyclictest1812604-21munin-run22:50:000
176331299611,14cyclictest1769429-21latency_hist21:05:020
1763326996023,5cyclictest1834732-21sh23:41:003
1763326996017,13cyclictest275-21dbus-daemon20:50:013
1763326996017,13cyclictest275-21dbus-daemon20:50:003
176332699601,17cyclictest1849264-21cron00:15:003
176331299602,13cyclictest1769215-21sshd21:03:140
176331299601,5cyclictest1767636-21sshd20:31:150
1763326995922,11cyclictest37-21ksoftirqd/322:42:103
1763312995918,37cyclictest3467602-21systemd-journal19:36:150
1763312995918,37cyclictest3467602-21systemd-journal19:36:150
176331299591,6cyclictest1768169-21sshd20:41:140
176331299591,4cyclictest0-21swapper/021:00:020
176331299591,2cyclictest0-21swapper/000:10:450
176331299591,14cyclictest1768467-21munin-run20:50:020
176331299591,14cyclictest1768467-21munin-run20:50:010
1763326995817,6cyclictest1806855-21sh22:36:233
1763326995811,24cyclictest1773759-1kworker/u9:022:35:033
1763326995811,24cyclictest1773759-1kworker/u9:022:35:023
176331299581,8cyclictest1767148-21sshd20:24:140
1763312995814,39cyclictest0-21swapper/020:19:220
1763326995724,29cyclictest1842406-21sh23:59:113
1763326995724,29cyclictest1842406-21sh23:59:103
1763326995722,5cyclictest1856549-21sh00:31:313
1763326995721,9cyclictest37-21ksoftirqd/321:14:073
1763326995721,9cyclictest37-21ksoftirqd/321:14:073
1763326995718,5cyclictest1777163-21sh21:27:333
1763326995716,4cyclictest1782380-21tr21:40:003
1763326995716,13cyclictest1816876-21sendmail23:00:013
1763326995622,30cyclictest1774110-21sh21:20:073
1763326995621,7cyclictest3467602-21systemd-journal00:35:023
1763326995617,8cyclictest1801118-1kworker/u9:223:37:173
176331299562,49cyclictest1765804-21modprobe19:55:150
176331299562,48cyclictest1763810-21munin-run19:20:010
176331299561,10cyclictest364-21snmpd20:00:120
176332699555,14cyclictest1764049-21munin-plugin-st19:25:013
176332699555,14cyclictest1764049-21munin-plugin-st19:25:003
1763326995524,27cyclictest37-21ksoftirqd/323:20:073
176332699552,16cyclictest1778754-21ssh21:31:113
176332699552,16cyclictest1778754-21ssh21:31:103
176332699552,15cyclictest1796214-21ssh22:11:433
1763326995520,8cyclictest37-21ksoftirqd/300:18:013
1763326995517,9cyclictest37-21ksoftirqd/323:16:523
1763326995513,9cyclictest1855422-21sshd00:29:133
1763324995547,5cyclictest1764328-21kworker/2:221:55:532
1763319995546,7cyclictest23-21ksoftirqd/119:40:021
1763319995537,13cyclictest275-21dbus-daemon20:45:011
176331299552,49cyclictest1765097-21munin-run19:45:010
176331299551,49cyclictest1765589-21munin-plugin-st19:55:010
176331299551,48cyclictest1763306-21cyclictest19:10:010
176331299551,48cyclictest1763306-21cyclictest19:10:010
176331299551,3cyclictest48-21kcompactd020:07:190
1763326995420,5cyclictest1852830-21sh00:22:573
1763324995445,5cyclictest1764328-21kworker/2:221:15:112
1763324995434,15cyclictest1-21systemd00:40:012
176331299541,49cyclictest364-21snmpd19:31:280
1763326995324,26cyclictest1763249-21latency_hist19:10:033
1763326995324,26cyclictest1763249-21latency_hist19:10:023
176332699532,18cyclictest1773847-21sendmail-msp21:20:013
1763326995317,5cyclictest1765400-21latency_hist19:50:013
1763326995314,12cyclictest275-21dbus-daemon20:00:013
1763326995312,10cyclictest1821172-21diskmemload23:10:013
176332699531,17cyclictest1766376-21modprobe20:10:003
1763324995345,5cyclictest1797494-21kworker/2:223:22:322
1763324995345,5cyclictest1797494-21kworker/2:222:50:582
1763324995345,5cyclictest1797494-21kworker/2:222:40:342
1763324995345,5cyclictest1782401-21kworker/2:323:47:342
176331299531,47cyclictest1764023-21sshd19:22:150
176331299531,47cyclictest1764023-21sshd19:22:150
176332699528,9cyclictest0-21swapper/321:09:593
176332699525,13cyclictest275-21dbus-daemon21:55:013
176332699522,18cyclictest1838071-21ssh23:49:003
176332699522,18cyclictest1791006-21munin-run22:00:003
1763326995221,5cyclictest1812193-21sh22:49:023
1763326995221,11cyclictest37-21ksoftirqd/323:30:563
1763326995219,5cyclictest1769493-21diskmemload22:23:473
1763326995218,10cyclictest37-21ksoftirqd/322:25:043
1763326995214,12cyclictest37-21ksoftirqd/319:15:423
176332699521,18cyclictest1765608-21cat19:55:013
1763324995244,5cyclictest1764328-21kworker/2:219:50:022
176332699513,18cyclictest1818681-21ssh23:04:053
1763326995122,9cyclictest37-21ksoftirqd/320:20:013
1763326995122,25cyclictest1795487-21latency_hist22:10:023
176332699512,19cyclictest1812968-21ssh22:50:323
1763326995120,6cyclictest1844110-21rm00:02:483
1763326995114,9cyclictest37-21ksoftirqd/322:04:473
1763324995142,5cyclictest1782401-21kworker/2:322:01:502
1763324995141,6cyclictest1782401-21kworker/2:300:05:502
1763326995015,6cyclictest0-21swapper/321:05:023
1763326995015,6cyclictest0-21swapper/321:05:013
1763326995014,5cyclictest0-21swapper/319:15:013
1763326995011,9cyclictest1764287-21cron19:29:593
1763324995040,6cyclictest1764328-21kworker/2:221:44:262
1763324995040,6cyclictest1764328-21kworker/2:221:44:252
176332499502,3cyclictest131rcu_preempt00:35:002
176331999505,3cyclictest131rcu_preempt23:29:341
176331299502,5cyclictest48-21kcompactd020:11:440
176332699492,21cyclictest1822778-21ssh23:13:393
1763324994941,5cyclictest1797494-21kworker/2:222:25:272
1763324994941,5cyclictest1782401-21kworker/2:323:53:402
1763324994940,6cyclictest1782401-21kworker/2:323:06:062
1763324994940,5cyclictest1764328-21kworker/2:221:54:202
1763324994939,6cyclictest1797494-21kworker/2:223:20:002
1763324994939,6cyclictest1782401-21kworker/2:300:15:022
1763319994936,9cyclictest1834129-21cron23:39:591
1763319994934,12cyclictest23-21ksoftirqd/122:17:101
1763319994933,12cyclictest3467602-21systemd-journal20:05:011
1763326994832,11cyclictest0-21swapper/320:35:013
1763326994813,11cyclictest3467718-21lldpd20:15:023
176332499484,4cyclictest131rcu_preempt22:31:422
176332499484,4cyclictest131rcu_preempt22:31:412
1763324994840,5cyclictest1782401-21kworker/2:323:58:132
1763324994840,5cyclictest1782401-21kworker/2:323:58:122
1763324994839,6cyclictest1797494-21kworker/2:223:42:322
1763324994839,5cyclictest1797494-21kworker/2:200:30:022
1763324994839,5cyclictest1782401-21kworker/2:300:01:132
1763319994830,14cyclictest3467602-21systemd-journal19:35:001
1763319994827,15cyclictest3467602-21systemd-journal21:55:021
1763326994718,5cyclictest1766115-21latency_hist20:05:023
1763324994739,5cyclictest1782401-21kworker/2:323:30:362
1763324994739,5cyclictest1782401-21kworker/2:323:12:542
1763324994739,5cyclictest1782401-21kworker/2:323:01:242
1763324994739,5cyclictest1782401-21kworker/2:322:16:422
176331999476,28cyclictest3467602-21systemd-journal20:20:001
1763324994638,5cyclictest1797494-21kworker/2:223:38:252
1763324994638,5cyclictest1797494-21kworker/2:222:22:162
1763324994638,5cyclictest1791580-21kworker/2:022:11:092
1763324994638,5cyclictest1764328-21kworker/2:221:10:152
1763324994638,5cyclictest1764328-21kworker/2:221:10:142
1763324994638,5cyclictest1757258-21kworker/2:019:22:042
1763324994638,5cyclictest1757258-21kworker/2:019:22:032
1763324994631,12cyclictest30-21ksoftirqd/223:25:332
1763324994629,12cyclictest1-21systemd00:25:012
1763319994629,13cyclictest275-21dbus-daemon23:55:001
176331999462,3cyclictest131rcu_preempt21:32:591
176331999462,3cyclictest131rcu_preempt21:32:581
176331999461,3cyclictest131rcu_preempt21:23:381
1763319994613,11cyclictest131rcu_preempt23:19:591
1763324994537,5cyclictest1764328-21kworker/2:221:05:292
1763324994537,5cyclictest1764328-21kworker/2:219:55:032
176331999459,24cyclictest1765836-21modprobe19:59:591
1763319994529,11cyclictest309-21in:imuxsock20:10:011
1763319994527,9cyclictest23-21ksoftirqd/121:45:011
1763319994527,9cyclictest23-21ksoftirqd/121:45:001
176332699443,15cyclictest3467602-21systemd-journal20:29:593
1763326994418,4cyclictest1769493-21diskmemload22:19:023
1763324994436,5cyclictest1764328-21kworker/2:221:31:392
1763324994436,5cyclictest1764328-21kworker/2:221:31:382
1763324994436,5cyclictest1764328-21kworker/2:221:29:442
176332499441,3cyclictest131rcu_preempt22:05:182
1763324994410,6cyclictest131rcu_preempt21:20:582
176331999445,26cyclictest1855748-21modprobe00:29:591
1763319994432,9cyclictest1-21systemd23:05:001
1763319994430,7cyclictest309-21in:imuxsock22:15:001
1763319994430,10cyclictest23-21ksoftirqd/123:08:321
1763319994427,12cyclictest1-21systemd20:30:011
1763324994334,5cyclictest1797494-21kworker/2:222:37:452
1763319994332,8cyclictest23-21ksoftirqd/119:20:021
1763319994330,9cyclictest1765470-21date19:50:031
1763319994330,10cyclictest23-21ksoftirqd/122:50:011
1763319994330,10cyclictest23-21ksoftirqd/100:35:141
1763319994326,13cyclictest3467602-21systemd-journal22:55:011
1763319994325,14cyclictest275-21dbus-daemon21:05:021
1763319994325,14cyclictest275-21dbus-daemon21:05:011
1763319994324,15cyclictest1768968-21systemctl21:00:001
1763319994315,18cyclictest275-21dbus-daemon21:40:001
176331999431,3cyclictest131rcu_preempt23:43:491
1763326994219,7cyclictest1764971-21cat19:40:033
1763326994219,7cyclictest1764971-21cat19:40:023
1763324994234,5cyclictest1764328-21kworker/2:220:45:012
176331999425,31cyclictest1767681-21unin-run20:35:011
1763319994228,10cyclictest23-21ksoftirqd/122:26:511
176331999422,3cyclictest131rcu_preempt22:35:561
1763319994219,18cyclictest1-21systemd22:00:011
176331999421,3cyclictest131rcu_preempt00:31:101
1763324994130,9cyclictest30-21ksoftirqd/220:55:012
1763324994127,10cyclictest3467602-21systemd-journal20:50:012
1763324994127,10cyclictest3467602-21systemd-journal20:50:002
176332499412,5cyclictest131rcu_preempt20:20:012
176332499412,32cyclictest1782492-21ssh21:40:012
1763319994129,9cyclictest23-21ksoftirqd/122:00:071
1763319994126,11cyclictest23-21ksoftirqd/121:15:591
1763319994125,9cyclictest3467602-21systemd-journal20:40:011
1763319994125,12cyclictest23-21ksoftirqd/121:14:271
1763319994125,12cyclictest23-21ksoftirqd/121:14:261
1763319994120,8cyclictest23-21ksoftirqd/122:45:011
1763319994118,14cyclictest275-21dbus-daemon21:30:001
176331999411,4cyclictest131rcu_preempt22:07:421
1763324994032,5cyclictest1782401-21kworker/2:322:45:092
1763324994032,5cyclictest1764328-21kworker/2:220:22:032
1763324994032,5cyclictest1757258-21kworker/2:019:13:332
1763324994028,9cyclictest1-21systemd20:10:012
1763324994027,9cyclictest3467602-21systemd-journal21:00:012
1763324994024,12cyclictest30-21ksoftirqd/219:29:592
176331999406,24cyclictest1838601-21cat23:50:011
1763319994031,5cyclictest1852465-21sh00:22:051
1763319994029,6cyclictest1769505-21modprobe21:10:001
1763319994025,12cyclictest23-21ksoftirqd/123:30:141
1763319994025,12cyclictest23-21ksoftirqd/120:51:151
176331999402,3cyclictest131rcu_preempt00:14:591
1763326993930,5cyclictest1768956-21sh21:00:003
1763324993927,8cyclictest3467602-21systemd-journal21:00:032
1763324993927,8cyclictest3467602-21systemd-journal21:00:022
176331999394,3cyclictest131rcu_preempt22:57:181
1763319993928,8cyclictest23-21ksoftirqd/123:11:231
1763319993924,12cyclictest23-21ksoftirqd/120:50:021
1763319993924,12cyclictest23-21ksoftirqd/120:50:011
1763319993916,16cyclictest275-21dbus-daemon19:40:011
1763319993916,16cyclictest275-21dbus-daemon19:40:001
176331999391,3cyclictest131rcu_preempt00:00:501
176332499385,4cyclictest131rcu_preempt19:40:012
176332499385,4cyclictest131rcu_preempt19:40:002
1763324993829,6cyclictest30-21ksoftirqd/220:35:012
1763324993827,9cyclictest30-21ksoftirqd/220:30:002
1763324993823,11cyclictest30-21ksoftirqd/219:45:002
176331999382,8cyclictest1842403-21ssh23:59:101
176331999382,8cyclictest1842403-21ssh23:59:091
1763319993825,8cyclictest309-21in:imuxsock00:20:001
1763319993825,10cyclictest23-21ksoftirqd/122:30:431
1763319993825,10cyclictest23-21ksoftirqd/122:30:431
1763319993822,10cyclictest1786714-21systemd-run21:50:001
1763324993728,6cyclictest1767937-21sendmail20:40:002
1763324993727,8cyclictest30-21ksoftirqd/200:18:272
176331999379,20cyclictest1764041-21modprobe19:25:001
176331999379,20cyclictest1764041-21modprobe19:25:001
176331999373,3cyclictest131rcu_preempt00:07:331
1763319993729,5cyclictest1-21systemd19:15:001
1763319993727,7cyclictest23-21ksoftirqd/122:24:101
1763319993726,9cyclictest23-21ksoftirqd/123:20:111
1763319993716,16cyclictest3467602-21systemd-journal19:29:591
1763319993715,15cyclictest1765588-21munin-plugin-st19:55:001
1763326993626,5cyclictest309-21in:imuxsock20:25:023
176332699357,5cyclictest131rcu_preempt20:40:013
176332499353,25cyclictest1763935-21grep19:20:022
1763324993426,5cyclictest1763415-21latency_hist19:10:032
1763324993426,5cyclictest1763415-21latency_hist19:10:022
1763326993319,10cyclictest1-21systemd19:45:013
176332699331,3cyclictest131rcu_preempt20:50:023
176332499331,3cyclictest131rcu_preempt20:05:002
1763319993315,13cyclictest275-21dbus-daemon20:15:011
1763324993223,6cyclictest30-21ksoftirqd/220:15:012
176331999323,18cyclictest1763355-21cat19:10:031
176331999323,18cyclictest1763355-21cat19:10:021
1763319993219,8cyclictest1767224-21cat20:25:021
1763324993120,9cyclictest30-21ksoftirqd/219:35:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional