You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-06 - 00:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Tue May 05, 2026 12:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1769399997535,9cyclictest3467602-21systemd-journal12:20:000
1769399997535,9cyclictest3467602-21systemd-journal12:20:000
1769399997140,22cyclictest12-21ksoftirqd/010:40:030
1769403997018,11cyclictest23-21ksoftirqd/107:35:001
1769399997022,30cyclictest12-21ksoftirqd/011:10:020
176941499695,47cyclictest1814079-21ssh10:30:023
1769399996911,33cyclictest1780524-21latency_hist09:20:030
1769399996821,30cyclictest12-21ksoftirqd/012:34:250
1769414996711,10cyclictest37-21ksoftirqd/310:45:013
1769403996718,8cyclictest23-21ksoftirqd/109:10:001
176940399662,27cyclictest1824038-21ssh10:50:321
176940399646,24cyclictest1790926-21ssh09:41:451
176939999647,43cyclictest16850irq/134-800000012:10:020
1769403996318,11cyclictest23-21ksoftirqd/108:00:011
1769403996317,11cyclictest23-21ksoftirqd/111:08:521
1769403996314,20cyclictest1871661-21ssh12:29:501
1769403996314,20cyclictest1871661-21ssh12:29:501
1769399996327,4cyclictest177557850diskmemload12:30:000
1769399996327,4cyclictest177557850diskmemload12:30:000
176940399627,15cyclictest1874777-21ssh12:35:571
1769403996215,10cyclictest23-21ksoftirqd/110:11:511
1769399996233,3cyclictest12-21ksoftirqd/009:23:060
176941099613,28cyclictest1762570-21kworker/2:109:25:022
176941099613,28cyclictest1762570-21kworker/2:109:25:022
176940399619,15cyclictest1864305-21ssh12:14:411
176940399617,19cyclictest1830083-21sh11:03:181
1769403996115,10cyclictest23-21ksoftirqd/110:40:291
176939999611,42cyclictest131rcu_preempt09:19:210
1769414996013,12cyclictest37-21ksoftirqd/311:24:353
1769414996013,12cyclictest37-21ksoftirqd/311:24:353
176940399608,15cyclictest1853763-21ssh11:52:171
1769403996012,9cyclictest23-21ksoftirqd/109:29:511
1769403996012,9cyclictest23-21ksoftirqd/109:29:511
1769399996016,28cyclictest12-21ksoftirqd/011:51:030
1769399996015,4cyclictest12-21ksoftirqd/009:33:160
1769414995914,8cyclictest37-21ksoftirqd/308:50:013
1769414995911,9cyclictest37-21ksoftirqd/312:20:353
1769403995912,8cyclictest23-21ksoftirqd/112:05:001
1769403995910,19cyclictest1864751-21ssh12:15:091
1769403995910,19cyclictest1864751-21ssh12:15:091
1769410995840,8cyclictest1762570-21kworker/2:111:31:292
176940399585,42cyclictest1840566-21ssh11:25:021
176940399585,42cyclictest1840566-21ssh11:25:021
1769403995813,9cyclictest23-21ksoftirqd/111:36:331
1769403995813,8cyclictest23-21ksoftirqd/112:30:211
1769403995812,15cyclictest1836107-21ssh11:15:361
1769399995820,34cyclictest12-21ksoftirqd/012:13:260
1769399995815,38cyclictest12-21ksoftirqd/010:17:160
176940399578,17cyclictest1773212-21date08:20:031
1769403995730,23cyclictest23-21ksoftirqd/111:35:021
1769403995712,15cyclictest1818574-21ssh10:39:311
1769399995747,5cyclictest16950irq/224-800000011:25:020
1769399995747,5cyclictest16950irq/224-800000011:25:020
176939999573,37cyclictest131rcu_preempt09:28:030
176939999573,37cyclictest131rcu_preempt09:28:030
1769399995727,15cyclictest1804030-21ssh10:09:190
1769399995714,38cyclictest1834488-21sshd11:12:260
176941499567,12cyclictest37-21ksoftirqd/310:18:243
1769414995616,8cyclictest37-21ksoftirqd/312:35:023
1769414995611,9cyclictest37-21ksoftirqd/311:08:513
176940399567,14cyclictest1787337-21rm09:34:361
176940399566,15cyclictest1801840-21ssh10:04:481
1769403995614,8cyclictest23-21ksoftirqd/111:12:351
1769403995613,14cyclictest1808419-21sh10:18:191
1769403995613,10cyclictest23-21ksoftirqd/111:30:011
1769403995612,9cyclictest23-21ksoftirqd/109:50:511
1769403995612,8cyclictest23-21ksoftirqd/111:40:531
176939999561,36cyclictest131rcu_preempt11:41:180
176939999561,36cyclictest131rcu_preempt10:41:100
1769399995610,42cyclictest12-21ksoftirqd/010:47:280
1769410995547,5cyclictest1762570-21kworker/2:110:00:462
1769403995518,10cyclictest23-21ksoftirqd/111:56:491
1769403995518,10cyclictest23-21ksoftirqd/111:56:491
1769403995512,8cyclictest23-21ksoftirqd/109:19:061
1769403995512,8cyclictest23-21ksoftirqd/109:19:051
1769403995511,10cyclictest23-21ksoftirqd/110:30:001
176940399550,15cyclictest1826377-21ssh10:55:281
1769399995527,13cyclictest1828166-21ssh10:59:340
1769399995523,6cyclictest12-21ksoftirqd/011:03:540
1769399995519,18cyclictest16850irq/134-800000011:33:200
1769399995519,17cyclictest16850irq/134-800000010:10:270
176939999551,26cyclictest131rcu_preempt09:37:420
1769414995425,9cyclictest37-21ksoftirqd/311:48:093
1769414995418,6cyclictest37-21ksoftirqd/310:53:023
1769410995446,5cyclictest1762570-21kworker/2:108:45:022
176940399544,17cyclictest1792911-21ssh09:45:511
1769403995414,9cyclictest23-21ksoftirqd/109:39:421
1769403995412,9cyclictest23-21ksoftirqd/109:13:201
1769399995428,11cyclictest1792017-21ssh09:44:210
1769399995420,6cyclictest16850irq/134-800000011:19:150
1769399995419,18cyclictest16850irq/134-800000010:26:050
1769399995413,23cyclictest12-21ksoftirqd/009:52:120
176941499539,8cyclictest37-21ksoftirqd/309:39:443
176941499536,8cyclictest37-21ksoftirqd/310:23:343
1769414995344,5cyclictest361rcuc/312:00:093
1769414995319,8cyclictest37-21ksoftirqd/307:35:013
1769414995310,7cyclictest37-21ksoftirqd/311:27:453
1769410995345,5cyclictest1801913-21kworker/2:311:46:162
1769410995344,6cyclictest1801913-21kworker/2:311:12:472
1769410995344,5cyclictest1801913-21kworker/2:311:16:162
1769410995343,5cyclictest1801913-21kworker/2:311:45:012
1769410995342,7cyclictest1762570-21kworker/2:107:35:332
1769410995342,7cyclictest1762570-21kworker/2:107:35:002
1769410995336,11cyclictest1-21systemd10:35:012
1769403995314,9cyclictest23-21ksoftirqd/112:06:211
1769399995327,22cyclictest12-21ksoftirqd/010:00:110
1769399995325,13cyclictest1776703-21ssh09:12:220
1769399995318,18cyclictest16850irq/134-800000012:02:570
1769399995318,18cyclictest16850irq/134-800000009:48:270
176941499528,8cyclictest37-21ksoftirqd/311:03:163
176941499528,10cyclictest0-21swapper/310:02:013
176941499526,8cyclictest361rcuc/312:09:263
1769414995215,8cyclictest37-21ksoftirqd/311:30:303
1769414995214,9cyclictest37-21ksoftirqd/309:41:403
1769414995214,9cyclictest0-21swapper/310:47:173
1769410995244,5cyclictest1762570-21kworker/2:109:52:072
1769410995243,6cyclictest1762570-21kworker/2:112:06:432
1769410995243,6cyclictest1762570-21kworker/2:109:32:322
1769410995243,5cyclictest1801913-21kworker/2:311:04:562
176940399526,6cyclictest23-21ksoftirqd/110:06:311
176940399525,14cyclictest1814344-21ssh10:30:191
1769399995219,17cyclictest16850irq/134-800000011:27:370
1769399995218,18cyclictest16850irq/134-800000010:24:220
1769399995217,18cyclictest16850irq/134-800000011:39:310
1769399995217,17cyclictest16850irq/134-800000012:35:210
176939999521,33cyclictest131rcu_preempt10:54:090
1769399995213,25cyclictest12-21ksoftirqd/010:30:120
1769414995112,10cyclictest37-21ksoftirqd/311:43:413
1769414995111,9cyclictest0-21swapper/311:40:013
1769410995144,5cyclictest1801913-21kworker/2:312:20:122
1769410995143,5cyclictest1801913-21kworker/2:312:35:002
1769410995143,5cyclictest1762570-21kworker/2:110:09:042
1769410995143,5cyclictest1762570-21kworker/2:108:30:012
1769410995136,10cyclictest3467602-21systemd-journal09:40:032
1769410995132,14cyclictest1-21systemd12:40:012
1769410995131,14cyclictest30-21ksoftirqd/212:20:002
1769410995131,14cyclictest30-21ksoftirqd/212:20:002
176940399518,9cyclictest23-21ksoftirqd/110:20:421
1769403995112,7cyclictest23-21ksoftirqd/111:47:111
1769403995111,7cyclictest23-21ksoftirqd/110:45:591
1769399995119,15cyclictest12-21ksoftirqd/011:46:260
1769399995118,17cyclictest16850irq/134-800000011:58:240
1769399995118,17cyclictest16850irq/134-800000011:58:240
176941499505,8cyclictest37-21ksoftirqd/312:11:263
176941499502,14cyclictest1797108-21munin-run09:55:003
1769414995013,8cyclictest37-21ksoftirqd/310:56:203
1769410995042,5cyclictest1762570-21kworker/2:109:55:252
1769410995042,5cyclictest1762570-21kworker/2:109:39:462
1769410995040,6cyclictest1801913-21kworker/2:312:13:222
176940399505,13cyclictest1781514-21ssh09:22:141
1769403995037,10cyclictest1-21systemd07:45:011
1769403995037,10cyclictest1-21systemd07:45:001
176940399503,15cyclictest1867450-21ssh12:20:421
1769403995012,8cyclictest23-21ksoftirqd/108:20:011
176941499496,7cyclictest1865203-21ssh12:16:163
176941499496,7cyclictest1865203-21ssh12:16:163
176941499495,9cyclictest37-21ksoftirqd/311:17:323
1769414994933,11cyclictest3467602-21systemd-journal07:15:033
1769414994910,8cyclictest37-21ksoftirqd/309:13:403
1769414994910,7cyclictest0-21swapper/312:27:543
1769414994910,7cyclictest0-21swapper/312:27:543
1769410994940,6cyclictest1762570-21kworker/2:108:05:072
1769410994940,5cyclictest1762570-21kworker/2:110:47:152
1769410994939,6cyclictest1762570-21kworker/2:110:42:172
1769410994933,13cyclictest1840454-21cron11:25:002
1769410994933,13cyclictest1840454-21cron11:25:002
1769399994922,12cyclictest1867521-21ssh12:20:510
1769399994920,14cyclictest1798497-21ssh09:57:390
176941499484,8cyclictest37-21ksoftirqd/310:15:013
1769414994823,17cyclictest37-21ksoftirqd/309:20:143
1769414994817,9cyclictest37-21ksoftirqd/311:53:473
1769410994843,3cyclictest1801913-21kworker/2:310:17:502
1769410994842,3cyclictest1801913-21kworker/2:311:37:132
1769410994841,4cyclictest1762570-21kworker/2:111:56:102
1769410994841,4cyclictest1762570-21kworker/2:111:56:102
1769410994840,5cyclictest1801913-21kworker/2:311:25:132
1769410994840,5cyclictest1801913-21kworker/2:311:08:412
1769410994840,5cyclictest1801913-21kworker/2:310:54:442
1769410994840,5cyclictest1801913-21kworker/2:310:22:532
1769410994840,5cyclictest1762570-21kworker/2:110:40:002
1769410994840,5cyclictest1762570-21kworker/2:109:12:282
1769410994839,5cyclictest1762570-21kworker/2:108:40:022
1769410994838,5cyclictest1762570-21kworker/2:107:20:012
1769410994832,12cyclictest275-21dbus-daemon08:35:002
1769403994834,9cyclictest3467602-21systemd-journal07:40:011
1769403994831,13cyclictest3467602-21systemd-journal07:25:001
1769399994836,10cyclictest12-21ksoftirqd/007:15:020
176941499478,10cyclictest37-21ksoftirqd/310:10:033
176941499474,7cyclictest0-21swapper/309:46:223
176941499472,3cyclictest131rcu_preempt08:30:013
1769414994713,7cyclictest0-21swapper/309:05:273
1769414994712,8cyclictest37-21ksoftirqd/309:59:193
1769410994742,3cyclictest1762570-21kworker/2:111:52:182
1769410994739,5cyclictest1762570-21kworker/2:109:16:372
1769410994739,5cyclictest1762570-21kworker/2:109:16:372
1769410994738,6cyclictest1801913-21kworker/2:312:25:042
1769410994738,6cyclictest1801913-21kworker/2:312:25:042
1769399994738,5cyclictest16850irq/134-800000009:10:000
176941499461,13cyclictest1833729-21ssh11:10:373
1769410994640,4cyclictest1762570-21kworker/2:109:47:332
1769410994638,5cyclictest1801913-21kworker/2:310:25:062
1769410994637,6cyclictest1801913-21kworker/2:310:55:482
176940399466,7cyclictest23-21ksoftirqd/107:53:591
176941499454,6cyclictest37-21ksoftirqd/310:31:133
1769414994531,9cyclictest1-21systemd08:20:013
1769414994517,6cyclictest37-21ksoftirqd/309:26:363
1769414994517,6cyclictest37-21ksoftirqd/309:26:353
1769414994510,8cyclictest0-21swapper/312:37:253
176941499450,14cyclictest1856013-21ssh11:57:023
176941499450,14cyclictest1856013-21ssh11:57:013
1769410994537,5cyclictest1762570-21kworker/2:109:24:112
1769410994536,6cyclictest1762570-21kworker/2:110:10:332
1769410994536,5cyclictest1762570-21kworker/2:107:45:032
1769410994536,5cyclictest1762570-21kworker/2:107:45:022
1769410994531,9cyclictest30-21ksoftirqd/209:10:022
176940399458,6cyclictest131rcu_preempt08:50:011
1769403994527,14cyclictest1797368-21ssh09:55:061
1769399994538,4cyclictest16950irq/224-800000009:00:010
1769399994534,8cyclictest12-21ksoftirqd/008:00:020
176939999452,14cyclictest1774050-21munin-run08:40:000
176941499443,12cyclictest37-21ksoftirqd/310:36:143
1769410994436,5cyclictest1762570-21kworker/2:112:01:322
1769410994436,5cyclictest1762570-21kworker/2:108:11:112
1769410994436,5cyclictest1762570-21kworker/2:107:30:012
176940399447,4cyclictest131rcu_preempt08:35:011
176940399442,6cyclictest1-21systemd09:00:011
1769403994411,8cyclictest23-21ksoftirqd/109:05:011
1769399994433,8cyclictest12-21ksoftirqd/007:45:020
1769399994433,8cyclictest12-21ksoftirqd/007:45:020
176941499437,23cyclictest1770606-21kworker/u8:007:45:023
176941499437,23cyclictest1770606-21kworker/u8:007:45:023
176941499434,9cyclictest37-21ksoftirqd/307:55:013
1769414994311,7cyclictest0-21swapper/309:17:173
1769414994311,7cyclictest0-21swapper/309:17:173
1769410994335,5cyclictest1762570-21kworker/2:108:18:272
1769410994335,5cyclictest1762570-21kworker/2:107:23:032
1769410994322,16cyclictest1-21systemd08:49:592
176940399434,9cyclictest23-21ksoftirqd/108:05:021
176940399434,9cyclictest23-21ksoftirqd/108:05:021
176940399434,7cyclictest23-21ksoftirqd/107:13:521
1769403994325,14cyclictest1773988-21sshd08:39:261
176939999432,38cyclictest131rcu_preempt08:19:270
176940399426,6cyclictest23-21ksoftirqd/108:44:251
1769403994222,14cyclictest275-21dbus-daemon08:15:001
1769399994220,17cyclictest1770936-21unin-run07:40:000
176941499416,7cyclictest361rcuc/307:49:593
1769414994132,6cyclictest1770138-21munin-plugin-st07:25:003
1769414994120,16cyclictest1775335-21munin-run09:05:013
176941499411,7cyclictest0-21swapper/309:30:183
1769399994134,4cyclictest16950irq/224-800000007:20:020
176939999412,8cyclictest1770824-21latency_hist07:35:020
1769399994127,11cyclictest12-21ksoftirqd/008:25:000
1769399994127,10cyclictest0-21swapper/008:55:020
1769399994126,11cyclictest1773482-21sshd08:29:270
1769399994125,12cyclictest3467718-21lldpd08:05:020
176939999412,35cyclictest1775303-21sshd09:04:260
1769410994033,5cyclictest1762570-21kworker/2:107:50:262
1769410994030,5cyclictest309-21in:imuxsock08:05:012
1769410994030,5cyclictest309-21in:imuxsock08:05:012
1769410994025,9cyclictest30-21ksoftirqd/208:00:002
1769403994027,10cyclictest23-21ksoftirqd/108:54:251
176940399402,6cyclictest23-21ksoftirqd/108:05:031
1769403994012,24cyclictest1769405-21latency_hist07:10:011
1769399994027,10cyclictest1-21systemd07:25:000
1769399994026,10cyclictest12-21ksoftirqd/007:55:020
1769414993926,10cyclictest275-21dbus-daemon07:15:003
1769414993924,12cyclictest275-21dbus-daemon08:35:023
1769410993932,5cyclictest1762570-21kworker/2:109:05:002
176940399393,6cyclictest231ksoftirqd/107:47:261
176941499383,8cyclictest37-21ksoftirqd/308:06:273
176941499383,7cyclictest37-21ksoftirqd/308:24:063
1769414993825,9cyclictest3467602-21systemd-journal07:37:263
1769410993828,7cyclictest1775027-21sshd08:59:532
1769410993828,7cyclictest1762570-21kworker/2:108:25:012
1769399993823,12cyclictest1772195-21cron08:05:000
1769399993823,12cyclictest1772195-21cron08:05:000
176941499374,6cyclictest37-21ksoftirqd/308:00:013
1769410993727,6cyclictest1762570-21kworker/2:107:15:002
176940399376,7cyclictest23-21ksoftirqd/107:28:541
176940399375,4cyclictest131rcu_preempt07:19:271
1769399993723,11cyclictest3467718-21lldpd07:47:020
176939999371,6cyclictest284-21systemd-logind08:35:010
176941499363,8cyclictest37-21ksoftirqd/308:40:013
1769414993621,11cyclictest1-21systemd08:15:003
1769403993623,10cyclictest1773503-21munin-run08:30:001
176939999364,7cyclictest12-21ksoftirqd/007:30:020
1769399993627,6cyclictest12-21ksoftirqd/008:50:020
176939999361,31cyclictest0-21swapper/008:44:510
1769414993526,5cyclictest1775131-21latency_hist09:00:023
176941099351,3cyclictest131rcu_preempt08:55:022
1769399993528,4cyclictest16950irq/224-800000008:11:260
1769414993422,9cyclictest37-21ksoftirqd/307:10:023
1769414993422,9cyclictest1774267-21sshd08:44:263
176941499341,3cyclictest131rcu_preempt08:45:033
1769410993323,8cyclictest30-21ksoftirqd/207:50:002
176941499323,7cyclictest37-21ksoftirqd/308:05:013
176941499323,7cyclictest37-21ksoftirqd/308:05:003
176941499322,4cyclictest37-21ksoftirqd/307:30:013
176939999321,3cyclictest150-21pr/ttyS207:10:020
1769410992916,10cyclictest0-21swapper/207:10:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional