You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-25 - 21:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Sat Apr 25, 2026 12:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2839894991003,93cyclictest2869877-21ssh10:05:022
2839884999822,37cyclictest2900240-21latency_hist11:15:020
2839884999729,26cyclictest0-21swapper/009:52:280
2839884999626,26cyclictest2855968-21ssh09:32:430
283988499961,64cyclictest2910633-21ssh11:39:320
283989899911,69cyclictest2857041-21sh09:35:013
2839884998729,41cyclictest2847870-21ssh09:14:040
2839884998619,58cyclictest16850irq/134-800000010:01:050
2839890998511,32cyclictest2913079-21cat11:45:021
2839890998511,32cyclictest2913079-21cat11:45:021
2839884998247,31cyclictest16850irq/134-800000009:30:020
2839884998230,34cyclictest2849610-21ssh09:18:000
283988499802,31cyclictest2919823-21kworker/u8:012:15:030
283988499802,31cyclictest2919823-21kworker/u8:012:15:030
2839884997917,34cyclictest2894825-21rm11:02:370
2839884997671,3cyclictest150-21pr/ttyS210:40:020
283988499752,6cyclictest2880623-21munin-run10:30:000
2839884997514,52cyclictest16850irq/134-800000011:07:340
283988499745,63cyclictest2884941-21cron10:40:010
2839884997430,28cyclictest2908105-21ssh11:33:370
2839884997332,37cyclictest16850irq/134-800000010:25:010
2839884997330,27cyclictest2920678-21ssh12:02:340
283988499722,53cyclictest2911350-21ssh11:40:440
283988499722,53cyclictest2911350-21ssh11:40:430
2839884997124,31cyclictest2850802-21ssh09:20:250
283988499705,58cyclictest2846141-21unin-run09:10:010
283988499705,58cyclictest2846141-21unin-run09:10:000
2839884997029,24cyclictest2922033-21ssh12:05:320
283988499702,3cyclictest0-21swapper/012:25:030
283988499701,65cyclictest0-21swapper/011:20:030
283988499701,65cyclictest0-21swapper/010:45:050
2839884997015,37cyclictest2870313-21ssh10:05:420
2839884996962,4cyclictest16850irq/134-800000010:55:020
2839884996923,29cyclictest2875376-21ssh10:17:440
2839884996918,35cyclictest16850irq/134-800000011:58:460
283988499691,5cyclictest2881072-21ssh10:30:350
283988499691,11cyclictest2842321-21latency_hist07:55:030
2839884996864,2cyclictest0-21swapper/007:35:010
283988499681,63cyclictest0-21swapper/010:00:030
2839894996717,15cyclictest30-21ksoftirqd/209:20:012
283988499678,44cyclictest16850irq/134-800000012:28:550
283988499672,19cyclictest2893432-21sshd10:59:420
2839884996717,30cyclictest16850irq/134-800000012:33:430
2839884996717,30cyclictest16850irq/134-800000012:33:430
283988499662,7cyclictest2844561-21sendmail08:40:010
2839884996623,26cyclictest2905703-21ssh11:27:550
2839884996619,43cyclictest2857679-21ssh09:36:350
2839884996522,26cyclictest2903802-21ssh11:23:340
2839884996522,26cyclictest2903802-21ssh11:23:330
283988499651,7cyclictest2917485-21latency_hist11:55:030
283988499651,7cyclictest2917485-21latency_hist11:55:030
2839884996428,33cyclictest2845585-21cron09:00:000
283988499642,6cyclictest2863374-21munin-run09:50:010
2839884996416,41cyclictest0-21swapper/007:40:010
283988499641,5cyclictest2914728-21ssh11:48:580
283988499641,59cyclictest0-21swapper/012:15:140
283988499641,14cyclictest2935426-21ssh12:36:420
2839898996314,5cyclictest131rcu_preempt09:20:333
283989499636,11cyclictest30-21ksoftirqd/208:55:022
283988499632,57cyclictest2839878-21cyclictest07:10:020
283988499623,52cyclictest284-21systemd-logind07:15:000
283988499622,54cyclictest48-21kcompactd008:15:330
283988499622,54cyclictest48-21kcompactd008:15:330
283988499621,9cyclictest48-21kcompactd008:40:300
283988499621,12cyclictest2842670-21sshd08:02:430
283989899614,9cyclictest37-21ksoftirqd/310:30:033
2839890996113,10cyclictest23-21ksoftirqd/107:20:011
283988499612,44cyclictest2843955-21kworker/u8:008:50:020
283988499612,16cyclictest2840323-21sshd07:19:440
2839890996016,7cyclictest23-21ksoftirqd/109:13:031
2839890996015,8cyclictest23-21ksoftirqd/110:13:291
283988499601,5cyclictest2872868-21sshd10:11:420
283988499601,4cyclictest2841893-21sshd07:49:420
283988499601,16cyclictest2859745-21ssh09:41:100
283989899594,7cyclictest37-21ksoftirqd/311:05:013
2839898995912,6cyclictest131rcu_preempt11:05:113
2839894995917,5cyclictest131rcu_preempt12:39:592
283988499593,51cyclictest3467602-21systemd-journal07:30:010
283988499593,51cyclictest3467602-21systemd-journal07:30:000
283988499592,6cyclictest2843741-21munin-run08:24:590
2839894995841,7cyclictest2845354-21kworker/2:309:12:112
2839894995812,7cyclictest131rcu_preempt10:48:322
2839894995812,5cyclictest131rcu_preempt10:25:172
2839890995813,9cyclictest23-21ksoftirqd/109:30:031
283988499583,48cyclictest2845318-21cron08:55:010
2839898995719,5cyclictest131rcu_preempt10:25:003
2839898995711,6cyclictest131rcu_preempt12:29:473
2839894995730,10cyclictest30-21ksoftirqd/210:08:162
283989099572,21cyclictest2898706-21ssh11:11:321
283988499572,11cyclictest2842457-21munin-run08:00:000
283989899569,6cyclictest131rcu_preempt09:56:523
283989899564,14cyclictest37-21ksoftirqd/309:55:013
2839898995613,5cyclictest131rcu_preempt11:42:513
2839898995613,5cyclictest131rcu_preempt11:42:513
2839898995612,6cyclictest131rcu_preempt10:17:183
2839894995615,5cyclictest131rcu_preempt09:00:012
2839894995611,5cyclictest131rcu_preempt11:36:532
283989099569,12cyclictest23-21ksoftirqd/108:20:011
283989099569,12cyclictest23-21ksoftirqd/108:20:001
283989099567,10cyclictest23-21ksoftirqd/110:55:001
2839890995614,9cyclictest23-21ksoftirqd/111:08:011
2839890995611,11cyclictest23-21ksoftirqd/108:45:021
283988499562,12cyclictest2840621-21munin-run07:25:000
283989899557,9cyclictest37-21ksoftirqd/311:27:053
283989899557,8cyclictest37-21ksoftirqd/311:35:013
283989899556,9cyclictest37-21ksoftirqd/311:20:033
283989899555,8cyclictest37-21ksoftirqd/309:15:003
2839898995519,8cyclictest131rcu_preempt12:16:433
2839898995516,5cyclictest131rcu_preempt10:32:023
2839898995515,11cyclictest37-21ksoftirqd/310:45:133
2839898995514,5cyclictest131rcu_preempt12:08:283
283989499555,28cyclictest2875241-21kworker/u8:111:50:022
283989499555,28cyclictest2875241-21kworker/u8:111:50:012
2839894995540,6cyclictest2878502-21kworker/2:111:34:032
283989499552,4cyclictest131rcu_preempt10:20:012
283989099559,7cyclictest2894383-21ssh11:01:341
283989099552,18cyclictest2855850-21ssh09:32:181
2839890995510,10cyclictest23-21ksoftirqd/111:36:561
283988499552,6cyclictest364-21snmpd07:43:350
283988499552,13cyclictest1-21systemd09:05:010
283989899548,8cyclictest37-21ksoftirqd/311:10:413
283989899547,8cyclictest37-21ksoftirqd/309:25:203
283989899546,8cyclictest37-21ksoftirqd/309:15:123
283989899546,11cyclictest37-21ksoftirqd/310:14:543
283989899546,11cyclictest37-21ksoftirqd/309:48:543
283989899545,11cyclictest37-21ksoftirqd/310:56:443
2839898995436,13cyclictest1-21systemd07:55:013
2839898995417,17cyclictest37-21ksoftirqd/312:03:083
2839898995415,5cyclictest131rcu_preempt11:39:583
2839898995411,6cyclictest131rcu_preempt10:39:163
283989499548,11cyclictest30-21ksoftirqd/209:34:592
2839894995440,6cyclictest2846137-21kworker/2:009:39:242
2839894995439,7cyclictest2846137-21kworker/2:009:44:382
2839894995414,5cyclictest131rcu_preempt12:22:222
2839894995413,5cyclictest131rcu_preempt12:19:202
2839894995411,6cyclictest131rcu_preempt11:45:322
283989499541,15cyclictest3467602-21systemd-journal07:45:002
283989099549,9cyclictest23-21ksoftirqd/112:22:491
283989099549,14cyclictest2881567-21ssh10:31:541
2839890995413,8cyclictest23-21ksoftirqd/110:39:241
2839890995411,11cyclictest23-21ksoftirqd/111:27:521
283988499543,46cyclictest3467602-21systemd-journal08:10:010
283988499542,48cyclictest2844235-21md.daily08:34:430
283989899538,9cyclictest37-21ksoftirqd/311:57:083
283989899538,6cyclictest131rcu_preempt09:42:073
283989899536,6cyclictest131rcu_preempt12:37:173
2839898995336,13cyclictest3467602-21systemd-journal08:00:023
2839898995311,6cyclictest131rcu_preempt10:02:073
2839898995311,5cyclictest131rcu_preempt09:35:273
2839898995310,5cyclictest131rcu_preempt11:22:493
2839898995310,5cyclictest131rcu_preempt11:22:483
283989499536,6cyclictest131rcu_preempt12:35:022
283989499536,6cyclictest131rcu_preempt12:35:022
283989499536,29cyclictest2842911-21kworker/u8:109:05:022
2839894995345,5cyclictest30-21ksoftirqd/211:05:012
2839894995344,6cyclictest2878502-21kworker/2:111:18:472
2839894995312,5cyclictest131rcu_preempt11:58:282
2839890995316,8cyclictest23-21ksoftirqd/110:57:431
2839890995313,8cyclictest0-21swapper/112:15:451
283989899527,9cyclictest37-21ksoftirqd/310:40:223
283989899526,8cyclictest37-21ksoftirqd/311:48:123
283989899523,9cyclictest37-21ksoftirqd/309:10:023
283989899523,9cyclictest37-21ksoftirqd/309:10:013
2839898995220,10cyclictest37-21ksoftirqd/312:23:003
2839898995213,5cyclictest131rcu_preempt12:30:143
2839898995213,5cyclictest131rcu_preempt12:30:133
2839898995213,5cyclictest131rcu_preempt11:52:233
2839898995213,5cyclictest131rcu_preempt11:52:233
283989499529,6cyclictest131rcu_preempt11:26:252
2839894995244,5cyclictest2878502-21kworker/2:111:21:432
2839894995244,5cyclictest2878502-21kworker/2:111:21:422
2839894995244,5cyclictest2846137-21kworker/2:011:11:592
2839894995244,5cyclictest2831119-21kworker/2:207:20:012
2839894995243,6cyclictest2900089-21kworker/2:312:09:212
2839894995210,6cyclictest131rcu_preempt10:32:502
2839890995213,9cyclictest23-21ksoftirqd/110:08:461
2839890995212,8cyclictest23-21ksoftirqd/111:52:271
2839890995212,8cyclictest23-21ksoftirqd/111:52:271
283989899518,8cyclictest37-21ksoftirqd/310:06:253
2839898995112,5cyclictest131rcu_preempt07:15:003
283989499514,6cyclictest131rcu_preempt12:10:352
283989499514,6cyclictest131rcu_preempt12:10:342
2839894995143,5cyclictest2900089-21kworker/2:312:00:132
2839894995143,5cyclictest2846137-21kworker/2:010:13:322
2839894995143,5cyclictest2845354-21kworker/2:309:25:052
2839894995136,7cyclictest2845354-21kworker/2:309:52:162
2839894995117,19cyclictest30-21ksoftirqd/211:06:202
2839894995113,5cyclictest131rcu_preempt10:40:012
283989099517,7cyclictest23-21ksoftirqd/109:57:461
283989099513,16cyclictest2921100-21ssh12:03:441
283989099512,10cyclictest23-21ksoftirqd/112:31:561
283989099512,10cyclictest23-21ksoftirqd/112:31:551
2839890995110,9cyclictest23-21ksoftirqd/109:49:461
283988499511,4cyclictest2843167-21sshd08:10:420
2839898995018,5cyclictest131rcu_preempt10:53:133
2839898995016,5cyclictest131rcu_preempt12:10:363
2839898995016,5cyclictest131rcu_preempt12:10:353
2839894995042,5cyclictest2846137-21kworker/2:010:44:412
2839894995042,5cyclictest2846137-21kworker/2:010:23:572
2839894995042,5cyclictest2845354-21kworker/2:309:56:412
2839894995042,5cyclictest2845354-21kworker/2:309:22:102
2839894995017,12cyclictest30-21ksoftirqd/210:51:062
2839894995012,6cyclictest131rcu_preempt07:40:002
2839894995011,5cyclictest131rcu_preempt10:56:522
2839890995035,11cyclictest3467602-21systemd-journal10:20:581
283988499503,6cyclictest2844071-21date08:30:020
283988499503,6cyclictest2844071-21date08:30:010
283989899496,10cyclictest37-21ksoftirqd/308:15:003
2839898994910,6cyclictest131rcu_preempt07:40:003
2839894994941,5cyclictest2878502-21kworker/2:112:27:202
2839894994941,5cyclictest2878502-21kworker/2:111:41:002
2839894994941,5cyclictest2878502-21kworker/2:111:41:002
2839894994941,5cyclictest2846137-21kworker/2:009:46:512
2839894994941,5cyclictest2842718-21kworker/2:108:49:352
2839894994941,5cyclictest2842718-21kworker/2:108:44:152
2839894994941,5cyclictest2842718-21kworker/2:108:20:072
283989099497,8cyclictest2875144-21ssh10:17:031
2839890994915,7cyclictest23-21ksoftirqd/110:45:121
283989899488,23cyclictest2555041-1kworker/3:2H07:50:013
2839894994840,5cyclictest2842718-21kworker/2:108:29:192
2839894994840,5cyclictest2842718-21kworker/2:108:29:182
2839894994840,5cyclictest2831119-21kworker/2:208:12:112
2839894994838,6cyclictest2842718-21kworker/2:108:30:172
283989099484,11cyclictest2924000-21modprobe12:10:021
283989099484,11cyclictest2924000-21modprobe12:10:011
283989099483,15cyclictest2879746-21ssh10:27:511
283989899474,8cyclictest37-21ksoftirqd/309:00:433
283989899472,12cyclictest37-21ksoftirqd/308:55:023
2839894994739,5cyclictest2845354-21kworker/2:309:05:292
2839894994739,5cyclictest2845354-21kworker/2:309:05:282
2839894994739,5cyclictest2842718-21kworker/2:108:38:342
2839894994739,5cyclictest2832490-21kworker/2:307:33:162
2839894994739,5cyclictest2831119-21kworker/2:208:19:262
2839894994739,5cyclictest2831119-21kworker/2:208:19:252
2839894994739,5cyclictest2831119-21kworker/2:208:02:212
2839894994739,5cyclictest2831119-21kworker/2:207:45:472
2839894994739,5cyclictest2831119-21kworker/2:207:25:032
2839894994739,5cyclictest2831119-21kworker/2:207:25:022
2839894994739,5cyclictest2831119-21kworker/2:207:20:042
2839894994739,5cyclictest2831119-21kworker/2:207:10:162
2839894994711,6cyclictest131rcu_preempt07:59:592
283989099477,9cyclictest2851053-21ssh09:21:001
283989099474,9cyclictest23-21ksoftirqd/112:07:321
283989099474,13cyclictest23-21ksoftirqd/111:21:541
283989099474,13cyclictest23-21ksoftirqd/111:21:531
283989899463,12cyclictest37-21ksoftirqd/309:00:013
283989899461,8cyclictest37-21ksoftirqd/308:41:433
2839894994638,5cyclictest2831119-21kworker/2:208:06:572
283989099469,8cyclictest0-21swapper/111:55:281
283989099467,7cyclictest23-21ksoftirqd/111:33:181
283989099467,6cyclictest2886251-21ssh10:42:501
283989099466,7cyclictest23-21ksoftirqd/112:37:431
2839890994629,13cyclictest2842400-21cron08:00:001
2839890994613,6cyclictest0-21swapper/107:20:131
2839890994611,8cyclictest131rcu_preempt09:52:401
283989899456,8cyclictest37-21ksoftirqd/308:17:013
283989899456,8cyclictest37-21ksoftirqd/308:17:003
283989899453,7cyclictest37-21ksoftirqd/308:50:013
2839898994532,10cyclictest1-21systemd07:35:003
2839894994538,4cyclictest2831119-21kworker/2:207:51:372
283989099459,9cyclictest0-21swapper/109:06:481
283989099459,9cyclictest0-21swapper/109:06:481
283989099454,7cyclictest23-21ksoftirqd/111:15:221
2839890994535,6cyclictest2846021-21latency_hist09:05:011
2839890994528,12cyclictest275-21dbus-daemon07:45:021
2839890994513,25cyclictest2841385-21munin-run07:40:001
2839890994510,7cyclictest23-21ksoftirqd/109:20:011
283989899447,6cyclictest37-21ksoftirqd/307:23:313
283989099448,8cyclictest0-21swapper/109:43:081
283989099446,10cyclictest364-21snmpd08:08:531
283989099445,5cyclictest2846116-21diskmemload10:02:171
283989099443,6cyclictest23-21ksoftirqd/112:27:161
2839890994429,10cyclictest23-21ksoftirqd/108:30:021
2839890994429,10cyclictest23-21ksoftirqd/108:30:011
283989099442,7cyclictest23-21ksoftirqd/111:48:291
2839890994413,5cyclictest0-21swapper/107:53:281
2839890994412,5cyclictest0-21swapper/109:38:381
2839890994410,5cyclictest0-21swapper/108:32:101
283989899434,8cyclictest37-21ksoftirqd/308:40:033
283989899434,5cyclictest131rcu_preempt08:29:403
283989899434,5cyclictest131rcu_preempt08:29:393
283989899431,8cyclictest0-21swapper/308:08:473
2839890994320,18cyclictest275-21dbus-daemon07:35:011
2839890994311,6cyclictest0-21swapper/107:47:431
283989899421,6cyclictest0-21swapper/308:24:123
2839890994230,9cyclictest2843167-21sshd08:10:431
283989099421,9cyclictest0-21swapper/108:58:391
2839898994122,15cyclictest3467602-21systemd-journal08:00:003
2839890994112,6cyclictest0-21swapper/108:38:481
283989899403,10cyclictest0-21swapper/308:31:093
2839898994010,5cyclictest131rcu_preempt07:42:443
283989899400,6cyclictest150-21pr/ttyS207:27:563
283989899400,6cyclictest150-21pr/ttyS207:27:553
283989099401,8cyclictest0-21swapper/107:12:561
2839898993910,5cyclictest131rcu_preempt07:10:023
2839898993824,11cyclictest275-21dbus-daemon07:20:003
283989099385,8cyclictest23-21ksoftirqd/108:23:001
2839890993822,12cyclictest1-21systemd07:30:001
2839890993822,12cyclictest1-21systemd07:30:001
2839894993525,6cyclictest30-21ksoftirqd/207:10:012
2839890993525,7cyclictest23-21ksoftirqd/108:04:261
283989099351,6cyclictest0-21swapper/108:48:571
2839890993426,5cyclictest2839822-21latency_hist07:10:021
2839890993411,5cyclictest0-21swapper/108:52:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional