You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-19 - 00:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sat Jul 18, 2026 12:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
58425599884,6cyclictest131rcu_preempt08:20:023
584240998458,11cyclictest16950irq/224-800000009:40:030
584248998172,4cyclictest23-21ksoftirqd/108:40:021
58424099738,38cyclictest16950irq/224-800000012:00:030
584255996829,13cyclictest150-21pr/ttyS212:18:123
584240996818,44cyclictest12-21ksoftirqd/009:25:010
58424899677,46cyclictest644691-21ssh11:05:011
584248996724,38cyclictest647020-21latency_hist11:10:031
584248996644,18cyclictest1-21systemd11:25:011
584255996552,10cyclictest596509-21ssh09:23:023
584255996547,14cyclictest594577-21ssh09:18:583
584255996547,14cyclictest594577-21ssh09:18:583
584255996543,18cyclictest654904-21ssh11:26:433
584255996543,18cyclictest654904-21ssh11:26:433
584255996447,14cyclictest621227-21ssh10:15:133
584255996447,14cyclictest621227-21ssh10:15:123
584255996447,13cyclictest674401-21ssh12:08:033
584255996446,14cyclictest667087-21ssh11:52:283
584255996444,16cyclictest683488-21ssh12:27:093
584248996419,10cyclictest684431-1kworker/u9:012:35:021
58424099646,39cyclictest12-21ksoftirqd/011:04:400
584255996348,12cyclictest687624-21ssh12:35:403
584255996348,12cyclictest643318-21ssh11:02:053
584255996348,12cyclictest630759-21ssh10:35:343
584255996348,12cyclictest625358-21ssh10:24:263
584255996348,12cyclictest592056-21ssh09:13:413
584255996347,12cyclictest600948-21ssh09:32:273
584255996346,13cyclictest664900-21ssh11:47:483
584255996346,13cyclictest619041-21ssh10:10:273
584240996316,4cyclictest12-21ksoftirqd/010:55:090
584255996250,9cyclictest660240-21ssh11:38:053
584255996250,9cyclictest660240-21ssh11:38:053
584255996249,10cyclictest647454-21ssh11:10:493
584255996249,10cyclictest628637-21ssh10:31:033
584255996249,10cyclictest614144-21ssh10:00:083
584255996247,12cyclictest637856-21ssh10:50:263
584255996247,12cyclictest602673-21ssh09:35:523
584255996246,13cyclictest680440-21ssh12:20:353
584255996246,13cyclictest652033-21ssh11:20:303
584255996246,12cyclictest676774-21ssh12:13:073
584255996246,12cyclictest662498-21ssh11:42:483
584255996246,12cyclictest608249-21ssh09:47:533
584255996245,13cyclictest649953-21ssh11:16:073
584255996245,13cyclictest644868-21ssh11:05:143
58424899622,19cyclictest680082-21latency_hist12:20:021
584240996228,13cyclictest6325191ssh10:39:510
584240996227,19cyclictest12-21ksoftirqd/011:50:020
584255996148,10cyclictest640040-21ssh10:55:033
584255996148,10cyclictest610389-21ssh09:52:263
584255996147,11cyclictest685314-21ssh12:30:483
584255996147,11cyclictest670381-21ssh11:59:423
584255996146,12cyclictest633099-21ssh10:40:313
584255996146,12cyclictest597660-21ssh09:25:083
584255996145,13cyclictest657132-21ssh11:31:213
584248996124,10cyclictest23-21ksoftirqd/111:55:231
584248996115,12cyclictest623224-21systemctl10:20:011
584248996115,12cyclictest623224-21systemctl10:20:001
584255996047,10cyclictest605084-21ssh09:41:053
584255996046,12cyclictest635404-21ssh10:45:243
584255996045,12cyclictest670833-21ssh12:00:083
584255996045,12cyclictest612623-21ssh09:57:073
584255996043,13cyclictest626550-21ssh10:26:393
584255996043,13cyclictest626550-21ssh10:26:383
584255996043,13cyclictest616904-21ssh10:05:563
584248996020,7cyclictest609367-21latency_hist09:50:021
58424899593,21cyclictest661043-21sh11:40:011
58424899593,21cyclictest661043-21sh11:40:001
58424899593,15cyclictest589128-21cut08:45:001
584248995925,30cyclictest658617-21sh11:34:501
584248995924,9cyclictest23-21ksoftirqd/109:29:111
584248995921,10cyclictest23-21ksoftirqd/110:39:071
584248995919,5cyclictest629469-21sh10:32:581
584240995927,16cyclictest637995-21ssh10:50:430
584255995844,9cyclictest48-21kcompactd008:08:403
58425299583,33cyclictest620008-21kworker/u8:310:15:032
58425299583,33cyclictest620008-21kworker/u8:310:15:022
584248995824,8cyclictest23-21ksoftirqd/112:11:391
584248995820,4cyclictest590420-21diskmemload09:35:591
584255995744,10cyclictest585979-21sshd07:40:193
584248995724,9cyclictest23-21ksoftirqd/112:25:011
584248995724,30cyclictest673483-21sh12:06:001
584248995717,7cyclictest595668-21kworker/u8:210:22:011
58424899571,20cyclictest635008-21ssh10:45:011
58424099574,12cyclictest3467602-21systemd-journal10:45:000
584240995729,13cyclictest645072-21ssh11:05:470
584255995643,9cyclictest48-21kcompactd008:44:103
584248995623,8cyclictest23-21ksoftirqd/111:17:121
584248995620,5cyclictest605773-21sh09:42:421
584248995613,11cyclictest585826-21cat07:40:031
584248995613,11cyclictest585826-21cat07:40:021
584240995638,14cyclictest1-21systemd08:10:010
584240995636,15cyclictest1-21systemd10:48:000
584240995624,15cyclictest666166-21ssh11:50:180
58424099561,25cyclictest131rcu_preempt12:02:350
584255995543,8cyclictest48-21kcompactd007:53:413
584255995542,9cyclictest48-21kcompactd008:45:183
58424899553,17cyclictest588596-21grep08:35:001
58424899553,13cyclictest586760-21modprobe07:59:591
584248995524,12cyclictest23-21ksoftirqd/112:28:361
58424899552,18cyclictest586514-21munin-plugin-st07:55:001
584248995521,8cyclictest23-21ksoftirqd/109:55:531
584248995521,7cyclictest23-21ksoftirqd/109:24:311
584248995519,4cyclictest617180-21sh10:06:391
584248995516,8cyclictest23-21ksoftirqd/109:04:141
584248995513,9cyclictest642222-21systemd-run11:00:011
584248995510,12cyclictest275-21dbus-daemon08:10:011
58424099553,46cyclictest590420-21sh09:10:010
58424099553,46cyclictest590420-21sh09:10:000
584240995529,22cyclictest656029-21ssh11:29:210
584240995529,22cyclictest656029-21ssh11:29:200
584240995528,12cyclictest657821-21ssh11:32:580
584240995528,12cyclictest648136-21ssh11:12:240
584240995528,11cyclictest610062-21ssh09:51:340
584240995526,13cyclictest593651-21ssh09:16:540
584240995526,13cyclictest593651-21ssh09:16:530
584255995442,9cyclictest48-21kcompactd007:14:193
584255995442,9cyclictest48-21kcompactd007:14:183
584252995444,6cyclictest623270-21kworker/2:310:35:412
58424899543,16cyclictest595008-21sendmail09:20:001
58424899543,16cyclictest595008-21sendmail09:20:001
584248995422,5cyclictest668295-21latency_hist11:55:031
584248995420,4cyclictest590420-21diskmemload12:03:511
584248995419,4cyclictest590420-21diskmemload11:45:571
584248995417,9cyclictest23-21ksoftirqd/110:11:331
584240995432,14cyclictest609183-21modprobe09:50:010
584240995427,11cyclictest602460-21ssh09:35:260
584240995417,31cyclictest12-21ksoftirqd/009:34:240
584255995341,9cyclictest48-21kcompactd007:35:233
584255995341,9cyclictest48-21kcompactd007:35:233
584252995345,5cyclictest649235-21kworker/2:112:40:022
584252995345,5cyclictest649235-21kworker/2:112:32:342
584252995345,5cyclictest623270-21kworker/2:310:52:502
584252995345,5cyclictest623270-21kworker/2:310:48:172
584252995345,5cyclictest589390-21kworker/2:211:51:312
584252995341,9cyclictest653948-21ssh11:25:002
584248995324,25cyclictest590420-21diskmemload09:12:241
584248995317,7cyclictest23-21ksoftirqd/111:12:271
584248995316,6cyclictest587610-21cat08:15:021
584240995326,11cyclictest615074-21ssh10:02:140
584240995324,14cyclictest627665-21ssh10:29:190
584240995324,14cyclictest627665-21ssh10:29:180
584255995240,9cyclictest48-21kcompactd007:29:173
584252995244,5cyclictest649235-21kworker/2:111:59:452
584252995244,5cyclictest623270-21kworker/2:310:24:502
584252995244,5cyclictest589390-21kworker/2:209:36:352
584252995243,5cyclictest589390-21kworker/2:211:03:492
58424099524,6cyclictest620969-21latency_hist10:15:010
584240995234,14cyclictest12-21ksoftirqd/012:31:510
584240995231,17cyclictest16850irq/134-800000012:38:080
584240995222,15cyclictest592610-21ssh09:14:580
584255995139,9cyclictest364-21snmpd07:19:003
584255995137,9cyclictest48-21kcompactd009:03:033
584252995143,5cyclictest589390-21kworker/2:211:09:252
584252995143,5cyclictest589390-21kworker/2:210:00:322
584252995142,5cyclictest589390-21kworker/2:209:25:042
584252995142,5cyclictest589390-21kworker/2:209:11:172
584252995142,5cyclictest586287-21kworker/2:108:10:022
58425299513,12cyclictest642188-21munin-run11:00:002
584248995120,5cyclictest23-21ksoftirqd/109:31:201
584248995120,4cyclictest590420-21diskmemload10:28:181
584248995120,4cyclictest590420-21diskmemload10:28:171
584240995143,5cyclictest16850irq/134-800000012:30:010
584240995112,36cyclictest12-21ksoftirqd/009:25:580
584255995043,3cyclictest48-21kcompactd008:28:383
584255995034,13cyclictest589636-21/usr/sbin/munin08:55:013
584252995042,5cyclictest649235-21kworker/2:112:13:202
584252995042,5cyclictest623270-21kworker/2:310:40:262
584252995042,5cyclictest585787-21kworker/2:308:45:022
584252995041,6cyclictest649235-21kworker/2:112:22:292
584252995041,6cyclictest649235-21kworker/2:112:19:262
584252995041,6cyclictest649235-21kworker/2:112:09:362
584252995041,6cyclictest589390-21kworker/2:209:16:042
584252995041,6cyclictest589390-21kworker/2:209:16:042
58425299502,44cyclictest364-21snmpd12:30:032
58424899502,17cyclictest590417-21sh09:10:011
58424899502,17cyclictest590417-21sh09:10:001
584248995018,7cyclictest0-21swapper/108:25:011
584248995018,7cyclictest0-21swapper/108:25:011
58424099502,43cyclictest677593-21cron12:15:000
58424099502,42cyclictest3467602-21systemd-journal10:34:590
584240995022,13cyclictest617159-21ssh10:06:350
584240995014,26cyclictest584235-21cyclictest07:10:010
584255994938,9cyclictest48-21kcompactd008:55:163
584255994938,9cyclictest364-21snmpd09:09:203
584255994938,9cyclictest364-21snmpd09:09:203
584255994936,10cyclictest37-21ksoftirqd/308:15:033
584252994941,5cyclictest589390-21kworker/2:211:30:182
584252994941,5cyclictest589390-21kworker/2:211:27:312
584252994941,5cyclictest589390-21kworker/2:211:27:302
584252994941,5cyclictest589390-21kworker/2:211:12:212
584252994941,5cyclictest589390-21kworker/2:209:23:012
584252994941,5cyclictest589390-21kworker/2:209:03:282
584252994939,6cyclictest589390-21kworker/2:209:35:002
584252994932,13cyclictest30-21ksoftirqd/208:20:002
584248994924,8cyclictest23-21ksoftirqd/109:51:381
584248994920,4cyclictest586008-21latency_hist07:45:011
584248994917,4cyclictest655361-21sh11:27:501
584248994917,4cyclictest655361-21sh11:27:501
584240994941,5cyclictest16950irq/224-800000010:20:010
584240994941,5cyclictest16950irq/224-800000010:20:000
584240994941,4cyclictest16950irq/224-800000011:25:010
584240994937,8cyclictest679973-21cron12:20:010
584240994932,12cyclictest1-21systemd12:25:000
584240994932,12cyclictest1-21systemd07:40:010
584240994932,12cyclictest1-21systemd07:40:010
584240994931,12cyclictest587620-21cut08:15:030
58424099491,8cyclictest584235-21cyclictest07:25:030
58424099491,8cyclictest584235-21cyclictest07:25:020
584255994840,4cyclictest48-21kcompactd008:04:473
584255994834,11cyclictest588599-21/usr/sbin/munin08:35:003
584252994840,5cyclictest589390-21kworker/2:210:05:472
584252994839,6cyclictest589390-21kworker/2:209:42:172
584252994838,7cyclictest623270-21kworker/2:310:25:082
584252994838,7cyclictest623270-21kworker/2:310:25:082
584252994834,10cyclictest1-21systemd11:50:012
584252994833,11cyclictest672185-21sshd12:03:192
584248994823,8cyclictest23-21ksoftirqd/110:01:451
584248994822,8cyclictest23-21ksoftirqd/111:45:011
584248994817,7cyclictest23-21ksoftirqd/108:55:021
584240994844,2cyclictest0-21swapper/008:25:030
584240994844,2cyclictest0-21swapper/008:25:020
584240994827,4cyclictest6631261ssh11:44:170
584240994826,15cyclictest275-21dbus-daemon09:59:590
58424099482,14cyclictest675412-21latency_hist12:10:010
584252994739,5cyclictest649235-21kworker/2:111:36:102
584252994739,5cyclictest649235-21kworker/2:111:36:092
584252994739,5cyclictest589390-21kworker/2:209:59:052
584252994738,5cyclictest576538-21kworker/2:007:25:022
584252994738,5cyclictest576538-21kworker/2:007:25:022
584252994736,7cyclictest364-21snmpd07:45:002
58424099472,39cyclictest625626-21latency_hist10:25:010
584252994639,5cyclictest589390-21kworker/2:211:42:092
584252994637,5cyclictest589390-21kworker/2:211:15:222
584252994636,6cyclictest623270-21kworker/2:310:35:022
584240994627,13cyclictest3467602-21systemd-journal07:29:590
584240994625,15cyclictest3467602-21systemd-journal08:35:010
584240994621,21cyclictest660006-21ssh11:37:360
584240994621,21cyclictest660006-21ssh11:37:350
584240994618,13cyclictest650178-21ssh11:16:480
584255994533,9cyclictest150-21pr/ttyS207:30:113
584255994532,10cyclictest150-21pr/ttyS208:21:563
584255994532,10cyclictest150-21pr/ttyS208:21:553
584252994537,5cyclictest576538-21kworker/2:007:27:402
584252994531,10cyclictest120-21kswapd009:55:012
584248994526,13cyclictest584681-21cron07:20:001
584248994515,11cyclictest23-21ksoftirqd/110:52:521
584248994513,10cyclictest589957-21cut09:00:011
584240994531,9cyclictest589896-21systemctl09:00:000
58424099452,6cyclictest131rcu_preempt07:50:000
58424099452,38cyclictest151rcuc/008:30:010
584252994427,13cyclictest1-21systemd07:55:012
584240994428,12cyclictest3467602-21systemd-journal08:40:000
58424099441,39cyclictest590135-21sshd09:01:190
584255994332,9cyclictest150-21pr/ttyS207:55:103
584255994332,9cyclictest150-21pr/ttyS207:45:093
584255994331,9cyclictest150-21pr/ttyS208:35:193
584248994327,12cyclictest1-21systemd07:30:001
584240994339,2cyclictest0-21swapper/008:40:020
584255994232,8cyclictest150-21pr/ttyS207:20:353
584255994232,8cyclictest150-21pr/ttyS207:20:343
584252994234,5cyclictest585787-21kworker/2:308:36:202
584252994228,10cyclictest30-21ksoftirqd/209:10:022
584252994228,10cyclictest30-21ksoftirqd/209:10:012
584252994223,14cyclictest3467602-21systemd-journal07:59:592
584240994230,9cyclictest12-21ksoftirqd/007:55:020
584240994230,7cyclictest309-21in:imuxsock08:05:000
584252994132,6cyclictest589390-21kworker/2:210:14:472
584252994129,9cyclictest589714-21sed08:55:012
584252994126,12cyclictest275-21dbus-daemon08:30:002
584252994125,12cyclictest30-21ksoftirqd/208:02:002
584252994125,12cyclictest275-21dbus-daemon07:40:022
584252994125,12cyclictest275-21dbus-daemon07:40:012
584240994128,9cyclictest586753-21cron07:59:590
584240994121,16cyclictest275-21dbus-daemon07:45:010
58424099411,37cyclictest0-21swapper/007:35:010
584252994027,9cyclictest588597-21cut08:35:012
584252994022,14cyclictest3467602-21systemd-journal07:20:012
584252994022,14cyclictest275-21dbus-daemon07:15:022
584252994022,14cyclictest275-21dbus-daemon07:15:012
584240994028,9cyclictest12-21ksoftirqd/007:15:010
584240994028,9cyclictest12-21ksoftirqd/007:15:010
584240993932,4cyclictest16950irq/224-800000008:20:000
584240993924,9cyclictest589381-21grep08:50:010
584252993830,5cyclictest609192-21sh09:50:012
58425299381,3cyclictest131rcu_preempt07:50:002
584252993627,6cyclictest585476-21latency_hist07:35:002
584252993624,9cyclictest588221-21sed08:25:032
584252993624,9cyclictest588221-21sed08:25:022
584248993626,7cyclictest23-21ksoftirqd/107:15:001
584248993626,7cyclictest23-21ksoftirqd/107:15:001
584248993625,9cyclictest23-21ksoftirqd/110:47:381
584248993624,9cyclictest23-21ksoftirqd/107:32:441
584248993624,10cyclictest23-21ksoftirqd/112:40:011
584248993622,11cyclictest23-21ksoftirqd/107:25:031
584248993622,11cyclictest23-21ksoftirqd/107:25:021
584248993618,13cyclictest586249-21cron07:50:011
58424099361,10cyclictest1-21systemd08:54:190
58424899355,3cyclictest131rcu_preempt08:20:031
584248993520,11cyclictest1-21systemd08:05:021
58424099351,31cyclictest0-21swapper/007:17:000
584252993421,8cyclictest0-21swapper/207:10:012
584248993419,10cyclictest309-21in:imuxsock08:46:191
584252993322,8cyclictest30-21ksoftirqd/209:00:022
584255993222,7cyclictest584346-21sed07:10:023
58425299321,3cyclictest131rcu_preempt08:50:002
584248993023,4cyclictest588516-21latency_hist08:30:031
584252992923,4cyclictest3468525-21sendmail-mta08:10:292
584248992617,6cyclictest23-21ksoftirqd/107:10:031
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional