You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-10 - 05:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sun May 10, 2026 00:45:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3258217991012,4cyclictest131rcu_preempt22:50:021
325821499803,55cyclictest0-21swapper/021:10:000
325821499787,39cyclictest0-21swapper/021:05:010
3258214997810,31cyclictest1-21systemd23:59:250
3258214997810,31cyclictest1-21systemd23:59:240
325821499771,73cyclictest0-21swapper/020:00:030
3258214997571,2cyclictest0-21swapper/000:05:020
3258214997571,2cyclictest0-21swapper/000:05:010
3258214997317,26cyclictest3323981-21ssh23:14:340
3258214997114,20cyclictest3258262-21sed19:10:020
3258214997013,21cyclictest3305616-21ssh22:35:450
3258214997010,23cyclictest3331328-21ssh23:29:570
3258214997010,23cyclictest3331328-21ssh23:29:570
3258219996946,14cyclictest3269169-21kworker/2:123:19:172
3258214996811,18cyclictest3290666-21date22:05:010
3258214996811,18cyclictest3290666-21date22:05:010
325821499680,34cyclictest0-21swapper/021:36:380
325821499680,34cyclictest0-21swapper/021:36:380
3258214996715,16cyclictest3304031-21ssh22:32:380
3258214996631,32cyclictest3312584-21ssh22:50:170
3258214996616,46cyclictest3319916-21ssh23:05:410
325821499658,23cyclictest3349531-21ssh00:07:480
3258214996531,31cyclictest3341764-21ssh23:51:250
3258214996530,31cyclictest3309242-21ssh22:43:310
3258214996430,31cyclictest3291338-21ssh22:06:050
3258214996429,4cyclictest3276074-21sshd21:34:240
3258214996415,14cyclictest3334150-21ssh23:35:220
3258214996319,16cyclictest3353148-21sh00:15:080
3258214996317,15cyclictest3327299-21ssh23:21:060
3258214996315,15cyclictest3318630-21sh23:03:090
3258214996310,37cyclictest3294599-21ssh22:13:090
3258214996310,37cyclictest3294599-21ssh22:13:080
3258219996244,7cyclictest3269169-21kworker/2:121:54:262
3258219996244,7cyclictest3269169-21kworker/2:121:54:262
3258214996241,6cyclictest16850irq/134-800000022:30:010
3258214996238,19cyclictest16850irq/134-800000023:00:010
3258214996232,4cyclictest3258378-21sshd19:13:250
3258214996232,4cyclictest3258378-21sshd19:13:240
3258214996215,13cyclictest3360940-21ssh00:31:280
325821499621,24cyclictest0-21swapper/021:47:230
325821499619,35cyclictest3358189-21ssh00:25:420
3258214996117,9cyclictest3273875-21ssh21:29:580
3258214996117,9cyclictest3273875-21ssh21:29:570
3258214996111,45cyclictest3331682-21ssh23:30:130
3258214996111,45cyclictest3331682-21ssh23:30:130
3258214996010,19cyclictest3338874-21ssh23:45:080
3258214996010,19cyclictest3338874-21ssh23:45:070
3258219995936,14cyclictest3246273-21kworker/2:322:59:142
3258214995836,5cyclictest16850irq/134-800000000:15:000
3258214995818,9cyclictest3312166-21ssh22:49:520
3258219995750,3cyclictest3246273-21kworker/2:322:30:022
3258219995748,6cyclictest3329045-21kworker/2:000:26:482
325821499576,30cyclictest131rcu_preempt00:25:010
325821499576,30cyclictest131rcu_preempt00:25:000
325821499574,17cyclictest3364938-21latency_hist00:40:020
3258214995717,12cyclictest0-21swapper/023:43:450
3258219995647,6cyclictest3329045-21kworker/2:023:47:452
3258219995647,6cyclictest3329045-21kworker/2:023:47:442
3258214995622,29cyclictest16850irq/134-800000023:19:370
3258214995613,6cyclictest12-21ksoftirqd/021:21:030
3258214995613,13cyclictest3286382-21sh21:55:470
3258219995546,5cyclictest3246273-21kworker/2:322:06:212
3258219995510,32cyclictest0-21swapper/222:00:012
3258224995437,12cyclictest1-21systemd22:10:013
3258219995445,6cyclictest3269169-21kworker/2:122:04:502
3258219995445,6cyclictest3269169-21kworker/2:122:04:492
3258219995440,6cyclictest3246273-21kworker/2:322:15:442
3258219995427,9cyclictest3329045-21kworker/2:023:52:142
3258214995435,13cyclictest3467602-21systemd-journal19:35:000
3258214995427,11cyclictest3284089-21ssh21:50:560
3258214995427,11cyclictest3284089-21ssh21:50:560
3258224995317,6cyclictest131rcu_preempt21:20:003
3258224995317,6cyclictest131rcu_preempt21:20:003
3258219995346,5cyclictest3246273-21kworker/2:322:31:442
3258219995345,5cyclictest3269169-21kworker/2:123:01:022
3258219995343,6cyclictest3329045-21kworker/2:000:09:302
3258219995343,6cyclictest3329045-21kworker/2:000:00:082
3258219995343,6cyclictest3329045-21kworker/2:000:00:082
3258219995335,8cyclictest3246273-21kworker/2:322:24:402
3258219995334,8cyclictest3269169-21kworker/2:122:47:042
3258214995315,6cyclictest12-21ksoftirqd/020:15:010
3258219995244,5cyclictest3246273-21kworker/2:322:36:452
3258219995244,5cyclictest3246273-21kworker/2:321:14:022
3258219995243,5cyclictest3246273-21kworker/2:322:44:242
3258219995242,7cyclictest3269169-21kworker/2:121:28:382
3258219995242,7cyclictest3269169-21kworker/2:121:28:372
3258219995242,6cyclictest3329045-21kworker/2:023:56:402
3258219995242,6cyclictest3329045-21kworker/2:023:56:392
3258219995227,9cyclictest3269169-21kworker/2:121:45:082
3258217995212,10cyclictest131rcu_preempt20:20:001
325821499527,26cyclictest3259216-21kworker/0:220:00:020
3258214995233,11cyclictest3297831-21modprobe22:19:590
325821499521,36cyclictest3299100-21ssh22:22:190
3258224995131,15cyclictest1-21systemd00:00:053
3258224995131,15cyclictest1-21systemd00:00:043
3258219995145,4cyclictest3329045-21kworker/2:000:10:192
3258219995144,4cyclictest3246273-21kworker/2:322:53:462
3258219995143,6cyclictest3269169-21kworker/2:123:34:432
3258219995143,6cyclictest3269169-21kworker/2:123:34:422
3258219995143,5cyclictest3329045-21kworker/2:000:33:382
3258219995143,5cyclictest3329045-21kworker/2:000:16:482
3258219995143,5cyclictest3269169-21kworker/2:123:13:292
3258219995143,5cyclictest3269169-21kworker/2:121:44:592
3258219995142,6cyclictest3329045-21kworker/2:023:40:362
3258219995142,6cyclictest3329045-21kworker/2:023:38:172
3258219995142,6cyclictest3269169-21kworker/2:123:20:372
3258219995142,5cyclictest3246273-21kworker/2:321:16:142
3258219995142,5cyclictest3246273-21kworker/2:321:16:142
3258219995141,6cyclictest3269169-21kworker/2:121:32:322
3258219995129,8cyclictest3246273-21kworker/2:322:11:262
3258219995129,8cyclictest3246273-21kworker/2:322:11:262
325821499513,40cyclictest3261777-21latency_hist20:20:010
3258214995110,13cyclictest0-21swapper/021:15:170
3258214995110,13cyclictest0-21swapper/021:15:170
325822499504,3cyclictest131rcu_preempt21:51:513
325822499504,3cyclictest131rcu_preempt21:51:503
3258219995043,4cyclictest3246273-21kworker/2:319:15:032
3258219995043,4cyclictest3246273-21kworker/2:319:15:022
3258219995042,5cyclictest3269169-21kworker/2:123:28:202
3258219995042,5cyclictest3269169-21kworker/2:123:28:202
3258219995042,5cyclictest3269169-21kworker/2:123:08:322
3258219995041,5cyclictest3246273-21kworker/2:320:35:022
325821799504,3cyclictest131rcu_preempt00:00:011
325821799504,3cyclictest131rcu_preempt00:00:001
3258214995034,11cyclictest3467602-21systemd-journal21:40:020
3258214995030,15cyclictest3265824-21sshd21:12:580
3258224994932,4cyclictest33266991ssh23:20:013
325822499492,6cyclictest131rcu_preempt20:15:013
325822499492,4cyclictest131rcu_preempt00:00:033
325822499492,4cyclictest131rcu_preempt00:00:023
325822499491,6cyclictest131rcu_preempt00:20:003
325821999495,7cyclictest30-21ksoftirqd/200:40:012
3258219994941,5cyclictest3329045-21kworker/2:000:21:362
3258219994941,5cyclictest3329045-21kworker/2:000:21:352
3258219994941,5cyclictest3269169-21kworker/2:121:35:362
3258219994941,5cyclictest3269169-21kworker/2:121:35:352
3258217994934,12cyclictest1-21systemd19:25:011
325821799492,4cyclictest131rcu_preempt00:14:081
3258214994933,11cyclictest3260195-21sshd19:47:240
3258214994933,11cyclictest275-21dbus-daemon20:30:010
3258214994925,18cyclictest3259679-21cron19:40:010
325822499482,3cyclictest131rcu_preempt23:42:043
3258224994820,20cyclictest0-21swapper/323:20:343
3258219994840,5cyclictest3246273-21kworker/2:319:20:252
3258219994839,5cyclictest3246273-21kworker/2:321:00:002
3258219994833,11cyclictest275-21dbus-daemon20:00:012
325821799482,7cyclictest131rcu_preempt21:40:001
325821799482,7cyclictest131rcu_preempt21:40:001
3258214994835,10cyclictest3258910-21sshd19:22:240
3258214994833,10cyclictest3263570-21sshd20:53:240
325822499475,3cyclictest131rcu_preempt21:48:433
325822499474,4cyclictest131rcu_preempt22:45:023
325822499474,3cyclictest131rcu_preempt23:00:233
325822499474,3cyclictest131rcu_preempt21:28:033
325822499474,3cyclictest131rcu_preempt21:28:023
325822499473,3cyclictest131rcu_preempt23:06:003
3258224994733,11cyclictest3278696-21readlink21:40:013
3258224994733,11cyclictest3278696-21readlink21:40:003
325822499472,4cyclictest131rcu_preempt21:41:003
325822499471,5cyclictest131rcu_preempt19:55:003
325822499471,3cyclictest131rcu_preempt00:38:263
3258219994739,4cyclictest3246273-21kworker/2:321:20:512
3258219994732,11cyclictest1-21systemd19:45:012
325821799479,6cyclictest131rcu_preempt22:55:001
325821799474,3cyclictest131rcu_preempt22:22:091
325821799472,3cyclictest131rcu_preempt20:50:021
325821799472,3cyclictest131rcu_preempt20:00:011
325821799471,4cyclictest131rcu_preempt23:01:251
325821499473,6cyclictest3467602-21systemd-journal21:00:010
3258214994733,10cyclictest3261263-21munin-run20:10:010
3258214994721,22cyclictest3262787-21sshd20:38:240
3258214994721,22cyclictest3262787-21sshd20:38:240
325822499465,3cyclictest131rcu_preempt23:13:133
325822499464,5cyclictest131rcu_preempt22:10:023
325822499464,5cyclictest131rcu_preempt22:10:023
325822499462,4cyclictest131rcu_preempt23:34:383
325822499462,4cyclictest131rcu_preempt23:34:383
3258224994621,10cyclictest37-21ksoftirqd/322:32:513
3258224994620,5cyclictest37-21ksoftirqd/322:20:013
325822499461,4cyclictest131rcu_preempt00:25:303
325821999464,3cyclictest131rcu_preempt20:15:022
3258219994638,5cyclictest3246273-21kworker/2:320:45:012
3258219994628,12cyclictest275-21dbus-daemon20:50:012
325821799464,4cyclictest131rcu_preempt22:17:151
325821799464,3cyclictest131rcu_preempt23:19:211
325821799464,3cyclictest131rcu_preempt23:08:201
3258217994636,7cyclictest23-21ksoftirqd/121:10:001
3258217994633,9cyclictest1-21systemd19:40:011
3258217994633,10cyclictest1-21systemd22:15:021
3258217994633,10cyclictest1-21systemd22:15:011
3258217994631,10cyclictest3467602-21systemd-journal23:12:241
325821799462,4cyclictest131rcu_preempt00:30:011
325821799461,4cyclictest131rcu_preempt23:30:061
325821799461,4cyclictest131rcu_preempt23:30:051
325821799461,3cyclictest131rcu_preempt22:58:541
325821799461,3cyclictest131rcu_preempt00:30:071
325821499466,36cyclictest0-21swapper/020:31:250
3258214994627,14cyclictest3258679-21tr19:20:000
3258214994620,20cyclictest3260490-21unin-run19:55:020
325822499454,3cyclictest131rcu_preempt22:56:283
325822499453,3cyclictest131rcu_preempt23:47:043
325822499453,3cyclictest131rcu_preempt23:47:033
325822499453,3cyclictest131rcu_preempt23:26:373
325822499453,3cyclictest131rcu_preempt23:26:363
325822499453,3cyclictest131rcu_preempt00:31:233
3258224994530,11cyclictest3467602-21systemd-journal19:40:003
325822499452,3cyclictest131rcu_preempt23:36:013
325822499452,3cyclictest131rcu_preempt22:46:443
3258224994521,6cyclictest37-21ksoftirqd/319:50:003
3258224994520,5cyclictest37-21ksoftirqd/321:23:073
325822499451,4cyclictest131rcu_preempt22:05:013
325822499451,4cyclictest131rcu_preempt22:05:013
325822499451,3cyclictest131rcu_preempt22:38:233
3258219994537,5cyclictest3246273-21kworker/2:319:40:002
3258219994537,5cyclictest3246273-21kworker/2:319:20:022
325821799455,3cyclictest131rcu_preempt19:42:451
325821799455,3cyclictest131rcu_preempt00:23:541
325821799455,3cyclictest131rcu_preempt00:23:531
325821799454,3cyclictest131rcu_preempt22:05:021
325821799454,3cyclictest131rcu_preempt22:05:011
325821799453,3cyclictest131rcu_preempt21:52:131
325821799453,3cyclictest131rcu_preempt21:52:131
325821799452,4cyclictest131rcu_preempt21:40:491
3258217994524,11cyclictest23-21ksoftirqd/123:52:321
325821799452,3cyclictest131rcu_preempt23:27:111
325821799452,3cyclictest131rcu_preempt23:27:111
325821799452,3cyclictest131rcu_preempt21:32:121
3258217994516,4cyclictest23-21ksoftirqd/122:30:001
325821799451,5cyclictest131rcu_preempt20:25:001
325821799451,3cyclictest131rcu_preempt00:00:061
325821799451,3cyclictest131rcu_preempt00:00:061
325822499444,4cyclictest131rcu_preempt19:40:023
325822499444,3cyclictest150-21pr/ttyS221:01:093
325822499444,3cyclictest131rcu_preempt22:21:073
325822499444,3cyclictest131rcu_preempt20:42:243
325822499444,3cyclictest131rcu_preempt00:22:583
325822499444,3cyclictest131rcu_preempt00:22:573
325822499443,3cyclictest131rcu_preempt22:50:313
325822499443,3cyclictest131rcu_preempt21:59:173
325822499443,3cyclictest131rcu_preempt21:14:173
325822499443,3cyclictest131rcu_preempt20:00:013
3258224994426,14cyclictest275-21dbus-daemon20:30:003
325822499442,4cyclictest131rcu_preempt00:05:533
325822499441,4cyclictest131rcu_preempt00:10:533
325822499441,3cyclictest131rcu_preempt22:26:233
325822499441,3cyclictest131rcu_preempt21:33:113
325821999446,10cyclictest30-21ksoftirqd/220:16:252
3258219994437,4cyclictest3246273-21kworker/2:321:00:142
3258219994436,5cyclictest3246273-21kworker/2:320:36:592
3258219994436,5cyclictest3246273-21kworker/2:320:36:592
325821799444,3cyclictest131rcu_preempt21:20:141
325821799443,3cyclictest131rcu_preempt21:57:131
325821799443,3cyclictest131rcu_preempt21:26:381
325821799443,3cyclictest131rcu_preempt21:26:381
325821799443,3cyclictest131rcu_preempt21:13:171
325821799443,3cyclictest131rcu_preempt00:37:151
3258217994428,13cyclictest1-21systemd19:15:011
3258217994428,13cyclictest1-21systemd19:15:011
325821799442,5cyclictest131rcu_preempt22:45:001
325821799442,3cyclictest131rcu_preempt22:37:471
3258217994421,4cyclictest23-21ksoftirqd/100:06:221
3258217994419,5cyclictest23-21ksoftirqd/123:21:091
325821799441,4cyclictest131rcu_preempt21:47:401
325821799441,3cyclictest131rcu_preempt21:16:141
325821799441,3cyclictest131rcu_preempt21:16:131
3258214994432,9cyclictest3262018-21sshd20:20:230
3258214994431,9cyclictest48-21kcompactd020:45:390
325822499432,4cyclictest131rcu_preempt23:50:183
3258219994336,4cyclictest3246273-21kworker/2:320:05:022
3258219994336,4cyclictest3246273-21kworker/2:319:50:142
3258219994335,5cyclictest3246273-21kworker/2:320:55:032
3258219994334,5cyclictest3246273-21kworker/2:321:10:022
325821799434,3cyclictest131rcu_preempt00:16:411
325821799433,3cyclictest131rcu_preempt22:06:361
325821799432,3cyclictest131rcu_preempt23:36:251
325821799431,3cyclictest131rcu_preempt23:41:591
325821799431,3cyclictest131rcu_preempt22:30:441
3258214994330,9cyclictest3467602-21systemd-journal20:45:000
3258214994329,10cyclictest3259170-21sshd19:28:240
3258214994326,14cyclictest48-21kcompactd019:40:110
3258224994226,12cyclictest3467602-21systemd-journal20:40:013
3258224994226,12cyclictest3467602-21systemd-journal20:40:003
325822499421,3cyclictest131rcu_preempt20:25:043
325822499421,3cyclictest131rcu_preempt19:15:033
325822499421,3cyclictest131rcu_preempt19:15:023
3258219994235,4cyclictest3246273-21kworker/2:320:25:372
3258217994218,6cyclictest23-21ksoftirqd/119:50:011
325821799421,3cyclictest131rcu_preempt23:47:101
325821799421,3cyclictest131rcu_preempt23:47:091
325822499419,27cyclictest3258687-21modprobe19:20:003
325822499415,3cyclictest131rcu_preempt20:20:003
325822499413,3cyclictest131rcu_preempt20:55:013
325822499413,3cyclictest131rcu_preempt19:30:023
325822499412,3cyclictest131rcu_preempt21:10:013
325822499412,3cyclictest131rcu_preempt20:50:033
3258219994134,4cyclictest3246273-21kworker/2:320:06:082
3258219994128,9cyclictest3467602-21systemd-journal20:20:022
3258219994121,14cyclictest1-21systemd19:35:002
325821799412,3cyclictest131rcu_preempt19:30:011
325821799411,3cyclictest131rcu_preempt20:00:241
325822499404,3cyclictest131rcu_preempt20:05:253
325822499404,3cyclictest131rcu_preempt19:25:023
325822499401,3cyclictest131rcu_preempt21:00:013
3258219994024,11cyclictest275-21dbus-daemon19:30:012
325821799404,3cyclictest131rcu_preempt20:40:021
325821799404,3cyclictest131rcu_preempt19:10:021
325821799402,5cyclictest3262307-21munin-run20:30:001
3258217994023,13cyclictest275-21dbus-daemon20:45:001
325822499392,3cyclictest131rcu_preempt20:32:233
325822499392,3cyclictest131rcu_preempt19:34:023
325822499391,3cyclictest131rcu_preempt20:00:563
325822499391,3cyclictest131rcu_preempt19:10:023
325821799392,3cyclictest131rcu_preempt20:35:151
325821799391,4cyclictest131rcu_preempt19:55:001
325821799391,3cyclictest131rcu_preempt20:32:511
325821799391,3cyclictest131rcu_preempt19:35:001
325821799382,3cyclictest131rcu_preempt19:20:001
3258217993816,5cyclictest23-21ksoftirqd/121:00:011
3258219993718,4cyclictest30-21ksoftirqd/219:48:252
325821799372,3cyclictest131rcu_preempt20:52:141
325821799371,3cyclictest131rcu_preempt21:01:101
325821799361,6cyclictest131rcu_preempt20:10:001
325821799361,3cyclictest131rcu_preempt20:11:241
325821999334,4cyclictest131rcu_preempt19:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional