You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-21 - 20:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sat Mar 21, 2026 12:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
54427099942,7cyclictest131rcu_preempt08:30:010
54427099942,7cyclictest131rcu_preempt08:30:010
54428299864,70cyclictest629940-21ssh12:15:023
54427099824,10cyclictest131rcu_preempt09:35:020
544270997340,16cyclictest623447-21ssh12:00:010
54427099726,15cyclictest615005-21cat11:40:030
544282997121,7cyclictest584748-21ssh10:30:023
544270997117,33cyclictest12-21ksoftirqd/009:45:010
544270997025,42cyclictest12-21ksoftirqd/011:28:540
54427099693,46cyclictest625532-21ssh12:05:000
54427099693,46cyclictest625532-21ssh12:04:590
544282996733,11cyclictest37-21ksoftirqd/312:32:583
54427099674,46cyclictest284-21systemd-logind09:25:010
544270996619,15cyclictest131rcu_preempt12:09:070
544279996516,31cyclictest564955-21kworker/u8:409:45:022
54427999631,41cyclictest3476162-21ntpd11:55:022
54427999631,41cyclictest3476162-21ntpd11:55:022
544270996321,22cyclictest12-21ksoftirqd/012:40:020
544270996314,4cyclictest12-21ksoftirqd/011:35:020
544279996133,9cyclictest571906-21kworker/2:310:41:262
544270996141,17cyclictest16850irq/134-800000010:45:010
544279996036,11cyclictest3467602-21systemd-journal11:40:002
544270996048,8cyclictest16850irq/134-800000010:55:450
544270996041,16cyclictest12-21ksoftirqd/010:05:010
544270996020,20cyclictest16850irq/134-800000010:25:010
544270995926,16cyclictest592409-21ssh10:47:470
544270995918,8cyclictest12-21ksoftirqd/009:10:020
544270995910,29cyclictest12-21ksoftirqd/010:00:020
544270995829,13cyclictest637910-21ssh12:33:390
544270995828,14cyclictest562195-21ssh09:37:200
544270995827,15cyclictest631749-21ssh12:19:200
544270995820,6cyclictest16850irq/134-800000011:13:090
54427099581,52cyclictest620713-21ssh11:53:470
54427099581,52cyclictest620713-21ssh11:53:460
544279995717,23cyclictest0-21swapper/211:48:262
544270995734,5cyclictest12-21ksoftirqd/009:19:590
544270995729,13cyclictest575541-21ssh10:08:220
544270995724,17cyclictest616952-21ssh11:45:010
544270995724,17cyclictest616952-21ssh11:45:000
544270995719,18cyclictest16850irq/134-800000012:10:340
544270995718,6cyclictest16850irq/134-800000010:35:240
544270995718,19cyclictest16850irq/134-800000011:17:020
544270995718,19cyclictest16850irq/134-800000009:48:570
544270995717,18cyclictest16850irq/134-800000009:11:230
54427099566,14cyclictest549911-21modprobe09:00:010
544270995618,18cyclictest16850irq/134-800000012:22:310
544270995618,18cyclictest16850irq/134-800000011:46:570
544270995618,18cyclictest16850irq/134-800000011:01:330
544270995618,18cyclictest16850irq/134-800000009:53:390
544270995617,19cyclictest16850irq/134-800000010:17:110
544270995616,6cyclictest16850irq/134-800000010:31:330
54427599551,41cyclictest547838-21kworker/u8:408:30:021
54427599551,41cyclictest547838-21kworker/u8:408:30:021
544270995526,14cyclictest577202-21ssh10:12:050
544270995519,16cyclictest16850irq/134-800000010:27:350
544279995437,12cyclictest1-21systemd07:40:002
54427599546,5cyclictest131rcu_preempt12:00:001
544270995419,17cyclictest16850irq/134-800000010:54:070
544270995417,6cyclictest16850irq/134-800000011:20:090
544279995343,6cyclictest571906-21kworker/2:310:34:062
544279995339,5cyclictest547901-21kworker/2:210:00:032
544270995316,6cyclictest16850irq/134-800000012:28:220
544279995243,5cyclictest571906-21kworker/2:311:45:032
544279995241,6cyclictest616993-21kworker/2:012:20:012
54427099523,44cyclictest559091-21ls09:30:020
544279995143,5cyclictest571906-21kworker/2:310:46:562
544279995143,5cyclictest547901-21kworker/2:210:05:212
544279995142,6cyclictest571906-21kworker/2:312:05:362
544270995115,18cyclictest16850irq/134-800000011:09:450
544279995042,5cyclictest571906-21kworker/2:311:56:192
544279995042,5cyclictest531934-21kworker/2:007:22:472
544279995041,6cyclictest616993-21kworker/2:012:34:102
544279995041,6cyclictest586862-21kworker/2:111:30:462
544279995041,6cyclictest586862-21kworker/2:111:05:312
544279995040,6cyclictest586862-21kworker/2:110:58:582
544275995022,10cyclictest23-21ksoftirqd/110:05:001
544270995043,4cyclictest16950irq/224-800000007:10:020
544279994941,5cyclictest586862-21kworker/2:111:10:052
544279994941,5cyclictest547901-21kworker/2:209:29:142
544279994936,8cyclictest3467602-21systemd-journal12:15:002
54427999491,44cyclictest3467602-21systemd-journal11:15:032
544275994919,6cyclictest23-21ksoftirqd/110:14:431
54427099494,3cyclictest131rcu_preempt08:35:020
544270994922,21cyclictest275-21dbus-daemon08:15:010
544270994922,21cyclictest275-21dbus-daemon08:15:010
544282994833,11cyclictest1-21systemd10:50:013
54427999487,5cyclictest131rcu_preempt09:38:182
54427999484,8cyclictest30-21ksoftirqd/207:29:452
544279994841,4cyclictest571906-21kworker/2:310:19:482
544279994840,5cyclictest582675-21kworker/2:010:29:242
544279994840,5cyclictest571906-21kworker/2:311:26:012
544279994840,5cyclictest547901-21kworker/2:210:13:392
544279994840,5cyclictest547901-21kworker/2:209:47:222
544279994840,5cyclictest547901-21kworker/2:209:13:022
54427599482,3cyclictest131rcu_preempt09:11:381
544275994814,5cyclictest131rcu_preempt10:54:591
544270994810,21cyclictest544725-21modprobe07:20:000
544279994738,6cyclictest571906-21kworker/2:311:01:512
544279994738,5cyclictest547901-21kworker/2:209:19:392
544270994739,5cyclictest16950irq/224-800000008:00:010
54428299465,4cyclictest131rcu_preempt11:55:003
54428299465,4cyclictest131rcu_preempt11:55:003
544282994632,11cyclictest37-21ksoftirqd/311:45:013
544282994632,11cyclictest37-21ksoftirqd/311:45:013
544279994640,4cyclictest571906-21kworker/2:311:41:282
544279994638,5cyclictest586862-21kworker/2:110:51:402
544279994638,5cyclictest547901-21kworker/2:208:37:112
544279994638,5cyclictest547901-21kworker/2:208:30:532
544279994638,5cyclictest537936-21kworker/2:108:18:462
544279994628,13cyclictest3467602-21systemd-journal12:25:022
544279994623,5cyclictest30-21ksoftirqd/212:24:292
54427599463,3cyclictest131rcu_preempt09:20:591
544275994629,10cyclictest0-21swapper/112:40:011
544270994628,9cyclictest309-21in:imuxsock07:40:000
54427099462,3cyclictest131rcu_preempt08:40:020
544282994528,12cyclictest3467602-21systemd-journal12:40:013
544282994526,14cyclictest275-21dbus-daemon10:25:013
544282994524,11cyclictest37-21ksoftirqd/310:12:143
544282994518,17cyclictest37-21ksoftirqd/309:39:443
54427999454,3cyclictest131rcu_preempt07:10:022
544279994529,12cyclictest1-21systemd08:15:012
544279994529,12cyclictest1-21systemd08:15:002
54427599454,3cyclictest131rcu_preempt10:41:311
544275994532,10cyclictest23-21ksoftirqd/110:22:181
544275994529,12cyclictest1-21systemd07:15:011
544270994538,4cyclictest16950irq/224-800000008:25:020
544270994532,9cyclictest549256-21wc08:45:020
544270994532,9cyclictest549256-21wc08:45:010
544270994525,14cyclictest547842-21cron08:19:590
544270994513,22cyclictest545503-21modprobe07:35:000
544270994511,6cyclictest131rcu_preempt08:05:020
54428299443,3cyclictest131rcu_preempt09:48:053
544282994432,9cyclictest1-21systemd09:45:013
544282994418,17cyclictest37-21ksoftirqd/309:12:583
54428299441,4cyclictest131rcu_preempt10:45:033
54428299441,4cyclictest131rcu_preempt09:51:473
54427999448,5cyclictest131rcu_preempt12:36:182
544279994437,5cyclictest547901-21kworker/2:210:02:372
544279994436,5cyclictest571906-21kworker/2:312:01:202
544279994436,5cyclictest571906-21kworker/2:312:01:202
544279994436,5cyclictest546322-21kworker/2:309:33:412
544279994436,5cyclictest546322-21kworker/2:307:58:562
544279994429,10cyclictest3467602-21systemd-journal07:35:012
54427599442,3cyclictest131rcu_preempt10:35:031
54427599441,4cyclictest131rcu_preempt09:34:591
54428299433,3cyclictest131rcu_preempt11:20:013
544282994331,8cyclictest1-21systemd11:25:013
544282994318,16cyclictest632237-21cat12:20:043
544282994317,15cyclictest625552-21kworker/3:112:05:363
54428299431,3cyclictest131rcu_preempt10:00:003
54428299431,3cyclictest131rcu_preempt09:23:053
54427999435,5cyclictest131rcu_preempt07:55:012
544279994335,5cyclictest571906-21kworker/2:311:21:252
544279994335,5cyclictest571906-21kworker/2:310:24:042
544279994335,5cyclictest547901-21kworker/2:209:23:042
544279994329,10cyclictest3467602-21systemd-journal08:54:442
544279994329,10cyclictest30-21ksoftirqd/209:54:592
544279994315,18cyclictest30-21ksoftirqd/207:50:032
544275994329,8cyclictest638445-21grep12:35:001
54427599432,3cyclictest131rcu_preempt09:37:111
544275994322,15cyclictest275-21dbus-daemon10:10:021
54427599431,3cyclictest131rcu_preempt12:20:051
54427099434,14cyclictest546104-21latency_hist07:45:020
544270994327,12cyclictest1-21systemd09:05:000
544270994326,13cyclictest549654-21cron08:55:010
544270994326,13cyclictest545246-21cron07:30:000
54428299424,3cyclictest131rcu_preempt07:25:023
54428299423,3cyclictest131rcu_preempt12:05:023
54428299423,3cyclictest131rcu_preempt12:05:023
544282994232,6cyclictest580421-21sendmail10:20:003
544282994229,10cyclictest546516-21sshd07:51:463
544282994228,7cyclictest309-21in:imuxsock10:35:013
544282994228,10cyclictest37-21ksoftirqd/311:37:003
54428299422,3cyclictest131rcu_preempt08:00:013
54428299422,3cyclictest131rcu_preempt07:20:003
544279994234,5cyclictest531934-21kworker/2:007:15:002
54427999421,5cyclictest0-21swapper/208:10:022
54427599424,4cyclictest131rcu_preempt07:10:011
54427599424,3cyclictest131rcu_preempt11:25:091
54427599423,6cyclictest23-21ksoftirqd/111:54:581
54427599423,6cyclictest23-21ksoftirqd/111:54:581
544275994235,4cyclictest597599-21sh11:00:001
544275994232,7cyclictest23-21ksoftirqd/109:20:021
544275994231,8cyclictest3467602-21systemd-journal11:42:451
544275994231,8cyclictest3467602-21systemd-journal11:42:451
544275994230,10cyclictest23-21ksoftirqd/107:20:011
544275994229,10cyclictest23-21ksoftirqd/110:48:091
544275994226,12cyclictest275-21dbus-daemon07:30:021
544275994225,13cyclictest275-21dbus-daemon12:25:001
54427599422,3cyclictest131rcu_preempt09:51:201
544275994223,14cyclictest275-21dbus-daemon11:50:011
544275994221,15cyclictest1-21systemd08:50:011
54427599421,3cyclictest131rcu_preempt09:27:141
544275994211,19cyclictest131rcu_preempt09:10:011
54427099425,11cyclictest12-21ksoftirqd/008:10:000
544270994225,10cyclictest309-21in:imuxsock07:15:000
544282994126,10cyclictest309-21in:imuxsock08:50:003
544282994117,18cyclictest275-21dbus-daemon11:15:013
54428299411,4cyclictest131rcu_preempt12:29:363
54428299411,3cyclictest131rcu_preempt10:06:493
54428299411,3cyclictest131rcu_preempt10:04:243
54428299411,3cyclictest131rcu_preempt09:28:473
544279994133,5cyclictest547901-21kworker/2:209:08:142
544279994126,12cyclictest30-21ksoftirqd/209:00:002
54427999411,10cyclictest544956-21sed07:20:022
54427599414,3cyclictest131rcu_preempt11:09:591
544275994125,12cyclictest3467602-21systemd-journal08:00:001
544275994124,13cyclictest275-21dbus-daemon07:40:011
54427599412,3cyclictest131rcu_preempt10:40:031
54427599411,5cyclictest131rcu_preempt12:08:511
54427599411,3cyclictest131rcu_preempt10:15:141
54428299405,3cyclictest131rcu_preempt09:34:043
544282994028,9cyclictest596709-21sshd10:57:453
544282994027,11cyclictest37-21ksoftirqd/311:09:503
544282994027,10cyclictest37-21ksoftirqd/311:03:003
54428299402,4cyclictest131rcu_preempt08:19:593
544282994024,12cyclictest37-21ksoftirqd/308:55:013
54428299402,3cyclictest131rcu_preempt09:15:553
544279994020,4cyclictest30-21ksoftirqd/207:45:012
544279994011,19cyclictest30-21ksoftirqd/208:25:022
54427599405,3cyclictest131rcu_preempt08:55:001
54427599403,3cyclictest131rcu_preempt12:05:001
54427599403,3cyclictest131rcu_preempt12:05:001
544275994027,9cyclictest309-21in:imuxsock09:05:011
544275994027,10cyclictest23-21ksoftirqd/108:11:501
544275994027,10cyclictest23-21ksoftirqd/108:11:491
544275994026,11cyclictest1-21systemd11:15:011
544275994025,12cyclictest547968-21wc08:20:021
544275994025,11cyclictest614795-21sh11:39:591
544275994025,10cyclictest626557-1kworker/u9:012:15:011
54427599402,3cyclictest131rcu_preempt09:00:031
544275994018,9cyclictest23-21ksoftirqd/108:10:011
544275994015,5cyclictest23-21ksoftirqd/111:17:111
54427099405,28cyclictest3467602-21systemd-journal07:55:000
54427099403,3cyclictest131rcu_preempt07:50:030
544270994022,14cyclictest549374-21sshd08:45:550
544282993926,10cyclictest37-21ksoftirqd/308:25:023
544282993925,9cyclictest3467602-21systemd-journal07:44:593
544282993924,10cyclictest275-21dbus-daemon07:50:023
54428299392,3cyclictest131rcu_preempt07:15:003
54428299391,34cyclictest610668-21sed11:30:023
544282993910,4cyclictest131rcu_preempt11:30:573
544279993930,5cyclictest571906-21kworker/2:310:36:492
544279993924,11cyclictest275-21dbus-daemon08:45:012
544279993924,11cyclictest275-21dbus-daemon08:45:012
54427599394,3cyclictest131rcu_preempt10:28:471
544275993924,11cyclictest23-21ksoftirqd/108:40:011
54427599391,3cyclictest131rcu_preempt10:00:011
54428299384,5cyclictest131rcu_preempt11:48:453
54428299384,3cyclictest131rcu_preempt10:53:263
544282993827,5cyclictest309-21in:imuxsock08:45:013
544282993827,5cyclictest309-21in:imuxsock08:45:003
54428299382,3cyclictest131rcu_preempt09:10:023
544282993823,12cyclictest548962-21wc08:40:023
54428299381,3cyclictest131rcu_preempt11:59:313
544279993829,7cyclictest30-21ksoftirqd/208:05:022
54427599383,3cyclictest131rcu_preempt07:34:591
544275993828,7cyclictest608360-21pluginstate11:25:011
544275993828,6cyclictest546103-21cat07:45:011
544275993826,9cyclictest545089-21sed07:25:011
54427599382,4cyclictest131rcu_preempt12:28:061
54427599381,3cyclictest131rcu_preempt09:45:451
544270993825,10cyclictest12-21ksoftirqd/007:25:020
54428299375,4cyclictest131rcu_preempt09:05:003
54428299374,3cyclictest131rcu_preempt08:10:023
544282993725,9cyclictest589054-21systemd-run10:40:013
54427999377,6cyclictest131rcu_preempt08:50:002
54427999371,4cyclictest131rcu_preempt08:30:012
54427999371,4cyclictest131rcu_preempt08:30:002
544275993725,10cyclictest23-21ksoftirqd/109:42:581
544275993723,11cyclictest23-21ksoftirqd/111:02:451
54427599371,4cyclictest131rcu_preempt08:45:021
54427599371,3cyclictest131rcu_preempt07:55:021
544282993627,6cyclictest545927-21latency_hist07:40:023
544282993624,9cyclictest550024-21sed09:00:023
544282993623,10cyclictest37-21ksoftirqd/308:15:023
544282993623,10cyclictest37-21ksoftirqd/308:15:013
54427599363,5cyclictest131rcu_preempt08:05:031
54427599363,3cyclictest131rcu_preempt08:25:011
54427599362,3cyclictest131rcu_preempt11:31:021
54427599361,4cyclictest131rcu_preempt08:45:001
54427599361,3cyclictest131rcu_preempt07:50:021
544282993524,9cyclictest37-21ksoftirqd/308:30:013
544282993524,9cyclictest37-21ksoftirqd/308:30:003
544282993522,9cyclictest37-21ksoftirqd/307:30:003
544282993522,10cyclictest37-21ksoftirqd/308:05:023
544282993517,9cyclictest275-21dbus-daemon08:35:013
544275993527,5cyclictest548622-21latency_hist08:35:011
544282993425,6cyclictest545554-21latency_hist07:35:023
54428299342,4cyclictest131rcu_preempt12:20:473
54427999333,4cyclictest131rcu_preempt09:05:012
54428299321,3cyclictest131rcu_preempt07:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional